Why would you?

That's the whole problem.

Do you hire people to do work or pretend to do work?

Give them the tools. Measure normally i.e. did they work faster with new tools?

If things are good and there's support people will use it to make their lives easier. That should be natural. Forcing them on it with arbitrary rules will just give you fake results.

Fair - but I did not mean this as in "control the team". It's more around how can we can push the boundaries. We came up with a framework internally. But I was asking muyself if that is the right / best way. Maybe we share what we have.

> It's more around how can we can push the boundaries.

Is it the right boundaries though?

Person A hates AI but is the fastest developer.

Person B loves AI uses the tools but isn't as good.

Both develop as fast. A might have somewhat higher salaries. B costs you in tokens.

What are you pushing? The best results. Not if they use AI.

So why are you framing this as benchmark AI? Benchmark what can be done? AI is the driver / enhancer not the result.