Measuring Whether AI Helps

A starting question

An assistant writes a feature in ten minutes instead of an hour. Then the team spends two days fixing edge cases. Was that faster?

The initial typing was faster; the whole change may not have been. Generated lines, accepted suggestions, and prompt counts are activity measures. They do not tell you whether users received a reliable improvement.

Start with practical observations

For a repeated type of task, record a rough baseline before using AI. Then compare similar work: time to a reviewed change, number of defects found before and after release, rework caused by the change, and whether another developer can understand it later. Do not treat one task as proof; tasks differ in difficulty.

A small personal log can be enough: “The assistant helped draft tests in 20 minutes; I spent 15 minutes correcting assertions; the tests caught the original bug.” This is more useful than “AI wrote 300 lines.” Do not collect sensitive prompts or private source code merely to count usage.

Balance speed and stability

Teams may also watch software-delivery measures such as change lead time and change failure rate. DORA’s guide groups delivery measures into throughput and instability and includes deployment rework.These are team-level signals, not a score for one person or one prompt.¹ DORA’s history of the metrics explains why definitions have evolved.Choose a stable definition before comparing periods.²

An AI tool can be useful even when it writes little code: explaining a confusing trace, finding the right test boundary, or reducing time spent searching. Conversely, a large patch can increase review and maintenance cost. GitHub’s agent guidance notes that output can be inaccurate and needs verification.A speed claim should include that verification work.³

Avoid turning metrics into targets

If people are rewarded for lines generated, they can produce more lines without improving the product. If they are rewarded only for faster merges, they may skip review. Treat measures as questions: Did the change help users? Did it create more failures or rework? Can the next developer safely change it?

Recap and try it

AI helps when it improves the full development outcome, including understanding and reliability. Measure useful work and costs together, with enough context to avoid false comparisons.