Measuring AI Productivity: the Confirmed Lead Time framework

Every CTO I talk to is trying to answer the same question: is AI actually making the team more productive, and how do I prove it to the people who signed the invoice?

Most of the answers I see are measuring the wrong thing. So here is the framework I am shifting to, why the usual numbers fail, how to adopt it this week, and the part I have not solved yet. I call it Confirmed Lead Time. It is not clever. That is the point.

The vanity metrics

Walk into most engineering orgs right now and the “AI productivity” dashboard looks like this: token usage, commits per day, lines of code shipped, maybe story points if someone is feeling nostalgic.

Every one of those measures cost, not value. An agent can burn two million tokens and 4,000 lines of code producing something nobody asked for. That is not productivity. That is expensive motion.

Token usage is not useless. It is a fair signal for who is adopting the tools and who needs re-training. Just not a productivity number. Put it on a dashboard and it gets gamed by Friday. I have watched it happen.

These are all throughput metrics. They measure how busy the team is, not whether it served anyone. That was a weak proxy before AI. It is a broken one now, because an agent inflates all of them at once. The numbers go up and to the right while the business feels no different.

The reframe: measure the response, not the output

Stop measuring what engineering produces. Start measuring how fast engineering responds to the people who need it.

Every unit of engineering capacity exists for a reason, and the reason usually has a name attached. A PM files a feature requirement. A customer reports a bug. Sales asks for a demo. The CISO flags a vulnerability. An internal engineer needs a shared utility.

Each is a stakeholder ask. The metric is the time from when the ask is filed to when it is fit for use. That is Confirmed Lead Time, and the word doing the work is confirmed: the clock stops when the stakeholder says it works, not when engineering says it is done.

“Fit for use” is not perfect, and not even deployed. The bar is different per ask: a PM ask is fit for use when the hypothesis can be validated; a customer ask when their workflow is unblocked; a sales ask when it runs for one prospect for thirty minutes (fragile is fine); a CISO ask when it is mitigated with rigor; an internal ask when the dev loop is unblocked. None of that shows up in a token count.

This is not a replacement for DORA

If you already run DORA, keep it. DORA’s lead time stops at deploy. Confirmed Lead Time keeps running until the stakeholder confirms. Everything DORA structurally cannot see lives in that gap: shipped-but-not-actually-useful, the re-opened “fix,” the demo nobody showed.

Think of it as an outer loop. DORA measures the factory. CLT measures whether the customer got what they needed. You are not ripping anything out. You are adding the clock that runs past deploy.

The framework: one metric, five rules

A metric is easy to state and easy to game. These five rules keep it honest.

1. The stakeholder stops the clock, not engineering. This is the load-bearing move. It removes “shipped-but-broken counted as a fast cycle.” The agent generates code in four hours, we mark it complete, and three days later the PM tells us the workflow still does not work. So the timer stops when the stakeholder says “this works for me.”
2. Segment before you average. A single CLT number hides everything. A sales demo and a CISO vulnerability do not belong in the same average. Segment by ask type and size, or mix-shift masquerades as improvement.
3. p50 and p90, never the mean. The median is the typical experience. The p90 is the tail, where the reputation damage lives. One ticket that sat for three weeks destroys an average.
4. Run two clocks. Engineering time (opened to deployed) and wall-clock (opened to confirmed). The gap between them is an intake problem, not an engineering one. Separating the two stops you blaming the team for a slow approver, and stops the team hiding behind one.
5. Pair speed with re-open rate. The anti-gaming guardrail. A real improvement is p50 and p90 falling while re-open rate stays flat. Faster without breaking more.

Tagging the asks: a POC is not a CISO request

All five rules depend on one thing the dashboard cannot guess: what kind of ask this is. Tag by intent and durability, not by who asked. A proof-of-concept and a security request sit at opposite ends of one axis.

Never average across classes. A two-hour throwaway and a three-week regulated ask are two healthy numbers, not a 1.5-week average. The guardrail flips too: a re-opened POC is fine; a re-opened regulated ask is an incident.

Two disciplines keep tagging honest. The requester or a triage rotation sets the class, never the engineers being measured, or hard work quietly gets relabeled “POC” to look fast. And the tag freezes once work starts, so a reclassification is a logged decision, not a silent edit.

Faster only counts if it does not break more

Speed metrics get gamed. Always. So every speed number gets a quality number stapled to it: the re-open rate, the percentage of asks re-opened within fourteen days of a fit-for-use confirmation.

AI-agnostic, on purpose

You may have noticed CLT never mentions AI. That is deliberate, and it is a feature. An outcome metric that cannot be fooled by AI vanity numbers is exactly what you want when the whole problem is AI vanity numbers.

It also means CLT will not tell you “AI made us 3x faster.” Neither can anyone honestly quoting that number, because the old system never measured the thing that mattered, so there is no clean before. What CLT can tell you is sharper and more honest. Assume your team already uses AI to generate content, code, docs, images, reports. AI compresses generation. So the testable hypothesis is this: CLT gains should show up on generation-bound asks and stay muted on judgment- and coordination-bound asks. Tag each ask generation-bound or judgment-bound and watch where the speedup survived to the outcome.

If a generation-heavy ask type is not improving, generation was never the bottleneck, and now you know where to actually look: review, integration, the rework from plausible-but-wrong output, or the intake gap. “We got 10x faster at drafting and 0x faster at shipping” is a finding worth the entire framework.

Adopting it: crawl, walk, run

The fastest way to kill a measurement system is to require behavior change from people you do not control. So start with what a single CTO can run alone.

• Crawl (you, this week). Do not ask anyone to confirm anything. Use signals you already have: the re-open event (the customer came back, so it was not fit for use) and the existing “thanks, works” comment as the timestamp. Internal-engineer asks you control fully. This produces a real CLT number with nobody outside engineering changing a thing.
• Walk. Add a one-click confirm for your highest-value classes only, durable and regulated, where the stakeholder already has skin in the game.
• Run. Per-class confirmation across the board, plus an LLM classifier to remove the tagging friction.

A starter kit (issue-form template, labels, and a read-only metrics script for the crawl) lives in the companion repo, Hypermindz-AI/clt-starter, so this is a clone-and-go, not a build.

Automating the classification

At the run stage, an LLM removes the tagging friction without surrendering the anti-gaming controls.

The AI proposes, a human disposes. Engineering still never sets its own bar, and the named confirmer stays human. The classifier gets its own quality metric, human override rate: when overrides climb, the prompt and examples get refreshed. And the classifier is itself a stakeholder ask, measured by CLT.

Proving it improved

You do not have a clean baseline and you cannot fully reconstruct one. The discipline is: baseline forward, approximate backward, and label which is which.

Baseline forward. Run CLT for about six weeks before you change anything else. That window is your real baseline, because it is the only data captured the same way you will measure the gains. Pick a representative window, no holiday weeks, no crunch.

Approximate backward. Legacy orgs can reconstruct the shape of the past, an AI classifier can re-tag historical tickets by class, but it cannot invent a confirmation timestamp nobody ever wrote. So historical durations stay a range, never a benchmark. The method, including a hindsight-calibration check, is in the appendix.

The part I have not solved

I am not going to pretend this is finished. How long the forward baseline needs to run before I trust the trend, I am guessing at six weeks. Asks with no single clear stakeholder, like cross-team infrastructure, do not fit cleanly. Whether to weight asks by business value would be more honest and would add another knob to game. And the clock runs while a stakeholder sits on an ask, which I would rather see than hide, but I do not have a clean way to hold the intake gap accountable yet.

None of this is AI-specific, by the way. It is how you would measure any engineering team, and Lean Software Development was talking about cycle time and fit for use twenty years ago. AI did not invent the right metric. It just made the wrong ones so obviously bankrupt, and the bills so obviously large, that we finally have to fix it.

If your team is trying to prove the AI investment is paying off, do not reach for token counts. Measure the response to the people who needed something. I will report back in a few weeks with the first real numbers from our own crawl. In the meantime I would genuinely like to know: what does “done” mean on your team today, and who gets to decide?

Appendix: reconstructing a baseline with AI

For legacy orgs that want a head start, an LLM can re-classify your backlog, but mind the limit: it recovers the class, not the clock.

• Composition, not durations. Use AI to reconstruct the historical mix of classes and task-shapes (so you can control for mix-shift). Express historical durations only as confidence-tagged ranges from proxies (open date, the “works” comment, re-open dates). Flag every reconstructed data point retro and never blend it silently with forward-measured data.
• Hindsight calibration. When AI reads an old ticket it sees the ending, so it classifies with information the intake moment did not have. Measure that bias: have the classifier label a sample of recent tickets twice, once blind (title and body only) and once with the full resolution thread. The delta is how much hindsight inflates the retro classification, so you can discount the backlog by a known amount instead of guessing.
• Survivorship. Tracked-and-closed tickets look healthier than the ones silently abandoned. The reconstructed baseline flatters the past; treat it as directional.