Token dashboards should act as smoke detectors for unusual usage, never as leaderboards that reward people for spending more.
2
Ironclad measures value alongside token cost, moving from lines of code and open pull requests to merged pull requests weighted by complexity.
3
AI has shifted the engineering bottleneck toward human review and CI, so teams need to reduce flaky tests, limit retry loops, and measure time from ready to merged.
Summary
Mingsheng Hong argues that engineering teams should stop treating token usage as a target. A dashboard can reveal adoption gaps, sudden usage bursts, or unusual differences between teams, but high usage is not proof of value. Ironclad first measured lines of code, then open pull requests, then merged pull requests. Its current measure weights merged pull requests by an AI-generated complexity score. Hong calls the broader goal trusted throughput: work that passes objective checks, receives human review, and survives contact with customers. AI makes code generation abundant, which moves pressure downstream into review and CI. Large pull requests can reduce review quality, while flaky tests and slow pipelines waste both engineer time and tokens. Hong recommends AI as an initial review layer, human judgment for architecture and security, budgets and anomaly alerts, capped agent loops, prompt caching, and context pruning.
Token dashboards should detect problems instead of rewarding spend
Hong describes voluntary token dashboards that turned into competitions, with engineers trying to reach the top by maximizing usage. His alternative is to track every team and individual, then treat the dashboard as a smoke detector. Low usage in one team may signal an adoption problem, while a sudden burst may need investigation. The dashboard should not create a direct or indirect incentive to burn more tokens. Teams also need context when comparing usage, because a platform infrastructure team may use AI differently from a UI team. The purpose is to extract lessons and feed them back into shared engineering practices.
Teams need adoption before they try to control cost
Hong says cost controls make more sense after engineers have easy access to AI and have started using it. Ironclad had only recently moved past that adoption phase. For teams still struggling, he recommends speaking with resistant engineers about their concerns. Some people took pride in handcrafting code and now feel that their work has become reviewing AI-generated code. Leaders need to find technical work that still gives engineers room to grow and has high impact. This concern matters because adoption is not complete when a tool is provisioned. Engineers also need a worthwhile role in the new workflow.
Ironclad builds legal contracting AI, so trust already shapes how its customers adopt product features. Lawyers often test AI on contracts they know well before using it for less familiar work. Hong applies a similar progression to engineering AI. The goal is not austerity or simply reducing token spend. Teams should measure cost, measure the value produced, and then find the bottlenecks that limit return on that spend. For organizations using several coding tools, Ironclad combines vendor data into dashboards and pipelines that show usage by team and individual across tools. This creates a shared view of cost without making cost the objective.
Lines of code and token usage are metrics that become harmful goals
Hong compares token spend with lines of code. Lines of code can be useful to track, but optimizing for more lines can reward the wrong outcome because removing code may be better engineering work. Ironclad's measurement evolved in stages. The team moved from lines of code to open pull requests as AI made code generation more abundant. It then moved to merged pull requests, since shipped code matters more than work that remains open. The same warning applies throughout: a count can provide information without being a suitable target. More tokens or more code do not automatically mean more useful work.
Ironclad found that merged pull requests still differ greatly in value. A ten-line change that finds and fixes a concurrency bug may matter more than a thousand lines of boilerplate. The team therefore began tagging merged pull requests with a complexity score. Hong says there is no traditional definition they adopted. Instead, they use a carefully designed prompt and ask one or two models to assign a T-shirt-size score. A more complex AI-generated pull request receives more weight in their measure. He presents this as a pragmatic metric that will continue to change as the team learns how to approximate the value created by AI.
Trusted throughput requires checks, judgment, and customer contact
Hong defines trusted throughput as high-quality output that is trusted by engineering and leadership inside the company and by customers outside it. The first part comes from objective checks such as test coverage, security checks, and canarying. Human review adds judgment about code quality, clarity, maintainability, and architectural fit. The final test is what happens in production. Customer complaints, usability friction, bugs, incidents, and rollbacks show whether the change held up outside internal review. These three forms of evidence work together. A merged pull request is therefore only an intermediate signal, not proof that the work delivered value.
AI-generated code moves the bottleneck into review and CI
When AI makes pull request creation abundant, review and merging take more of the time. Hong warns against responding to slow CI by submitting one large pull request instead of several smaller ones. If each run takes an hour, splitting work can feel expensive, but large pull requests increase the human review burden and can spread attention too thin. Ironclad uses AI as the first review layer for coding style issues and missing test coverage. Human reviewers then focus on architecture, code quality, and security design. Engineers retain final accountability. The workflow only works if CI can process the smaller pull requests without making people wait or babysit them.
Developer experience work should remove flaky tests and limit wasteful loops
Slow or flaky CI makes engineers rerun tests by hand, wait for pull requests, or ask an AI agent to keep trying. The latter can waste tokens while neither approach solves the underlying infrastructure problem. Ironclad invests in developer experience work to remove flaky tests and improve CI. Hong recommends measuring the time from a pull request being ready to submit until it is submitted, then comparing that with the normal CI runtime. Repeated test retries are another signal. The team also sets budgets, quotas, usage tracking, and anomaly alerts. Agentic loops need a cap on retry steps so an unsuccessful task does not consume tokens indefinitely.
"The bottleneck from kind of the whole life cycle perspective gets shifted onto review and they're subsequently merging the PR."13:53
Who should watch
You lead an engineering organization that has moved past AI adoption and now needs to understand whether rising token usage is producing useful shipped work.
Your team is seeing more pull requests but review queues, flaky tests, or slow CI are making delivery harder.
You are designing internal AI usage dashboards and want to avoid turning them into incentives for engineers to spend more.
