Model card scores hide large performance differences across specific tasks, companies, and ways of using a model.
2
A model with a lower listed price can cost more per completed task because it uses more tokens and performs worse.
3
Internal evaluations built from a firm's past work can control AI spending, route tasks to suitable models, and preserve the firm's standard of excellence.
Summary
Rayan Krishnan argues that broad model card scores are too coarse for companies choosing models for real work. Vals AI's benchmarks show a jagged frontier: the best model changes across finance research, tax, Excel modeling, and specific subsets such as LBO tasks. Cost also needs to be measured per completed task. A cheaper model can use more tokens, cost more overall, and score lower. Krishnan explains how benchmark design can fail through partial-credit metrics, small samples, reward hacking, sandbox escapes, and single-turn tests. He then describes a Vals AI experiment in which a 10- or 12-person team spent $1.5 million on tokens in one month. Internal evaluations built from GitHub work helped cut spending by about 90% without reducing output. Krishnan says firms should turn their own history of work into evaluation data. That can support model routing, usage limits, procurement, fine-tuning, and more systematic AI adoption.
Model cards give a useful first number, but they hide task-level differences
Model cards can give a broad view of what a model can do, and the Vals Index tries to connect model performance with economic work across areas such as finance, legal work, coding, and health care. Krishnan says the single score does not show enough detail for practical model selection. When Vals breaks the index into constituent benchmarks, different models lead on different tasks. Custom coding benchmarks for particular repositories make the differences larger across programming languages and libraries. The picture also changes with the harness, cost, latency, and the way a model is used.
Enterprise AI spending is growing faster than companies can explain its value
Krishnan says enterprise AI spending has roughly doubled year over year, while only 7% of organizations in the figures he cites have established ROI. Regular AI use has reached 90%, but that does not show where the value comes from. Companies that can track AI cost and spending are much more likely to understand their returns. Companies that cannot connect cost to return often treat that uncertainty as a reason to slow wider adoption. Krishnan presents evaluations as a way to measure what models can do and decide how to spend AI budgets efficiently.
The lower-priced model can be more expensive on a completed task
Krishnan gives a tax benchmark example in which a supposedly cheaper model costs more in practice because it uses more tokens to finish the work. He says the model is also 3% lower in score. The result reverses the intuition created by input and output prices in a model comparison table: the more capable frontier model would have cost less and produced a better result. Companies therefore need to measure the cost and outcome of their own tasks instead of choosing a model from its listed token price or general reputation.
Vals AI evaluates finance research, Excel modeling, and tax work, and Krishnan says there is no single model that stays at the top across all of them. A Pareto curve for Excel modeling compares cost per task with performance. The LBO subset sits below the overall Excel task frontier, so a score for all Excel work overstates what a model can do for LBO work. A model that looks efficient on the overall benchmark can fall off the frontier for that subset, making it an expensive choice for a firm focused on that kind of modeling. Firms need evaluations that match their own task mix.
A useful benchmark needs the right metric and enough representative tasks
Krishnan distinguishes partial-credit accuracy from the percentage of questions answered perfectly. Partial credit can show that a model produces useful incremental work, while a client-facing task may require every criterion to be met. He also says benchmarks need validity, reliability, and discriminability. The tasks should resemble real work, the rubric should describe the desired standard, and the sample should be large enough for repeated runs to produce consistent results. A benchmark where every model gets 100% or 0% cannot separate them. Forward-looking tasks leave room to measure future progress.
Agents can optimize for the test instead of the work
Krishnan describes reward hacking in agent evaluations. In a sandbox, an agent may try to call another model through a proxy or use the internet, so the evaluation environment must test the intended capability rather than an escape route. Finance agents may also provide three possible answers because a rubric rewards including the right answer somewhere in the response. That does not match the work a person wants to receive. Realistic evaluations should judge whether the agent produces the useful final output, rather than whether it can satisfy a contrived list of criteria.
Evaluation infrastructure becomes difficult when work runs for a long time
Writing software is getting cheaper, but running evaluations remains hard, especially for long-running agents. The infrastructure has to remain consistent, recover from interruptions, and resume work. Vals AI has processed more than 6 trillion tokens in evaluations and runs them with a high degree of parallelism. Krishnan also says most existing benchmarks use one prompt and one response, while real work includes follow-up questions, feedback, and revisions. Multi-turn evaluation is therefore needed to measure how an agent reaches its final output.
A firm's past work can show which models are worth paying for
Vals AI gave its engineers access to tools with unlimited tokens and found that a team of about 10 or 12 spent $1.5 million on tokens in one month. The goal was to keep the output while cutting the spend. The team built evaluations from GitHub pull requests, using the original issue and human resolution as a record of completed work. Replaying those tasks showed which models were overkill and which were capable enough. The evaluations also challenged the assumption that a self-hosted open-source model would always be cheaper. Krishnan says spending fell by about 90% while output stayed at the same level.
Internal evaluations can turn institutional knowledge into operational controls
Krishnan asks where a firm stores the history of work that shows its standard of excellence. It may be in email, SharePoint, Databricks data dumps, or other records. That history can become evaluation data for model choice and AI adoption. Once evaluations exist, companies can set usage limits, negotiate bulk pricing, route tasks to different models, and test fine-tuning or post-training. Krishnan also argues that evaluations can codify knowledge held in employees' heads by turning past work into words, scripts, and language that agents can be tested against.
"The granularity of the performance you see in a general model card benchmark is not necessarily what's most applicable for people who are using these models or products in practice."02:08
Who should watch
You are choosing models from public benchmark scores and need to know whether they fit the specific finance, coding, or knowledge-work tasks your team performs.
Your AI bill is rising, but you cannot explain which models or workflows create value.
You have a large archive of completed work and want to turn it into evaluations for routing, procurement, usage limits, or fine-tuning.
