Witan Labs improved spreadsheet benchmark accuracy from 50% to 92% by changing the agent interface and adding better verification.
2
A persistent JavaScript REPL worked better than many separate tools because the agent could combine operations, keep state, and interleave reasoning with actions.
3
Spreadsheet agents need high-fidelity formula and rendering engines, along with deterministic evaluation where possible, because infrastructure errors can look like model failures.
Summary
Nuno Campos describes four months of work teaching coding agents to handle spreadsheets. The team moved from about 50% to 92% accuracy on an internal financial analysis benchmark. A rigid multi-agent design and many spreadsheet representations failed to work on their own. The largest improvement came from replacing roughly 15 separate tools with a persistent JavaScript REPL. The agent could combine operations in one call, preserve variables between calls, and reason between shorter scripts. Spreadsheet work also required a formula engine and a rendering engine so the agent could calculate results, inspect layout, and correct its work. Campos is direct about the limits: verification is only useful when the underlying engines are accurate, and apparent reasoning failures often come from bugs in prompts, tools, or infrastructure. He closes with lessons about interfaces, feedback loops, domain prompts, planning, evaluation, and trace inspection.
Spreadsheet work is difficult because structure and meaning are visual
Campos starts with the gap between opening an unfamiliar Excel file and giving that file to an LLM. A person immediately sees a revenue table, assumptions, and a chart. An LLM has to determine which revenue is meant, including whether it is net or gross and which quarter or year applies. It also has to tell whether a number is an input or the result of a formula. The task looks like structured data access, but the spreadsheet's layout and formatting carry information that a plain text representation does not capture.
A multi-agent workflow made planning errors easier to fix, but it was too rigid
The team first split the work across three agents. The central edit agent followed a five-step process: define the end state, plan, execute, and verify, with the other steps implied by that workflow. This changed when errors happened. Instead of making mistakes while building a financial model, the agent often made them during planning, which was easier to correct. The design still failed because discovery ran only once at the beginning, could not be revisited, and did not pass context cleanly between agents.
Spreadsheet representations became useful only as parts of a larger interface
The team tried many ways to represent spreadsheets to an LLM. SQL seemed promising because models see a lot of it in training and it fits structured data. XML seemed plausible because Excel files use it on disk. Neither worked as a complete representation. CSV and TSV views of selected spreadsheet areas were useful inside the eventual REPL, even though they were insufficient as the only interface. HTML helped expose layout and formatting, which led the team to build a rendering engine that showed the agent an image of the rendered spreadsheet.
A persistent JavaScript REPL replaced a collection of separate tools
The largest improvement came when the team replaced about 15 accumulated tools with one Node.js REPL. The former tools became JavaScript functions that the agent could combine in a single call. JavaScript was chosen because it is familiar to language models and relatively easy to sandbox, while the spreadsheet implementation remained in C#. Before the change, exploration commonly required 10 to 15 sequential tool calls and often timed out. With the REPL, the agent could request several operations together and receive the results in one call.
Persistent state lets the agent reason between smaller actions
Campos distinguishes a REPL from ordinary code mode by its persistent state. The agent can define variables in one call, inspect the results, spend more reasoning tokens, and reuse those variables in a later call. Without persistent state, agents often wrote scripts around 50 lines long. With the REPL, they wrote shorter scripts and interleaved reasoning with actions more often. That made the process less static and often helped the agent reach a better answer faster. Adding a new spreadsheet method also became simpler: the team exposed it in the JavaScript REPL and supplied TypeScript definitions in the prompt.
The interface change drove most of the benchmark improvement
On the internal benchmark, accuracy rose from 50% before the REPL to 74% after it, then reached 92% as the team added further changes. Later work included better fuzzy search, formula-tracing functions for dependencies, system-prompt improvements, and bug fixes. Campos says none of those later changes was as dramatic as the REPL, but their effects accumulated. The same design also eliminated the usual five-minute task timeouts in the team's runs, because the agent could explore and operate on spreadsheets more efficiently.
Spreadsheet agents need a calculation loop and a visual verification loop
Campos compares spreadsheet work with coding agents that can run a compiler, linter, or tests and then revise their code. For spreadsheets, Witan Labs built a formula engine to calculate formulas and a rendering engine to turn a formatted range into an image. These engines give the agent a source of truth for checking its work. The agent can detect a wrong result, formula, or formatting choice and revise it. The approach depends on high fidelity. An incomplete formula engine can produce worse behavior because the agent may reject a correct formula after receiving an incorrect result or an unsupported-function error.
Evaluation and trace inspection separate model errors from system errors
The team began with an LLM acting as the judge, but scores could change because the evaluator changed its output rather than because the agent improved. Where possible, they replaced that process with deterministic comparisons. One example uses a golden spreadsheet with defined inputs and outputs. The evaluator puts the same inputs into the model-produced spreadsheet and checks whether the outputs match. Campos also warns that infrastructure bugs often look like reasoning failures. A bad prompt example, a tool failure, or a skill bug can make the model appear confused. Examining traces helps identify whether the model misunderstood the task or was responding faithfully to a broken system.
The durable lessons are feedback loops, suitable interfaces, and deterministic checks
Campos generalizes the spreadsheet work to other agent tasks. If an agent makes many sequential or parallel tool calls, the developer may have created a poor scripting language and should consider giving it code mode or a REPL. Domains without existing feedback loops may justify building a renderer, calculator, or equivalent checking system. Interfaces need deliberate design and later revision as model capabilities change. Planning still matters, as do simple fixes and domain-specific prompts that remind the model what to attend to. Deterministic evaluation is preferable when available, while LLM judges remain useful when no other evaluation method exists.
"If your agent is making many sequential tool calls or even parallel tool calls, then you've kind of invented a bad scripting language."16:31
Who should watch
You are building an agent that needs to work with spreadsheets or another structured domain where layout carries meaning.
Your agent makes many tool calls, times out, or struggles to combine results across calls.
You are evaluating agent improvements with an LLM judge and need a more dependable way to distinguish model changes from evaluator and infrastructure bugs.
