# We Cut 94% of AI Coding Tokens With a Local Code Index Rajkumar Sakthivel, Tesco | AI Engineer World's Fair 2026 | 10:43 Source: https://www.youtube.com/watch?v=dRmWYHuIJxM Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks. Page: https://aietalks.com/talks/we-cut-94-of-ai-coding-tokens-with-a-local-code-index Published: 2026-06-28 Tags: agents, coding-agents, context-engineering, cost, rag ## TL;DR - AI coding costs were driven mainly by sending irrelevant input context, rather than by model output. - A local retrieval layer reduced the tested context from 83,000 tokens to 4,900 per question, with further compression reaching 523 tokens. - Combining meaning search, keyword search, code relationships, and a simple relevance heuristic produced a practical shared index for multiple coding tools. ## Summary Rajkumar Sakthivel describes how his team responded to a sudden increase in its AI coding bill. Their investigation found that coding tools were sending about 45,000 tokens per query even though roughly 5,000 were useful. Prompt changes, model settings, and shorter outputs did not address the input cost. The team built a local code index that splits code into AST-aware units, combines semantic and keyword search, follows relationships between functions, compresses retrieved code, and rejects low-scoring results. On a FastAPI test set, the system reduced context from 83,000 to 4,900 tokens per question, or 523 tokens with compression, while finding the right code in 90% of cases. Sakthivel is candid about weaker recall on large, complicated files and the trade-off of using a smaller, faster search model. The same index can support Cloud Code, Cursor, and Copilot, with project knowledge shared between sessions. ## Key ideas ### Most coding-tool cost comes from irrelevant input context [00:27](https://www.youtube.com/watch?v=dRmWYHuIJxM&t=27s) Rajkumar says the team's bill rose even though its project and tools had not changed. They found that the tools were sending much more code than the model needed. A typical query sent 45,000 tokens, while only about 5,000 mattered. The unused 40,000 tokens were still paid for on every request. He compares this with ordering a pizza and paying for nine extra pizzas that go uneaten. The team first tried shorter prompts, model settings, and output compression. Shorter prompts came after the context had already been sent, model settings changed the response rather than the input, and output was only about 10% of the cost. ### Reducing input has a much larger effect than shortening output [02:26](https://www.youtube.com/watch?v=dRmWYHuIJxM&t=146s) The talk separates AI cost into input and output. Rajkumar gives the team's estimate that input accounts for 90% of the cost, including files, search results, and other context. Output accounts for the remaining 10%. Cutting output by 75% therefore saves only about 8% overall. Cutting input by 94% can save about 61% overall under the same calculation. This led the team to put a local search layer between the codebase and the AI tool. Instead of sending complete files, the layer searches an index and returns smaller pieces of code that match the request. ### The index uses structured code chunks and two complementary searches [03:25](https://www.youtube.com/watch?v=dRmWYHuIJxM&t=205s) The system reads code and splits it into functions, classes, and methods rather than arbitrary text chunks. It then runs meaning-based search and exact keyword search at the same time. Semantic search can find related ideas but miss exact names, such as returning a different authentication function for a search for 'authenticate user'. Keyword search finds names precisely but can miss related wording, such as 'sign in' when the query says 'login flow'. Rajkumar says each method misses about one in four results on its own, while the combined approach misses about one in ten. ### Retrieval quality depends on rejecting results that only look relevant [05:39](https://www.youtube.com/watch?v=dRmWYHuIJxM&t=339s) The difficult part was deciding when retrieval was wrong. The search could return ten results with none of them useful, which could lead to a confident but incorrect answer. The team tried asking another AI model to judge the results, but that added two or three seconds to each request. A fixed score threshold also failed because short questions could receive low scores despite being exact matches. Their final heuristic combines a 50% meaning score, a 30% keyword score, and a 20% recency score. The threshold changes with the current result set. Rajkumar says it runs in 0.4 milliseconds without another AI call. ### Compression reduces retrieved code after search has found the right area [03:51](https://www.youtube.com/watch?v=dRmWYHuIJxM&t=231s) The retrieval layer can shrink its results after finding relevant code. It may keep only a function name and description, reducing a 50-line function to five lines. The system also follows relationships between functions, such as which function calls which other function. This lets it expand from one retrieved code unit to connected code when the request requires it. Finally, every result receives a score, and results below the threshold are left out. The whole process runs locally, so the code and retrieval requests do not go to the cloud. ### The benchmark shows large savings with a stated accuracy trade-off [06:37](https://www.youtube.com/watch?v=dRmWYHuIJxM&t=397s) Rajkumar reports a test on FastAPI, an open-source project with 53 files and 20 real developer questions. Without the tool, the test used 83,000 tokens per question. The retrieval system reduced that to 4,900 tokens, a 94% reduction. Additional compression brought it to 523 tokens per question, while the system still found the right code in 90% of cases. He qualifies the headline result because the baseline read full files every time, while tools such as Cloud Code can already use smarter context selection. The public test is available to run. ### The approach works best when files have focused responsibilities [07:38](https://www.youtube.com/watch?v=dRmWYHuIJxM&t=458s) The 94% figure is not presented as a universal production result. Rajkumar says real savings are lower when existing tools already select context intelligently. On larger projects with 396 files, recall dropped close to zero. The system worked well when each file did one thing, but struggled when files contained many unrelated responsibilities. The team also used a small, fast model for search. Re-indexing took under a second, while a larger model might find more. They chose speed over maximum retrieval quality and kept the surrounding infrastructure small. ### One local index can share code knowledge across different AI tools [08:38](https://www.youtube.com/watch?v=dRmWYHuIJxM&t=518s) Rajkumar says the team uses Cloud Code for difficult problems, Cursor for quick edits, and Copilot for small completions. These tools normally start each session without shared project knowledge, so developers have to explain the same codebase repeatedly. The team built one shared index that all of the tools can use. It also stores project knowledge learned by one tool so another tool can use it in a later session. On a real project, 247 queries saved 12.4 million tokens, and the team calculated that nearly £186 was not spent. The search layer produced 84% of those savings, with compression producing the rest. ## Notable quotes - "Most of the money was not the AI thinking. Most of it was sending too much context." (00:27) - "Fix the input. That's where your money goes." (02:56) - "The lesson we learned, simple formula beats the complex model most of the time." (06:04) - "The answer was not a better model. The answer was sending less." (10:05) - "We explain the code base once. Every tool remembered." (09:06) ## Tools & references mentioned - Foss - Cloud Code - Cursor - Copilot - Code X - FastAPI - tree-sitter - CCE - Opus - Sonnet ## Who should watch - You are paying for AI coding requests and suspect that irrelevant files or search results make up most of the input. - You are building retrieval for a codebase and need practical choices for chunking, hybrid search, scoring, and local execution. - Your developers use several AI coding tools and want shared project context without sending source code to a cloud service.