# Benchmarking Semantic Code Retrieval on Claude Code

Kuba Rogut, Turbopuffer | AI Engineer Europe 2026 | 16:08

Source: https://www.youtube.com/watch?v=zKk7sDMGDEQ
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/benchmarking-semantic-code-retrieval-on-claude-code
Published: 2026-06-03
Tags: agents, benchmarks, coding-agents, embeddings, rag

## TL;DR
- Claude Code's default exploratory file reading produces lower precision than windowed reads, and adding semantic search raises file precision further.
- Semantic search finds behaviorally related files without shared keywords, while grep works better when a task involves tracing explicit imports or keywords.
- Claude Code gains less from semantic search than Cursor because Claude Code does not know when or why to call the extra tool.

## Summary
Kuba Rogut presents a 50-task benchmark of code retrieval in Claude Code. The benchmark measures whether the agent found the files, lines, and symbols needed during a task, rather than whether it ultimately solved the task. He compares default Claude Code, reads capped at 50 lines, and capped reads with a semantic search tool backed by Turbopuffer. Windowed reads reduce irrelevant file access, while semantic search raises file precision from 65% to 87%. The two retrieval methods help with different tasks. Semantic search finds related files whose names share few keywords, while grep is stronger for direct import tracing. Rogut explains that Claude Code treats semantic search as an extra tool, so it does not choose it as effectively as Cursor's Composer model. He also describes semantic retrieval as cached computation, with the upfront work of chunking, embedding, and indexing code reused across sessions.

## Key ideas
### Claude Code defaults to exploratory grep-based search
[00:47](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=47s)
Kuba says Claude Code does not use semantic code search by default. Boris, whom he describes as a founding figure behind Claude Code, has said that early versions used a local vector database, but agentic search through the file system proved simpler and worked better for Claude Code. Cursor took a different path and indexes code bases for semantic search. Rogut connects that choice to Cursor's reported production gains, including a 24% relative improvement in answer accuracy for its Composer model and a 2.6% increase in code retention in large code bases. He also cautions that these numbers look modest because many requests do not need semantic retrieval.

### Semantic indexing turns repeated search work into reusable computation
[02:31](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=151s)
Rogut describes embeddings as cached computation. With ordinary file-system search, an agent greps, reads files, and searches again whenever it needs to understand a code base. That work repeats across sessions and across agents, even when they ask similar questions. Semantic retrieval has an upfront cost: the code base must be chunked, embedded, and indexed. After that, an agent can query the stored meaning, such as asking how metadata filtering works, and retrieve relevant chunks directly. Rogut says the savings from one request may be small, but they accumulate when several agents repeatedly work against the same code base.

### Turbo Grep adds semantic retrieval to Claude Code
[04:08](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=248s)
Turbopuffer built a command-line tool called Turbo Grep for Claude Code. It walks the file system, parses and chunks the code with a tree splitter library, embeds it with Voyage's code model, and uploads the results to Turbopuffer. Rogut shows a tool trace from the Django repository in which Claude Code calls the search tool for a password reset token generator. The returned contents give the agent enough context to explain the relevant code. He presents this example as a reason to measure the approach rather than assume that the tool works better.

### The benchmark measures retrieval behavior instead of task completion
[05:09](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=309s)
The benchmark is based on ContextBench, a human-labeled dataset that identifies the files, lines, and symbols an agent should inspect while completing a task. It does not score only whether the coding agent reached a working solution. Rogut tests 50 tasks under three conditions: unmodified Claude Code, Claude Code with reads capped at 50 lines, and capped reads plus the Turbopuffer search tool. He limits reads because long files made the comparison noisy. If an agent reads a thousand-line file, the benchmark cannot distinguish useful context from a large amount of irrelevant context as clearly.

### Windowed reads and semantic search raise file precision
[06:37](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=397s)
Rogut defines precision as the share of files read that were actually needed. The default condition reaches 65% file precision, 33% line precision, and 43% symbol precision. Adding windowed grep improves the results, and adding semantic search reaches 87% file precision. In plain terms, default Claude Code wastes about one in every three file reads. Windowed grep reduces that to about one in five, and windowed grep with semantic search reduces it to about one in eight. Rogut attributes the weak default precision partly to Claude Code's tendency to explore broadly and read as much as possible.

### Semantic search improves precision without improving every recall measure
[07:27](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=447s)
Recall measures how much of the needed context the agent found. Default Claude Code wins file recall because it explores many files, although its line recall falls because much of that exploration concerns files without the labeled context. Windowed grep and windowed grep plus semantic search have similar recall in the aggregate results. Rogut does not treat that as evidence that semantic search failed. When he separates the tasks, the methods show different strengths. Semantic search helps with behavior-adjacent files that share few keywords, while grep helps when the task follows imports and the relevant keyword appears immediately.

### Claude Code uses semantic search less effectively than Cursor
[09:58](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=598s)
Rogut says the benchmark gains are smaller than Cursor's because Claude Code is built around grep and receives semantic search as another item in its tool list. The model is not given a strong understanding of when or why to use it. Cursor's Composer model has semantic search built in and knows when and how to call it. Rogut links that tool-selection difference to Cursor's reported 23.5% performance gain in the discussion, while the earlier slide cites a 24% relative improvement for Composer. His conclusion is that agents need lightweight ways to retrieve different kinds of context, rather than relying on one search method.

### Inline comments make code easier for semantic retrieval
[12:07](https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=727s)
In the questions, Rogut says semantic search works best when code contains comments or inline documentation. Comments above a function give both the embedding model and the agent more information about the code's meaning. The benchmark used raw code, without a more elaborate parent-child representation. Rogut says a production system could attach higher-level descriptions, such as an authentication flow, to lower-level code. He also says Cursor has its own embedding model and may create artificial comments above code before embedding it. That can translate raw implementation details into queries closer to how a person describes a behavior.

## Notable quotes
- "The process really matters for understanding like are agents actually looking for the right files?" (05:27)
- "You can't just grep through everything unfortunately in a file system." (10:47)
- "The embedding it and is not the hard part. It's like figuring out what meaning really is of that chunk." (12:52)
- "People love grepping because it's zero cost." (14:28)

## Tools & references mentioned
- Turbopuffer
- Claude Code
- Cursor
- Anthropic
- Boris
- ContextBench
- Voyage Code 3
- Django
- Notion
- Composer

## Who should watch
- You are building coding-agent retrieval and need a benchmark that scores the files, lines, and symbols an agent inspects.
- Your agent reads too broadly and you want to compare capped file reads, grep, and semantic search.
- You are deciding whether semantic retrieval should supplement grep, especially for code bases with sparse naming signals or useful inline documentation.

## Related talks

- [Claude Code & the evolution of agentic coding](https://aietalks.com/talks/claude-code-the-evolution-of-agentic-coding) (Boris Cherny, Anthropic, 18:12)
- [How Claude Code Works](https://aietalks.com/talks/how-claude-code-works) (Jared Zoneraich, PromptLayer, 1:05:43)
- [How Codeium Breaks Through the Ceiling for Retrieval](https://aietalks.com/talks/how-codeium-breaks-through-the-ceiling-for-retrieval) (Kevin Hou, Codeium (Exafunction), 18:42)
- [Boris explains Claude Code](https://aietalks.com/talks/boris-explains-claude-code) (Boris, Anthropic, 00:56)
- [How We Taught Agents to Use Good Retrieval](https://aietalks.com/talks/how-we-taught-agents-to-use-good-retrieval) (Hanna Lichtenberg, Mixedbread AI, 14:28)