# How Codeium Breaks Through the Ceiling for Retrieval Kevin Hou, Codeium (Exafunction) | AI Engineer World's Fair 2024 | 18:42 Source: https://www.youtube.com/watch?v=DuZXbinJ4Uc Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks. Page: https://aietalks.com/talks/how-codeium-breaks-through-the-ceiling-for-retrieval Published: 2024-07-31 Tags: evals, latency, rag, reranking ## TL;DR - Codeium argues that embedding search struggles with code tasks because useful generation often requires several related files rather than one relevant passage. - The company evaluates retrieval with Recall@50 and datasets built from real pull requests, commit messages, and modified files. - Codeium's M-Query runs parallel model calls over candidate files, then combines those rankings with editor activity, directory context, and recent commits. ## Summary Kevin Hou explains why Codeium treats retrieval as the central problem in AI coding tools. Long context can read more code, but it brings latency and cost. Fine-tuning requires ongoing updates and separate models for customers. Embeddings are cheap and fast, yet they have trouble reasoning across several related files. Hou says common benchmarks reward finding one relevant passage, while code generation often needs a set of files, such as components, style rules, and documentation for a form. Codeium built a pull-request-based evaluation set and uses Recall@50 to measure how much of the relevant file set appears in retrieval results. Its M-Query system makes parallel language-model calls over candidate items, then adds signals such as active files, neighboring directories, and recent commits. Codeium supports this with custom models and its own infrastructure, which Hou says makes the extra computation affordable and fast enough for interactive use. ## Key ideas ### Context has to match the developer's codebase [01:31](https://www.youtube.com/watch?v=DuZXbinJ4Uc&t=91s) Hou says a generic request to ChatGPT for a React contact form will not fit a moderately large codebase. The system needs to retrieve local buttons and inputs, match other forms in the repository, follow the project's style guide, and use documentation for packages and dependencies. Tailwind conventions are one example of local context that affects how the result should look. The retrieval problem is therefore deciding which pieces to collect and how to rank them before sending them to the language model. Codeium's product goal is context awareness inside the IDE, where the same need applies to autocomplete, chat, and search. ### Long context and fine-tuning each impose practical costs [03:02](https://www.youtube.com/watch?v=DuZXbinJ4Uc&t=182s) Hou describes three ways teams try to improve context. Long-context models accept more input, but they add latency and financial cost. He gives Gemini's processing of 325,000 tokens as taking 36 seconds, while a small or moderate repository can exceed one million tokens. Fine-tuning changes model weights toward a customer's data distribution, but it needs continuous updates, significant computation, and potentially one model per customer. Embeddings are cheaper to compute and store, but Hou says they have limited dimensional space and struggle to reason over multiple items. Each approach handles part of the problem while leaving a constraint that matters for interactive coding. ### Embedding benchmarks measure the wrong retrieval task [04:34](https://www.youtube.com/watch?v=DuZXbinJ4Uc&t=274s) Hou argues that existing retrieval evaluations often resemble a needle-in-a-haystack test. They ask whether one relevant item can be found in a large corpus. Code generation usually needs several needles. His contact-form example requires components, related forms, style guidance, and documentation. Codeium uses Recall@50 instead, measuring what fraction of the ground-truth files appears among the top 50 retrieved items. This metric is intended to capture multi-document context in large codebases. Hou's criticism is about the shape of the task: semantic distance between vectors does not necessarily equal the relevance of a function to a coding request. ### Codeium builds evaluations from real pull requests [07:01](https://www.youtube.com/watch?v=DuZXbinJ4Uc&t=421s) To make retrieval testing resemble production, Codeium created its own dataset from pull requests. A pull request is broken into commits, and each commit can be matched with the files it changed. The commit message provides an English description, while the modified files provide a relevant-file set. Codeium can then test whether retrieval finds those files and run the evaluation at scale. Hou calls this a product-led benchmark because it connects retrieval changes to the experience of a code-generation product. He says publicly available models performed worse on this task because they struggled to reason about code and the real-world language found in commit messages. ### M-Query ranks candidates with parallel model reasoning [10:03](https://www.youtube.com/watch?v=DuZXbinJ4Uc&t=603s) M-Query takes a retrieval query and a codebase containing many candidate items. Instead of comparing only vectors, Codeium makes parallel language-model calls to reason over each item. Hou compares this to asking a model to answer yes or no about whether an item fits the query. The resulting scores provide a higher-confidence ranking, which Codeium combines with active files, neighboring directories, recent commits, and the current ticket. The system can then stream more relevant code into generation and chat. This design spends much more computation per request than ordinary embedding search, so its feasibility depends on Codeium controlling its models and serving stack. ### Vertical integration pays for the extra computation [08:59](https://www.youtube.com/watch?v=DuZXbinJ4Uc&t=539s) Hou gives three parts of Codeium's approach. The company trains its own models for its workflows. It builds custom infrastructure down to the metal through its connection with Exafunction, an ML infrastructure company. It also judges changes through real product usage rather than relying only on research benchmarks. He says this combination makes computation one hundredth the cost of competitors that use APIs, allowing Codeium to give users one hundred times as much compute. The product still has to be fast, powerful, and easy to use. Hou says M-Query runs thousands of language models in parallel so code can begin streaming within seconds, and often milliseconds. ### Production feedback closes the retrieval iteration loop [12:27](https://www.youtube.com/watch?v=DuZXbinJ4Uc&t=747s) Codeium rolled M-Query out to a small percentage of its large installed user base and tested it on monorepos and remote repositories. Hou says the system selected files relevant to the requested generation while remaining fast. The team saw more positive chat feedback, more accepted generations, and more code written for users. Codeium's iteration cycle starts with product data and evaluations, applies additional compute to improve the product, and then checks the result in production through signals such as thumbs up and thumbs down. Hou says the same approach will support future features such as documentation, commit messages, code reviews, code scanning, and converting Figma designs into interfaces built from a user's components. ### AI-first products should plan for more available compute [15:48](https://www.youtube.com/watch?v=DuZXbinJ4Uc&t=948s) Hou compares retrieval to autonomous driving. In 2015, teams compensated for limited hardware with sensor fusion, lower polling rates, and offboard models. As more compute became available in vehicles, larger models could process more sensors and data. He uses that history to argue that embedding-only retrieval should be treated as a temporary heuristic for AI products. Codeium's position is to plan for larger models and make AI a first-class part of the system. He also says ideas matter less than shipping them in a product, so the company tests its retrieval approach through the Codeium extension rather than presenting the theory alone. ## Notable quotes - "The short answer is context awareness." (01:31) - "Code search requires multiple different needles." (06:10) - "We are literally taking models and running them on each one of those items." (10:38) - "Ideas are cheap." (17:17) ## Tools & references mentioned - Codeium - Exafunction - Gemini - ChatGPT - GitHub Copilot - Tailwind - Recall@50 - React - ShadCN - TechCrunch - Stack Overflow ## Who should watch - You are building code search or retrieval for an AI coding product and need an evaluation that reflects several relevant files rather than one passage. - Your retrieval system depends on embeddings, and you want to understand the tradeoff between vector search, long context, fine-tuning, and model-based reranking. - You run models in an interactive developer tool and need ideas for controlling the cost and latency of heavier retrieval computation. ## Related talks - [Move Fast Break Nothing](https://aietalks.com/talks/move-fast-break-nothing) (Dedy Kredo, CodiumAI, 13:01) - [Benchmarking Semantic Code Retrieval on Claude Code](https://aietalks.com/talks/benchmarking-semantic-code-retrieval-on-claude-code) (Kuba Rogut, Turbopuffer, 16:08) - [We Cut 94% of AI Coding Tokens With a Local Code Index](https://aietalks.com/talks/we-cut-94-of-ai-coding-tokens-with-a-local-code-index) (Rajkumar Sakthivel, Tesco, 10:43) - [AI-powered entomology: Lessons from millions of AI code reviews](https://aietalks.com/talks/ai-powered-entomology-lessons-from-millions-of-ai-code-reviews) (Tomas Reimers, Graphite, 10:21) - [What Data from 20m Pull Requests Reveal About AI Transformation](https://aietalks.com/talks/what-data-from-20m-pull-requests-reveal-about-ai-transformation) (Nick Arcolano, Jellyfish, 17:57)