# AI Engineering at Jane Street

John Crepezzi, Jane Street | AI Engineer Summit 2025 | 16:57

Source: https://www.youtube.com/watch?v=0ML7ZLMdcl4
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/ai-engineering-at-jane-street
Published: 2025-03-28
Tags: agents, coding-agents, developer-experience, evals, reinforcement-learning

## TL;DR
- Jane Street built its own OCaml coding assistants because public models have too little OCaml training data and do not understand the firm's unusual tools and workflows.
- Workspace snapshots paired with build states provide training examples that connect developer context, natural-language requests, and small code changes.
- A shared sidecar called Aid keeps editor integrations thin, lets Jane Street swap models and prompts centrally, and supports evaluation through real build and test results.

## Summary
John Crepezzi describes how Jane Street builds LLM coding tools for an OCaml-heavy development environment. Public coding assistants perform poorly because OCaml has little training data, while Jane Street also uses custom systems, a Mercurial monorepo, its own build infrastructure, and editors such as Emacs. The team first defined a concrete task: generate an applicable, type-checking multifile diff from a developer's request. Existing feature descriptions and commits were too large or poorly shaped for this task, so the team collected workspace snapshots and build states. Red-to-green transitions exposed real fixes, and a language model supplied human-like descriptions. Jane Street then used a code evaluation service to apply generated diffs and check parsing, type checking, compilation, and tests. The same system supports training and held-out evaluation. A shared Aid sidecar handles context, prompts, models, and build status, while thin integrations support VS Code, Emacs, and Neovim.

## Key ideas
### Jane Street's OCaml environment makes standard coding assistants a poor fit
[00:50](https://www.youtube.com/watch?v=0ML7ZLMdcl4&t=50s)
Jane Street uses OCaml across much of its development work, including web applications through Js_of_ocaml, Vim plugins through Vaml, and FPGA code through HardCaml. Crepezzi says the language is powerful but obscure, with far less public training data than mainstream languages. The firm's internal OCaml code may exceed the total amount available outside the company. Its workflow also differs from common developer setups: Jane Street has its own build systems, distributed build environment, Iron code review system, Mercurial-based monorepo, and heavy Emacs usage. The team wants assistants that can work across this environment and support tasks such as merge-conflict resolution and reviewer selection.

### The team trained toward a specific task instead of merely exposing the model to company code
[05:10](https://www.youtube.com/watch?v=0ML7ZLMdcl4&t=310s)
Jane Street initially expected that showing an off-the-shelf model its code would produce a model familiar with the firm's libraries and idioms. Crepezzi says that approach failed because training examples must match the shape of the questions the model will receive. The team chose a concrete goal: a developer writes a request in an editor, and the model produces a potentially multifile diff. The diff should apply cleanly and have a good chance of type-checking. The target was changes of up to about 100 lines. Each example therefore needed three parts: the prior context, a prompt written in a human-like style, and a diff that accomplishes the request.

### Workspace snapshots produce small, realistic examples of developer changes
[08:13](https://www.youtube.com/watch?v=0ML7ZLMdcl4&t=493s)
Feature descriptions and commits did not provide the right training data. Features often contain several paragraphs and changes of 500 or 1,000 lines. Jane Street's commits are mainly checkpoints during development rather than isolated changes, and they lack useful descriptions. The team instead snapshots developer workstations throughout the day, roughly every 20 seconds, along with the current build status. A green-to-red-to-green sequence can reveal an isolated change. A red-to-green transition can show a developer correcting a type or compilation error. The team captures the error at the red state and the diff that returns the build to green. A language model then creates a detailed description, which is filtered down to something closer to what a person might write.

### Code quality can be turned into checks that guide both training and evaluation
[09:34](https://www.youtube.com/watch?v=0ML7ZLMdcl4&t=574s)
For OCaml, the team treats parsing, type checking, compilation, and passing tests as increasingly strong evidence that generated code is good. Its Code Evaluation Service, or CES, starts from a pre-warmed green build and has workers apply model-generated diffs. CES reports whether the resulting build is red or green, along with the error or success. Jane Street uses those outcomes during reinforcement learning so the model is pushed toward code that compiles and passes tests. The same setup can evaluate a held-out set: give the model a problem, apply its proposed change, and check whether the resulting code works. This gives the team a concrete test instead of relying only on subjective judgments.

### Meaningful evaluations prevent expensive training failures
[11:29](https://www.youtube.com/watch?v=0ML7ZLMdcl4&t=689s)
Crepezzi describes a code-review model that produced the response "I'll do it tomorrow" after months of training. The behavior came from human examples in which people wrote phrases such as "I'll do this tomorrow," rather than from an intentional review strategy. The incident shows why a model can appear plausible while failing its actual task. Jane Street treats evaluations as a way to catch this kind of drift before spending more time and money. The model must be judged by useful behavior, including whether generated changes work and whether review output resembles what developers need. Crepezzi is direct about the possibility of training producing results that are both costly and funny.

### A shared sidecar keeps editor integrations thin and centrally changeable
[12:40](https://www.youtube.com/watch?v=0ML7ZLMdcl4&t=760s)
Jane Street supports Neovim, VS Code, and Emacs, but the team did not want separate implementations of context construction, prompting, and model selection. It built Aid, a sidecar service on each developer's machine. Aid handles prompt and context construction, model calls, and build status, while each editor has a thin integration layer. When the team changes Aid, it can restart the service rather than wait for developers to update or restart their editors. The design also preserves flexibility. Jane Street can replace the model, change prompting, add editors, and provide domain-specific tools without rebuilding every integration.

### Editor design follows the habits of each editor's users
[14:07](https://www.youtube.com/watch?v=0ML7ZLMdcl4&t=847s)
The VS Code integration presents multifile diffs in a sidebar, in a style familiar to users of coding assistants. The Emacs integration takes a different approach because Emacs users work in text buffers and expect to move, copy, and edit text in their normal way. Aid appears in a Markdown buffer where users can ask questions, while key bindings append additional content at the bottom. The underlying service remains shared, but the interface fits each editor. This also lets Jane Street measure practical use. The team tracks latency and whether diffs apply, then can run experiments such as sending different model versions to different groups and comparing acceptance rates.

### The platform is intended to support future tools without rewriting the integrations
[15:39](https://www.youtube.com/watch?v=0ML7ZLMdcl4&t=939s)
Crepezzi says the shared architecture is useful because model and editor needs keep changing. Jane Street can add retrieval-augmented generation, multi-agent workflows, reasoning models, and domain-specific tools while keeping the same general structure. Different parts of the company can supply tools that become available across the editors through Aid. The team can also change a model or context-building method in one place and expose that change everywhere. The design reflects a practical priority: build a foundation that other engineers can extend, rather than hard-coding one assistant experience into each editor. Crepezzi says the same approach applies across these efforts: keep components pluggable, then give the rest of the company ways to add tools.

## Notable quotes
- "We wanted to be able to generate diffs given a prompt." (05:25)
- "Context, prompt, diff and we need a bunch of these examples." (06:30)
- "The real test of models is whether or not they work for humans." (12:24)
- "Every time something changes in large language models we're able to change it in one place downstream of the editors and then have it available everywhere." (15:39)

## Tools & references mentioned
- Jane Street
- OCaml
- Js_of_ocaml
- Vaml
- HardCaml
- Iron
- Mercurial
- Emacs
- Neovim
- VS Code
- Meta
- Code Compose
- Hack
- Code Evaluation Service
- Aid

## Who should watch
- You are building coding assistants for a language or codebase with little public training data and need a way to create task-shaped examples.
- Your team supports several editors and wants model, prompt, context, and evaluation logic to be shared instead of duplicated.
- You need generated code to be judged by real compilation and test outcomes rather than by output quality alone.

## Related talks

- [AI Engineering 201: The Rest of the Owl](https://aietalks.com/talks/ai-engineering-201-the-rest-of-the-owl) (Charles Frye, Full Stack LLM Bootcamp, 56:57)
- [Self Coding Agents](https://aietalks.com/talks/self-coding-agents) (Colin Flaherty, Augment Code, 17:23)
- [AIE Europe Keynotes & Coding Agents](https://aietalks.com/talks/aie-europe-keynotes-coding-agents) (Tejas Kumar, AI Engineer & Omar Sanseviero, Google DeepMind & David Soria Parra, Anthropic & Ido Salomon, MCP Apps & Mario Zechner, Pi & Armin Ronacher & Cristina Poncela Cubeiro, Earendil & Benjamin Dunphy, AI Engineer & David Gomes, Cursor & Matthias Luebken, TAVON & Sarah Chieng, Cerebras & Lawrence Jones, Incident.io & Luke Alvoeiro, Factory & Ben Burtenshaw, Hugging Face & Michael Richman, Cmd+Ctrl & Liam Hampton, Microsoft & Tuomas Artman, Linear & Gergely Orosz, The Pragmatic Engineer & Jacob Lauritzen, Legora & Peter Gostev, Arena AI & swyx, AI Engineer, 9:09:51)
- [Building a Chess Coach](https://aietalks.com/talks/building-a-chess-coach) (Anant Dole & Asbjørn Steinskog, Take Take Take, 18:22)
- [Claude Code & the evolution of agentic coding](https://aietalks.com/talks/claude-code-the-evolution-of-agentic-coding) (Boris Cherny, Anthropic, 18:12)