Booking.com measures AI coding tools through developer speed, review time, code quality, test coverage, and progress on replatforming its codebase.
2
Agents helped Booking.com investigate large GraphQL schemas and legacy functions by searching for relevant context and breaking migrations into smaller tasks.
3
Developer education had a direct effect on adoption. People who stopped using Cody often lacked the knowledge to provide the right prompt and context.
Summary
Bruno Passos and Beyang Liu describe a partnership between Booking.com and Sourcegraph to automate repetitive work across a large software organization. Booking.com has more than 3,000 developers, a bloated codebase shaped by years of experiments and feature flags, and a replatforming effort expected to take years without automation. The company moved beyond the weak metric of self-reported time saved and defined measures for lead time, code quality, codebase health, and developer experience. Booking.com found that daily Cody users were more than 30% faster and shipped 30% more merge requests. The teams built agents for a GraphQL schema that exceeds a million tokens, large-scale migration of legacy functions, and organization-specific code review. They also discuss declarative service rules that could move CI feedback into the editor and eventually help services repair themselves. The speakers are honest that the work remains experimental. They place major weight on training developers and helping whole teams test the tools in focused workshops and hackathons.
Booking.com's codebase accumulated toil from years of experimentation
Booking.com runs more than 3,000 developers, over 250 merge requests, and 2.5 million CI jobs in a year. Its data-driven product process created many experiments and feature flags. Bruno Passos says those flags and dead code often stayed in the codebase, which became bloated over decades. As the codebase grew, cycle times increased and developers spent more than 90% of their time on toil. Developer surveys showed that working in the codebase was becoming harder. The business needed to clear away this maintenance work so developers could spend more time on new features and user problems.
The team replaced weak adoption stories with measurable operating metrics
Booking.com initially gave all 3,000 developers access to Cody and tracked whether people said they had saved time. Bruno calls broad claims about thousands or hundreds of thousands of hours saved "semi BS" when they are based on only a few developers. The team defined four KPIs in October that could show results within a year: lead time for change, code quality, codebase health, and developer experience. By the end of November, they had traces showing that daily Cody users were more than 30% faster. Those users also shipped 30% more merge requests, and their merge requests contained less code.
Model choice and training changed whether developers kept using Cody
Booking.com first used Sourcegraph Search to find context in its large codebase, then started experimenting with Cody because Cody could use that search context. At first, the company had one available language model and token limits. Sourcegraph helped provide multiple models, which mattered because the team found that models had different strengths. One model could work better on codebase excavation, while another was better for developing a new service. Training also changed adoption. Developers who had tried Cody and stopped often did not know how to provide the right prompt or context. After training, some became daily users.
Agents grew out of developers composing repeated model calls
The teams noticed that developers who used the coding assistant were also experimenting with its underlying APIs. They began composing prompts and API calls into longer automations, which they called agents. Early attempts were difficult because people needed a clear understanding of what a language model could and could not do. Sourcegraph and Booking.com then held a weeklong joint hackathon in Amsterdam. The first result was an agent for generating GraphQL code from Booking.com's very large API schema. The teams used search to find relevant schema nodes, walked up the schema tree for parent context, and then generated a response.
Large migrations became manageable when the system divided the codebase
Booking.com explored an agent for migrating legacy code, including functions with more than 10,000 lines. The approach combined code search, structured metadata, a meta-prompt, and a divide-and-conquer process. Before automating the work, developers spent months understanding the size of the migration and identifying its call sites. During a hackathon, the teams defined the problem and found smaller pieces where they could begin. Bruno says that this made it possible to collect available low-hanging fruit instead of treating the entire replatforming effort as one task. The work was still experimental, but it gave the team a clearer view of the codebase.
Organization-specific rules make code review more useful
Beyang Liu says code review is a common enterprise problem, but the rules differ across organizations. Booking.com and Sourcegraph found that many off-the-shelf review tools were not customizable enough for the long tail of local standards and guidelines. Their review agent lets a team define rules in a simple flat-file format. The agent selects the rules relevant to the files changed in a pull request and posts comments that follow those rules. The team optimized for precision over recall, so the agent would provide selective feedback instead of creating noise.
Declarative service rules could move fixes from CI into the editor
The speakers propose declaring the rules that must hold for each service. Instead of waiting for a CI pipeline to report an error, an agent could anticipate the problem in the editor and provide a fix. They describe this as a possible path toward self-healing services. Senior engineers, architects, and other technical leaders could define constraints that apply across a codebase. Those rules could be enforced during review and inside the editor for code written by people or by AI. Compliance requirements and other service-specific knowledge could also be expressed as rules.
Education was the strongest adoption practice they found
After a year of work, Bruno says education was the most important lesson. The teams trained developers and worked with entire business units. They then gave those groups focused workshops and hackathons where people could experiment with the tools in two days. Developers came out of those sessions excited about what the agents could do and more likely to become daily users. Bruno connects that daily use to the observed increase in speed. He advises teams adopting these tools to teach developers how to work with language models instead of assuming that access alone will produce value.
"We came within two days within a hackathon we were able to really define and understand where the call sites were coming from and being able to define how big the problem is was important for us to be able to have a start point."15:02
Who should watch
You are responsible for AI adoption in a large engineering organization and need measures that executives can understand.
Your team is considering agents for code migration, code review, or internal developer tools in a large, difficult codebase.
Developers have access to coding assistants but adoption is uneven, and you need practical ideas for training and evaluation.
