# Move Fast Break Nothing

Dedy Kredo, CodiumAI | AI Engineer Summit 2023 | 13:01

Source: https://www.youtube.com/watch?v=oJz4wveQ4oM
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/move-fast-break-nothing
Published: 2023-11-08
Tags: code-generation, evals, security, testing

## TL;DR
- Code generation needs a separate critic that checks whether generated code matches the developer's intent.
- Behavior coverage can reveal what code does more usefully than line-based code coverage.
- CodiumAI uses generated tests, automated fixes, code suggestions, and pull request review across the development lifecycle.

## Summary
Dedy Kredo argues that code generation needs a critic alongside the generator. Drawing on the idea behind generative adversarial networks, he describes CodiumAI's Code Integrity component, which analyzes generated code, maps its behaviors, identifies edge cases, and creates tests. He presents behavior coverage as a more useful measure than code coverage because it focuses on what the code actually does. In a live VS Code demonstration, Codium analyzes a 600-line scraping class with no tests, creates tests for happy paths and edge cases, runs them, and tries to fix a failing test. Kredo then uses the generated test suite while applying a performance suggestion, before showing commit-message generation and pull request review. The review finds an exposed API key and suggests a fix. The demo presents testing as protection for code changes, especially as AI agents produce more code.

## Key ideas
### Code generation needs a separate critic
[00:42](https://www.youtube.com/watch?v=oJz4wveQ4oM&t=42s)
Kredo says Transformers made generative models state of the art for many uses, but code has more nuance. For code that works as intended, he prefers a system modeled on the structure of a generative adversarial network. One component generates code, while another critic analyzes the result. CodiumAI calls this critic the Code Integrity component. It reviews generated code, looks for edge cases, and checks the result against the developer's intent. The focus of CodiumAI's product is this reviewing and testing side rather than code generation alone.

### Behavior coverage describes what the code does
[01:35](https://www.youtube.com/watch?v=oJz4wveQ4oM&t=95s)
CodiumAI maps the behaviors of a codebase and uses those behaviors to create tests. Kredo says behavior coverage is more useful than ordinary code coverage because it focuses on the cases the code handles, rather than only which lines executed. The product helps developers understand behavior, generate tests, improve code, and review changes. Kredo says these capabilities are available through extensions for JetBrains and VS Code, a GitHub plugin, and planned APIs for embedding the functions in other agents.

### Generated tests protect a project with no existing tests
[03:10](https://www.youtube.com/watch?v=oJz4wveQ4oM&t=190s)
Kredo demonstrates Codium on AO AutoScraper, an open-source scraping project with more than 5,000 GitHub stars. The class he opens has about 600 lines and no tests, which makes changes risky because nothing protects the project from regressions. Codium analyzes the class, maps its different behaviors, streams generated tests, and produces a detailed explanation of how the class works. The generated cases include ordinary successful inputs and edge cases. Kredo can inspect each behavior, view sub-behaviors, and choose individual cases for additional tests.

### Test failures trigger an automated reflection loop
[05:39](https://www.youtube.com/watch?v=oJz4wveQ4oM&t=339s)
After selecting tests, Kredo runs the suite and shows that a complex test fails. Codium analyzes the failure, reflects on what went wrong, and generates a fix automatically. The revised test passes on the second attempt. Kredo describes this as a chain-of-thought or reflection process aimed at producing a higher-quality test suite. He then saves the tests into the project, where they can protect later code changes.

### The test suite makes code improvements safer
[06:49](https://www.youtube.com/watch?v=oJz4wveQ4oM&t=409s)
Kredo next asks Codium to improve the class's build method, which contains much of its functionality. The suggestions go beyond linting and include performance, security, best practices, and readability. One recommendation replaces a hash function with Blake3 for performance. Kredo applies the change and runs the generated test suite. A test breaks after the modification, showing how the tests expose the effect of a code change, although he moves on from that failure during the demo.

### Review continues through the pull request
[08:59](https://www.youtube.com/watch?v=oJz4wveQ4oM&t=539s)
The workflow continues after testing and code changes. Codium generates a commit message, then reviews the pull request. The review summarizes the change, gives it a score, and checks for issues the developer may have missed. In the demonstration, the review identifies a possible security vulnerability: an API key was left in the code. The improve action suggests a fix and links Kredo to the location where the key appears. The GitHub plugin provides similar functions inside GitHub.

## Notable quotes
- "Code is very, very nuanced." (00:42)
- "We believe that behavior coverage is a more useful metric than actual code coverage." (01:35)
- "Now I have a test suite that protects me." (06:49)
- "This is a chain of thought, this reflection process in order to get to a high quality test suite." (06:05)

## Tools & references mentioned
- CodiumAI
- Code Integrity
- Generative adversarial networks
- Transformers
- AO AutoScraper
- VS Code
- JetBrains
- GitHub
- Blake3

## Who should watch
- You are adding AI-generated code to an existing project and need tests that describe its actual behavior.
- Your codebase has little or no test coverage, so routine changes can introduce regressions without being noticed.
- You want one workflow for generating tests, improving code, and checking pull requests for issues such as exposed API keys.

## Related talks

- [Don't get one-shotted: Use AI to test, review, merge, and deploy code](https://aietalks.com/talks/dont-get-one-shotted-use-ai-to-test-review-merge-and-deploy-code) (Tomas Reimers, Graphite, 05:45)
- [How Codeium Breaks Through the Ceiling for Retrieval](https://aietalks.com/talks/how-codeium-breaks-through-the-ceiling-for-retrieval) (Kevin Hou, Codeium (Exafunction), 18:42)
- [Beyond the Prototype: Using AI to Write High-Quality Code](https://aietalks.com/talks/beyond-the-prototype-using-ai-to-write-high-quality-code) (Josh Albrecht, Imbue, 17:59)
- [Code Generation and Maintenance at Scale](https://aietalks.com/talks/code-generation-and-maintenance-at-scale) (Morgante Pell, Grit, 18:54)
- [Guide, Verify, Solve](https://aietalks.com/talks/guide-verify-solve) (Anirban Chatterjee, Sonar, 22:31)