Code generation needs a separate critic that checks whether generated code matches the developer's intent.
2
Behavior coverage can reveal what code does more usefully than line-based code coverage.
3
CodiumAI uses generated tests, automated fixes, code suggestions, and pull request review across the development lifecycle.
Summary
Dedy Kredo argues that code generation needs a critic alongside the generator. Drawing on the idea behind generative adversarial networks, he describes CodiumAI's Code Integrity component, which analyzes generated code, maps its behaviors, identifies edge cases, and creates tests. He presents behavior coverage as a more useful measure than code coverage because it focuses on what the code actually does. In a live VS Code demonstration, Codium analyzes a 600-line scraping class with no tests, creates tests for happy paths and edge cases, runs them, and tries to fix a failing test. Kredo then uses the generated test suite while applying a performance suggestion, before showing commit-message generation and pull request review. The review finds an exposed API key and suggests a fix. The demo presents testing as protection for code changes, especially as AI agents produce more code.
Kredo says Transformers made generative models state of the art for many uses, but code has more nuance. For code that works as intended, he prefers a system modeled on the structure of a generative adversarial network. One component generates code, while another critic analyzes the result. CodiumAI calls this critic the Code Integrity component. It reviews generated code, looks for edge cases, and checks the result against the developer's intent. The focus of CodiumAI's product is this reviewing and testing side rather than code generation alone.
CodiumAI maps the behaviors of a codebase and uses those behaviors to create tests. Kredo says behavior coverage is more useful than ordinary code coverage because it focuses on the cases the code handles, rather than only which lines executed. The product helps developers understand behavior, generate tests, improve code, and review changes. Kredo says these capabilities are available through extensions for JetBrains and VS Code, a GitHub plugin, and planned APIs for embedding the functions in other agents.
Generated tests protect a project with no existing tests
Kredo demonstrates Codium on AO AutoScraper, an open-source scraping project with more than 5,000 GitHub stars. The class he opens has about 600 lines and no tests, which makes changes risky because nothing protects the project from regressions. Codium analyzes the class, maps its different behaviors, streams generated tests, and produces a detailed explanation of how the class works. The generated cases include ordinary successful inputs and edge cases. Kredo can inspect each behavior, view sub-behaviors, and choose individual cases for additional tests.
Test failures trigger an automated reflection loop
After selecting tests, Kredo runs the suite and shows that a complex test fails. Codium analyzes the failure, reflects on what went wrong, and generates a fix automatically. The revised test passes on the second attempt. Kredo describes this as a chain-of-thought or reflection process aimed at producing a higher-quality test suite. He then saves the tests into the project, where they can protect later code changes.
Kredo next asks Codium to improve the class's build method, which contains much of its functionality. The suggestions go beyond linting and include performance, security, best practices, and readability. One recommendation replaces a hash function with Blake3 for performance. Kredo applies the change and runs the generated test suite. A test breaks after the modification, showing how the tests expose the effect of a code change, although he moves on from that failure during the demo.
The workflow continues after testing and code changes. Codium generates a commit message, then reviews the pull request. The review summarizes the change, gives it a score, and checks for issues the developer may have missed. In the demonstration, the review identifies a possible security vulnerability: an API key was left in the code. The improve action suggests a fix and links Kredo to the location where the key appears. The GitHub plugin provides similar functions inside GitHub.
"This is a chain of thought, this reflection process in order to get to a high quality test suite."06:05
Who should watch
You are adding AI-generated code to an existing project and need tests that describe its actual behavior.
Your codebase has little or no test coverage, so routine changes can introduce regressions without being noticed.
You want one workflow for generating tests, improving code, and checking pull requests for issues such as exposed API keys.
