# Enhancing Quality and Security in CI

Gunjan Patel, Palo Alto Networks | AI Engineer World's Fair 2024 | 18:27

Source: https://www.youtube.com/watch?v=prttz9_1XDc
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/enhancing-quality-and-security-in-ci
Published: 2024-11-27
Tags: human-in-the-loop, security, testing, workflows

## TL;DR
- AI tasks that need patience and iteration can move from the developer's editor into CI.
- A sequence of prompts can clarify code intent, generate unit tests, and use project history to cover relevant edge cases.
- Security review can combine several AI roles with company context, followed by human approval of proposed fixes.

## Summary
Gunjan Patel describes a CI workflow for code that improves its readability, tests, and security after a developer checks it in. He distinguishes fast coding assistants, which need to answer within seconds, from a slower system that can iterate during a pull request. The workflow first improves variable names and comments so the code's intent is clearer. It then establishes expected behavior, asks an adversarial model to list edge cases, adds tests based on team priorities and historical bugs, and runs them in a loop. For security, the system uses project context and simulates a red-team engineer, a developer, and an engineering manager who debate the risks and effort of proposed fixes. The system suggests changes rather than applying them automatically. A human reviews the reasoning, policies, and code changes after baseline tests run. Patel presents this as a side project with a GitHub repository and planned GitHub Actions and GitLab CI files.

## Key ideas
### CI can take over repetitive work that interrupts coding
[01:43](https://www.youtube.com/watch?v=prttz9_1XDc&t=103s)
Patel begins with the difference between the work developers enjoy and the work required to ship software. Writing code is only a small part of the full flow. Developers also build Kubernetes manifests and Dockerfiles, deploy systems, write unit tests, and maintain documentation. A coding assistant helps with immediate technical hurdles, but Patel wants slower, repeatable work to happen offline in CI. He calls this approach a "ghost pilot". A coding copilot helps with quick, just-in-time decisions, while the CI system has time to reflect, gather context, and iterate during a pull request.

### Clear names and comments give AI a better statement of intent
[10:26](https://www.youtube.com/watch?v=prttz9_1XDc&t=626s)
The first CI step improves variable names and comments. Patel gives the example of a finance application where a variable named "rev" may mean revenue to the original developer. Changing it to "annual revenue" gives later models more useful context from finance-related knowledge. Comments also become stale when code changes under deadline pressure. The workflow updates comments to describe the intended behavior of a code block, even when the implementation no longer fully matches that intent. In his example, a vague port value becomes more explicit because it could refer to a port name or a port number.

### Unit-test generation works better as a sequence of smaller prompts
[09:01](https://www.youtube.com/watch?v=prttz9_1XDc&t=541s)
Patel does not ask an AI model to generate tests immediately from a function. The workflow first asks the model to describe the expected behavior using the code, comments, README, and possibly a product requirements document. Another prompt takes an adversarial view and lists boundary cases, including unusual characters and minimum and maximum values. Team priorities from a context.yaml file and bugs from the project's history add application-specific cases. Only then does the system generate the tests. It can repeat the process until the tests pass. The baseline behavior is established before the later security review, so a reviewer can check proposed changes against known behavior.

### Project context determines which tests and security checks matter
[11:08](https://www.youtube.com/watch?v=prttz9_1XDc&t=668s)
The context file describes the environment around the code. It can include the cloud provider, deployment location, whether the service is internal or customer-facing, whether it handles personal data, whether it is a frontend or backend application, the networking setup, the product requirements document, the issue tracker, and a project Slack channel. AI can add technical context, such as whether the code is infrastructure, backend, shell, or another kind of code. Historical issues and escalations can set focus areas such as SQL injection, timeout errors, or concurrent users. Patel also mentions a possible integration with OpenContext.org and a future option for the system to ask a developer for missing context.

### AI security review needs business context and a debate over findings
[13:26](https://www.youtube.com/watch?v=prttz9_1XDc&t=806s)
Patel separates this workflow from static analysis and SAST tools. It looks for security practices and logical-flow problems that automated pattern checks may miss. To reduce the tendency to report every possible concern, he simulates three AI employees: a red-team engineer, a Python developer representing the code author, and an engineering manager. The red-team role receives security policies and code. The developer receives the product requirements, while the manager receives business context from requirements, Slack conversations, and meeting transcripts. They debate each issue, its risk, whether it needs a fix, and the effort involved. A further model can review whether their answer matches the original intent.

### The workflow proposes security fixes while leaving the decision to a human
[16:51](https://www.youtube.com/watch?v=prttz9_1XDc&t=1011s)
The security stage produces a recommendation instead of changing the code automatically. Each issue includes its reasoning, risk or effort score, a code suggestion, and the conversation among the simulated roles summarized for the reviewer. The system also cites relevant security policies. Before a human reviews the proposed fix, it runs the unit tests that established the baseline behavior. Patel's example shows a hardcoded API key with an explanation of the risk, an effort estimate, a recommendation, and a code snippet. The reviewer decides which findings should actually be fixed.

## Notable quotes
- "The key part in this talk is about outsourcing a boring task to CI." (02:52)
- "Co-pilots are made for quick just in time decisions." (03:19)
- "It doesn't do a very good job, and this is where having iterations comes into play." (09:24)
- "It's finding security best practices that are not followed." (13:26)
- "At the end, human reviewer comes in and decides which ones to actually fix." (16:51)

## Tools & references mentioned
- Palo Alto Networks
- The Flow
- Thinking, Fast and Slow
- GitHub Copilot
- Sourcegraph Cody
- Kubernetes
- Docker
- context.yaml
- OpenContext.org
- Perplexity Pro
- GPT-3.5
- GitHub Actions
- GitLab CI

## Who should watch
- You maintain a CI pipeline and want AI to handle documentation, test generation, or review work after code is checked in.
- Your team needs tests based on historical bugs and application context rather than generic examples from a language model.
- You are evaluating AI-assisted security review and want proposed changes to remain subject to human approval.

## Related talks

- [Don't get one-shotted: Use AI to test, review, merge, and deploy code](https://aietalks.com/talks/dont-get-one-shotted-use-ai-to-test-review-merge-and-deploy-code) (Tomas Reimers, Graphite, 05:45)
- [Guide, Verify, Solve](https://aietalks.com/talks/guide-verify-solve) (Anirban Chatterjee, Sonar, 22:31)
- [Beyond the Prototype: Using AI to Write High-Quality Code](https://aietalks.com/talks/beyond-the-prototype-using-ai-to-write-high-quality-code) (Josh Albrecht, Imbue, 17:59)
- [How to Kill the Code Review](https://aietalks.com/talks/how-to-kill-the-code-review) (Ankit Jain, Aviator, 16:26)
- [Shipping complex AI applications](https://aietalks.com/talks/shipping-complex-ai-applications) (Giran Moodley, Braintrust & Mayank Soni & Oussama Hafferssas, Trainline, 1:38:34)