# Your Coding Agent Should Do AI System Engineering

Ben Burtenshaw, Hugging Face | AI Engineer Europe 2026 | 18:25

Source: https://www.youtube.com/watch?v=JomVvNDjGb8
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/your-coding-agent-should-do-ai-system-engineering
Published: 2026-05-21
Tags: agent-skills, agents, coding-agents, multi-agent, synthetic-data

## TL;DR
- Coding agents can write and benchmark CUDA kernels when they have hardware context, examples, and reusable project files.
- Skills turn difficult zero-shot engineering tasks into few-shot workflows by giving agents versioned instructions, examples, and tests.
- Open repositories, compute, storage, and tracking let multiple agents run machine learning experiments in parallel as an automated research lab.

## Summary
Ben Burtenshaw argues that coding agents should move beyond application code into AI systems engineering. He presents three increasingly autonomous examples. First, an agent writes and benchmarks CUDA kernels, then distributes them through Hugging Face's kernels repositories. Second, an agent fine-tunes a language model end to end from a prompt using Hugging Face infrastructure. Third, a research lab uses separate researcher, planner, worker, reviewer, and reporter agents to propose, run, assess, and report training experiments. Burtenshaw's main technique is to give agents file-based skills containing instructions, examples, scripts, and references. This changes a zero-shot task into a few-shot workflow. He also argues that open primitives matter because agents need access to the underlying data and tools. Trackio stores experiment data in an open format, while Hugging Face provides storage, tracking, and compute for the jobs.

## Key ideas
### Coding agents can move into AI systems engineering
[00:07](https://www.youtube.com/watch?v=JomVvNDjGb8&t=7s)
Burtenshaw says coding agents can now tackle difficult engineering work in systems engineering and machine learning engineering. He proposes moving closer to the silicon as agents become common in ordinary software development. The talk has three progressively more autonomous examples. The first is an interactive agent that writes a CUDA kernel. The second is a zero-shot prompt that asks an agent to train a language model on Hugging Face. The third is a multi-agent research setup that operates like an automated AI lab. The examples shift from a human working with one agent to several agents running experiments with less direct intervention.

### Custom kernels improve model speed by using the GPU more efficiently
[01:57](https://www.youtube.com/watch?v=JomVvNDjGb8&t=117s)
A model running on a GPU executes its work through kernels written for particular hardware. Custom kernels can target a specific math operation and use hardware features that are unavailable elsewhere, but writing and installing them requires knowledge of CUDA, hardware generations, software versions, and a large compatibility matrix. Burtenshaw divides deep learning efficiency into compute, memory, and overhead. He says memory is usually the bottleneck, because a modern GPU can perform enormous amounts of computation while data still has to move through memory. Kernels such as FlashAttention increase arithmetic intensity by doing more sums for each read and write, keeping more of the GPU's capacity in use.

### The Hugging Face kernels library turns optimized kernels into shareable repositories
[05:16](https://www.youtube.com/watch?v=JomVvNDjGb8&t=316s)
Hugging Face's kernels library provides a way to distribute custom kernels as repositories on the Hub. Each repository has a TOML file that records supported hardware, CUDA versions, and other software requirements. This lets users check whether a kernel matches their GPU instead of dealing with compatibility by guesswork. Burtenshaw describes kernel writers as a category alongside model publishers. In his example, a kernel generated for Qwen3 8B on an H100 produced a 94% speedup. He presents that result as a compatibility demonstration rather than a state-of-the-art result, since a model may have a kernel that is not tuned for the hardware available to a user.

### Skills give agents examples and tools for specialized work
[06:08](https://www.youtube.com/watch?v=JomVvNDjGb8&t=368s)
Burtenshaw describes skills as file-based context. Files can be opened when needed, versioned, and stored in source control, and agents can use them in the same way. A kernel skill can contain examples of how to write and use kernels, plus scripts for benchmarking and testing them. He says this changes a task from zero-shot to few-shot because the agent receives examples of the expected work. Hugging Face is placing maintained skills inside projects, where project maintainers can update them, while the separate Hugging Face skills repository contains more experimental examples. The project-owned approach gives agents instructions that track the software they are meant to operate.

### Upskill compares models on the same skill
[08:34](https://www.youtube.com/watch?v=JomVvNDjGb8&t=514s)
Burtenshaw introduces Upskill as an open source library for generating skills, generating evaluations for those skills, and comparing models on the same task. This creates a way to test whether a skill works and to choose a cheaper or smaller model for repeated use. He gives comparisons in which GPT-OSS, Kimi, and Haiku differ in accuracy and token use. The point is to measure model behavior under the same skill rather than choosing a model without testing. A team can use the results to iterate on the skill and reduce the cost of an agent workflow while keeping the task definition fixed.

### An agent can fine-tune a language model from a single instruction
[09:26](https://www.youtube.com/watch?v=JomVvNDjGb8&t=566s)
The second example asks an agent to fine-tune Qwen3 6B on a chain-of-thought dataset and improve the model's chain-of-thought behavior. Burtenshaw says the workflow is integrated with the Hugging Face Hub, including the ability to run the needed GPUs there, and it uses Hugging Face CLI skills. He also mentions an Onslaught version that uses optimized models and can be cheaper. The example is presented as an end-to-end task: the user provides the instruction, while the agent handles the training pipeline. Burtenshaw points viewers to a blog post and existing examples rather than describing a custom system built from scratch.

### A research lab can divide model experiments among specialized agents
[10:16](https://www.youtube.com/watch?v=JomVvNDjGb8&t=616s)
Burtenshaw builds on Andre Karpathy's Auto Research project, where Claude Code changes a nanoGPT training script, runs experiments, and keeps improvements that raise the measured result. Burtenshaw distributes that single-agent loop across several roles. A researcher searches HF Papers or arXiv and turns papers into hypotheses. A planner keeps a queue of jobs. Workers implement hypotheses as training-script changes. A reporter monitors the jobs and maintains a dashboard. The experiments run as separate branches and jobs, with scores stored in the main branch. This structure lets agents explore several proposed changes instead of having one agent inspect and modify the project one step at a time.

### Open data and compute let agents run and inspect experiments
[12:09](https://www.youtube.com/watch?v=JomVvNDjGb8&t=729s)
The research system is implemented in OpenCode in the example, with similar implementations in Codex and Claude available in the repository. The planner, researcher, reviewer, and worker agents use templates containing the current state, previous jobs, successful and failed experiments, and parameters they can change. Workers launch jobs through HF Jobs and submit patches back to the repository. The reporter uses Trackio to collect metrics, events, warnings, and notifications. Trackio stores data in Parquet, so an agent can read the underlying data directly and create a different view, such as a Gantt chart. Burtenshaw says the open data layer matters because agents are not limited to the dashboard's built-in visualizations.

### Agents need exposed primitives rather than fully hidden abstractions
[16:57](https://www.youtube.com/watch?v=JomVvNDjGb8&t=1017s)
Burtenshaw's final point is that agents work well with open primitives. He names Trackio and kernels as tools that expose enough of their internals for agents to control them directly. Abstract APIs can be useful, but a layer that agents cannot access becomes a ceiling. He argues that the Hugging Face Hub already has the basic pieces for these workloads: storage, tracking, and compute. With verifiable experiments such as model training or CUDA kernel generation, an agent system can run work, measure it, and preserve the results in repositories. The claim depends on having inspectable files and data, rather than handing the agent an opaque service.

## Notable quotes
- "My proposal is that we need to go kind of closer to the silicon and tackle harder problems." (01:10)
- "I like to say that it takes a task from being zero-shot to being few-shot." (06:45)
- "In short, most people might assume that the compute is the bottleneck here because it's doing most of the math, right? That's not correct." (04:14)
- "So, you can go and just have your kind of own AI lab, and you can try it out." (16:57)
- "Even though abstracted APIs are really useful, if we have a layer that we can't necessarily get behind, that is a ceiling." (17:15)

## Tools & references mentioned
- Hugging Face
- Hugging Face Hub
- Hugging Face kernels
- GPU mode
- AMD hackathon
- KernelBench
- FlashAttention
- Qwen3
- Upskill
- GPT-OSS
- Kimi
- Haiku
- Onslaught
- Andre Karpathy
- Auto Research
- nanoGPT
- nanochat
- Claude Code
- HF Papers
- OpenCode
- Codex
- Gastown
- HF Jobs
- Trackio
- Parquet

## Who should watch
- You are building coding-agent workflows and want to apply them to CUDA, model training, or other work that needs tests and hardware access.
- Your team has specialized engineering knowledge that could be captured in versioned skills, examples, and reusable scripts.
- You want agents to run machine learning experiments overnight and need open storage, metrics, and dashboards that the agents can inspect directly.

## Related talks

- [AI Engineer Code Summit 2025](https://aietalks.com/talks/ai-engineer-code-summit-2025) (Jed Borovik, Google & Swyx, AI Engineer & Barry Zhang & Mahesh Murag, Anthropic & Dex Horthy, HumanLayer & Lee Robinson & Naman Jain, Cursor & Jacob Kahn, Meta & Rhythm Garg & Linden Li, Applied Compute & Will Brown, Prime Intellect & Will Hang & Cathy Zhou, OpenAI & Kitze, Independent & Kath Korevec, Google Labs & Eno Reyes, Factory AI & Beyang Liu, Amp Code / Sourcegraph & Natalie Serrino, Gimlet Labs & Jake Nations, Netflix & Eiso Kant & Jason Warner, Poolside & Aparna Dhinakaran, Arize & Nik Pash, Cline & Joel Becker, METR & Kevin Hou, Google DeepMind & Benjamin Dupy & Leah McBride, AI Engineer, 8:57:06)
- [Self Coding Agents](https://aietalks.com/talks/self-coding-agents) (Colin Flaherty, Augment Code, 17:23)
- [Agentic Engineering: Working With AI, Not Just Using It](https://aietalks.com/talks/agentic-engineering-working-with-ai-not-just-using-it) (Brendan O'Leary, Kilo Code, 27:03)
- [Your Coding Agent Just Got Cloned And Your Brain Isn't Ready](https://aietalks.com/talks/your-coding-agent-just-got-cloned-and-your-brain-isnt-ready) (Rustin Banks, Google Labs, 13:40)
- [The Emerging Skillset of Wielding Coding Agents](https://aietalks.com/talks/the-emerging-skillset-of-wielding-coding-agents) (Beyang Liu, Sourcegraph / Amp, 35:06)