An AI Research Agent That Runs Your Experiments

Tim Sweeney, Weights & Biases21:16 · Sept 2026 · 8,035 views
Thumbnail for An AI Research Agent That Runs Your Experiments Watch on YouTube
TL;DR
  1. 1

    ARIA can launch, monitor, and analyze machine learning experiments from inside Weights & Biases.

  2. 2

    The team uses Weave traces, live LLM judges, YAML tasks, and nightly evals to improve the agent.

  3. 3

    Sweeney recommends treating agent evals as CI while keeping humans involved in reviewing behavior.

Summary

Tim Sweeney demonstrates ARIA, Weights & Biases' AI research and iteration agent, running Karpathy's autoresearch project on real GPUs. ARIA launches batches of experiments, changes code and hyperparameters, summarizes runs, finds patterns, and creates reports and workspaces. Sweeney then explains the system behind it: a client, API server, database, worker harness, sandbox, model provider, long-running job system, and Weave observability. The team logs all ARIA traces, reviews conversations, runs LLM judges on live traffic, turns recurring problems into YAML tasks, and evaluates candidate models nightly. Sweeney's production advice is practical. Log sessions, tool calls, errors, and feedback so behavioral bugs can be found. Treat tasks and evals as go-no-go checks. Keep humans reviewing traces because automated judges miss some behavioral details. Add domain context and useful tools before adding complexity to the agent harness. The live experiment produced a score of 5.833, just short of the previous 5.831 result, after 12 experiments.

Key ideas
02:33

ARIA turns a Weights & Biases project into an interactive research workspace

Sweeney opens ARIA inside a Weights & Biases workspace containing more than 200 training jobs from Karpathy's autoresearch project. The agent can download code, configure launch jobs and GPUs, then iterate on code and hyperparameters. It also answers questions about completed work. For example, it can summarize the highest-performing runs for someone joining a project or returning from time away. When asked to find patterns, ARIA identifies a new model family, a strong relationship with batch size, and a promising architectural recipe. Sweeney says these findings could take him hours or days to discover manually.

04:33

The live demo connects agent decisions to real GPU jobs

Sweeney asks ARIA to conduct another batch of experiments while the audience watches. ARIA decides against a large architecture change and chooses hyperparameter modifications, then issues a shell call to run the loop. Weights & Biases Launch connects the agent to a Kubernetes cluster and lets agents or people submit long-running jobs that use GPUs. Sweeney shows the cluster output and says the experiments are executing live. While the jobs run, ARIA polls for results, queries the project, applies patches, and writes code. The demo makes the agent's research loop visible instead of presenting a precomputed result.

05:57

ARIA generates analysis and visual reports alongside experiment results

The agent does more than return prose. It can summarize the best runs, identify patterns across the research, and create a Weights & Biases report with embedded plots, charts, and graphics. Sweeney shows ARIA using a parameter-importance chart to examine correlations among training parameters. The agent also builds workspaces with plots and the platform's built-in charts. This gives researchers a way to move from an experiment result to a report that colleagues can inspect. Sweeney presents the feature as a data science companion inside the interface where the research already happens.

10:22

The system separates the agent harness from sandboxes, models, jobs, and observability

Sweeney describes ARIA's architecture as a web client and iOS client connected to an API server, database, and worker harness. The harness can call a sandbox for shell commands and Python data science, use an LLM provider, and hand long-running training jobs to Weights & Biases Launch and CoreWeave GPUs. Weave provides the observability layer. The team logs 100% of ARIA's traces, including sessions, turns, tool calls, and errors. Offline analysis turns production behavior into tasks and evals. Evaluation results feed a shared dashboard, where the team decides whether prompt or architecture changes should move forward.

13:02

The team uses Weave to inspect ARIA's behavior and improve ARIA itself

The Weave agent dashboard shows span volume, conversation volume, token tracking, and a live conversation feed. Its spans view displays the shape of each trace, including tool calls, LLM calls, and thinking blocks. Researchers can open a conversation, inspect prompts and shell calls, then add notes, feedback, and emojis. Those observations become tasks for future iterations. ARIA can also summarize the conversation being viewed. Sweeney describes this as using ARIA to analyze its own conversations and recommend improvements within the same interface. The dashboard connects operational monitoring with hands-on review of individual interactions.

15:12

Live LLM judges expose recurring behavior such as user frustration

Weave runs LLM judges against live traffic to detect signals such as user frustration, low-quality responses, and requests to ask the user for more information. Sweeney shows a frustration signal whose reasoning says the user was unhappy with the loss curve. The team uses these signals to find clusters of behavior that need attention in the next iteration. The judges do not replace human review. Sweeney later says that some behavioral nuances will not be caught by an LLM, so the team must still inspect the best and worst traces manually.

16:07

YAML tasks turn observed failures into repeatable tests

ARIA's tasks are written as YAML files and treated as unit tests for the model. Sweeney shows a task in which the agent must compare two runs and explain what can be learned from their differences. The task defines an LLM judge for correctness, another judge for whether the insight is interesting, and a rule-based judge that checks whether the result was produced within six tool calls. The team has about 200 such tasks grouped into a nightly eval suite. This gives production observations a repeatable form that can be run against candidate models.

17:17

Nightly evals provide a shared decision point for model changes

The team tracks each night's evaluation in Weave. Sweeney gives an example in which a candidate model scored 73% on the eval suite while the production model scored 72%. That result was enough for the team to plan a push forward. He describes the dashboard as a place where the team can make decisions about prompt and architecture changes, then promote a chosen model through the registry. The process links live traffic, human observations, task definitions, candidate implementations, and evaluation results into one improvement loop.

18:47

Production agents need human review and useful context before more harness complexity

Sweeney's production advice starts with agent-focused observability. Teams should log sessions, turns, tools, and feedback so they can find behavioral bugs, not only exceptions or performance problems. He says tasks and evals should become the new CI, with researchers and software engineers using metrics as go-no-go decisions. Humans still need to review traces because automated judges miss some behavior. His final advice is to add business context, domain information, available primitives, and business data before over-engineering memory or the harness. In his demo, the live batch ran 12 experiments and reached 5.833, close to the earlier 5.831 result.

"ARIA's great at orchestrating jobs, understanding GPU workloads, responding to events within the W&B ecosystem, and listening to researchers, looking up arXiv papers, and collaborating on hypotheses."09:57
Who should watch
  • You are building an agent that needs to run long-lived jobs, call tools, and work with experiment data.
  • Your team has production traces but no repeatable way to turn bad interactions into evals and release decisions.
  • You want practical guidance on observability, human review, and agent evals rather than another agent architecture overview.