AutoResearch gives a language model editable code, a repeatable evaluation and a goal, then keeps changes that improve the score.
2
A useful search depends on a fast, cheap, low-noise evaluation that cannot be easily reward-hacked.
3
Human steering matters because people can introduce ideas that move a search past a plateau.
Summary
The workshop explains AutoResearch as a repeated code-editing and evaluation loop. An agent proposes changes, runs a fixed evaluator, keeps improvements and discards weaker attempts. The presenters use Mini Parameter Golf to make this process small enough to watch during a workshop. They describe the source file, evaluation command, metric, search budget, time limit and starting guidance that define the search. Weco's dashboard shows attempts as an experiment tree, with scores and code diffs, and lets a person branch from promising work or inject a new direction. The talk also covers Aiden's work in OpenAI's Parameter Golf challenge, where the system searched for better language-model training code under model-size and runtime constraints. The final exercise applies the same setup to a Flappy Bird controller. Multiple game environments, hidden validation data and restrictions on editable code make the evaluation design part of the task.
AutoResearch keeps code changes that improve a fixed score
Zhengyao Jiang describes AutoResearch as a loop in which a language model receives code and a replayable evaluation script. It tries ideas, tests them, keeps changes that improve the score and discards the rest. The metric can measure accuracy, speed or cost. The presenters say the same pattern has moved beyond model training into backend engineering, prompt and agent engineering, and inference speed. Because the agent runs the loop on its own, the initial prompt and evaluation setup have a much larger effect than they do in ordinary step-by-step coding assistance.
Aiden searched Parameter Golf under strict size and time limits
Parameter Golf asks participants to train the best language model they can while keeping four artifacts, including model weights and training code, within 16 megabytes and finishing training within 10 minutes. Submissions use OpenAI's fixed data and evaluation scripts, with lower validation bits per byte being better. Aiden automated the research loop by gathering papers, implementations and its own past logs, then proposing, testing and checking ideas before creating pull requests. The presenters report that Aiden landed seven leaderboard records, produced work used by later pull requests and used a fraction of the competition's GPU compute.
Mini Parameter Golf makes the research loop visible
Mini Parameter Golf keeps the same basic task as the larger challenge, including real language-model training, a fixed evaluator and a single validation bits-per-byte metric. It shrinks the dataset and evaluation so an experiment takes only a few minutes. Workshop participants clone the Mini Parameter Golf repository, give it to their coding agent and run experiments on Modal. The smaller setup lets participants see several search steps during the session instead of waiting for a long training run. The presenters frame it as a scale model for understanding how the larger research process works.
The evaluation determines how much useful search fits in a budget
The agent modifies a defined source file, which limits the search space. In Mini Parameter Golf, solution.py contains the model configuration and training dynamics. The evaluation command runs the candidate and prints one scalar metric. The presenters want that evaluation to be fast, cheap and low-noise, because those properties allow more trials and make score changes easier to trust. They also warn that agents search for holes in an evaluation, so the protocol must resist reward hacking. For Mini Parameter Golf, the goal is to minimize validation bits per byte.
Time limits and starting guidance shape the search
The workshop setup uses 20 search steps so participants can watch the agent explore. The presenters say the right number depends on the available budget, and more attempted ideas can create more chances for improvement. Each evaluation is limited to 210 seconds, with 60 seconds for training, 60 seconds for testing on unseen text and 90 seconds of overhead for compute startup and other delays. Initial guidance can be supplied in a Markdown file. It can include domain knowledge, constraints and ideas such as compression, quantization and weight sharing.
An experiment tree makes each code change inspectable
Weco presents AutoResearch as a tree-based search over code. The root is the baseline program, and each later node is a modified version with an evaluation score. Promising branches can be extended while weaker ones are pruned. The dashboard lets users open a node, read the exact diff and compare the change with the score movement. Users can also add an idea in plain English, which creates a new branch, and assign a compute budget to a node. This gives people a way to alter the search while it is running.
The presenters use a Parameter Golf example to explain why steering matters. Aiden tried new attention methods, compression and a richer stack for five days, but its score plateaued. A human participant then released a new tokenizer. Within eight hours of finding that change, the system combined it with work it had already explored and produced the largest score jump in weeks. The presenters describe this as collaboration rather than a process that runs without supervision. People can bring outside ideas into the search when the current branches stop making progress.
A good multi-objective evaluation reduces several aims to one signal
In response to a question about reinforcement learning, the presenters say AutoResearch can be applied to reinforcement-learning training scripts, including systems that train an agent to play a game. When several objectives matter, they recommend combining them into a scalar metric, either by weighting the objectives or by treating some as constraints and optimizing another. Fraud detection is one example: accuracy can be held as a constraint while precision is optimized. Parameter Golf follows the same pattern by constraining model size and optimizing model quality.
The final exercise replaces model training with a controller for Flappy Bird. The editable should_flap function decides whether the bird flaps on each frame, and the score is the average across normal play, moving pipe gaps, wind and a mixed environment. The leaderboard uses a hidden validation set, so a controller cannot simply overfit the visible game. Participants can edit only controller.py, must preserve the function signature and cannot change the game or evaluator. External libraries, file access and parallel compute are disallowed. The rules make the desired behavior and the limits of the search explicit.
"You give a language model your code and a replayable evaluation script. It will keep trying ideas and test them against the eval and it keeps the ideas that improve the score and discard everything else."Zhengyao Jiang01:06
Who should watch
You are building an agent that needs to improve code against a repeatable test and want a concrete way to define the search space and score.
Your evaluations are slow, noisy or easy to exploit, and you need practical constraints for making an automated research loop useful.
You want to inspect agent-generated experiments, branch from good attempts or inject your own ideas while a search is running.