Scientific agents reach a plateau when they can implement experiments but cannot generate new hypotheses.
2
A hierarchy of linked documents can decompose a scientific ML problem into components that an agent can inspect and modify.
3
Agents can improve their loops by adding multimodal review, adversarial critique, and models with more test-time compute.
Summary
Sina Shahandeh argues that autonomous agents for scientific work need more than the hill-climbing loops used in coding benchmarks and toy ML tasks. Those agents can change code, run experiments, and search hyperparameters, but they often stop improving because they run out of useful hypotheses. He describes a system at Radicait for generating PET images from CT scans, where a major improvement came from considering a 3D model instead of treating stacked CT slices as a 2D problem. The system decomposes the codebase and scientific task into a hierarchy of linked documents covering data, architecture, losses, metrics, and supporting scripts. Agents can then search across those components and propose larger changes. Shahandeh also describes adding image-review skills and other models to critique results. He is clear that scientific observation remains a major limitation, especially for subtle findings in medical images.
Scientific agents plateau when they stop producing useful hypotheses
Shahandeh contrasts coding tasks with open-ended scientific work. Coding agents can implement code, change models, and run experiments over large datasets, but they often saturate at a certain level because they run out of ideas. He describes this as a lack of what people call research taste. Human researchers keep improving because they continue to form hypotheses about how a model or physical system could be changed. In a scientific loop, the system observes a situation, asks a question, proposes a hypothesis, runs an experiment, learns from the result, and repeats. Organizing mistakes and activity helps with memory, but generating a good hypothesis is harder.
The PET-from-CT problem requires choices across a long research loop
At Radicait, the goal is to generate a PET image from a CT scan. PET scans show tissue activity through radioactive tracer uptake, and tumors often absorb more tracer. A machine learning model can translate the structure seen in CT into an inferred PET image, but the overall research problem has many parts. Shahandeh focuses on training the model within that longer process. The example begins with an encoder-decoder or GAN-style architecture, a prepared dataset, and metrics for comparing synthetic PET with real PET. The agent must then improve the initial setup through repeated experiments.
A useful hypothesis can change the model's view of the problem
In one real run, the initial model treated CT slices as a 2.5D input. It used 2D convolutions while stacking several slices as channels. Ordinary parameter searches might adjust hyperparameters or make small code changes, yet miss a fundamental alternative. Shahandeh describes manually prompting the agent to consider a 3D convolutional approach, along with reading relevant papers. This kind of architectural change is the example of a hypothesis that can move the system beyond a plateau. The agent needs a way to search for changes to the problem formulation, rather than only optimize the existing implementation.
A hierarchy of linked documents gives the agent places to search
Shahandeh's approach explicitly decomposes the problem into subcomponents. For the PET task, the hierarchy includes data, the core learning problem, model architecture, training loss, operational ML code, metrics and evidence, peripheral scripts, and data preparation. A coding agent can inspect the codebase and generate linked documents at several levels. The top document contains the problem statement, lower levels describe components such as the model architecture, and the leaves connect back to the actual code. The hierarchy gives the reasoning model a structured view of the system and lets it consider changes within individual components.
Structured hypothesis search supports larger changes than a plain optimization prompt
With only a codebase and an optimization objective, Shahandeh says the agent tends to saturate. With the generated hierarchy, it can work through the components and propose many different solutions. He describes a process that can create 100 solutions, with the scaffold guiding decisions about each part of the system. One resulting idea is to replace the 2D treatment with a three-dimensional structure, a change the agent might otherwise miss. Another model or agent can review the proposed plan either adversarially or collaboratively. The resulting loop is: inspect the code and metric, decide whether the goal was achieved, hypothesize a change, implement it, and measure the outcome.
Image registration needs models that can inspect intermediate results
Shahandeh gives image registration as a second example. CT and PET scans taken at different times can be misaligned because of respiration, patient movement, changes in the body, or different scanners and positioning. Registration must align them, but success involves several metrics and can fail in different ways. A researcher might inspect images to decide whether a lung mask was applied correctly or whether a scan was cropped or truncated properly. These checks can become their own loops. The harness can call a model with stronger multimodal ability to review an image and judge whether the scans are aligned, then use that review during hypothesis generation and critique.
Scientific observation remains a larger limitation than code execution
Shahandeh says implementation is already fairly well handled when a task can be placed in a basic simulated environment. Real science depends on better observations, and current multimodal models struggle with scientific images and data. He uses the example of detecting tiny changes, such as the presence of a lung nodule. Models have not been trained well enough on these images to identify such findings reliably. Future systems may need fine-tuning or other adjustments so the models can observe a scientific result in a way closer to a trained scientist. Better hypothesis generation can be built with explicit hierarchies for now, although newer models may need fewer such scaffolds.
"No LLM today is able to identify these very well because they're just not simply trained on scientific images and scientific data."17:10
Who should watch
You are building an agent that has to improve a real ML or scientific workflow after the obvious code changes stop working.
Your experiments involve data preparation, model architecture, metrics, and domain-specific judgment that a single optimization prompt does not cover.
You want a concrete pattern for adding decomposition, image review, and agent critique to an autonomous research loop.
