# Real-time Experiments with an AI Co-Scientist

Stefania Druga, Independent researcher, formerly Google DeepMind | AI Engineer World's Fair 2025 | 18:42

Source: https://www.youtube.com/watch?v=wNH3q9pqn0U
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/real-time-experiments-with-an-ai-co-scientist
Published: 2025-07-28
Tags: context-engineering, multimodal, open-source, robotics

## TL;DR
- An AI co-scientist can combine sensor readings, camera feeds, voice, and experiment protocols to give feedback while a physical experiment is running.
- Stefania Druga built an open-source lab-in-a-box system for under $300, using a React app, Gemini, webcams, and micro:bit-connected sensors.
- Real-time experiments can produce findings that are easy to miss in static analysis, such as crystal growth happening in bursts after critical saturation is reached.

## Summary
Stefania Druga presents an AI co-scientist that works with live physical experiments rather than only analysing existing papers and datasets. Her prototype combines webcams, microscope images, temperature and other sensor readings, voice, text, and experiment protocols in a React application backed by Gemini. A context assembly layer checks which modalities are available and builds the prompt for each interaction. She demonstrates the system with crystal growth and fermentation experiments assembled from inexpensive, open-source hardware at home. The crystal experiment produced sensor logs and microscope recordings that revealed sudden bursts of growth after critical saturation, rather than steady growth. Druga also shows a tracking camera and discusses applications in robotics and synthetic biology. She argues that live experimental data can inform simulations, which can then suggest lab conditions for further testing. The talk is practical about the constraints: limited hardware, latency, data formats, safety, and the need to make tools usable by scientists and learners.

## Key ideas
### A co-scientist connects an AI assistant to a running experiment
[00:34](https://www.youtube.com/watch?v=wNH3q9pqn0U&t=34s)
Druga compares a science co-scientist with a coding copilot. The assistant receives data from a micro:bit board connected to a sensor system, including a temperature reading from a heat pad experiment. It can also receive microscope images. A user defines a protocol and gives the assistant the experiment's conditions and constraints, so later feedback is interpreted in context rather than as an isolated measurement. For repeated work, the system can monitor and plot readings in real time. Druga also describes leaving an experiment running while a camera tracks an object or a custom target such as crystal growth.

### AI can help scientists search hypotheses across large bodies of work
[03:50](https://www.youtube.com/watch?v=wNH3q9pqn0U&t=230s)
Druga says science has too much data and too much complexity for researchers to inspect everything manually. AI can analyse information at scale, generate hypotheses, identify blind spots, and let researchers test many possibilities instead of working through one hypothesis at a time. She was inspired by an AI co-scientist system built from multiple agents, with separate roles for analysing and summarising papers, examining data, ranking hypotheses, and searching online sources. The agents used Gemini 2.0 and produced a plan for a researcher. Druga describes reported tests involving gene transfer mechanisms and liver fibrosis treatment discovery.

### Real-time feedback changes the co-scientist from a planning tool into an experiment partner
[06:47](https://www.youtube.com/watch?v=wNH3q9pqn0U&t=407s)
The usual AI co-scientist workflow analyses data that already exists and returns a plan later. Druga's proposed system formulates and revises hypotheses while empirical data arrives from the lab. The same pattern could apply to a robot that is visibly breaking down during operation. She connects this idea to the paper 'Welcome to the Era of Experience' by Rich Sutton and David Silver, which describes systems learning from continuous interaction with an environment. Images, sensors, and audio streams give the model information about what is happening now, rather than only information collected in an earlier dataset.

### Context assembly has to adapt to whichever modalities and protocols are active
[08:43](https://www.youtube.com/watch?v=wNH3q9pqn0U&t=523s)
The prototype is a React app with inputs for Jackd-connected sensors over USB, multiple webcams, text, and voice. These inputs become webhooks and are sent to a backend that communicates with Gemini. Druga's context assembly layer checks whether text, voice, images, and chat history are available each time a message is sent. It then builds the context dynamically, using the selected experiment protocol and its conditions. She presents this context injection step as a central engineering part of the system, because different experiments expose different sensors and require different constraints.

### Cheap open hardware can support useful experiments outside a conventional lab
[10:34](https://www.youtube.com/watch?v=wNH3q9pqn0U&t=634s)
Druga designed experiments around the hardware she had available, while limiting herself to work that was safe to perform at home, portable, and measurable in real time. Her lab-in-a-box included cameras, boards, cables, and sensors, and she says the parts cost under $300 and took two weeks to assemble. For crystal growth, she dissolved salt in hot water, then cooled the solution gradually. The setup recorded microscope footage and sensor data, including temperature and humidity. She compared samples held in different environments and used the resulting CSV files to plot growth and temperature.

### The data showed that crystal formation happens in bursts
[13:50](https://www.youtube.com/watch?v=wNH3q9pqn0U&t=830s)
The crystal experiment gave Druga an observation she had not expected. Crystal formation did not progress as a smooth, gradual process. Once critical saturation was reached, growth appeared in sudden bursts. The system recorded the experiment over an extended period, with microscope footage and changing temperature and humidity readings. Druga exported the sensor values to CSV and wrote a separate script to create visualisations. The live platform did not yet include those plots, so this part still required an additional analysis step. The example shows why camera and sensor streams need to be correlated rather than treated as separate records.

### An open-source ecosystem can extend the prototype toward automated laboratory work
[15:59](https://www.youtube.com/watch?v=wNH3q9pqn0U&t=959s)
Druga places the prototype within a wider open-source effort to recreate lab equipment and automate physical procedures. She mentions systems for pipetting, liquid handling, mixing, and analysis, including the Jubilee motion platform and an open bioreactor. A workshop at the University of Washington brought people together to build tools for droplet manipulation and robotic handling of liquids and vials. She also presents an education version of her system that runs on a phone and lets people test the camera, with a micro:bit as an optional input. The full lab system was not yet deployed, but she planned to release its code.

### Live experiments can provide the data needed for more realistic simulations
[17:09](https://www.youtube.com/watch?v=wNH3q9pqn0U&t=1029s)
Druga's proposed next step is to use camera, sensor, and voice recordings from physical experiments to build simulations. A crystal-growth simulation could reproduce the conditions observed in the real setup, while more useful cases might model bacterial colony growth. The simulation could run different lab conditions, identify promising settings, and then provide those conditions to a real experiment. In this design, physical experiments supply evidence for the simulation, and the simulation helps choose what to try next in the lab. The loop depends on collecting enough structured information from the real environment.

## Notable quotes
- "So, think about pair programmers. How many of you use any form of copilot for coding? Right. So it's just like that but for doing things in the real world like science experiments." (00:34)
- "So instead of testing one hypothesis at a time, you could test a 100 hypothesis." (04:08)
- "The insight I got from this is that the crystal formation is actually not gradual, that it happens in bursts." (13:50)
- "So we need not be limited by the experiments we're doing in real life but those experiments that we're doing in real life are going to inform realistic simulations." (17:09)

## Tools & references mentioned
- Google DeepMind
- Gemini 2.0
- AI co-scientist
- Rich Sutton
- David Silver
- Welcome to the Era of Experience
- Jackd
- micro:bit
- REC camera
- Jubilee motion platform
- University of Washington
- open bioreactor
- SO Arm 100
- LeRobot

## Who should watch
- You are building an AI system that needs to interpret camera feeds, sensor values, or voice while a physical process is still running.
- You want practical design ideas for assembling a low-cost scientific or educational lab from open-source hardware.
- You are interested in connecting foundation-model reasoning with robotics, laboratory automation, or simulation.

## Related talks

- [Imagination Engineering: "Live in the future and then build what's missing."](https://aietalks.com/talks/imagination-engineering-live-in-the-future-and-then-build-whats-missing) (Eve Bouffard, Y Combinator, 16:04)
- [Milliseconds to Magic: Real-Time Workflows Using the Gemini Live API and Pipecat](https://aietalks.com/talks/milliseconds-to-magic-real-time-workflows-using-the-gemini-live-api-and-pipecat) (Kwindla Kramer, Daily & Shrestha Basu Mallick, Google DeepMind, 21:43)
- [Invisible Users, Invisible Interfaces: Accelerating Design Iteration with AI Simulation](https://aietalks.com/talks/invisible-users-invisible-interfaces-accelerating-design-iteration-with-ai) (Alex Liss, Huge, 12:37)
- [Autonomous Agents for Scientific Tasks](https://aietalks.com/talks/autonomous-agents-for-scientific-tasks) (Sina Shahandeh, Radicait, 19:23)
- [How Anthropic Builds: Lessons from Labs](https://aietalks.com/talks/how-anthropic-builds-lessons-from-labs) (Mike Krieger, Anthropic, 26:11)