# Einstein Arena: Harnessing Collective Agent Intelligence for Open Science James Zou, Together AI | AI Engineer World's Fair 2026 | 16:56 Source: https://www.youtube.com/watch?v=mMNkdYnIVC4 Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks. Page: https://aietalks.com/talks/einstein-arena-harnessing-collective-agent-intelligence-for-open-science Published: 2026-08-25 Tags: agents, benchmarks, multi-agent, open-models, reinforcement-learning ## TL;DR - AI agents can produce better scientific results when they work inside an environment with problems, incentives, communication, verification, and live feedback. - Agents in Einstein Arena improved the known eleven-dimensional kissing number construction from 593 to 604 spheres by building on one another's solutions. - DS Gym tests data science agents on tasks that require access to the data, then uses execution-verified runs to fine-tune small models that can run on laptops. ## Summary James Zou argues that agent systems should be designed around environments rather than fixed workflows. A workflow tells an agent which steps, tools, and prompts to follow. An environment defines the problems, resources, incentives, guardrails, and feedback available to the agent while leaving its method open. Einstein Arena applies this idea to open scientific problems. Agents enter by solving a puzzle, choose curated problems, share work in forums, submit solutions to deterministic verifiers, and see scores and other solutions on a live leaderboard. In one example, agents improved the eleven-dimensional kissing number construction from 593 to 604 spheres. The same setup produced GPU kernels with speedups of more than two times, including kernels now used in production at Together AI. Zou also presents DS Gym, a data science evaluation and training environment designed to prevent shortcut solutions. Execution-verified trajectories from the gym can train small open-source models for local use. ## Key ideas ### Agent environments leave methods open while defining the conditions for work [00:12](https://www.youtube.com/watch?v=mMNkdYnIVC4&t=12s) James Zou contrasts agent workflows with agent environments. Workflows specify steps, prompts, tools, and instructions that tell an agent how to operate. An environment specifies where the agent works and supplies incentives, infrastructure, guardrails, and resources. His thesis is that increasingly capable agents can be constrained by workflows designed in advance. A well-designed environment leaves more room for agents to find their own methods. The talk presents Einstein Arena and DS Gym as examples of this approach, with the environment controlling access to problems, feedback, execution, and evaluation rather than prescribing one fixed sequence of actions. ### Einstein Arena lets agents collaborate and compete on open scientific problems [01:54](https://www.youtube.com/watch?v=mMNkdYnIVC4&t=114s) Einstein Arena is an agent-native environment for open-ended scientific problems. Humans are intentionally kept out by a puzzle that participants must solve to prove they are AI agents, while any agent can participate freely. Agents see curated problems chosen for scientific interest and for the availability of a deterministic verifier. Inside each problem space, they can read a precise description, ask for help in a discussion forum, submit solutions, and receive live scores. The leaderboard also exposes other agents' solutions for download. This creates a setting where agents can share ideas while competing to improve the measured result. ### The arena produced better-known solutions to eleven problems within weeks [04:34](https://www.youtube.com/watch?v=mMNkdYnIVC4&t=274s) Zou says Einstein Arena found new solutions to eleven problems within a few weeks of its launch in March. The solutions were better than previous human solutions and results from more specialized AI tools. The arena is designed around problems that already interest human researchers, so its verifiers can measure progress on questions with existing scientific value. The results support Zou's claim that agents can make progress when they have access to a shared problem space, other agents' work, a way to test submissions, and immediate public feedback. The system does not require a central workflow that dictates how each agent should reason. ### Agent collaboration raised the eleven-dimensional kissing number from 593 to 604 [05:16](https://www.youtube.com/watch?v=mMNkdYnIVC4&t=316s) The kissing number problem asks how many equal spheres can touch a central sphere without overlapping one another. The answer is easy in one dimension, where it is two, and in two dimensions, where it is six. Higher dimensions remain difficult. In eleven dimensions, known constructions progressed from 440 spheres in the 1980s to 582, then 592 in 2022, and 593 after a DeepMind result the following year. Agents collaborating in Einstein Arena constructed a solution with 604 spheres within a few days. Zou connects better high-dimensional sphere packings with improved coding systems, including error correction for information transfer. ### A shared leaderboard can turn agent competition into faster GPU kernels [09:02](https://www.youtube.com/watch?v=mMNkdYnIVC4&t=542s) The group reused the arena structure for machine learning kernel optimization. The backend compiles, benchmarks, tests, and verifies each kernel, then reports speed and quality through a live leaderboard. Agents with different roles inspect different aspects of the work. One focuses on profiling, another on memory consumption, and another on precision and tensor computations. They can compare submissions, collaborate, and refine one another's changes. Zou reports speedups of more than two times for some production kernels, including page attention kernels across particular shapes and other hardware settings. These agent-designed kernels are already used in production at Together AI. ### DS Gym evaluates data science agents through a common execution environment [10:47](https://www.youtube.com/watch?v=mMNkdYnIVC4&t=647s) DS Gym is a unified environment for evaluating and training data science agents. It brings together datasets and tasks from different settings through a common interface and code execution. Agents can start multiple Docker containers to test algorithms in parallel. The task collection covers scientific analysis, discovery, and predictive modeling. Zou describes tasks drawn from recently published papers, reviewed by human scientists and experts, along with selected Kaggle competitions that have strong data and evaluation. The gym contains more than a dozen tasks across dozens of scientific domains, including biology, physics, and economics, with several data types and modalities. ### Many popular data science benchmarks allow agents to succeed without using the data [12:10](https://www.youtube.com/watch?v=mMNkdYnIVC4&t=730s) Zou found that popular data science benchmarks often allow shortcut solutions. Across several existing benchmarks, agents could solve as much as 20 to 50 percent of tasks without accessing the underlying datasets. They could answer through reasoning or other shortcuts, which means the benchmark did not always test whether an agent could perform the intended data science work. DS Gym was built to address this problem by curating tasks from scientific papers and predictive modeling competitions, then checking the tasks and their execution. Zou says frontier models still achieve less than 50 percent accuracy on DS Gym, so he considers the benchmark unsaturated. ### Execution-verified trajectories can train small models for local data science work [14:30](https://www.youtube.com/watch?v=mMNkdYnIVC4&t=870s) DS Gym records trajectories produced by agents and verifies them by executing the agents' code. These execution-verified trajectories can then be used to fine-tune open-source models. Zou says the resulting models are among the best open-source models for these data science tasks and are small enough to run on laptops and personal computers. The environment also supports synthetic data generation, which gives users another way to create training material for their own data science agents. This makes DS Gym both an evaluation system and a source of checked examples for model training. ## Notable quotes - "Our thesis here is that as agents become more and more powerful, if we try to design workflows that often can limit the capabilities and creativity of the agents." (01:17) - "But the collaboration among multiple agents is really critical." (07:52) - "By shortcut, I mean here is that ... the agents can actually solve without actually using the data sets themselves." (12:26) - "What we really want to do is to design environments, which is a set of infrastructure and incentives that motivates the agents to solve more and more challenging problems." (16:06) ## Tools & references mentioned - Together AI - Stanford - Einstein Arena - DS Gym - Isaac Newton - DeepMind - AlphaZero - GPT 5.5 - Docker - Kaggle ## Who should watch - You are building agent harnesses and want to see an alternative where the system defines the task, feedback, and resources while agents choose their own process. - You work on scientific discovery or optimization and need a way for multiple agents to share attempts, verify results, and improve submissions over time. - You evaluate data science agents and worry that benchmark tasks can be solved without touching the data.