AGI benchmarks should compare AI with humans on tasks that require learning skills it has not seen before.
2
ARC-AGI-3 will use hidden, game-like environments where agents must explore, infer rules, and complete objectives without detailed instructions.
3
ARC Prize will measure skill acquisition efficiency through human baselines, action counts, and completion time rather than relying only on whether an agent wins a known game.
Summary
Greg Kamradt argues that beating familiar games does not establish general intelligence. Agents that played Pokemon became stuck, hallucinated actions, needed interventions, and benefited from existing training data. ARC Prize instead uses tasks that humans can solve but AI systems have not encountered. ARC-AGI-3 extends this approach into interactive environments. Agents will enter unfamiliar games, explore them, infer their rules, form goals, and act under sparse rewards. The benchmark will include public training games and a private evaluation set with no internet access. The evaluation games will be new to both the agent and its developer. ARC Prize plans to focus on core knowledge priors such as basic math, geometry, agentness, and objectness, while avoiding language, trivia, and specialist knowledge. Performance will be compared with hundreds of human participants using completion time and action count. A five-game sandbox and agent competition are planned before a target launch of about 120 games in Q1 2026.
Beating a familiar game does not prove general intelligence
Kamradt opens with agents playing Pokemon, including Claude, OpenAI's system, and Gemini. The demonstrations are engaging, but he points out that Claude stayed stuck in the same place for three days, needed interventions, and hallucinated actions. The model also had access to a large amount of Pokemon training data. For Kamradt, these systems show that an agent can explore a world, but they do not settle whether it can learn unfamiliar skills. A benchmark must separate generalization from performance on a known game.
ARC Prize uses humans as the reference point for AGI
ARC Prize designs problems that humans can solve while AI systems struggle with them. That gap gives researchers something measurable to work on. Kamradt calls humans the only proof point of general intelligence that we know about. He quotes John McCarthy's definition of AI as making machines do tasks they have never seen or prepared for. He also uses François Chollet's phrase "skill acquisition efficiency": intelligence involves learning new skills and learning them efficiently. ARC-AGI is built around that distinction between acquiring a skill and repeating memorized patterns.
ARC-AGI measures whether a newly learned transformation generalizes
In ARC-AGI-1, the training side shows an input grid and an output grid. The participant has to infer the transformation, then demonstrate the learned skill on another grid using an editor. Success means the participant has applied what they inferred rather than only copying the example. ARC-AGI-2 contains over 1,000 tasks, and each task is novel and unique. A skill required for one task is not reused in another, so the benchmark tests a collection of small reasoning abilities instead of repeated application of the same trick.
Interactive benchmarks test abilities that single-turn tasks leave out
Kamradt says single-turn ARC tasks provide all the needed information at test time. They do not require an agent to explore, discover goals, or learn from rewards. A more human-like evaluation should place an agent in an open world with defined rules and sparse rewards. The agent must collect observations, build a world model, and make its own rules. He connects this idea to Rich Sutton's paper "The Era of Experience," which argues that adaptable agents need experience in the open world rather than relying only on the human data available in their training.
Games provide a controlled setting for unfamiliar interactive reasoning
Kamradt presents games as a useful medium because they combine complex rules, a defined scope, and flexibility in how environments are created. Earlier Atari benchmarks had dense rewards, inconsistent reporting, no hidden test set, and games that developers already knew. That allowed developer knowledge to be built into the systems. ARC-AGI-3 aims for a stronger claim: an AI should handle games that neither the system nor its developer has seen before. Winning known games such as chess or Go is different from generalizing across many unseen environments.
ARC-AGI-3 hides evaluation games from both models and developers
ARC-AGI-3 will have a public training set of roughly 40 novel games so developers and agents can learn the interface and understand the type of environment. Scores will come from a private evaluation set with no internet access. The evaluation games will be unseen by the AI and by the developer. This separation is intended to support an assertion about generalization to open domains. Kamradt also says the benchmark will force understanding through exploration. In a game called Locksmith, the agent must discover pickups, walls, goals, and objectives instead of receiving an explanation of the rules.
The benchmark strips away specialist knowledge and tests four basic priors
ARC-AGI-3 will require only core knowledge priors. It will avoid language, text, symbols, and trivia. Kamradt names four priors: basic math, such as counting to ten; basic geometry, including shapes and topology; agentness, or understanding that other agents exist and interact; and objectness, or grouping parts that move together as one object. This design shifts attention away from specialist knowledge, where AI can already outperform people, and toward reliable learning of simple structures that humans acquire early.
Skill acquisition efficiency will be measured against human baselines
ARC Prize plans to test hundreds of humans on the same ARC-AGI-3 tasks. It will measure how long they take and how many actions they use, then compare AI systems with that baseline. The evaluation will ask whether an agent can explore, infer what is happening, create its own goals, and complete an objective efficiently. Kamradt says that if machines still cannot solve problems humans can solve, he would assert that AGI has not been reached. For interactive evaluations, action count and completion time provide practical proxies, while energy and training-data requirements are harder to measure for closed models.
ARC Prize plans an early sandbox before the full benchmark
ARC Prize plans to release five games as a sandbox preview and run a small agent competition before the full benchmark. The target is about 120 games by Q1 2026. Kamradt says each game can contain multiple levels and substantial depth, so the number of games alone does not describe the evaluation's richness. The project moved away from Unity because it was too large for the intended 2D games. ARC Prize is building a lightweight Python engine and is looking for game developers, game designers, and adversarial testers.
"We're going to drop AI and humans into this world and they won't know what's going on until they start exploring."10:15
Who should watch
You are building or evaluating agents and need tests that separate memorization from learning unfamiliar tasks.
You want to understand why a hidden, interactive benchmark might say more about generalization than performance on known games.
You are interested in contributing games, agents, adversarial tests, or engineering work to ARC-AGI-3.
