# Local Agentic Theory For Mobile Games Shafik Quoraishee & Joanne Song, The New York Times | AI Engineer World's Fair 2026 | 18:04 Source: https://www.youtube.com/watch?v=418t26CVz-w Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks. Page: https://aietalks.com/talks/local-agentic-theory-for-mobile-games Published: 2026-07-23 Tags: computer-use, edge, planning, reinforcement-learning ## TL;DR - Running game agents on the phone reduces latency, keeps computation private, supports offline play, and allows more local personalization. - A mobile agent must fit its model, state, planning work, rendering, and energy use into tight device budgets, including a 16-millisecond frame at 60 Hz. - Accessibility can become a live set of controls that an agent adjusts from signals such as gaze, shaky taps, focus loops, and target size. ## Summary Shafik Quoraishee and Joanne Song present experimental work on agents that run inside mobile games and accessibility systems. They distinguish reinforcement learning, which changes model weights through repeated training, from agentic systems that reason over the current game state through a loop of perception, prediction, decisions, and actions. Their Space Invaders example watches the scene, predicts attacks, and moves to avoid bullets. A crossword agent uses a constraint graph and backtracking to recover from fills that stop fitting. The practical limit is the phone itself. Models, state history, planning artifacts, rendering, frame timing, and battery use all compete for resources. Song argues that accessibility should move beyond fixed menus and binary toggles. An on-device agent could resize controls, detect motor difficulty, recognize handwriting, and escape keyboard focus traps. The speakers describe a future requiring faster models, prediction, long-term memory, shared game-state languages, better chips, and honest benchmarks. ## Key ideas ### The talk concerns experimental game assistance, not AI-generated New York Times puzzles [00:01](https://www.youtube.com/watch?v=418t26CVz-w&t=1s) Shafik Quoraishee says New York Times puzzles are made by people and have no AI features in the games. The work presented is experimental. It explores whether agents can help with playability and related features on a local device, rather than generate games. He points to earlier work on a Connections solver as a deeper example of reinforcement-learning game solvers. This framing matters because the demos use crossword and game mechanics, while the speakers explicitly separate them from the Times' actual puzzle production. ### Local inference makes mobile agents faster, private, offline, and more personal [02:49](https://www.youtube.com/watch?v=418t26CVz-w&t=169s) Most practical mobile AI uses cloud calls, which adds a round trip and creates latency. Running inference on the phone can compress that work into the device's frame computation. Quoraishee also says local computation can keep gameplay data inside the device's security zone, without sending it upstream for telemetry when that is unnecessary. A local agent can keep working in places such as a subway tunnel. The device can also use local features to tailor the game experience to the individual player. ### Agentic play reasons over the current game state instead of retraining game-specific weights [05:01](https://www.youtube.com/watch?v=418t26CVz-w&t=301s) Quoraishee contrasts reinforcement learning with agentic systems. Reinforcement learning runs many iterations that change a model's weights until it becomes tuned to a particular game. He cites EfficientZero and EfficientZero V2 as sample-efficient reinforcement-learning models that learn with less data. An agentic system instead uses in-context reasoning to understand the game state, then calls local tools and adapts to a new situation. Its loop perceives the scene, predicts what may happen, chooses an action, acts, and observes the result again. ### The Space Invaders agent repeats perception, prediction, decision, and action on the phone [07:03](https://www.youtube.com/watch?v=418t26CVz-w&t=423s) The Space Invaders demo uses a local agentic framework. The agent tries to destroy the spaceships as they approach and avoid incoming bullets. Its internal states show the loop: perceive what is happening, predict the spaceships' next move, decide what to do, and act when a bullet or spaceship is close. The speaker presents this as a simple agent, but it illustrates the difference between an agentic loop and a model trained only to play one fixed game. ### A phone agent has to share limited space, time, and energy with the game [08:18](https://www.youtube.com/watch?v=418t26CVz-w&t=498s) The device budget includes model weights, compressed state history, planning artifacts, and enough render headroom for the game itself. At a 60 Hz refresh rate, each frame arrives about every 16 milliseconds, so planning must fit inside that interval or the player sees jank. Battery life adds another limit because current phones are not yet optimized for sustained agentic workloads. Quoraishee describes a constraint graph that manages these tradeoffs, penalizing excessive time more heavily when it threatens the user experience while allowing softer limits elsewhere. ### Constraint graphs let a crossword agent backtrack when a fill breaks the puzzle [11:23](https://www.youtube.com/watch?v=418t26CVz-w&t=683s) The mini crossword demo uses a constraint satisfaction agent. A satisfaction graph represents where words can fit and how their letters must agree. When the constraints become too crowded or incorrect, the agent backtracks rather than continuing with a bad fill. It can use natural language from the graph to find another word and recover a valid crossword layout. The example shows a different kind of agentic game interaction from Space Invaders, with graph structure guiding search instead of rapid movement. ### Accessibility works better as graded, live controls than as a fixed easy-mode switch [12:33](https://www.youtube.com/watch?v=418t26CVz-w&t=753s) Joanne Song grounds the design in WCAG 2.2, using the framework's perceivable, operable, understandable, and robust pillars. She describes problems that static settings miss, such as tiny crossword cells, invisible active-cell highlights, manual logins, and timers that do not fit a player's processing speed. She connects this to the draft direction of WCAG 3.0, which moves from binary pass/fail toward bronze, silver, and gold scores. In gameplay, the equivalent is a set of dials. Input tolerance can make physical interaction more forgiving, while step granularity can reduce the operations needed for a sequence. ### On-device agents can detect difficulty and change the interface while a person is playing [14:40](https://www.youtube.com/watch?v=418t26CVz-w&t=880s) Song describes models that read eye gaze to detect search friction and analyze shaky taps when a player struggles with controls. The system could also allow handwriting recognition. In her keyboard example, an agent monitors focus order, detects an endless loop in a broken dialogue, and injects an exit route on the device. It can act as a live layout auditor, measure the interface, and resize small controls for a player with limited dexterity or inaccurate taps. The game changes around the person's current needs instead of forcing the person to adapt to a fixed layout. ### Local game agents still need prediction, memory, shared state, faster chips, and real tests [16:46](https://www.youtube.com/watch?v=418t26CVz-w&t=1006s) The speakers say agents must make plans within 16 milliseconds to avoid stuttering. They also need to predict the result of a layout change before applying it, remember one person's habits and needs over time, and use a shared game-state language so the same agent can work across multiple games. Better chips are part of the hardware answer, but Quoraishee and Song also call for honest testing and real benchmarks that show whether the agent improves the experience. Their closing picture is billions of local agents shaped by the people using them. ## Notable quotes - "Our puzzles are made by people. They're not made by AI." (00:31) - "You need to actually fit the agent planning within that, and if you go outside of that budget, what will happen in the agent execution, you're going to see jank." (08:53) - "Accessibility and challenge stop being treated separately and become two ends of one dial constantly tuning for the moment." (16:23) - "The future of AI doesn't have to be one giant centralized brain. It can be billions of small local brains, each running on a personal device, each shaped entirely by the individual it serves." (17:22) ## Tools & references mentioned - The New York Times - Wordle Bot - Connections - Pac-Man - AlphaGo - AlphaZero - EfficientZero - EfficientZero V2 - WCAG 2.2 - WCAG 3.0 - Space Invaders - Google Gemini - Arc AGI - WCAG ## Who should watch - You are building mobile games and need AI behavior that continues to work without a cloud connection. - You are designing game accessibility and want interface changes to respond to motor, visual, cognitive, or navigation difficulty as it happens. - You are evaluating on-device agents and need to account for frame timing, memory, rendering capacity, and battery use together.