The chess coach uses Stockfish and specialized detectors to produce structured analysis, while the LLM translates that analysis into English.
2
User reports flow into Slack and a running Claude Code session, which can investigate commentary, change prompts or detectors, regenerate the result, and submit a pull request.
3
The team targets sub-three-second feedback with Gemini 3 Flash and tests models against scenarios drawn from real games.
Summary
Anant Dole and Asbjørn Steinskog explain how Take Take Take built a chess coach for its consumer app. The system does not ask an LLM to calculate chess positions. Stockfish evaluates the game, detectors identify tactics and positional features, and the LLM turns those signals into commentary. A second chess engine, Maya, estimates which moves a player at a given rating would likely choose, helping the coach explain why a strong move is difficult to find. The team also built a feedback loop around Slack and a running Claude Code session. User reports can trigger investigation, prompt or detector changes, new commentary, and a pull request for human review. The product targets feedback that feels almost instant, with average end-to-end latency around three seconds. Their evaluation set currently contains 16 scenarios, and Gemini Flash performs better than the tested Claude and GPT-5 variants on that set.
The product turns played games into explanations and learning suggestions
After a user finishes a game in the Play Magnus app, the game review shows labels such as "brilliant move" alongside generated commentary. The system explains why a move works, including the tactic, the threats, and the detectors that fired. The app also exposes broader patterns, such as accuracy during a game phase, current rating, or depth in an opening. These insights become opportunities for the coach to return to the player. The goal is to help users understand their games and improve rather than only showing whether a move was good or bad.
LLMs are unreliable chess players because language prediction does not calculate positions
The speakers trace chess software from Claude Shannon's 1949 proposal through Deep Blue's 1997 match against Garry Kasparov and DeepMind's AlphaGo and AlphaZero. When people began testing language models at chess, the models sometimes played reasonable openings but soon hallucinated and lost track of the position. The speakers attribute this to models being trained on language rather than direct calculation. They also point out that transformer architectures can play chess when trained to predict evaluations from chess positions. Such systems can reach grandmaster-level strength, but they do not explain their moves in natural language.
The pipeline gives the LLM structured chess facts instead of asking it to reason from scratch
The review first runs Stockfish through the game to identify the best move in each position. Detectors then extract the surrounding context, including threats, plans, forks, pins, skewers, doubled pawns, and other structural themes. Maya adds a human-level perspective by predicting the moves players at a given online rating would choose. That lets the system distinguish a best move from a best move that is especially hard for a 1,500-rated player to find. The LLM receives this information and translates it into English. The team deliberately limits its independent reasoning because unsupported reasoning quickly produces hallucinations.
Concrete detectors make a bad-move label useful to the player
A bad-move indicator alone does not explain how to respond. In the example, an opponent's F5 threatens to trap the player's queen and also creates a line involving Bishop G5. The detectors identify those threats, then find that the player can capture the central pawn. That capture defends the relevant square and gives the queen a way out. The generated explanation can therefore describe the danger, the defensive resource, and the reason the move matters. This is the kind of detail the team wants the coach to provide instead of a bare engine verdict.
Reported commentary can trigger an autonomous investigation with human approval
When a user reports commentary as bad, the event is posted to Slack and injected into a running Claude Code session through Channels, an MCP server feature in research preview. Claude Code invokes a commentary triage skill, examines the position, and runs scripts that can regenerate the text. It may alter the prompt, change existing detectors, or create new ones. It then checks its own output and asks questions back in Slack when it needs guidance. If the result looks correct, Anant can tell it to submit a pull request and review or merge that pull request from a mobile phone. The demo showed the agent asking what specifically felt wrong, then determining that the commentary was already correct.
Consumer feedback requires a short, predictable generation time
Players expect to move through a game review quickly, so the app cannot leave them waiting while a reasoning model produces an uncertain-length response. The team aims for feedback in under three seconds. Gemini 3 Flash usually reaches its first token in about a second, and average end-to-end latency is about three seconds. More capable reasoning models can produce good analysis, but their completion time is less predictable. The team expects those models to fit better in a future conversational coach experience, where users will wait longer for an answer.
The team evaluates models with chess scenarios taken from real games
Anant and Asbjørn maintain 16 scenarios covering areas such as tactical patterns, blunders, and hallucination control. One scenario might test whether a model recognizes and mentions a knight fork after receiving the extracted chess context. The cases come from real games, and the team uses an LLM as a judge before comparing model outputs. OpenRouter lets them switch among new versions of Gemini, Claude, and GPT models without rebuilding the evaluation setup. Gemini Flash passes about 75 percent of their scenarios, while Claude with more thinking passes just under 60 percent and takes longer. GPT-5 mini has lower latency and lower accuracy in their tests.
Separating context extraction from language generation makes the system easier to improve
The speakers' main engineering advice is to keep the data pipeline separate from the language-generation step. Building the context model is slow at first. They describe it as a large JSON structure that the team gradually prunes while checking how output quality changes. Automated evaluations show whether changes help, and domain experts provide judgments that the builders may not be able to make alone. Their feedback loop also shortens iteration time because an agent can inspect failures, edit prompts or detectors, and prepare a code change for review.
"The LLM's job is only to translate this information into English, because we really don't want it to try to figure out too much on its own, because it quickly leads to hallucination."08:51
Who should watch
You are building an LLM feature that needs fast responses but depends on facts the model cannot reliably derive on its own.
Your application receives domain-expert feedback and you want an agent to investigate reports and prepare code changes without removing human review.
You need a practical evaluation setup for comparing models on narrow, real-world tasks rather than relying on general model impressions.
