Khanmigo uses Socratic tutoring, platform context, and specialized education interfaces to help students learn without simply revealing answers.
2
Khan Academy moved toward rapid prototyping, demo-driven development, and different software quality levels for prototypes, experiments, and widely released features.
3
Scaling Khanmigo requires work on math accuracy, evaluation, model routing, latency, cost, and classroom safety.
Summary
Shawn Jansepar describes Khan Academy's development of Khanmigo, an AI tutor for learners and assistant for teachers. The product grew from rapid prototypes made with OpenAI into a launched product built for classroom use. Jansepar explains how Khanmigo guides students with leading questions, checks their mathematical work, uses context from Khan Academy exercises, and supports writing and teacher workflows. Khan Academy changed its product process so teams could prototype uncertain ideas quickly, test them with smaller groups, and apply stricter engineering and accessibility checks before broad release. The talk also covers an AI platform team, math agents, retrieval, prompt-based tutoring steps, evaluation dashboards, human review, model-graded tests, and multiple models in production. Jansepar is direct about remaining problems. Model graders catch about 70% of math mistakes in their comparisons, multilingual testing is difficult, and dedicated capacity can sit idle outside peak demand. Classroom trust, affordability, speed, and accuracy shape the engineering choices.
Khanmigo is designed to guide learning instead of handing over answers
Khan Academy's goal is to use AI to make one-to-one tutoring available at scale. Jansepar contrasts Khanmigo with a general chatbot that immediately supplies an answer. Khanmigo asks leading questions and tries to make the student think through the problem. In a math example, it catches an error in distributing negative four, then backs up and helps the learner understand the mistake. The tutor also receives the exercise, the student's entered work, and the step-by-step solution when it is launched from Khan Academy's own platform. That context lets it respond to the student's actual attempt rather than treating the exchange as a disconnected chat.
Education products need interfaces built around classroom tasks
Jansepar argues that the product experience matters as much as the underlying model. Khanmigo includes a math input widget for complex equations and specialized graph rendering. Its writing coach breaks essay work into outlining, drafting, and revision. The system can help with a thesis and give feedback on evidence, structure, style, tone, and conclusion, but it refuses to write the next paragraph for the student. Teachers can inspect the conversation history and see how the essay developed, including words that appeared without a visible drafting process. Khan Academy is also applying the same approach to its coding platform.
Classroom integration turns safety concerns into product features
Khan Academy focused on classrooms to reach students who would not independently search for a learning product, including students who believed they were bad at math. Khanmigo connects student activity with teacher tools. A moderation feature flags inappropriate or off-track conversations and notifies a teacher. Classroom snapshots provide an overview of activity, with paths into skill reports and graphs. Teachers can use the available data to draft a progress report for a student's family. Other tools include lesson planning and exit tickets. Microsoft sponsored access to these teacher tools for teachers in the United States, with plans to extend it around the world.
Rapid prototypes and domain expertise replaced a heavyweight planning process
Jansepar says Khanmigo was built through rapid prototyping and iteration rather than a long requirements document and fixed Gantt chart. The team used demos to identify promising ideas and stopped work on demos that did not work well. He recommends pairing strong builders with domain experts who understand the customers, preferably within the product team. The OpenAI partnership began when OpenAI needed AP Biology questions for model testing. After an initial period with little progress, the teams prototyped and red-teamed together, narrowed the use cases, and prepared a launch alongside GPT-4. Khan Academy also held a companywide hackathon and embedded content staff with OpenAI for reinforcement learning using 100 math tutoring examples.
Software quality should match confidence in the feature
Khan Academy changed its product process to support uncertain AI ideas. A low-confidence idea can start as a scrappy prototype. A more promising idea can become an A/B test or a beta for a subset of users. A feature intended for everyone receives higher code quality and accessibility requirements, with checklists for that stage. Jansepar says this gives teams room to work differently while keeping stronger controls for broad releases. The organization also documented major decisions through architecture decision records, later called organization decision records, covering the problem, driver, approver, and contributors. Its stated values were changed to include trusting intuition and resisting perfectionism.
An AI platform team provides shared infrastructure for product teams
Khan Academy added an AI platform team between its infrastructure and product groups. The team owns the developer and prompt-engineer experience for Khanmigo. Its component interface in Go makes tracing and Langfuse available when developers conform to the interface. The team also owns an AI router, a chat component with APIs that product teams can extend, and consultation for teams building new features. Khan Academy was exploring a product called Khanmigo as a service, which would let third-party companies embed the tutor rather than build one from scratch.
Accuracy depends on agents, context, and ongoing evaluation
To improve tutoring accuracy, Khan Academy gives the system access to a math agent, a basic calculator, and Python for more advanced problems. Its prompting process asks one model call to reason about how the student reached an answer and how the student should have reached it, then passes that context to another call that generates the tutoring response. The team also supplies step-by-step solutions and retrieved context. A tutoring accuracy dashboard tracks quality as prompts, models, and other parts change. Khan Academy planned a math accuracy benchmark for evaluating whether a model is an accurate tutor, rather than simply whether it can solve math or reason.
Scale requires tradeoffs between quality, cost, speed, and capacity
Khanmigo has to be reliable for daily classroom use, affordable for lower-income districts, and fast enough that students and teachers do not become frustrated. Khan Academy uses different models for different workflows, falls back to shared capacity when dedicated capacity reaches its limits, and adds interface changes that make waiting feel clearer. It received a Microsoft donation of Azure language-model compute. Planned work included load balancing between OpenAI and Azure, evaluating GPT-4o and smaller models such as Phi-3, and working with Microsoft on Phi-3.14, described as a model optimized for math tutoring. The team also wanted dynamic scaling because dedicated capacity provisioned for peak demand sits underused at night.
"We need Khanmigo to be reliable enough for teachers and students to use it daily, cheap enough for lower income districts to afford it and fast enough such that it doesn't lead to user frustration."20:15
Who should watch
You are building an AI product for a regulated or high-trust setting and need examples of how safety concerns can become workflow and interface features.
Your team is deciding how much engineering effort to put into an uncertain AI feature before you have evidence that users want it.
You need practical ideas for improving model accuracy, evaluation, latency, and capacity management in a production application.
