# The Multimodal Future of Education

Stefania Druga, Google Gemini team | AI Engineer World's Fair 2024 | 20:05

Source: https://www.youtube.com/watch?v=qpmZID27t98
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/the-multimodal-future-of-education
Published: 2024-10-24
Tags: guardrails, human-in-the-loop, multimodal

## TL;DR
- Children need to understand how AI works so they can judge its answers and decide how much to trust it.
- AI learning tools work better when they help learners form hypotheses, test ideas, and keep ownership of the work.
- Multimodal AI can turn drawings, images, and spoken questions into activities for science, mathematics, curiosity, and creative coding.

## Summary
Stefania Druga argues that AI education should begin with tinkering. Children already interact with voice assistants and generative AI, but their assumptions about machine intelligence shape the questions they ask and how much they trust the answers. Her Cognimates platform lets learners program devices, train models, and inspect confidence levels. Studies with children showed that this work made them more skeptical of claims about AI intelligence. Druga then describes research on AI coding companions for families. These tools can suggest ideas and explain code while leaving the child responsible for the project, although families also need the ability to turn the assistant off. Her live Gemini demos use drawings and images to ask science and maths questions, encourage learners to continue solving problems, and prompt curiosity about objects. The demos are open source, with one implementation using fewer than 100 lines of code.

## Key ideas
### Children's beliefs about AI shape how they use it
[00:38](https://www.youtube.com/watch?v=qpmZID27t98&t=38s)
Druga says 70% of generative AI users in a recent Salesforce study are from Generation Z. Children have also grown up with voice assistants since around 2015. In her earlier research, younger children were skeptical about how smart Google Home and Alexa were, while school-age children often said that voice devices were smarter than they were. Those beliefs affect the queries children ask and how much they trust the answers. Druga argues that education should cultivate AI literacy and a critical understanding of how these systems work.

### Cognimates teaches AI literacy through making
[03:00](https://www.youtube.com/watch?v=qpmZID27t98&t=180s)
Druga built Cognimates, an open-source, free platform that expands Scratch with AI features. Children can program smart lights and voice assistants, train custom models with image or text examples, and use those models in games. One example trains a model to distinguish unicorns from narwhals, then shows the confidence of its prediction. Other projects examined food, implemented rock-paper-scissors, and made a robot speak like Shakespeare. The platform was translated into more than 30 languages.

### Training models made children less certain that AI was smarter than they were
[04:51](https://www.youtube.com/watch?v=qpmZID27t98&t=291s)
In a longitudinal study in public and private schools, Druga asked children what they thought about AI, gave them activities involving text training, image training, and smart-home programming, then asked the same questions again. Before the activities, children often said Google Home or a model was smarter than they were. After learning how the systems worked and training models themselves, they were much less certain. Druga attributes this change to giving children a sandbox where they could tinker, form hypotheses, and test them.

### A coding companion should help children continue their own ideas
[08:48](https://www.youtube.com/watch?v=qpmZID27t98&t=528s)
Druga's research with children and parents in ten US states examined what families wanted from an AI co-pilot for Scratch. Participants liked getting coding ideas when they were stuck, turning a game concept into suitable code constructs, and working with an AI friend that supported their identity as a creative coder. The child still did the coding. The assistant could also moderate family collaboration by prompting people to take turns. Families needed a way to stop the assistant because it could become distracting, and parents still had to explain concepts when the AI's explanations were too complex.

### Multimodal AI can connect learning to physical making and simulation
[11:34](https://www.youtube.com/watch?v=qpmZID27t98&t=694s)
Druga describes multimodal AI as a possible creative sidekick, learning companion, and coach. In games, Minecraft, physics simulations, or science simulations, it could respond to what a learner builds. A learner might photograph flowers and colors to get ideas for 3D models, or use AI to inspire knitting projects. Druga also worked with Nancy Ooto on a benchmark for common K-12 maths misconceptions and on evaluating whether generative AI models can identify those misconceptions in conversations with children.

### Her science demo turns drawings into questions for testing hypotheses
[13:00](https://www.youtube.com/watch?v=qpmZID27t98&t=780s)
In a live Gemini API demo, Druga shows drawings to a model and asks what might happen. A sketch of a balance scale with weights leads to a question about adding another five kilograms. She then changes the drawing to show ten kilograms on each side. Other examples involve mixing water and carbon dioxide and arrows striking the Earth. The system interprets the changing drawings and responds with questions, giving the learner a way to alter the hypothesis and try again.

### The maths demo gives prompts instead of completing the solution
[14:35](https://www.youtube.com/watch?v=qpmZID27t98&t=875s)
Druga's maths demo asks the learner to solve expressions with parentheses. When she reaches a step involving division, the system does not simply provide the answer. It gives another question so she can continue working. For a system of equations, it suggests subtracting seven from both sides and then dividing both sides by two. Druga presents this as a way to keep the learner involved rather than delegating the whole task to a chatbot.

### Small open-source experiments can move AI beyond chatbot interfaces
[19:02](https://www.youtube.com/watch?v=qpmZID27t98&t=1142s)
Druga says the demo code is open source and available as a template. It uses fewer than 100 lines of code and requires a free API key. Developers can change the instructions and build their own versions. She asks the audience to think beyond chatbot interfaces and delegated instructions, and to put these tools in the hands of young people so they can learn how the technology works.

## Notable quotes
- "We need to cultivate AI literacy and critical understanding of this technology." (03:00)
- "By providing this platform and allowing them to tinker and form hypothesis and test them, we basically allow them to engage in the scientific process." (07:13)
- "The bot was just helping them when they were stuck." (09:51)
- "I think that's my time." (19:44)

## Tools & references mentioned
- Salesforce
- Cognimates
- Scratch
- Google Home
- Alexa
- Microsoft Human-AI eXperience Team
- X Moonshot Factory
- Fixie.ai
- Hugging Face
- Gemini API
- AI Studio
- Vertex API
- LearnML
- Nancy Ooto

## Who should watch
- You are building educational tools and need examples of how an AI assistant can support a learner without taking over the work.
- You are designing child-facing AI products and want to think about trust, model confidence, family use, and the ability to turn assistance off.
- You want practical multimodal prototypes that connect images or drawings to maths and science activities.

## Related talks

- [Keynotes & Multimodality Track](https://aietalks.com/talks/keynotes-multimodality-track) (Benjamin Duny & Simon Sturmer, AI Engineer & Chris Lattner, Modular & Auntie Aart & Mike Chambers, AWS & Alex Albert, Anthropic & Harrison Chase, LangChain & Rob Chandra, Substrate & Vic Kapati, M87 Labs & Ben Hilac, Dawn & Karan Goel, Cartesia & Chang She, LanceDB & Noah Schnapp, Character.AI & Steph Duenas & Quinn Dombrowski, Daily & Roman Huet, OpenAI & Brian Bischof, Charles Fry, Hamza Hussein, Jason Liu, Shreya Shankar & Eugene Yan & Thomas Dohmke, GitHub, 8:34:14)
- [The Era of Unbounded Products: Designing for Multimodal IO](https://aietalks.com/talks/the-era-of-unbounded-products-designing-for-multimodal-io) (Ben Hylak, Dawn, 20:32)
- [See, Hear, Speak, Draw](https://aietalks.com/talks/see-hear-speak-draw) (Logan Kilpatrick & Simón Fishman, OpenAI, 18:43)
- [Second Order Effects of AI](https://aietalks.com/talks/second-order-effects-of-ai) (Cheng Lou, 21:46)
- [The Intelligent Interface](https://aietalks.com/talks/the-intelligent-interface) (Samantha Whitmore & Jason Yuan, New Computer, 18:56)