The Multimodal Future of Education

Stefania Druga, Google Gemini team20:05 · Oct 2024 · 3,015 views
Thumbnail for The Multimodal Future of Education Watch on YouTube
TL;DR
  1. 1

    Children need to understand how AI works so they can judge its answers and decide how much to trust it.

  2. 2

    AI learning tools work better when they help learners form hypotheses, test ideas, and keep ownership of the work.

  3. 3

    Multimodal AI can turn drawings, images, and spoken questions into activities for science, mathematics, curiosity, and creative coding.

Summary

Stefania Druga argues that AI education should begin with tinkering. Children already interact with voice assistants and generative AI, but their assumptions about machine intelligence shape the questions they ask and how much they trust the answers. Her Cognimates platform lets learners program devices, train models, and inspect confidence levels. Studies with children showed that this work made them more skeptical of claims about AI intelligence. Druga then describes research on AI coding companions for families. These tools can suggest ideas and explain code while leaving the child responsible for the project, although families also need the ability to turn the assistant off. Her live Gemini demos use drawings and images to ask science and maths questions, encourage learners to continue solving problems, and prompt curiosity about objects. The demos are open source, with one implementation using fewer than 100 lines of code.

Key ideas
00:38

Children's beliefs about AI shape how they use it

Druga says 70% of generative AI users in a recent Salesforce study are from Generation Z. Children have also grown up with voice assistants since around 2015. In her earlier research, younger children were skeptical about how smart Google Home and Alexa were, while school-age children often said that voice devices were smarter than they were. Those beliefs affect the queries children ask and how much they trust the answers. Druga argues that education should cultivate AI literacy and a critical understanding of how these systems work.

03:00

Cognimates teaches AI literacy through making

Druga built Cognimates, an open-source, free platform that expands Scratch with AI features. Children can program smart lights and voice assistants, train custom models with image or text examples, and use those models in games. One example trains a model to distinguish unicorns from narwhals, then shows the confidence of its prediction. Other projects examined food, implemented rock-paper-scissors, and made a robot speak like Shakespeare. The platform was translated into more than 30 languages.

04:51

Training models made children less certain that AI was smarter than they were

In a longitudinal study in public and private schools, Druga asked children what they thought about AI, gave them activities involving text training, image training, and smart-home programming, then asked the same questions again. Before the activities, children often said Google Home or a model was smarter than they were. After learning how the systems worked and training models themselves, they were much less certain. Druga attributes this change to giving children a sandbox where they could tinker, form hypotheses, and test them.

08:48

A coding companion should help children continue their own ideas

Druga's research with children and parents in ten US states examined what families wanted from an AI co-pilot for Scratch. Participants liked getting coding ideas when they were stuck, turning a game concept into suitable code constructs, and working with an AI friend that supported their identity as a creative coder. The child still did the coding. The assistant could also moderate family collaboration by prompting people to take turns. Families needed a way to stop the assistant because it could become distracting, and parents still had to explain concepts when the AI's explanations were too complex.

11:34

Multimodal AI can connect learning to physical making and simulation

Druga describes multimodal AI as a possible creative sidekick, learning companion, and coach. In games, Minecraft, physics simulations, or science simulations, it could respond to what a learner builds. A learner might photograph flowers and colors to get ideas for 3D models, or use AI to inspire knitting projects. Druga also worked with Nancy Ooto on a benchmark for common K-12 maths misconceptions and on evaluating whether generative AI models can identify those misconceptions in conversations with children.

13:00

Her science demo turns drawings into questions for testing hypotheses

In a live Gemini API demo, Druga shows drawings to a model and asks what might happen. A sketch of a balance scale with weights leads to a question about adding another five kilograms. She then changes the drawing to show ten kilograms on each side. Other examples involve mixing water and carbon dioxide and arrows striking the Earth. The system interprets the changing drawings and responds with questions, giving the learner a way to alter the hypothesis and try again.

14:35

The maths demo gives prompts instead of completing the solution

Druga's maths demo asks the learner to solve expressions with parentheses. When she reaches a step involving division, the system does not simply provide the answer. It gives another question so she can continue working. For a system of equations, it suggests subtracting seven from both sides and then dividing both sides by two. Druga presents this as a way to keep the learner involved rather than delegating the whole task to a chatbot.

19:02

Small open-source experiments can move AI beyond chatbot interfaces

Druga says the demo code is open source and available as a template. It uses fewer than 100 lines of code and requires a free API key. Developers can change the instructions and build their own versions. She asks the audience to think beyond chatbot interfaces and delegated instructions, and to put these tools in the hands of young people so they can learn how the technology works.

"By providing this platform and allowing them to tinker and form hypothesis and test them, we basically allow them to engage in the scientific process."07:13
Who should watch
  • You are building educational tools and need examples of how an AI assistant can support a learner without taking over the work.
  • You are designing child-facing AI products and want to think about trust, model confidence, family use, and the ability to turn assistance off.
  • You want practical multimodal prototypes that connect images or drawings to maths and science activities.