# Storyteller: Building Multi-modal Apps with TypeScript and ModelFusion

Lars Grammel, PhD, ModelFusion | AI Engineer Summit 2023 | 07:30

Source: https://www.youtube.com/watch?v=KnndhPSVF3E
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/storyteller-building-multi-modal-apps-with-typescript-and-modelfusion
Published: 2024-01-23
Tags: latency, multimodal, voice

## TL;DR
- Storyteller turns a short voice description into a roughly two-minute audio story with a title, image, character voices, and narration.
- The application improves responsiveness by streaming partially completed story structures and processing title, image, and audio work in parallel.
- ModelFusion handles partial parsing and supports multiple model providers, while the application code decides when a story section is ready for narration and which voice to use.

## Summary
Lars Grammel presents Storyteller, a TypeScript application for making short audio stories for preschool children. A React client records a spoken topic and sends it to a Fastify server, where OpenAI Whisper transcribes it. The server then creates a story outline, title, image, and narrated audio. The title, image, and audio generation run in parallel. The slowest part is expanding the story into dialogue and speaker sections with GPT-4, which takes about a minute and a half. Grammel avoids making the user wait by streaming and partially parsing the structured output. Each completed story section is sent to speech synthesis as soon as it is ready. Speaker descriptions are used to retrieve suitable voices, with gender filtering and duplicate-voice checks. The client receives results through server-sent events and starts playback when the first audio section is available, while later sections continue generating in the background.

## Key ideas
### Storyteller turns one voice input into a complete children's story
[00:01](https://www.youtube.com/watch?v=KnndhPSVF3E&t=1s)
The application needs only a voice input describing a topic. It generates an audio story of about two minutes, with a story, characters, images, and synthesized narration. Grammel opens with an example about Benny discovering that the forest is losing its color and that the trees are dying. The app is written in TypeScript and uses ModelFusion for AI orchestration. The example establishes that the system goes beyond a single text response: it coordinates transcription, writing, image generation, voice selection, and speech synthesis.

### The client gets incremental updates through a React and Fastify setup
[00:55](https://www.youtube.com/watch?v=KnndhPSVF3E&t=55s)
Storyteller has a React client and a custom Fastify server. The client shows a record topic button, captures audio, and sends the released recording to the server as a buffer. OpenAI Whisper transcribes a short topic in about 1.5 seconds. The server sends the result back through an event stream using server-sent events, and React updates its state when each event arrives. This lets the interface show progress instead of waiting for every model call to finish.

### The outline starts several generation tasks at the same time
[02:07](https://www.youtube.com/watch?v=KnndhPSVF3E&t=127s)
After transcription, Storyteller generates a story outline with GPT-3 Turbo Instruct, which Grammel found could produce one in about four seconds. Once the outline exists, the application starts title generation, image generation, and narrated-audio generation in parallel. The title uses GPT-3 Turbo Instruct again and is sent to the client as soon as it is available. The design reduces waiting by allowing independent pieces of the result to progress together.

### The image prompt is derived from the whole story for consistency
[02:56](https://www.youtube.com/watch?v=KnndhPSVF3E&t=176s)
Image generation first needs a prompt. Storyteller sends the complete story to GPT-4, which extracts representative keywords for an image prompt. That prompt is passed to Stability AI Stable Diffusion XL. The generated image is stored as a virtual file on the server. The server then sends a path to the client, which retrieves the image through a regular URL in an image tag. Using the whole story to create the prompt is intended to keep the generated image aligned with the story.

### Partial parsing lets narration begin before the full story is finished
[03:49](https://www.youtube.com/watch?v=KnndhPSVF3E&t=229s)
Generating the complete audio story is the slowest operation. A prompt sends the story to GPT-4, asks for dialogue and speakers, and extends the story. With a low temperature, the model takes about one and a half minutes, which Grammel considers too slow for an interactive client. Storyteller instead streams the structured output and repeatedly checks whether a new passage is complete. ModelFusion performs the partial parsing and returns an iterator over fragments. The application decides which finished story part can be narrated.

### Voice selection combines speaker descriptions with retrieval
[05:04](https://www.youtube.com/watch?v=KnndhPSVF3E&t=304s)
Each story part needs a voice. The narrator has a predefined voice, and speakers that already have assigned voices can proceed immediately. For a new speaker, Storyteller uses GPT-3.5 to produce a structured result containing gender and a voice description. The system had previously embedded voice descriptions, so it retrieves matching voices and filters them by gender. It then selects a voice while checking that two speakers do not receive the same voice. Speech synthesis uses the selected voice with supported providers such as Speech Synthesis and ElevenLabs.

### Playback starts while later story sections are still being generated
[06:05](https://www.youtube.com/watch?v=KnndhPSVF3E&t=365s)
Each synthesized section is stored virtually on the server, and the client receives a path that it reconstructs into a URL for a media element. When the first audio section is complete, playback can start. The server continues generating and synthesizing additional sections in the background while the user listens. Grammel's responsiveness strategy combines a multi-part loading state, streaming, parallel backend processing, early playback, and model choices aimed at reducing generation time.

## Notable quotes
- "Storyteller is implemented using TypeScript and ModelFusion, an AI orchestration library that I've been developing." (00:01)
- "The key idea is streaming the structure." (04:14)
- "We narrate each story part as it's getting finished." (04:40)
- "You can start listening while the processing is still going on." (06:52)

## Tools & references mentioned
- Storyteller
- ModelFusion
- TypeScript
- React
- Fastify
- OpenAI Whisper
- GPT-3 Turbo Instruct
- GPT-4
- GPT-3.5
- Stability AI Stable Diffusion XL
- ElevenLabs
- Speech Synthesis

## Who should watch
- You are building a TypeScript application that needs to coordinate transcription, text generation, images, and speech rather than calling one text model.
- Your users need an interactive experience even when a complete model-generated result takes much longer than a single request.
- You want an example of partial structured-output parsing, retrieval-based voice selection, and multi-provider model integration with ModelFusion.

## Related talks

- [From Text to Vision to Voice: Exploring Multimodality with OpenAI](https://aietalks.com/talks/from-text-to-vision-to-voice-exploring-multimodality-with-openai) (Romain Huet, OpenAI, 23:39)
- [Keynotes & Multimodality Track](https://aietalks.com/talks/keynotes-multimodality-track) (Benjamin Duny & Simon Sturmer, AI Engineer & Chris Lattner, Modular & Auntie Aart & Mike Chambers, AWS & Alex Albert, Anthropic & Harrison Chase, LangChain & Rob Chandra, Substrate & Vic Kapati, M87 Labs & Ben Hilac, Dawn & Karan Goel, Cartesia & Chang She, LanceDB & Noah Schnapp, Character.AI & Steph Duenas & Quinn Dombrowski, Daily & Roman Huet, OpenAI & Brian Bischof, Charles Fry, Hamza Hussein, Jason Liu, Shreya Shankar & Eugene Yan & Thomas Dohmke, GitHub, 8:34:14)
- [See, Hear, Speak, Draw](https://aietalks.com/talks/see-hear-speak-draw) (Logan Kilpatrick & Simón Fishman, OpenAI, 18:43)
- [Building Reactive AI Apps](https://aietalks.com/talks/building-reactive-ai-apps) (Matt Welsh, Fixie.ai, 17:02)
- [Multi model multimodal and multi agent innovations in Azure AI](https://aietalks.com/talks/multi-model-multimodal-and-multi-agent-innovations-in-azure-ai) (Cedric Vidal, Microsoft, 28:56)