# State Space Models for Realtime Multimodal Intelligence

Karan Goel, Cartesia | AI Engineer World's Fair 2024 | 14:26

Source: https://www.youtube.com/watch?v=U9DPRZ0lSIQ
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/state-space-models-for-realtime-multimodal-intelligence
Published: 2024-10-29
Tags: edge, long-context, multimodal, voice

## TL;DR
- Realtime AI needs streaming models that respond continuously instead of batch models that take seconds to return an answer.
- State space models compress incoming information into an internal memory, which gives them lower memory use and more efficient scaling on long sequences.
- Cartesia is applying state space models to voice generation and is working toward models that run with low latency and low power on devices such as Macs.

## Summary
Karan Goel argues that AI has focused too heavily on batch intelligence, where a model receives a request, spends time reasoning, and returns an answer. Many useful systems instead need to process audio, video, and sensor streams continuously. Voice assistants, generated worlds, and robots all need low-latency models that can run cheaply on phones, laptops, and other devices. Goel explains why Transformers become expensive as context grows, especially for multimodal data. State space models address this by updating a compact internal memory as each token arrives and discarding the token after it has been processed. This gives them a streaming interface, lower memory use, and roughly linear scaling with context length. Goel is honest that compression involves a quality trade-off, but argues it helps with very long inputs such as security-camera footage. He closes with Cartesia's voice-generation work and its efforts to bring similar performance to local devices.

## Key ideas
### Realtime applications need a different model workload from long-form reasoning
[00:00](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=0s)
Goel separates batch intelligence from streaming intelligence. Batch systems can take a few seconds to answer a difficult math or physics question. Streaming systems must return responses continuously and with low latency while they receive audio, video, or sensor data. He gives conversational voice, on-device assistants, real-time world generation, and robotics as examples. A voice agent could handle customer-support calls, answer health-insurance questions, or contact a vendor about a shipment. Robots would combine audio, video, and other sensor inputs and respond immediately. These applications require models that work constantly and cheaply on devices such as phones and laptops.

### Long multimodal sequences make compression a basic requirement
[04:52](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=292s)
Goel says practical data is often much longer than text. A day of security-camera footage contains a large amount of information, although only a small part may matter later. Humans process and compress this kind of input, retaining what they need to remember or understand. Current AI systems more often keep context available for retrieval and reasoning. Goel estimates that humans process about a billion text tokens, 10 billion audio tokens, and a trillion video tokens over a year, while the best AI models operate in a much smaller token range. He sees long-lived memory as a major unsolved capability.

### Transformer attention becomes expensive as multimodal context grows
[08:08](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=488s)
Goel explains that Transformers attend to past tokens, so the computation required for long context grows quadratically. Text sequences are often relatively short, but audio and video produce many more tokens. If future applications make inference happen thousands or even 100,000 times more often, the same scaling pattern could make them too expensive to run broadly. Data-center optimization alone does not solve the problem because these systems need to operate on phones, laptops, and robots. Cartesia's response is to investigate architectures that can handle long context with less computation and memory.

### State space models update compact memory instead of retaining every token
[10:29](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=629s)
Goel contrasts a Transformer's behavior with a state space model. To predict the next word in 'the quick brown fox jumped', a Transformer keeps the context and attends to its past tokens. A state space model receives a token stream, updates an internal memory, and then discards the token. The memory is a compressed state that influences future generation. Goel compares it to a zipped file inside the model. This recurrence creates a natural streaming interface and reduces the need to keep all past information in memory. He connects the approach to how humans handle information over time.

### Compression can improve answers when the context is extremely long
[12:20](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=740s)
Goel acknowledges a trade-off between compressing information and retaining every detail. He argues that compression is especially useful for large contexts. In the security-camera example, a model that compresses 24 hours of footage as it arrives may answer questions better than a system that examines all 24 hours from scratch each time. He says compression is less useful for short contexts, where preserving more information is easier. In his account, state space models already have good quality on long-context and multimodal tasks, although he does not present detailed benchmark results in this talk.

### Cartesia is applying state space models to low-latency voice generation
[12:52](https://www.youtube.com/watch?v=U9DPRZ0lSIQ&t=772s)
Goel describes Cartesia's recent voice-generation release as an application of state space models to multimodal data. The system performs text-to-speech and is designed to keep inference and training efficient. In the online playground, users receive generated voice almost immediately from the data center. Cartesia is also working on running the models on Macs and other devices, with the goal of giving local hardware a similar experience at low power. Goel ends by directing the audience to play with the model and send feedback, since he does not have time to run the planned live demo.

## Notable quotes
- "Our hypothesis is you need new architectures." (09:17)
- "You have a token stream in, they update an internal memory for the model, and then the token gets thrown away." (11:12)
- "Compression is super helpful for a large context, not as helpful for short context." (12:38)
- "You get instant voice back generated from the data center." (13:27)

## Tools & references mentioned
- Cartesia
- Mamba
- Transformers
- Jensen Huang
- Stanford
- AI Engineer World's Fair

## Who should watch
- You are building voice assistants, agents, robotics systems, or other products that must process streams with low latency.
- Your model handles long audio, video, or sensor context and the cost of retaining and attending to every token is becoming a problem.
- You want a practical introduction to why state space models differ from Transformers and where Cartesia is applying them.

## Related talks

- [Keynotes & Multimodality Track](https://aietalks.com/talks/keynotes-multimodality-track) (Benjamin Duny & Simon Sturmer, AI Engineer & Chris Lattner, Modular & Auntie Aart & Mike Chambers, AWS & Alex Albert, Anthropic & Harrison Chase, LangChain & Rob Chandra, Substrate & Vic Kapati, M87 Labs & Ben Hilac, Dawn & Karan Goel, Cartesia & Chang She, LanceDB & Noah Schnapp, Character.AI & Steph Duenas & Quinn Dombrowski, Daily & Roman Huet, OpenAI & Brian Bischof, Charles Fry, Hamza Hussein, Jason Liu, Shreya Shankar & Eugene Yan & Thomas Dohmke, GitHub, 8:34:14)
- [Scaling the Next Paradigm of Heterogeneous Intelligence](https://aietalks.com/talks/scaling-the-next-paradigm-of-heterogeneous-intelligence) (Adrian Bertagnoli, Callosum, 15:13)
- [See, Hear, Speak, Draw](https://aietalks.com/talks/see-hear-speak-draw) (Logan Kilpatrick & Simón Fishman, OpenAI, 18:43)
- [Multi model multimodal and multi agent innovations in Azure AI](https://aietalks.com/talks/multi-model-multimodal-and-multi-agent-innovations-in-azure-ai) (Cedric Vidal, Microsoft, 28:56)
- [Your Voice Agent Doesn't Need a Frontier Model](https://aietalks.com/talks/your-voice-agent-doesnt-need-a-frontier-model) (Joel Allou & Ornella Bahidika, Microsoft, 05:45)