# Substrate Launch: the API for modular AI

Rob Cheung, Substrate | AI Engineer World's Fair 2024 | 05:20

Source: https://www.youtube.com/watch?v=x8HbIJh2wpQ
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/substrate-launch-the-api-for-modular-ai
Published: 2025-02-06
Tags: inference, latency, workflows

## TL;DR
- The strongest AI products combine multiple inference runs and model types in a logical system instead of relying on one foundation model.
- Substrate pairs a developer SDK for describing computation graphs with an inference engine that runs those graphs efficiently.
- Substrate moves data between nodes in microseconds, which makes online applications with dozens of nodes feasible.

## Summary
Rob Cheung introduces Substrate, a system for building AI products from connected semantic tasks. He argues that a foundation model is rarely enough for useful applications, including document summarization and end-to-end coding. Multiple inference runs can be arranged into explicit graphs, making the system easier to understand, debug, extend, and evaluate. Substrate combines an SDK for describing graphs with an inference engine designed to execute them efficiently. Its infrastructure can optimize batching, caching, networking, concurrency, and physical placement across a coordinated compute cluster. Cheung contrasts this with separate API calls, where each step adds DNS resolution, proxy, authentication, and balance-check overhead. He says Substrate transfers data between processes in microseconds, around 10,000 times faster than that pattern, making applications with dozens of nodes practical. The talk also mentions investment in fast, reliable JSON decoding for multi-inference workflows.

## Key ideas
### Useful AI products combine several inference runs
[00:33](https://www.youtube.com/watch?v=x8HbIJh2wpQ&t=33s)
Cheung says successful products built with current AI usually use more than one inference run, often combining different model types for a specific task. He says developers quickly discover that a foundation model alone is insufficient. This applies to simple work such as summarizing a document and to harder work such as solving coding problems end to end. In his view, the product comes from arranging inference runs in a logical structure rather than treating one model call as the complete application.

### Modular systems are easier to inspect and change
[00:55](https://www.youtube.com/watch?v=x8HbIJh2wpQ&t=55s)
Substrate's approach is based on modular intelligence. Cheung says these systems are more legible because their structure can be understood directly. That makes them easier to debug and extend. Evaluations also become easier because the decision trees are explicit, so a developer can check each step and identify where something went wrong. The argument is practical: separating tasks gives developers places to inspect behavior instead of hiding the whole application inside one large model interaction.

### Substrate combines a graph SDK with an inference engine
[02:08](https://www.youtube.com/watch?v=x8HbIJh2wpQ&t=128s)
Cheung describes Substrate as two connected parts. The first is a developer SDK for describing a computation graph with any number of nodes. The nodes cover several modalities and tasks, including image generation, speech transcription, text generation, JSON, embeddings, and code execution. The second part is an inference engine built specifically to run those graphs efficiently. Together, they let developers describe relationships between tasks and run the resulting system as one coordinated computation.

### The engine optimizes how graph tasks run
[02:34](https://www.youtube.com/watch?v=x8HbIJh2wpQ&t=154s)
A graph represents multiple tasks and the relationships between them. Because Substrate runs on a coordinated compute cluster, it can optimize how those tasks execute. Cheung names batching, caching, networking, concurrency, and physical placement as areas for static and dynamic optimization. These details matter when one user request creates many dependent operations. The graph gives the engine information about the whole computation, rather than forcing it to treat every step as an unrelated API request.

### Separate API calls add latency at every step
[02:58](https://www.youtube.com/watch?v=x8HbIJh2wpQ&t=178s)
Cheung contrasts Substrate with frameworks that dispatch many API calls separately. Each step may require DNS resolution, proxy handling, authentication, and balance checks. He says this adds hundreds of milliseconds of latency to every step. Substrate instead transfers data from node to node and process to process in microseconds. He describes that as roughly 10,000 times faster and says the difference makes online applications with dozens of nodes feasible.

### JSON decoding is a central pattern in multi-inference systems
[04:01](https://www.youtube.com/watch?v=x8HbIJh2wpQ&t=241s)
Cheung says JSON decoding is one of the most useful patterns when several inference runs are combined. Substrate has invested in a JSON mode designed for reliability and speed. The feature fits the broader system because individual model outputs need to become structured inputs for later nodes. Reliable, fast decoding helps connect those tasks without turning the application into a chain of fragile parsing steps.

### The system is intended to support higher-quality AI applications
[04:25](https://www.youtube.com/watch?v=x8HbIJh2wpQ&t=265s)
Cheung presents Substrate as infrastructure for building AI applications from flexible, inspectable, and verifiable systems. His claim is that coordinating many specialized nodes can produce better outcomes than relying on a single monolithic intelligence. The SDK provides the graph abstraction, while the inference engine handles execution concerns such as data movement and scheduling. He closes by directing interested developers to the website, the Expo floor, and a QR code for credits.

## Notable quotes
- "Building with modular intelligence is always going to be more effective than building with a monolithic intelligence." (01:21)
- "These systems are inherently more legible, which means you can understand them structurally, which means that they're debuggable and they're extensible." (01:21)
- "We transfer data from node to node, process to process, on the order of microseconds, which is some 10,000 times faster." (03:41)
- "JSON decoding is one of the most useful patterns for multi-inference runs." (04:01)

## Tools & references mentioned
- Substrate
- JSON
- SDK
- embeddings

## Who should watch
- You are building an AI product that needs several model calls to complete one user request and are running into latency between steps.
- Your current orchestration code hides the structure of the workflow, making debugging and evaluation difficult.
- You want to connect text, image, speech, embedding, or code-execution nodes through a single computation graph.

## Related talks

- [More Nodes Is All You Need](https://aietalks.com/talks/more-nodes-is-all-you-need) (Nicolas Schlaepfer, Hyperspace, 05:48)
- [Designing AI-Intensive Applications](https://aietalks.com/talks/designing-ai-intensive-applications) (swyx, 13:02)
- [Infrastructure for the Singularity](https://aietalks.com/talks/infrastructure-for-the-singularity) (Jesse Han, Morph, 19:31)
- [Hypermode Launch](https://aietalks.com/talks/hypermode-launch) (Kevin Van Gundy, Hypermode, 05:03)
- [The Next Unicorns: 7 Top AI Startups from the HF0 Residency](https://aietalks.com/talks/the-next-unicorns-7-top-ai-startups-from-the-hf0-residency) (Diego Rodriguez, Krea & Josh, Coframe & Eugene, Featherless AI & Jonas Bauer, Upside & Lengyue, OpenAudio & Alex Atallah, OpenRouter, 22:16)