The strongest AI products combine multiple inference runs and model types in a logical system instead of relying on one foundation model.
2
Substrate pairs a developer SDK for describing computation graphs with an inference engine that runs those graphs efficiently.
3
Substrate moves data between nodes in microseconds, which makes online applications with dozens of nodes feasible.
Summary
Rob Cheung introduces Substrate, a system for building AI products from connected semantic tasks. He argues that a foundation model is rarely enough for useful applications, including document summarization and end-to-end coding. Multiple inference runs can be arranged into explicit graphs, making the system easier to understand, debug, extend, and evaluate. Substrate combines an SDK for describing graphs with an inference engine designed to execute them efficiently. Its infrastructure can optimize batching, caching, networking, concurrency, and physical placement across a coordinated compute cluster. Cheung contrasts this with separate API calls, where each step adds DNS resolution, proxy, authentication, and balance-check overhead. He says Substrate transfers data between processes in microseconds, around 10,000 times faster than that pattern, making applications with dozens of nodes practical. The talk also mentions investment in fast, reliable JSON decoding for multi-inference workflows.
Cheung says successful products built with current AI usually use more than one inference run, often combining different model types for a specific task. He says developers quickly discover that a foundation model alone is insufficient. This applies to simple work such as summarizing a document and to harder work such as solving coding problems end to end. In his view, the product comes from arranging inference runs in a logical structure rather than treating one model call as the complete application.
Substrate's approach is based on modular intelligence. Cheung says these systems are more legible because their structure can be understood directly. That makes them easier to debug and extend. Evaluations also become easier because the decision trees are explicit, so a developer can check each step and identify where something went wrong. The argument is practical: separating tasks gives developers places to inspect behavior instead of hiding the whole application inside one large model interaction.
Substrate combines a graph SDK with an inference engine
Cheung describes Substrate as two connected parts. The first is a developer SDK for describing a computation graph with any number of nodes. The nodes cover several modalities and tasks, including image generation, speech transcription, text generation, JSON, embeddings, and code execution. The second part is an inference engine built specifically to run those graphs efficiently. Together, they let developers describe relationships between tasks and run the resulting system as one coordinated computation.
A graph represents multiple tasks and the relationships between them. Because Substrate runs on a coordinated compute cluster, it can optimize how those tasks execute. Cheung names batching, caching, networking, concurrency, and physical placement as areas for static and dynamic optimization. These details matter when one user request creates many dependent operations. The graph gives the engine information about the whole computation, rather than forcing it to treat every step as an unrelated API request.
Cheung contrasts Substrate with frameworks that dispatch many API calls separately. Each step may require DNS resolution, proxy handling, authentication, and balance checks. He says this adds hundreds of milliseconds of latency to every step. Substrate instead transfers data from node to node and process to process in microseconds. He describes that as roughly 10,000 times faster and says the difference makes online applications with dozens of nodes feasible.
JSON decoding is a central pattern in multi-inference systems
Cheung says JSON decoding is one of the most useful patterns when several inference runs are combined. Substrate has invested in a JSON mode designed for reliability and speed. The feature fits the broader system because individual model outputs need to become structured inputs for later nodes. Reliable, fast decoding helps connect those tasks without turning the application into a chain of fragile parsing steps.
The system is intended to support higher-quality AI applications
Cheung presents Substrate as infrastructure for building AI applications from flexible, inspectable, and verifiable systems. His claim is that coordinating many specialized nodes can produce better outcomes than relying on a single monolithic intelligence. The SDK provides the graph abstraction, while the inference engine handles execution concerns such as data movement and scheduling. He closes by directing interested developers to the website, the Expo floor, and a QR code for credits.
"These systems are inherently more legible, which means you can understand them structurally, which means that they're debuggable and they're extensible."01:21
Who should watch
You are building an AI product that needs several model calls to complete one user request and are running into latency between steps.
Your current orchestration code hides the structure of the workflow, making debugging and evaluation difficult.
You want to connect text, image, speech, embedding, or code-execution nodes through a single computation graph.
