# From Mixture of Experts to Mixture of Agents with Super Fast Inference

Daniel Kim, Cerebras Systems & Daria Soboleva, Cerebras | AI Engineer World's Fair 2025 | 53:15

Source: https://www.youtube.com/watch?v=tzRvcTEapzo
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/from-mixture-of-experts-to-mixture-of-agents-with-super-fast-inference
Published: 2025-06-27
Tags: agents, inference, latency, multi-agent, prompt-engineering

## TL;DR
- Mixture-of-experts models increase parameter capacity by routing each token to selected feed-forward experts instead of activating one large feed-forward network every time.
- Mixture-of-agents systems combine responses from several prompted language models across parallel and sequential layers, then use a final model to produce one answer.
- Fast inference makes large multi-agent systems more practical, illustrated by an example that reduced a difficult math problem from 293 seconds to 7.4 seconds.

## Summary
Daria Soboleva explains how mixture-of-experts models split a transformer's feed-forward work among specialized networks, with a router selecting experts for each token. This lets a model grow in parameter count without activating every parameter for every token. Daniel Kim then introduces mixture of agents, which applies a related idea outside one trained model. Several language models receive custom roles and prompts, work in parallel within layers, and pass combined outputs to later layers and a final summarization model. The workshop lets attendees build and tune this system with Qwen3-32B and other Cerebras-hosted models. The speakers argue that fast inference matters because multi-agent reasoning can require many calls and hundreds of thousands of tokens. They also describe Cerebras hardware, whose distributed on-chip memory is designed to reduce data movement during inference. The talk is practical, including a Streamlit app and a code-generation challenge.

## Key ideas
### Mixture-of-experts lets models grow without activating every parameter
[06:54](https://www.youtube.com/watch?v=tzRvcTEapzo&t=414s)
Soboleva describes mixture of experts as a way to scale language models after simple parameter and data scaling become harder. In a standard transformer, the feed-forward network must handle many kinds of information, from different languages to domains such as biology and mathematics. MoE replaces one monolithic feed-forward network with several expert networks. A router selects the relevant expert for each token, so the model can contain more parameters while activating roughly the same number of parameters as a dense model. The intended result is greater model capacity and better skills without a matching increase in inference time.

### Inference-time compute trades latency for harder reasoning
[10:19](https://www.youtube.com/watch?v=tzRvcTEapzo&t=619s)
Kim contrasts ordinary generation with reasoning models on a math competition problem. GPT-4o took 45 seconds and produced the wrong answer in his test, while o3 took 293 seconds and produced the right answer. He uses the example to explain why more computation after training can improve answers, especially when a problem requires several sequential steps. The downside is latency. Waiting nearly five minutes can be unacceptable for interactive applications, even when the answer is correct.

### Mixture of agents combines specialised model calls into one answer
[12:29](https://www.youtube.com/watch?v=tzRvcTEapzo&t=749s)
A mixture-of-agents system sends the same input to multiple language models with custom system prompts. Each agent produces a response, and a final model combines those responses into one answer. Kim says the approach was introduced and benchmarked by Together AI, with results that surpassed frontier models on some benchmarks. The agents do not need to be the strongest individual models. Their different roles and combined outputs can produce a better result when the system is designed for the task.

### Parallel agent layers can turn many weaker calls into a faster result
[14:44](https://www.youtube.com/watch?v=tzRvcTEapzo&t=884s)
Kim describes a production example from Ninjate.ai that solved the same math problem in 7.4 seconds. A planning agent first produced eight possible solutions. A critique agent rejected them, so the planner generated another set with 16,000 context tokens of reasoning. The critique step then identified two possible candidates, and a summarization agent converted those into the final answer. The system used more than 500,000 generated tokens and 32 language-model calls, with some calls parallel and others sequential. Cerebras inference was presented as the factor that made this amount of work fit into a short response time.

### Cerebras reduces inference bottlenecks by keeping memory beside compute
[16:39](https://www.youtube.com/watch?v=tzRvcTEapzo&t=999s)
Kim compares a GPU design with Cerebras hardware. He says an H100 has about 17,000 compute cores while much of the model memory sits off-chip, so memory channels can become a bottleneck as models and KV caches grow. Cerebras instead has 900,000 cores and 900,000 distributed memory stores, with each core having direct access to its memory. The weights remain available on the chip, reducing waits for transfers. Across chips, Kim says Cerebras mainly transfers activations, while multi-GPU systems must move more activations and cache computations between layers.

### Prompted agent specialisation can replace repeated manual refinement
[18:58](https://www.youtube.com/watch?v=tzRvcTEapzo&t=1138s)
Soboleva frames mixture of agents around a common failure mode: a monolithic model often needs repeated prompting and refinement on a complex problem, eventually reaching a context limit and forcing a restart. In the MoA design, each agent handles one portion of the task through prompt engineering. She compares this to a surgery team in which different people specialise in different parts of the operation. The intended benefit is that one question can produce a complete answer instead of requiring several rounds of correction.

### The workshop makes system design the programming task
[28:51](https://www.youtube.com/watch?v=tzRvcTEapzo&t=1731s)
The hands-on exercise asks attendees to configure an MoA system rather than write the solution directly. The app lets users set a summarization agent, create agents with custom prompts, choose models, adjust temperature, rename or delete agents, and add layers or iterations. A code challenge asks the system to generate a Python function called calculate_user_matrix. Agents are assigned to find bugs and edge cases, improve performance, and combine the results. An automated grader scores the generated function, with 120 points as the target.

### MoA quality depends on engineering the full configuration
[38:07](https://www.youtube.com/watch?v=tzRvcTEapzo&t=2287s)
During Q&A, Kim is direct that a mixture-of-agents system does not automatically perform well. Poor prompts can make the system perform poorly, so users must tune the prompts, models, layers, and combination strategy for their use case. Soboleva adds that even similarly configured agents can sometimes improve a result through an ensemble effect. They also warn that adding too many agents can create redundancy and extra latency. If many agents produce poor answers while one agent is correct, the final system can become worse rather than better.

## Notable quotes
- "A mixture of agents is basically the ability to take advantage of these earthshattering speeds from our hardware and apply them into harder problems like this." (12:29)
- "So this whole process, even though it took 7 seconds, took over 500,000 tokens to be generated and 32 LLM calls, some of them in parallel, some of them sequential." (15:38)
- "The whole point though is that it can be better. It's just that you have to actually engineer it to be better." (42:27)
- "So if you have a very large model you want to serve, then Cerebras is the best position to do that." (50:43)

## Tools & references mentioned
- Cerebras Systems
- Cerebras Wafer-Scale Engine
- Qwen3-32B
- Llama 3.3-70B
- Mixture of Experts
- Mixture of Agents
- GPT-3
- Llama 3
- DeepSeek
- GPT-4o
- o3
- Together AI
- Ninjate.ai
- NVIDIA H100
- Streamlit
- GitHub
- SlimPajama
- BTLM-3B-8K
- Mistral
- state space models

## Who should watch
- You are building an LLM application that needs more reliable answers on multi-step problems and want to compare one model with a layered agent system.
- You want a working starting point for MoA rather than a purely theoretical explanation, including prompts, model selection, layers, and a Streamlit deployment.
- Your inference bill or response time makes large numbers of model calls impractical, and you want to understand the hardware and latency tradeoffs behind this design.

## Related talks

- [Scaling the Next Paradigm of Heterogeneous Intelligence](https://aietalks.com/talks/scaling-the-next-paradigm-of-heterogeneous-intelligence) (Adrian Bertagnoli, Callosum, 15:13)
- [Building Agents (the hard parts!)](https://aietalks.com/talks/building-agents-the-hard-parts) (Rita Kozlov, Cloudflare, 21:12)
- [Will Agent Evaluation via MCP Stabilize Agent Networks?](https://aietalks.com/talks/will-agent-evaluation-via-mcp-stabilize-agent-networks) (Ari Heljakka, Root Signals, 14:11)
- [MCP is all you need](https://aietalks.com/talks/mcp-is-all-you-need) (Samuel Colvin, Pydantic, 15:24)
- [Building Multi-agent Systems with Finite State Machines](https://aietalks.com/talks/building-multi-agent-systems-with-finite-state-machines) (Adam Terlson, 17:11)