# Accelerating Mixture of Experts Training With Rail-Optimized InfiniBand Networking in Crusoe Cloud

Ievgen, Crusoe | AI Engineer World's Fair 2024 | 17:45

Source: https://www.youtube.com/watch?v=tQTB4MU_z8w
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/accelerating-mixture-of-experts-training-with-rail-optimized-infiniband
Published: 2025-02-12
Tags: gpus, inference

## TL;DR
- Distributed training can spend about 25% of its time exchanging data between GPUs, leaving expensive hardware idle.
- Crusoe's rail-optimized InfiniBand design uses NVIDIA's PXN feature to move traffic across GPU rails with one network hop.
- On a fine-tuning run of a sparse mixture-of-experts model using 240 H100 GPUs, enabling PXN improved training time by 14%.

## Summary
Ievgen explains why GPU networking affects the cost and duration of distributed model training. Each training step requires GPUs to exchange their computed data, often through all-reduce and related operations. Without network optimization, this communication can take about 25% of training time, even after overlapping computation and communication. Crusoe separates ordinary VPC traffic from a high-bandwidth InfiniBand backend and uses a rail-optimized topology to reduce the hops between GPUs. NVIDIA's PXN feature lets an internal NVSwitch route traffic across rails before it reaches the leaf switch. Synthetic NCCL tests showed gains for both small and large messages, but Ievgen focuses on model-level results. In a fine-tuning run with a sparse mixture-of-experts model on 240 H100 GPUs, PXN produced a 14% improvement in training time. The talk also briefly describes Crusoe's compute, storage, networking, and climate-aligned infrastructure.

## Key ideas
### Distributed training leaves GPUs waiting on network exchanges
[09:47](https://www.youtube.com/watch?v=tQTB4MU_z8w&t=587s)
Ievgen says distributed training has a compute phase and a communication phase. When a training step finishes, GPUs must exchange the information they calculated, often through all-reduce operations. Without optimization, networking takes about 25% to 30% of training time. During that period, GPUs are idle and cannot continue computing, while customers continue paying for the infrastructure. Overlapping computation with communication helps, but in Crusoe's work with a customer it reduced the problem by only about 10%, leaving roughly 25% of training time spent on the network.

### Crusoe separates customer traffic from GPU-to-GPU traffic
[07:29](https://www.youtube.com/watch?v=tQTB4MU_z8w&t=449s)
Crusoe Cloud offers GPU-attached virtual machines, CPU instances, local NVMe storage, persistent block storage, and ordinary VPC networking for traffic from the Internet or a customer's environment. GPU-to-GPU communication uses a separate rail-optimized InfiniBand cluster network. Ievgen describes this backend as a high-performance, low-latency, high-bandwidth fabric whose topology is built around GPU communication. The separation keeps general-purpose customer traffic distinct from the data exchanges required during distributed training.

### A standard fat-tree can add failure domains and network hops
[12:08](https://www.youtube.com/watch?v=tQTB4MU_z8w&t=728s)
Ievgen describes the traditional fat-tree fabric as a scalable way to build a potentially non-blocking network, then points out two problems. If servers connect through one leaf switch, that leaf becomes a choke point. It also becomes a single failure domain, since losing the leaf disconnects every GPU attached to it. In a two-node example, GPUs on different nodes connected to the same leaf can communicate in one hop. Traffic between different rails must travel up to the spine and back down, adding another hop and potentially adding latency to all-reduce operations.

### PXN uses the host's internal switch to cross GPU rails
[13:19](https://www.youtube.com/watch?v=tQTB4MU_z8w&t=799s)
NVIDIA introduced PXN in a recent version of NCCL. The feature lets the internal NVSwitch inside a host act as a traffic hub across rails. When a GPU on one host needs to communicate with a GPU on another host, the traffic can pass through the internal switch and then go to the leaf where the destination connection resides. This keeps the path to one network hop while still allowing communication across different GPU rails. GPUs within one server can continue using their embedded NVLink and NVSwitch without leaving the host fabric.

### Synthetic NCCL tests showed gains for both message sizes
[14:48](https://www.youtube.com/watch?v=tQTB4MU_z8w&t=888s)
Crusoe ran NCCL tests after enabling PXN and saw a reported 50% improvement for small messages and a 50% improvement for large messages. Ievgen explains that the small-message result is mainly about latency, while large messages are more concerned with bandwidth. Latency tends to remain roughly stable as message size changes, so the two tests emphasize different network properties. He also warns that these are synthetic measurements. They describe the workload applied to the network rather than the time a customer needs to train a particular model.

### Sparse mixture-of-experts models create heavy all-to-all traffic
[15:34](https://www.youtube.com/watch?v=tQTB4MU_z8w&t=934s)
For a customer-facing test, Crusoe used a sparse mixture-of-experts model. The model routes traffic between different expert layers, so a large GPU cluster creates substantial network demand as GPUs send information to one another. The example used the open-source Mixtral model, described here as having eight feed-forward blocks and seven billion parameters, and fine-tuned it on 240 H100 GPUs. This workload gave Crusoe a way to measure whether the network design changed actual model training rather than only producing better benchmark scores.

### PXN reduced model training time by 14% in the reported run
[16:46](https://www.youtube.com/watch?v=tQTB4MU_z8w&t=1006s)
The fine-tuning comparison showed a 14% improvement when PXN was enabled. Ievgen connects that result directly to the time required to train the model and therefore to training cost. He treats this model-level number as more useful to customers than the synthetic NCCL results because it describes a workload they care about. The result comes from the stated Mixtral fine-tuning run on 240 H100 GPUs. The talk does not provide a wall-clock duration or dollar amount for the run.

## Notable quotes
- "Without any optimization, the networking part takes about 25 to 30% of the training time." (10:12)
- "The traffic communication will be passing through the one single leaf, just one hop away, and you will get to the destination." (13:42)
- "The 14% of improvement is something that can be directly connected to the time to train the model, that can be directly connected to cost of training the model." (17:07)
- "What customers care about is the time to train the particular model." (15:19)

## Tools & references mentioned
- Crusoe Cloud
- InfiniBand
- NVIDIA
- NCCL
- PXN
- NVLink
- NVSwitch
- Mixtral
- Together AI
- BOSON AI
- Cohere
- H100
- AI Engineer World's Fair

## Who should watch
- You are training mixture-of-experts or other distributed models across many GPUs and need to understand where network time enters the run.
- Your cluster design uses multiple GPU rails, leaf switches, and a spine, and you are comparing standard fat-tree networking with a rail-optimized layout.
- You want a concrete model-training result rather than only network microbenchmarks, including the reported 14% improvement from enabling PXN.

## Related talks

- [GPUs & Inference Track](https://aietalks.com/talks/gpus-inference-track) (Santos, Covalent & Dylan Patel, SemiAnalysis & S Madra, Groq & Yen Vinko, Crusoe & Kait, Google DeepMind & Dima, Fireworks AI & Scott Wu, Cognition AI, 4:08:41)
- [Hacking the Inference Pareto Frontier](https://aietalks.com/talks/hacking-the-inference-pareto-frontier) (Kyle Kranen, NVIDIA, 20:25)
- [Infra behind Krea 2: How to train and serve at scale](https://aietalks.com/talks/infra-behind-krea-2-how-to-train-and-serve-at-scale) (Gabriel Jorge Menezes, Krea.ai, 16:55)
- [Running LLMs Locally: Practical LLM Performance on DGX Spark](https://aietalks.com/talks/running-llms-locally-practical-llm-performance-on-dgx-spark) (Mozhgan Kabiri chimeh, NVIDIA, 10:16)
- [Mastering LLM Inference Optimization From Theory to Cost Effective Deployment](https://aietalks.com/talks/mastering-llm-inference-optimization-from-theory-to-cost-effective-deployment) (Mark Moyou, NVIDIA, 33:39)