AI training needs a dedicated backend network because synchronized GPUs can generate extreme, bursty traffic at the same time.
2
Enterprise AI racks need much more power and cooling than ordinary data center racks, with GPU servers drawing 10.2 kW and newer racks reaching 100 to 200 kW.
3
Network teams need lossless Ethernet controls, GPU-aware telemetry, careful load balancing, and no oversubscription on the backend network.
Summary
Paul Gilbert explains how AI infrastructure changes the work of enterprise network teams. Training uses a dedicated backend network that connects GPU servers and carries synchronized, bursty east-west traffic. A separate frontend network connects the training system to storage, while inference can use another network. Gilbert describes why ordinary data center assumptions fail: a single H100 server can place 4.8 terabytes of traffic on the network, GPU racks need water cooling and far more power, and one slow or failed GPU can affect the whole job. He recommends simple isolated designs with no backend oversubscription, careful load balancing, and Rocky v2 flow control through ECN and PFC. He also discusses RDMA error capture, GPU-to-switch telemetry, software upgrades without taking switches offline, and EVPN-VXLAN for multi-tenancy. The talk is aimed at executives and engineers planning self-managed AI networks.
AI networks need separate backend and frontend roles
Gilbert separates an AI data center into a backend network and a frontend network. The backend connects the GPU servers and is completely isolated because the GPUs are expensive, power-hungry, and difficult to obtain. A typical server has eight GPUs connected to high-speed leaf and spine switches. The frontend network connects the training system to storage and carries less intense traffic. Gilbert later adds a third network for inference, where the trained model is placed for serving. The GPU backend is where synchronized computation creates the strongest networking demands.
Synchronized GPUs create bursty traffic that requires a one-to-one design
In ordinary data centers, Gilbert has used oversubscription ratios such as 1:10 and 1:3. He says AI backend networks need no oversubscription because GPUs can all burst at once. An H100 server has eight 400-gigabit GPU ports and four 400-gigabit frontend ports, which Gilbert gives as 4.8 terabytes of possible traffic. He says future 800-gigabit systems could reach 9.6 terabytes per server. Since the GPUs synchronize, a slow component can hold back the rest of the job, making job completion time the practical measure of network performance.
Traditional network load balancing uses a five-tuple based on IP addresses, ports, and MAC addresses. GPU traffic can use a single IP address, causing flows to line up on one uplink and overload it. Gilbert describes a method that balances according to the percentage of bandwidth used on each uplink rather than relying only on the five-tuple. He says this can reach about 93 percent utilization across uplinks to the downlinks. He also discusses cluster load balancing, which considers the collective being run, because the communication pattern of the GPU collective affects the network design.
Gilbert compares ordinary data center racks, which he describes as using about 7 to 15 kW, with GPU servers that draw 10.2 kW for one eight-GPU system. A rack designed for ordinary servers may hold only one such GPU server. Enterprises therefore need new racks, and Gilbert says many are building racks in the 100 to 200 kW range. These racks require water cooling because air cooling is not sufficient. East-west traffic between GPUs is the most intense traffic path, while north-south traffic connects the GPUs to storage.
Congestion control needs both gradual feedback and an emergency stop
AI networks need congestion control because switch buffers can fill when synchronized GPU traffic arrives together. Gilbert describes Rocky v2 as using ECN and PFC. ECN marks packets when congestion appears, and the receiver tells the sender to slow down. The sender can speed up again when the congestion marks stop. PFC pauses traffic when a buffer is full and cannot accept more. Gilbert describes ECN as a gradual slowdown mechanism and PFC as a dead stop. Since all GPUs are synchronized, one slowdown can affect an entire pool.
GPU-aware telemetry helps separate network faults from GPU faults
Gilbert says network teams need to identify whether a failure comes from the GPU or the network. RDMA can expose multiple error conditions, so the switch can capture a dropped packet, its headers, RDMA information, and the reason it was dropped. Arista also provides an agent that runs on NVIDIA GPUs and communicates with the switch. The GPU reports its configuration and statistics such as packets sent, packets received, and RDMA errors. This lets operators compare GPU and switch information instead of treating every model failure as a generic network problem.
Operational uptime requires upgrades without stopping GPU jobs
Gilbert describes a smart system upgrade feature that allows switch software to be upgraded without taking the switch offline. He gives the example of a network with 1,024 GPUs and 64 switches, where the GPUs can continue working during the upgrade. This matters because AI networks are expensive to build and are expected to run continuously. Gilbert also recommends simple addressing with point-to-point /30 or /31 links, BGP for routing, and EVPN-VXLAN when multiple business units need to share the infrastructure.
The next network generation moves more work into the NICs
Gilbert introduces the Ultra Ethernet Consortium as an effort to improve congestion control, packet spraying, and communication between network interface cards. He says version 1.0 was expected to be ratified in the first quarter of 2025, with products likely later in the year. The approach puts more functionality into the NICs and removes some work from the network. Cloud-scale operators were interested because the network can focus on forwarding packets while the NICs handle more of the coordination.
"When we build these networks we don't build them over subscribe, we build them one to one."07:12
Who should watch
You are an AI executive planning a self-managed GPU cluster and need to understand the power, cooling, storage, and network consequences before buying hardware.
Your data center team knows conventional leaf-spine networks but needs guidance on GPU collectives, bursty east-west traffic, and lossless Ethernet.
You operate AI workloads and need better evidence about whether model failures come from GPUs, cables, optics, switches, or congestion.
