AI inference and agentic workloads increasingly depend on small coordination messages, making tail latency more important than raw throughput.
2
TCP and RDMA react to congestion from the sender and treat data as a byte stream, which can leave short messages waiting behind large transfers.
3
Homa uses message boundaries, receiver-issued grants, and switch priority queues to reduce short-message latency without slowing large messages.
Summary
John Ousterhout argues that networking requirements for AI are changing. Training still sends large transfers such as gradients, where TCP and RDMA can provide good throughput. Inference and agentic workloads increasingly exchange small pieces of metadata and coordination data, such as KV-cache lookups and barrier synchronization. These messages are sensitive to tail latency because one delayed exchange can leave every GPU waiting. Ousterhout explains that incast creates queues at the destination-side switch. TCP and RDMA then rely on delayed congestion signals at the sender, while their byte-stream model prevents them from recognizing or prioritizing individual messages. Homa is his clean-slate alternative. It carries message lengths through the transport, prioritizes short messages with shortest remaining processing time, and lets the receiver grant transmission to control congestion. It also uses switch priority queues so short messages can pass queued large transfers. In his benchmark, Homa reduces short-message P99 latency from over a millisecond with TCP to under 100 microseconds.
AI networking is moving from large transfers toward latency-sensitive exchanges
Ousterhout contrasts older training workloads, which moved gigabytes of gradients and mainly measured throughput, with newer inference and agentic workloads. These workloads exchange smaller pieces of metadata and coordination data, such as checking a distributed KV cache or performing barrier synchronization. For a small request and response, the important measure is the round-trip time. He says 99th percentile latency matters because a small number of slow messages can limit the throughput of the whole system.
One delayed synchronization message can idle every GPU
A distributed workload may run GPU computation on several nodes, then wait for a small exchange before starting the next computation phase. Every node must finish the exchange before the next phase can begin. When computation takes seconds, a few milliseconds of network delay may not matter much. Agentic workloads can have computation periods measured in milliseconds, so a synchronization delay can consume a significant share of available GPU time.
Incast builds the queue at the destination-side switch
Incast occurs when several nodes send to one destination at the same time. The destination link cannot receive all of that traffic at the combined sending rate, so packets accumulate at the egress port of the top-of-rack switch on the final hop. A short message sent to the same destination then waits behind packets from large messages. If the buffer fills, packet drops lead to timeouts and retransmissions.
Sender-side congestion control reacts too late and keeps oscillating
TCP and RDMA traditionally make the sender responsible for slowing down. Current systems use switch ECN markings instead of waiting only for packet loss, but the information still travels back to the sender after congestion has formed. Multiple senders then adjust their rates using delayed and incomplete information. Ousterhout says the result tends to oscillate between sending too much and too little, while queues and latency remain high.
Byte streams hide the boundaries needed to prioritize messages
TCP and RDMA expose a stream of bytes rather than independent messages. The transport does not know how large a message is or where one message ends. It therefore cannot prioritize a short message that follows large messages in the stream. This creates head-of-line blocking, where the short message waits behind data that could have been sent later.
Homa makes message boundaries part of the transport
Homa treats a remote procedure call as a request message followed by a response message. Once the receiver gets the first packet, it knows how much more data belongs to that message. Homa can use that information to predict remaining work and prioritize shorter messages with shortest remaining processing time. Independent messages can bypass long ones instead of being serialized into one stream.
Homa sends only an initial group of unscheduled packets. The receiver then issues grants for later scheduled packets, pacing them over time. Because the receiver sees the incoming messages and the destination-side congestion, it can delay grants when many messages are competing for the same queue. It can also grant shorter messages first.
Switch priority queues let short messages pass large transfers
Modern data center switches have multiple queues at each egress port. Homa places long-message traffic in lower-priority queues and uses higher-priority queues for short messages. In an incast situation, a short message can therefore bypass long packets already waiting in lower-priority queues. Ousterhout says this gives short messages lower latency without making long messages slower in his benchmark.
Homa improves both short-message tail latency and long-message performance in the benchmark
Ousterhout describes a benchmark with messages ranging from about 50 bytes to one megabyte, measuring round-trip time at the median and 99th percentile. For short messages, TCP's tail latency is above one millisecond, while Homa's is below 100 microseconds, which he describes as about 13 times faster. He also reports that Homa is nearly twice as fast as TCP for the largest messages in this test.
"What really matters is tail latency. That is, you'd like to know that if we send a whole lot of small messages, all of them will complete quickly."04:07
Who should watch
You run inference or agentic workloads where GPUs wait for small coordination messages between computation phases.
Your data center traffic mixes large transfers with short requests and your 99th percentile latency is much worse than median latency.
You are evaluating transport protocols and want to understand Homa's message-based, receiver-driven design.
