Python decorators can ship training and inference functions to selected GPU backends without requiring users to manage Docker or Kubernetes.
2
Large-model inference is split between compute-heavy prefill and memory-bandwidth-heavy decode, so serving systems need techniques such as continuous batching, disaggregated prefill, and context caching.
3
Open models and coding agents become more useful when they can be adapted to narrow domains, connected to tools, and run through production infrastructure.
Summary
This track covers the hardware and serving problems behind modern AI systems, then moves into open models and autonomous software agents. Santos demonstrates a Python-first workflow for fine-tuning many models and deploying the best one without directly managing Kubernetes or Docker. Dylan Patel explains why frontier-model inference is difficult: prefill consumes compute, decode consumes memory bandwidth, and large deployments need continuous batching, disaggregated prefill, and context caching. Groq presents a vision of much faster inference, where language models could become a central computing interface. Crusoe describes rail-optimized GPU networking and reports a 14% training improvement on a 240-GPU mixture-of-experts workload. Google presents Gemma 2, PaliGemma, and the surrounding open-model tooling. Fireworks AI focuses on serving and customizing open models for latency, cost, and function calling. Scott Wu shows how Cognition's Devon can plan, code, test, deploy, and revise software from natural-language instructions.
Python decorators can move training and inference work to chosen GPU backends
Santos describes Covalent as a way to write ordinary Python locally and send functions to selected compute. A decorator can specify a machine such as an H100 with a stated memory size and time limit. The same approach supports training, evaluation, model selection, and deployment. Users can attach their own cloud or on-premises compute, or use Covalent's GPU cluster. For inference, a Python initializer and endpoint can be deployed with one command. The service can scale to zero, scale with incoming requests, and follow custom rules based on time, GPU utilization, or request count.
Prefill and decode create different bottlenecks in large-model inference
Dylan Patel separates inference into prefill, which processes the prompt, and decode, which generates tokens one at a time. Prefill is compute-intensive. Decode is memory-bandwidth-intensive because the system repeatedly loads model weights for token generation. He uses a 1.8-trillion-parameter model to show the scale of the problem. Serving 30 tokens per second to 64 users would require far more memory bandwidth than an H100 can provide by itself. This makes inference a systems problem involving hardware, batching, networking, and serving software.
Continuous batching and disaggregated prefill are needed for affordable serving
Patel says batch size one is expensive in the cloud, while larger batches spread the cost across concurrent users. Since requests arrive at different times, serving systems need continuous batching that can add new users while existing requests are still decoding. He also describes disaggregated prefill, where one group of accelerators handles prompt processing and another handles token generation. The two workloads have different resource needs. Separating them can reduce interference between users and help control time to first token, token-rate variation, and service-level reliability.
Context caching can replace some expensive fine-tuning workflows
Patel presents context caching as an alternative for use cases that need to supply large documents to a strong model. Fine-tuning a leading closed model may be impossible, and fine-tuning a large open model can require many GPUs. With context caching, the system avoids recomputing the KV cache for the same long context. The cached data must be stored outside GPU memory, such as in CPU memory or storage, because it can be very large. Patel connects this approach to document-heavy applications such as legal and contract review, where input processing can dominate the cost.
GPU networks can spend a large share of training time exchanging data
Yen Vinko explains that distributed training includes a communication phase in which GPUs exchange results through operations such as all-reduce. Without optimization, networking can consume about 25% to 30% of training time, leaving GPUs idle while they wait. Computation and communication can overlap, but Vinko says this still left a substantial gap. Crusoe tested a rail-optimized topology using NVIDIA's PXN capability, which lets an internal NVLink switch move traffic across GPU rails while preserving a short path through the network. On a sparse mixture-of-experts fine-tuning workload using 240 H100 GPUs, the talk reports a 14% improvement.
Gemma 2 extends Google's open model family with larger and multimodal options
Google's Gemma presentation introduces Gemma 2 in 9-billion and 27-billion parameter sizes, along with PaliGemma for image and text tasks. The speaker says Gemma 2 keeps the same broad framework compatibility as Gemma 1 and is intended to be easy to fine-tune. PaliGemma combines a SigLIP vision encoder with the Gemma text decoder for image question answering, captioning, object detection, and segmentation. The talk also mentions CodeGemma and recurrent Gemma. Google provides model weights, documentation, a cookbook, Google AI Studio access for Gemma 2 27B, and a route for community contributions.
Open models can trade general capability for lower cost and better domain fit
Dima argues that a single proprietary model is often more capable than a narrow application needs. Smaller open models can be fine-tuned for a particular domain and can reduce latency and cost. Fireworks AI focuses on serving these models with custom runtime and deployment settings for different prompt lengths, throughput targets, and latency limits. The platform supports serverless inference, fine-tuning, LoRA adapters, dedicated deployments, and function-calling models. Dima describes compound AI systems in which a model is connected to retrieval, specialized models, external APIs, and other tools.
Coding agents shift engineers toward problem definition and review
Scott Wu demonstrates Devon building a name-game website from a short natural-language request, then revising it after feedback and deploying it. He says Devon can use a shell, edit code, browse the web, create pull requests, work with machine snapshots, and integrate with tools such as Slack and GitHub. Cognition uses multiple Devons in parallel, with engineers assigning tasks and reviewing the results. Wu expects software engineers to spend less time on implementation details and more time deciding what to build, selecting architectures, handling edge cases, and directing the work.
"You don't dockerize, you don't run kubernetes cluster, you don't do anything, you just say, hey, I have an initializer function and I have a need an endpoint called slash generate."26:44
Who should watch
You are deploying large language models and need to understand why prompt processing, token generation, batching, and memory bandwidth lead to different costs and latency.
Your distributed training jobs spend a noticeable amount of time waiting for GPU-to-GPU communication, and you want to compare network-topology approaches.
You are deciding between proprietary APIs, fine-tuned open models, and coding agents for a production application.
