SGLang is an open-source serving framework for large language models and vision-language models, with support for new model releases from labs such as Qwen and DeepSeek.
2
SGLang performance depends on choosing compatible configuration flags, including CUDA graph batch sizing and speculative decoding settings.
3
Eagle 3 speculative decoding must be tuned with prompts that match the production workload because token acceptance depends on the input and output content.
Summary
Philip Kiely and Yineng Zhang introduce SGLang as an open-source framework for serving language and vision-language models. They explain where it fits alongside vLLM, Ollama, and TensorRT-LLM, then walk through deploying a model with a SGLang server packaged in a Truss configuration on Baseten. The workshop focuses on performance tuning. Zhang demonstrates how a default CUDA graph maximum batch size can be too small for a real benchmark workload, then adjusts it so decoding requests continue to use CUDA graphs. They also configure Eagle 3 speculative decoding, explain how its draft model is derived from layers of the target model, and show how to search over draft depth, top-k, and verification-token settings. The speakers finish with SGLang's repository structure, contribution paths, supported features, and community channels. In the questions, Kiely explains why Baseten values SGLang's configurability, extensibility, and open-source code, especially for teams that need to inspect and lock down an inference stack.
SGLang is one production option for serving language and vision models
Philip Kiely describes SGLang as an open-source serving framework for large language models and large vision models. He places it alongside other production choices such as vLLM, Ollama, and TensorRT-LLM. The reasons he gives for using it are performance across a wide range of GPUs, production readiness, support for new model releases on launch day, and an active open-source community. Baseten uses SGLang in its inference stack, while Kiely says xAI uses it for Grok models. He also names inference providers, cloud providers, research labs, universities, and product companies as users.
SGLang grew from a paper into a widely used project quickly
Kiely says the SGLang arXiv paper appeared in December 2023. At the time of the workshop, he describes the project as having reached almost 15,000 GitHub stars in about 18 months. Yineng Zhang adds background on his own involvement. Before joining Baseten, he worked on click-through-rate ranking model optimization and inference optimization at Microsoft, where SGLang creator Lianmin Zheng contacted him. Zhang later worked closely with Zheng and Ying Sheng, and also worked on FlashInfer, which SGLang uses for attention and sampling kernels.
A first deployment is a server command packaged for a GPU
The deployment exercise starts with the basic SGLang server command. In the workshop environment, the command and dependencies are placed in a Truss YAML configuration, packaged, and sent to a GPU. The examples use NVIDIA L4 GPUs because they are available at lower cost and support FP8, while the same general approach applies to H100 and H200 GPUs. After deployment, participants call the model with sample Python code, a Jupyter notebook, or an OpenAI-compatible client. The model ID from the Baseten workspace is needed to construct the request URL.
SGLang tuning requires understanding how configuration flags interact
Kiely says SGLang usage involves knowing the available flags, configuration options, support matrix, and interactions between settings. He gives a concrete warning: enabling a major speculation algorithm while also raising the batch size substantially may produce poor results. Quantization, by contrast, can work well with some other optimizations. The workshop first runs a basic model server, then uses benchmark traffic to inspect prefill and decode behavior. The point is to adjust settings based on the workload and the server logs rather than treating every flag as independent.
CUDA graph performance can depend on the maximum captured batch size
Zhang demonstrates SGLang on an L4 GPU with the Llama 3 8B Instruct model and the FlashInfer attention backend. For this model and hardware, the default CUDA graph maximum batch size is eight. During a benchmark with ten running requests, the decode batch exceeds that value, so the log reports that CUDA graphs are not being used. Zhang then restarts the server with the CUDA graph maximum batch size set to 32. A later decode batch with 13 running requests uses a CUDA graph. The server logs expose whether CUDA graph execution is active, which helps decide whether the setting needs adjustment.
Eagle 3 builds its speculative draft from the target model
Zhang explains that SGLang supports Eagle 3 speculative decoding. Unlike a standard draft-target setup that runs a separate smaller model, Eagle uses multiple layers from the target model to build the draft model. The server configuration specifies the Eagle algorithm, a target model, a draft-model path, drafting depth, Eagle top-k, and the number of draft verification tokens. The speakers describe a tuning script that runs combinations of batch sizes, drafting steps, top-k values, and token counts, then reports speed and acceptance rate so a suitable combination can be selected for serving.
Speculative decoding must be benchmarked with representative prompts
Kiely stresses that speculative decoding is sensitive to the topic and content of the requests. The benchmark prompts should resemble the actual production inputs and outputs. If the test data is unrelated to the live workload, the selected Eagle 3 settings may be wrong. Zhang points to the benchmark script where users can replace the example prompts. The tuning process compares both serving speed and token acceptance rate. A configuration with a high acceptance rate is generally more useful because more of the proposed tokens can be kept instead of being discarded.
The SGLang repository offers several concrete contribution paths
The speakers describe SGLang as having a runtime, a domain-specific frontend language, and optimized kernels. Zhang points contributors to GitHub issues labeled 'good first issue' or 'help wanted', along with the development roadmap. The repository includes the SGLang kernel library for attention, normalization, and activation operations, the SGLang router with cache-aware routing, and the runtime's support for disaggregated prefill and decode, constrained decoding, function calling, an OpenAI-compatible server, and many model architectures. Someone adding a custom model can use an existing architecture such as Llama as a reference.
"If you're able to have a really deeply and well documented codebase like SGLang where you're able to really deeply understand all the different options that you have, that can get you a long way."36:55
Who should watch
You are serving open-source language or vision-language models and want a hands-on first deployment with SGLang.
Your current benchmark shows that decoding is missing CUDA graph execution, or you need a method for tuning Eagle 3 speculative decoding.
You want to inspect or modify an inference runtime, contribute CUDA kernels, add model support, or work with an internally hosted and locked-down serving stack.
