# Stop Shipping LLM Settings That Only Win on Paper

Rohan Pandey & Balaji Varadarajan, DigitalOcean | AI Engineer | 05:33

Source: https://www.youtube.com/watch?v=cWrYEBlWpcY
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/stop-shipping-llm-settings-that-only-win-on-paper
Published: 2026-11-12
Tags: agents, inference

## TL;DR
- The optimization agent uses Bayesian optimization through Optuna to search for better VLM serving recipes.
- It runs model-serving trials against four workloads, scores generated output tokens, and produces an HTML report and optimized YAML file.
- A GPT-OSS run improved generated-token throughput by 17.5%, while a DeepSeek R1 Distill run doubled token throughput and nearly halved ITL.

## Summary
Rohan Pandey demonstrates an optimization agent that searches for better serving configurations for a given model. The agent starts with a YAML recipe, hardware details, and a trial count. Optuna runs a Bayesian optimization loop, changing recipe values after each trial and scoring the result by generated output tokens. The demo uses a GPT-OSS 120B model and tests four regimes: chat short, chat long, RAG, and batch classify. Each regime has its own input token setup. After the run, the agent writes an HTML report and an optimized YAML recipe. One GPT-OSS run found a 17.5% improvement in generated-token throughput after 50 trials. A separate DeepSeek R1 Distill run used nine trials, doubled token throughput, and reduced ITL by almost half. Rohan presents the system as a way to reduce the manual time spent testing serving settings.

## Key ideas
### The agent searches for a serving recipe for each model and workload
[00:01](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=1s)
The agent is designed to find a good VLM serving configuration for a given model. Its purpose is to reduce the time spent manually testing serving recipes and parameters. The demo tests four regimes: chat short, chat long, RAG, and batch classify. Each regime has its own input token setup, and those inputs can be changed for the experiment. The starting recipe is the configuration received when the model weights are downloaded.

### Bayesian optimization reduces the number of configurations the agent must test
[00:48](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=48s)
The search uses Bayesian optimization to find better parameter values. Rohan explains that the method minimizes the search space, which reduces the time needed to find useful numbers in the serving recipe. Optuna handles the search loop. It receives the starting recipe, runs trials, assesses each result, assigns a score, and changes the recipe before running the next trial.

### The run starts with a model, a YAML file, hardware, and a trial count
[02:20](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=140s)
Once the model is loaded in a droplet, the operator runs tuner.py. The command identifies the study, points to the starting YAML file, names the hardware, and sets the number of trials. Rohan shows an example with 24 trials and mentions another earlier run with nine. He says these runs usually take a couple of hours. The GPT-OSS run took about four or five hours.

### The agent tests GPT-OSS 120B across four serving regimes
[02:18](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=138s)
The demo uses a droplet with the GPT-OSS 120B model loaded. The starting recipe covers chat short, chat long, RAG, and batch classify. These regimes use their own ISO OSL token inputs, which can be changed depending on what the operator wants to measure. The agent then searches for better values within that recipe instead of relying on one manually chosen configuration.

### The output includes both an experiment report and a deployable recipe
[03:44](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=224s)
When the agent finishes, it generates an HTML report showing which trial performed best and how many trials were run. It also writes an optimized YAML file containing the serving values found during the search. In the GPT-OSS example, the report identifies the best trial out of 50 and shows the values selected by the agent. The report also records how each individual trial performed.

### The GPT-OSS run improved generated-token throughput by 17.5%
[04:07](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=247s)
For the GPT-OSS model, the agent found a 17.5% improvement over the first configuration in tokens generated. Rohan describes the report as measuring the amount of generated output tokens and using that as the accuracy or performance signal. The result comes from the full search run rather than from a single manually adjusted setting.

### The DeepSeek R1 Distill run doubled token throughput and nearly halved ITL
[04:48](https://www.youtube.com/watch?v=cWrYEBlWpcY&t=288s)
A second HTML report covers the DeepSeek R1 Distill model. This run used nine trials, with the ninth trial producing the best result. Rohan says token throughput doubled and ITL improved by almost half, meaning the ITL value was reduced by almost half. The example shows that the agent can compare several recipes and select the strongest one from the completed trials.

## Notable quotes
- Rohan Pandey: "The goal is to build an agent that helps find the best VLM serving config for any given model." (00:24)
- Rohan Pandey: "Bayesian optimization is a technique that's used to find the most optimal numbers in any given situation." (00:48)
- Rohan Pandey: "Optuna basically takes each recipe after every trial, assesses it, gets a score, uses a strategy to change the recipe, improve and then it reruns it within the search loop until the trials is done." (01:33)
- Rohan Pandey: "Overall we found that from the first config we had an improvement of 17.5% in terms of tokens generated." (04:27)
- Rohan Pandey: "We found that token throughput doubled and ITL was improved by almost half aka it was reduced by almost half." (04:48)

## Tools & references mentioned
- Optuna
- GPT-OSS 120B
- DeepSeek R1 Distill
- Bayesian optimization
- tuner.py

## Who should watch
- You manually test vLLM serving parameters and want a repeatable search process for model-specific recipes.
- Your team needs to compare serving configurations across several workload regimes instead of optimizing for one benchmark case.
- You want experiment outputs that include both trial history and a YAML configuration you can use after the search.

## Related talks

- [Stop Guessing Your vLLM Config: Benchmarking LLMs on Runpod](https://aietalks.com/talks/stop-guessing-your-vllm-config-benchmarking-llms-on-runpod) (Chris Vela, Runpod, 16:29)
- [LLM Quality Optimization Bootcamp](https://aietalks.com/talks/llm-quality-optimization-bootcamp) (Thierry Moreau & Pedro Torruella, OctoAI, 53:05)
- [How fast are LLM inference engines anyway?](https://aietalks.com/talks/how-fast-are-llm-inference-engines-anyway) (Charles Frye, Modal, 16:07)
- [TLMs: Tiny LLMs and Agents on Edge Devices with LiteRT-LM](https://aietalks.com/talks/tlms-tiny-llms-and-agents-on-edge-devices-with-litert-lm) (Cormac Brick, Google, 1:20:58)
- [Mastering LLM Inference Optimization From Theory to Cost Effective Deployment](https://aietalks.com/talks/mastering-llm-inference-optimization-from-theory-to-cost-effective-deployment) (Mark Moyou, NVIDIA, 33:39)
