# Customized, production-ready inference with open source models

Dmytro (Dima) Dzhulgakov, Fireworks.ai | AI Engineer World's Fair 2024 | 18:55

Source: https://www.youtube.com/watch?v=ePMvfa8vgL8
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/customized-production-ready-inference-with-open-source-models
Published: 2025-02-16
Tags: deployment, inference, open-models, tool-use

## TL;DR
- Open models can reduce latency and inference cost when a general-purpose proprietary model has capabilities a narrow application does not need.
- Production inference requires tuning the serving stack for prompt length, latency, throughput, hardware, and model-specific requirements.
- Applications increasingly combine models with retrieval, function calls, and external tools, so Fireworks focuses on serving and customizing the whole system.

## Summary
Dmytro Dzhulgakov argues that open models are useful in production because teams can adapt them to a narrow domain, latency target, and cost budget. A customer-support bot may not need the broad capabilities of a large proprietary model, while a smaller tuned model can be faster and cheaper. The practical difficulty is operating these models: teams must find GPUs, choose frameworks, optimize serving settings, update deployments, and add production monitoring. Fireworks addresses this with a serving stack built from custom CUDA kernels through deployment orchestration, plus fine-tuning and model hosting. Dzhulgakov also describes applications as compound systems, where a model works with retrieval, function calls, and external APIs. He demonstrates a function-calling model that queries stock prices, creates charts, maintains context, and changes topics. The talk is also a product overview, covering serverless inference, dedicated hardware, fine-tuned models, and enterprise deployments.

## Key ideas
### Smaller open models can fit a narrow use case better than general-purpose models
[02:08](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=128s)
Dzhulgakov says proprietary models are strong across many domains, but they are often served in the same way for every application. A customer-support chatbot may not need to know 150 Pokémon or write poetry. A voice assistant or another real-time application may care more about response time. He says smaller models such as Llama or Gemma can reach similar or better quality in a narrow domain while being up to 10 times faster. He also gives cost as a reason to use smaller models, since repeated inference from a large model can become a major expense for a startup.

### Open model weights allow teams to specialize quality for their own domain
[04:27](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=267s)
The main benefit Dzhulgakov assigns to open models is domain adaptability. There are many fine-tuned variants for areas such as coding, medicine, and particular languages. Because the weights are open, a team can tune a model for its own use case instead of accepting the capabilities and serving setup of a single general model. He presents this as a way to spend less on capabilities an application does not need while improving performance on the work it actually performs.

### Operating open models requires work across the entire inference stack
[05:08](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=308s)
Dzhulgakov describes three practical difficulties. Teams must find GPUs, choose frameworks, download models, optimize performance, and repeat the process as models change. Serving settings also depend on the use case. Prompt length, latency targets, and throughput needs can vary widely, and he says choosing well across the stack can produce multiple-times improvements. Production adds more requirements, including scalable infrastructure, telemetry, and observability. Public-cloud GPUs are not always reliable, so reaching enterprise scale takes more than starting a model process on a rented machine.

### Inference performance depends on matching deployment settings to the workload
[05:47](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=347s)
Fireworks built a serving stack from its own CUDA kernels through service-level deployment and orchestration. Dzhulgakov says the stack can be tuned for a customer's model, cost, and latency requirements. Long prompts from retrieval applications need different runtime and deployment settings from short prompts. Teams can tune caching and other settings to raise throughput while keeping latency within a target. He frames the optimization problem as minimizing cost under a latency constraint, such as generating a required number of tokens within two seconds. Higher throughput can reduce the number of GPUs and the resulting cost.

### Fine-tuned adapters can support many model variants on shared GPUs
[09:52](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=592s)
Dzhulgakov says teams often want to fine-tune and deploy the resulting model efficiently. For LoRA tuning, he describes a serving approach that can place many model variants on the same GPU, potentially thousands of them. That allows serverless inference with per-token pricing even when many customized versions are deployed. The customer does not need to pay a fixed amount for separate infrastructure for every variant. Fireworks also offers fine-tuning with datasets brought from elsewhere or collected through feedback while a model is served.

### A production application combines a model with retrieval and external actions
[10:32](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=632s)
Dzhulgakov says the model alone is increasingly not the whole product because models can hallucinate and because applications often need several modalities. Retrieval can ground responses in external knowledge, while function calls connect the model to tools and actions. He uses the term compound AI system for this combination. In his description, a central function-calling model can coordinate domain-specific models and external tools such as a code interpreter or APIs. The application is built from these connected parts rather than from an isolated language model.

### Function calling requires planning, context management, and structured output
[12:32](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=752s)
The Fireworks demo combines general chat with external tools. The model can query stock quotes, plot charts, and generate images. To create a chart for the top cloud providers, it has to interpret the request, identify the providers, call functions for stock prices, and pass the gathered information to a plotting API. It also needs to retain context when the user asks to add Oracle to the existing graph. When the user changes subjects, it should discard less relevant history. Dzhulgakov says the open-source demo uses Fireworks function-calling models and structured generation with JSON model grammar.

### Fireworks offers a path from token-based experiments to dedicated enterprise deployments
[15:33](https://www.youtube.com/watch?v=ePMvfa8vgL8&t=933s)
Dzhulgakov describes serverless inference as the starting point, with per-token pricing and no need to manage hardware settings. Teams can fine-tune models on the platform or bring their own LoRA adapter. At higher production volume, they can use on-demand dedicated hardware with more configuration, then move to enterprise arrangements with customized hardware and performance tuning. He names Quora's Poe, Sourcegraph, Cursor, and Upstage as production users, and says Fireworks was serving more than 150 billion tokens per day when he last checked.

## Notable quotes
- "Open source models are the future also for Gen application." (01:21)
- "The model is not the product right? By itself you need a kind of bigger system in order to solve the target application." (10:32)
- "You need to have contextual awareness." (14:20)
- "You can start with serverless inference, which is not different from getting to OpenAI playground or something where you pay per token." (15:33)

## Tools & references mentioned
- Fireworks.ai
- PyTorch
- Meta
- Google AI
- GPT-4
- Claude
- Llama
- Gemma
- Artificial Analysis
- SDXL
- Stable Diffusion 3
- Hugging Face
- Fireworks Function V
- LoRA
- RAG
- Berkeley function-calling benchmark
- Quora
- Poe
- Sourcegraph
- Cursor
- Upstage
- LangChain
- LlamaIndex

## Who should watch
- You are deciding whether a smaller or fine-tuned open model can meet a latency or cost target that a large proprietary model does not fit.
- Your team can run models in a prototype but needs serving optimization, GPU sharing, observability, and a path to dedicated infrastructure.
- You are building an agent that needs retrieval, function calls, external APIs, and conversation context rather than plain text generation.

## Editor's note

From the pack [Inference: serving models efficiently](https://aietalks.com/packs/inference):

These talks make inference tuning a comparison between configurations, not a hunt for one universal benchmark. Kitaru can replay the same recorded model workload against two runtime setups so a team can compare outputs alongside time to first token, completion latency and token use. The replay holds the application inputs steady; it does not reproduce GPU scheduling, queue pressure or cache placement from the original deployment, so those still need measurements from the serving system itself.

Written by the AIE Talks editors (the Kitaru team), not by the speaker.

## Related talks

- [Making Open Models 10x Faster and Better for Modern Application Innovation](https://aietalks.com/talks/making-open-models-10x-faster-and-better-for-modern-application-innovation) (Dmytro (Dima) Dzhulgakov, Fireworks AI, 18:55)
- [The Rise of Open Models in the Enterprise](https://aietalks.com/talks/the-rise-of-open-models-in-the-enterprise) (Amir Haghighat, Baseten, 16:50)
- [AI Engineering 201: Inference](https://aietalks.com/talks/ai-engineering-201-inference) (Charles Frye, Full Stack Deep Learning, 1:43:16)
- [Open Models track](https://aietalks.com/talks/open-models-track-r0x7mpag) (Remy, txt & Sandra, Cohere & Leo, Gradient & Daniel, Unsloth & Maxim Labonne, Liquid AI, 3:34:55)
- [How fast are LLM inference engines anyway?](https://aietalks.com/talks/how-fast-are-llm-inference-engines-anyway) (Charles Frye, Modal, 16:07)