Small open-source models can match or exceed frontier performance on specific tasks while running on one affordable, older-generation GPU.
2
A fleet of small models needs workers to pull from shared queues and form their own batches, because top-down routers leave GPUs underused.
3
An open-source serving stack can combine several runtimes, pre-tuned configurations, model packing, and automated research loops to reduce infrastructure work.
Summary
Daniel Svonava argues that small open-source models are now practical for production when each model is assigned a specific task. A contract agent may use nine models, with separate models for OCR, question answering, SQL generation, code review, and other jobs. The models are available, but serving a wide fleet creates new infrastructure problems. Existing serving tools need substantial tuning, top-down routers struggle with many small requests, and LoRAs create repeated coordination between AI and infrastructure engineers. Svonava describes Superlinked's Apache 2.0 architecture, where a lightweight gateway annotates requests and places them in shared queues. Workers pull requests and build their own batches, which he says doubled cluster throughput. A Rust sidecar supports multiple runtimes and model adapters. He also describes packing small models onto GPUs, embedding throughput on a single GPU, and automated tuning loops that ship configurations with model support. One LoRA cost 80 cents to train and improved German legal retrieval by 18 percent.
Small models fit on older GPUs and can reach frontier performance for focused tasks
Svonava defines a small model as one that fits entirely on a GPU that is two or three generations old. That makes the hardware easier to find and cheaper to run. He argues that quality does not have to fall by much for a focused task. Frontier models are showing diminishing returns, while smaller open models are catching up. He gives Qwen 3 6 27B as an example that is around the performance of GPT-5.1. The practical benefit is lower cost, with potentially better latency and throughput.
A production agent should divide its workload among task-specific models
A 27-billion-parameter model should not be treated as a general system that can handle every request equally well. Svonava recommends splitting the workload into tasks, evaluating models for each task, and adapting them when needed. His contract review example uses nine different models. The same pattern can cover OCR, document question answering, image labeling, SQL generation, and code review. For a narrow task, a model trained on the relevant data can beat a much larger general model. He gives Vietnamese receipt OCR as an example, where the model that has seen the most Vietnamese receipts may perform best.
Open-source inference tools require model and hardware tuning before they are fast
Svonava says tools such as vLLM and SGLang are not tuned for every model, GPU, or traffic pattern. Teams must sweep parameters and tailor the system to their workload. Adopting one therefore becomes an open-ended research project rather than a short engineering task. He contrasts this with the expectation that a team can install a tool and have high-performance serving within a week. The problem becomes larger when a company runs many different small models, because each model and hardware combination may need its own configuration.
Top-down routing wastes capacity when requests are small and frequent
Large-model clusters often use a router that tracks worker state, including KV cache state, and decides where each request should go. Svonava says this approach struggles with many small, fast requests because the router's view of worker state is already out of date. The router must balance local queues correctly before the request reaches a worker. In experiments with vLLM and SGLang routers, GPU utilization was difficult to push beyond 20 to 30 percent under constant load. Batches ended up incorrectly sized because routing became the bottleneck.
LoRAs turn model serving into repeated coordination between teams
Small models benefit from LoRAs and other forms of adaptation, but those adaptations create deployment work. An AI engineer may arrive with ten LoRAs or a custom fine-tune made the previous night and ask the infrastructure team to serve it. Svonava describes the resulting back-and-forth as a major source of delay. His preferred setup lets infrastructure engineers operate the platform while AI engineers work on models without blocking each other. Supporting model adaptation without this conversation is therefore part of the serving problem.
Workers pulling from a shared queue avoids stale routing decisions
Superlinked's architecture puts a lightweight gateway in front of shared queues. The gateway inspects enough of a request to attach metadata, then places it in a central queue and side channels. Workers pull requests instead of having a router push them to a selected worker. Each worker can use its current state to form a batch and estimate the batch cost. Svonava also describes a local queue for machines with multiple GPUs, where processes can negotiate without adding network round trips. The stack uses MessagePack for binary data and NATS JetStream for queueing.
A Rust sidecar keeps several runtimes behind one serving abstraction
The serving system supports PyTorch, CUDA, and SGLang through a local socket and a Rust sidecar. Svonava says the sidecar now handles about 50 model adapters. PyTorch is the strongest runtime in their current work, while CUDA reduces worker image size but has not yet reached PyTorch's performance. SGLang provides a baseline, and the sidecar's batching logic can improve on its bare performance. The point is to avoid building many custom plugins for one runtime when a fleet of small models has diverse architectures and dependencies.
Model packing and automated tuning make small-model fleets easier to operate
Small models change the usual worker-pool model because loading one model does not need to occupy an entire GPU permanently. Svonava recommends combining pinned models with lazy loading and eviction based on memory pressure. He also describes automated research loops that add model support and test performance. When a model ships, its end-to-end cluster configuration is already tuned rather than left for each user to sweep. One resulting LoRA cost 80 cents to train and improved retrieval quality on German legal text by 18 percent.
