Small models can preprocess data and handle tool calls so agents work with less context and avoid context rot.
2
Putting one small model on each GPU wastes memory, so the inference engine hot-swaps models and evicts the least recently used model.
3
Supporting many model architectures requires changes to the forward pass, attention, padding, normalization, and positional embeddings, alongside routing and autoscaling infrastructure.
Summary
Filip Makraduli explains why small-model inference needs its own infrastructure. His starting point was a research mistake: understanding model internals and GPU limits did not tell him enough about scheduling, routing, and production operation. At Superlinked, he helped build an open-source inference engine for AI search and document processing. Small models can preprocess data, classify taxonomies, call tools, and reduce the context passed to agents. Since these models occupy only a few gigabytes, assigning one GPU to each model leaves memory idle. The engine instead hot-swaps models and uses least recently used eviction. Makraduli also describes the engineering needed to support different architectures, including BERT, Qwen, ColBERT, cross-encoders, and rerankers. The infrastructure layer adds routing, queues, GPU pools, spot instances, Prometheus metrics, KEDA autoscaling, Helm charts, Docker images, and Terraform configuration. The result is SIE, Superlinked's open-source attempt to combine model support with production cluster management.
Production inference was a blind spot even for an experienced ML engineer
Makraduli begins with an article about FlashAttention, model behavior, and whether workloads are memory-bound or compute-bound. Readers pointed out that he had missed a major part of real-world speed: inference. His previous work covered training, fine-tuning, applied ML, and academic research, but he had spent less time on how models run in production. Scheduling GPUs, routing requests, and automating deployments were unfamiliar areas. He joined Superlinked's infrastructure team to learn by building an inference system rather than studying the problem only in theory.
Small models can reduce the context that agents need to process
Makraduli connects small-model inference to context rot, where quality degrades as context increases. Small models can preprocess documents before an agent sees them, and they can handle tool calls that classify or retrieve information. He says preprocessing can improve code grepping and file-system workflows rather than replacing them. Examples include named-entity recognition for building ontologies and knowledge graphs, filtering input, and reducing token counts. Superlinked used its inference engine for taxonomy classification in an e-commerce tool-calling workflow, where small models examined the store's data.
One GPU can host several small models if the system can swap them quickly
Adding more GPUs does not solve the main problem when each small model uses only a few gigabytes of memory. Assigning a separate GPU to an embedding model, reranker, or named-entity recognition model leaves much of each GPU idle. Superlinked's engine loads several models onto one GPU over time and hot-swaps them as requests change. It also uses a least recently used eviction policy. This raises utilization and reduces cost while allowing different tools to select different models without deploying one container for every model.
Inference needs model support and infrastructure together
Makraduli describes inference as a combination of model support and infrastructure. Supporting the right models matters because open-source models are numerous and improving, while narrow-task models can beat managed services on benchmarks. Infrastructure matters because a model implementation alone does not provide routing, autoscaling, monitoring, queues, or GPU provisioning. The Superlinked system includes these production pieces instead of requiring users to connect them by hand. Its model configuration can be switched and applied with Terraform, with Helm charts and Docker images also available.
Different model architectures require different forward-pass implementations
A universal engine cannot treat BERT, Qwen, ModernBERT, ColBERT, cross-encoders, and rerankers as identical. Their attention implementations, positional embeddings, normalization, and output formats differ. ColBERT produces multiple vectors, while a reranker may produce scores rather than an embedding vector. Superlinked reimplements parts of the forward pass so it can adapt attention, padding, query-key-value fusion, and other model-specific behavior. Makraduli gives positional embeddings as an example: BERT can use absolute lookup, while Qwen uses rotary positional embeddings.
Variable-length batching avoids compute spent on empty tokens
Token-based batching can combine a short request with a much longer one. If both are padded to the longer length, the system performs work on empty tokens. Superlinked uses variable-length FlashAttention and adapts padding so batches do not waste that computation. Makraduli presents this as an important part of making many supported models run efficiently. The implementation has to account for the different ways architectures handle attention and their outputs, rather than applying one fixed optimization to every model.
The infrastructure layer routes requests across model and GPU pools
The system exposes three API primitives: encode, score, and extract. Behind them are routing and queuing mechanisms that respond to load, along with pools containing different GPUs and resources. The deployment can use spot instances and larger GPUs. Prometheus metrics feed KEDA autoscaling, which helps adjust capacity and move models without leaving hardware unused. Makraduli's goal is to provide the cluster and the model support together, so users can change a model configuration and run Terraform rather than writing the scheduling and provisioning layer themselves.
SIE packages model inference and cluster management as an open-source project
Superlinked open-sourced both parts of the system Makraduli discusses. The first is model inference, including forward-pass changes and attention handling for different architectures. The second is the cluster layer, which manages hardware, GPU provisioning, routing, queues, and autoscaling. He names the project SIE and points the audience to its repository. He closes by explaining the background image: a visualization of sinusoidal positional encodings used in transformers, with the visible pattern coming from the way positions are represented in embedding space.
