WEKA's open-source context platform engineering toolkit lets teams load-test agent swarms with configurable SLOs, prompt cycles, model parallelism, prefill and decode modes, and memory tiers.
2
KV-cache hit rate affects both the cost of API usage and the rate limits experienced by subscription users, so context systems need enough capacity and speed to keep useful tokens available.
3
Longer cache time-to-live and denser memory tiers can preserve more agent context as concurrency grows, with the toolkit comparing these configurations against latency and output-token targets.
Summary
Val Bercovici and Callan Fox describe context platform engineering as the work of sizing and configuring inference systems for agent swarms. Their open-source toolkit generates deterministic or random prompt workloads, models coding-agent users and subtasks, and tests time-to-first-token and output-token targets across memory configurations. Fox shows why agent workloads create a storage problem. Tool calls and responses make up much of the context, while human pauses can last minutes or hours. A short cache lifetime therefore causes repeated prefill work and lower hit rates. Longer lifetimes improve reuse, but require memory with enough capacity and fast writes and reads. The talk compares HBM, DRAM, and WEKA-backed storage as concurrency rises. The central operational idea is to turn agent service requirements into inference-platform objectives, then test how cache working sets, latency, concurrency, and throughput behave before deployment.
The open-source toolkit turns agent requirements into testable platform settings
WEKA announces an open-source context platform engineering toolkit for configuring agent swarms and subtasks with specific service-level objectives. Its load generator supports deterministic and random prompt cycles, model parallelism, aggregated or disaggregated prefill and decode, and memory-tier choices. Teams can download it from GitHub, run their own tests, fork the project, and provide changes. The toolkit is intended to connect the behavior an agent workload requires with the inference platform that runs it, rather than treating token usage as an abstract limit.
KV-cache hit rate determines how well production agents use their context
Bercovici cites Manus AI's statement that KV-cache hit rate is the single most important metric for production-grade AI agents. He presents context platform engineering as a way to increase cache reuse and reduce token anxiety. Without that work, teams may resort to what he calls context financial engineering: balancing cache writes, cache reads, input-token prices, output-token prices, and cache lifetimes. He argues that this requires predicting future access patterns, while a platform designed around the workload can preserve reusable context more directly.
Agent workloads contain much more reusable tool context than user text
Fox separates an agent loop into new tokens, reusable tokens, output tokens, and points where the user responds. Agents often consume context until they reach a model or provider limit, summarize it, and begin again. Summarization can reduce fidelity. In coding workloads, the user's direct input is a small part of the prompt. System instructions, tool calls, and tool responses occupy much of the rest. A coding agent may run a shell command, inspect its result, and continue, creating repeated context that could be cached.
Human pauses and agent activity create a cache timing mismatch
The median time between requests in the data Fox discusses can be around 10 or 15 seconds, depending on whether a human checks each tool use. The mean time can extend to minutes or hours because human response time is much slower. Meanwhile, agents and sub-agents continue work at a higher cadence, often in parallel. Multi-agent systems can use short-lived sub-agents whose context disappears between activations, or persistent ones whose context survives. This structure increases the amount of context and makes cache lifetime important.
Cache misses hurt both direct cost and subscription capacity
When a reusable token sequence misses the cache, an API customer pays the input-token price again. Fox says that can cost potentially 10 times more than a cached request in his example. Subscription users avoid that direct charge, but their plans are still rate-limited by cache usage and may reach those limits sooner. Providers therefore have an incentive to reduce misses, since repeated prefill work affects user experience and consumes provider capacity.
Longer cache lifetimes improve reuse when the working set fits
Fox compares one-minute, five-minute, and one-hour cache lifetimes against request timing. With a one-minute lifetime, gaps longer than a minute cause the system to drop and refresh context repeatedly. A five-minute lifetime preserves more hits, while an hour preserves still more when requests are separated by longer pauses. In the example, a one-minute lifetime can prefill the same tokens 15 or 16 times. Increasing the lifetime brings that number toward one, provided the memory system can hold the larger working set.
Token storage needs capacity, fast writes, and fast reads
Fox reduces the memory problem to three system needs. The memory tiers must have enough capacity to hold the cache up to the point where additional capacity gives diminishing returns. They must accept KV data quickly, or tokens may be dropped before storage or GPUs may wait. They must also fetch data quickly, so GPU work is not blocked. HBM is the preferred location but cannot hold every session in practice. DRAM is common and performs reasonably, though its capacity is tied closely to compute. WEKA's Augmented Memory Grid connects inference systems to denser NVMe-backed storage.
Benchmarking must expose both deterministic overflow and blended workloads
The toolkit can hold a fixed number of coding-agent users or increase the user pool over time. A sequential test starts with an initial working set and runs prompts deterministically. Slightly exceeding a memory tier can then produce a large performance drop. A second method grows concurrent users and samples prompts randomly from a pool, producing a blended result across memory tiers such as HBM and DRAM. Fox presents this as a more realistic way to compare configurations under changing working sets.
Dense storage preserves throughput as concurrency grows
Fox compares HBM with WEKA storage, HBM with DRAM, and a configuration that adds another storage system. All three benefit from HBM at first. As concurrent users increase, the DRAM-based configurations overflow and fall sharply. The WEKA-backed system also loses some HBM advantage, so the benchmark reduces concurrency, but it reaches a higher steady-state user count and output-token rate. In a prefill-focused disaggregated setup, the result is better because GPUs handle large prefill batches efficiently. Capacity alone is insufficient when storage cannot deliver cached data to the GPU fast enough.
