Chatbots run on human time. A person reads an answer, thinks about it, and types a follow-up, which takes anywhere from a few seconds to a few minutes per turn. Agents run on computer time. An agent reads a stack of documents, calls a tool, reads the result, and sends its next turn within seconds. It repeats that cycle dozens of times per task, at what looks, from the outside, like superhuman speed.

Human time is forgiving to the inference server; computer time is not. When a person takes two minutes to reply, a two-second rebuild of their conversation’s cache feels insignificant. For an agent running on the same server, each rebuild of the conversation can add up to more than two minutes of waiting over a 60-turn task. Chatbots could get away without a robust caching strategy because human pauses hid the cost of rebuilding. Agents, on the other hand, remove those pauses. Whether your agents stay fast depends on whether that work is still reusable when the agent comes back.

An enterprise inference server has to serve dozens or hundreds of these conversations at once on a single node. GPU memory cannot hold the KV cache for all of them. When memory fills, the engine evicts caches for conversations that are not actively in use, starting with the least recently used (LRU). When an evicted agent returns, the server has to rebuild its cache from scratch. With a handful of agents, rebuilds are rare, but with a few dozen, they happen on almost every turn. At that point, the delay stops being a cost to one request and spreads to every agent on the server.

We measured this on a single Gemma-4 26B endpoint serving 40 agents. We tracked time to first token (how long the model takes to start answering) on every returning turn:

Without a cache tier below GPU memory, returning turns started answering in about 0.1 seconds until roughly 22 agents were live.
Within the next 30 seconds, that median jumped to 17 seconds, and it rose to about 45 seconds by the end of the run.
With Pure Key-Value Accelerator (KVA) keeping evicted caches in host memory and on NFS storage, the median stayed between 0.11 and 0.30 seconds through all 40 agents, on the same hardware.
Matching Pure KVA’s latency without a second tier would take roughly 1.8 times the GPU memory, which in practice means close to twice the GPUs.

What Pure KVA does with an evicted cache

Pure KVA keeps evicted caches instead of discarding them. It’s a KV cache backend for inference engines, made up of a plugin in the engine, a service that runs beside it, and enterprise storage underneath. Pure KVA is available now, and it manages each conversation’s cache across three tiers:

GPU HBM (the high-bandwidth memory on the GPU package) holds the conversations the engine is actively serving. A turn whose cache is still here needs no rebuild at all. HBM is also the smallest and most contended tier.
Host memory is DRAM on the same server. When the engine evicts a conversation from HBM, Pure KVA keeps its cache here, one hop from the GPU, for the fastest restores.
NFS storage holds far more than any single server’s memory. It’s shared across the fleet, so a cache written on one node can be restored on another. Pure KVA also supports S3 object storage.

Eviction still happens with Pure KVA. HBM is the same size, and the engine still makes room by evicting the least recently used conversations. What changes is the cost of coming back. A returning agent’s cache is read back from host memory or storage instead of rebuilt on the GPU. In our tests, a restore from NFS was up to 32 times faster than a rebuild, and it added no rebuild work to the GPU’s queue. A restore still needs room in HBM, so it evicts another conversation, but that conversation can also come back with a read.

A faster restore on an idle server is easy to show and easy to overclaim. The case that matters for agents is dozens of long conversations competing for one pool of HBM. There, the important number is what one request’s delay does to everyone else’s, and our load test measures exactly that case.

We wanted to see what happens to a single inference server as agents pile onto it, so we built a load test that adds them one at a time. We served Gemma-4 26B from one endpoint and started a new agent every 15 seconds. No agent dropped out once it joined, so by the 10-minute mark all 40 agents were running at the same time.

Each agent behaves the way a real agent would. On its first turn, it loads 10 documents, which gives it a starting prompt of 22K to 33K tokens. The model answers, and the agent waits about 15 seconds before sending its next turn. That pause stands in for the tool calls and other work an agent does between turns. The agent never sends a new turn until the previous answer has arrived, and every new turn resends the whole conversation so far.

We ran this exact plan twice. The first run used vLLM with Pure KVA, and the second used the same engine without it, which we call “No KVA” in the graphs below. Both runs used the same random seed, so the agents arrived at the same moments, loaded the same documents, and paused for the same lengths of time. Each run lasted 750 seconds (12.5 minutes), and Appendix A below has the full rules if you want to reproduce the test.

The number we tracked is time to first token on returning turns, i.e., how long an agent waits for the model to start answering when it comes back. First turns don’t tell you much here. Neither endpoint has a cache to reuse on an agent’s first turn, so both compute it from scratch, at a median of 2.1 seconds with Pure KVA and 2.3 seconds without. The two endpoints only start to differ when agents come back. Figure 1 below shows the result as a rolling median, i.e., the median wait across a sliding window of recent returning turns.

Figure 1: Rolling median time to first token for returning turns, Pure KVA versus No KVA, across the 750-second run.

For the first five and a half minutes, you can’t tell the two versions apart. Both answer returning agents in about 0.1 seconds. Every live conversation still fits in HBM, so nothing has to be rebuilt, and Pure KVA adds nothing to the path because its first tier is that same HBM.

Then, in dramatic fashion, the latency for the agents not using Pure KVA jumps by two orders of magnitude. At 5.6 minutes, with about 22 agents running, No KVA answered its last returning turn in under half a second. Thirty seconds later, the median wait was 17 seconds, longer than the 15-second pause the agents take between turns. In other words, the agents were now spending more time waiting on the server than doing their own work. The wait kept growing, to 31 seconds at eight minutes and about 45 seconds by the end, and it never came back down.

Pure KVA stayed flat the whole way. Once the first few agents were past their opening turns, its median wait stayed between 0.11 and 0.30 seconds for the rest of the run. The one rise, to 0.30 seconds, came at the 10-minute mark, when the last agents arrived and computed their first turns. The median was back to 0.12 seconds within a minute. The slowest turn in the entire Pure KVA run took 4.6 seconds, and it was one of those first turns.

A fair question is whether agents without KVA slowed down because their prompts grew over the run. They barely did. All the inputs stayed between 21.8K and 34.1K tokens from start to finish, so no single request got harder to answer. The jump came entirely from HBM running out of room. Once HBM was full, every returning agent needed a rebuild, and the rebuilds piled up faster than the GPU could work through them.

What eviction costs agents, GPUs, and the teams that depend on them

The jump in Figure 1 shows up in three places: how long each agent waits, how much useful work the GPU gets done, and how long the work built on top of the agents takes to finish.

Let’s start with the agents themselves. The hardest part of the jump is how suddenly it arrives. No KVA ran at about 0.1 seconds for five and a half minutes with no sign of trouble, then passed 17 seconds within half a minute. A gradual slowdown gives you time to notice and add capacity; a sudden one doesn’t. It also means a pilot can mislead you. A deployment that looks perfect with 15 agents can run more than 100 times slower at 40.

Next, the GPU. Every rebuild is the GPU recomputing a cache it already had, and every second spent on a rebuild is a second not spent generating new answers. The agents make this worse because each one waits for its answer before sending the next turn, which is how real agents behave. Slower answers mean fewer turns, and fewer turns mean fewer useful tokens. High GPU utilization is not the same as good GPU utilization.

You can see the size of the effect with some quick arithmetic on our measured medians. An agent that pauses 15 seconds and waits 0.1 seconds for its answer completes about four turns a minute. The same agent waiting 45 seconds completes about one. Same agent, same GPU, but only a quarter of the output.

Finally, there’s the work that depends on the agents. Agents are rarely the end product because they’re steps inside engineering and business workflows. A coding agent working through a failing test suite holds up the developer waiting to merge. A support agent resolving a ticket holds up the customer who opened it.

Each of our test agents was planned for 60 turns. At four turns a minute, that task takes about 15 minutes, and at one turn a minute, it takes an hour. When one agent’s output feeds another agent, those delays add up from one end of the workflow to the other.

Why HBM runs out and rebuilds pile up

No KVA’s latency jumped for two reasons. Agent conversations are large enough that a few dozen of them fill GPU memory, and rebuilding an evicted conversation costs far more than reading it back. Put those two facts together on a busy server, and each slowdown snowballs the next one.

Start with size. When the model reads a prompt, a phase called prefill, it computes a key and a value for every token at every layer, and those keys and values make up the KV cache. The cache’s size depends on the model’s architecture and the conversation’s length:

KV bytes = 2 × layers × kv_heads × head_dim × tokens × dtype_bytes

To give a sense of scale, a 32B parameter model with 64 layers, 8 KV heads, and a head dimension of 128, stored at fp8, needs about 131KB per token. One 28K-token conversation, about the size of our agents’ prompts, comes to roughly 3.7GB, and 40 of them need about 150GB. The model weights take up a fixed amount of HBM no matter how many conversations are running, and every conversation’s cache has to fit in what remains. A GPU has enough compute to serve far more agents than its memory can hold at once, so the engine evicts.

Figure 2: How eviction compounds, simplified to an HBM with room for two conversations. Left: Without a tier below HBM, a returning agent’s cache is rebuilt with a full prefill, and making room for it evicts another agent. Right: With Pure KVA, the evicted cache moves to host memory or NFS, and the returning agent restores it with a read instead of a rebuild.

Next, cost. Rebuilding an evicted conversation means running prefill again over the whole conversation. Prefill time is dominated by attention, which grows as O(n²) with context length n, so doubling a conversation’s length roughly quadruples the attention work. Reading a saved cache back grows as O(n), in line with the number of bytes. The gap between the two has to widen as conversations get longer.

We measured how wide it gets. On an otherwise idle server, we took 40 prompts ranging from 5.5K to 118K tokens and ran each one twice: once from scratch, and once after discarding its cache and letting Pure KVA restore it from NFS. At the context lengths our agents use, the restore was about 10 times faster than the rebuild. At 118K tokens, it was 32 times faster.

Figure 3: Time to first token for the same prompt on Gemma-4 26B, rebuilt from scratch versus restored from NFS, across 40 prompts.

Size and cost explain why a single rebuild hurts, but they don’t explain the jump in Figure 1, where waits went from 0.1 seconds to 17 seconds in half a minute. The jump comes from what a rebuild does to every other agent on the server.

Once HBM is full, a rebuilt cache needs room, so the engine evicts another conversation to make it. That agent now needs a rebuild when it returns, which evicts a third. Meanwhile, every rebuild occupies the GPU, so the agents behind it wait in line. Once rebuilds arrive faster than the GPU can finish them, the line grows with every turn instead of draining, which is why the No KVA text never recovered. With Pure KVA, the evicted agent comes back with a read instead of a rebuild, so the chain stops after one step and the queue never forms.

Plan for eviction before your agents outgrow HBM

The usual case for KV caching is to increase speed. The tests we ran show something more useful for anyone running agents on-prem. With a caching tier below HBM, the cost of a returning conversation no longer depends on how full the GPU is. Without one, an agent’s next turn is fast only if HBM happened to have room for its conversation while the agent was away. Once HBM runs out of room, the agent has to wait for the prefill to finish before doing any useful work. With Pure KVA, your agents can focus on doing work instead of waiting in line.

That predictability is what makes an on-prem deployment easier to plan. Without a caching tier below HBM, you have two options. You can size the deployment so your agent population never crosses the point where HBM overflows, which was about 22 agents in our test. Or you can accept waits more than 100 times longer once it does.

Serving our 40 agents at the latency achieved with Pure KVA, without an external caching tier, would take enough HBM to hold every conversation at once. That’s roughly 1.8 times the HBM in our test server, which in practice means close to twice the GPUs. Pure KVA pushed the jump past all 40 agents on the same hardware. With GPUs supply-constrained and priced accordingly, that headroom is the practical benefit. Pure KVA allows you to do more with the same infrastructure.

Pure KVA enabled 2X concurrency, saving the extra provisioning.

The time to plan for eviction is when you size the deployment, not after your agents queue up and the workloads grind to a halt. Pure KVA is available now. Appendix A below lays out both tests in full.

Appendix A: Test methodology

Both tests ran on Gemma-4 26B with Pure KVA 1.2 over NFSv4.1. The agentic test plays the identical plan against both endpoints, one run each, with the same seed, arrival schedule, documents, and think-time draws.

A.1 Classic performance sweep

A deterministic ladder of 40 prompts from 5.5K to 118K input tokens is built by appending documents from a fixed corpus. Each prompt runs once, on an otherwise idle server, over NFSv4.1 with nconnect. For each prompt:

Cold prefill: Send the prompt with nothing cached and record time to first token.
Discard: Evict the prompt’s KV cache from HBM.
Restore: Send the same prompt again while Pure KVA restores its KV cache from NFS.

A.2 Agentic load harness

The harness does not replay a trace or fire requests at a fixed rate. It simulates agents, i.e., long-lived sessions that each hold a conversation, send a turn, wait for the answer, think, and send the next turn. Two things define the load: what goes into each prompt, and when each agent sends it.

Figure 4: Left: The prompt an agent sends at each turn. Right: Agents live over time. Turn 1 loads the agent’s 10 documents and has to be prefilled from scratch. Every later turn resends the whole conversation plus the previous 64-token answer and a new question—37 to 94 extra tokens, median ≈ 55. Agents join one every 15.4 seconds until all 40 are live at 10:00 minutes; the harness stops scheduling at 12:30 minutes.

Model: Gemma-4 26B
Arrivals: 40 agents, one every 15 seconds (40 waves over 600 seconds)
Documents: 10 distinct documents per agent, loaded at turn 1, drawn as 400 documents from a 6,119-document corpus
Turn 1 prompt: 21.8K to 32.8K tokens, median 27.6K
Later turns: Full history, previous answer, and a new question, adding 37 to 94 tokens per turn
Output: 64 tokens per turn
Think time: Drawn per turn from a normal distribution with mean 15 seconds and spread 10 seconds, floored at 0 (observed mean 14.7 seconds, median 14.3 seconds)
Loop: Closed, so the next turn is scheduled only after the previous answer arrives
Turns per agent: 60 planned, 2,400 in total
Client workers: 50, more than the number of agents, so no request ever waits in the harness
Stop rule: 750-second cap, in-flight turns drain, no respawn
Seed: 51,909, which fixes arrivals, document sets, and think-time draws
Endpoints: Pure KVA is vLLM with Pure KVA 1.2, a 32GiB host-memory tier, and an NFSv4.1 store. No KVA is vLLM with HBM only.
Runs: One run per endpoint, executed separately on the same plan
Recorded per turn: Send time, queue wait, time to first token (TTFT), total time, input and output tokens, requests in flight
Telemetry every 3 seconds: vLLM prefix-cache hit rate and KV usage, Pure KVA host-memory bytes and store entries