Our pace of innovation around enterprise KV cache continues to impact our customers’ inference stacks with the latest release of Pure Key-Value Accelerator (KVA) adding three-tiered architecture, S3 over RDMA, multi-tenancy, and support for TurboQuant. Today, let’s focus on the new features that impact time to first token (TTFT).
Increase GPU turns per time slice
The new three-tier architecture of Pure KVA (see Figure 1 below) improves KV cache load times and GPU utilization. In short, you get to run more turns in a given span of time on a fixed GPU footprint. This newly released feature enables you to leverage high-bandwidth memory (HBM), system memory, and storage to increase KV cache reuse with faster TTFT.
Figure 1: Three-tiered KV load path for Pure Key-Value Accelerator (KVA).
We tested Kimi-k3 on NVIDIA B300 GPUs, with “vLLM bench” queries of increasing context from 100k tokens to 200k tokens. Here’s an example of the command we used during testing:
vllm bench serve –model google/gemma-4-26B-A4B-it –num-prompts 50 –random-input-len 200000 –max-concurrency 1 –output-len 1
Comparing KV cache load TTFT shows Pure KVA is 5.4X to 11.5X faster than vLLM.
Figure 2: Time to first token running Kimi-k3 on B300 – vLLM vs. Pure KVA (NFS).
Comparing KV cache load TTFT shows Pure KVA is 1.7X to 2X times faster than LMCache. Both Pure KVA and LMCache are defined with a DRAM tier in addition to the storage tier (NFS or FS native).
Figure 3: Time to first token running Kimi-k3 on B300 – LMCache (fs native) vs. Pure KVA (NFS).
Let’s look at a theoretical example with a larger user base: We have eight B300 GPUs to serve 100 users, each with a 256K context, initiating a turn every 60s for 10 turns with 512 input tokens for 1,024 output tokens. Using Pure KVA, we can run more turns per hour (724) than without (313). Our workload finishes 2.3X faster—saving 108 minutes.
Figure 4: Time to complete all sessions—100 sessions each with 10 turns.
The reason is simple: recomputing. Large contexts take longer than reloading from a saved copy. In optimal conditions, a 256k context would take 7.25s to recompute on a B300 GPU. Note that these theoretical numbers, for this example use case, are 2X faster than our real-world results. The three-tiered approach of Pure KVA provides simple KV cache management from HBM to DRAM to storage, so the inference engine can start decoding new tokens instead of recomputing known “old” tokens.
Figure 5: Recompute vs. reload—optimal conditions.*
*Numbers are based on optimal maximum throughputs—example used bandwidth throughput of 55.6 GB/s for DRAM and 10 GB/s.
With our introduction of S3 over RDMA, using object storage on FlashBlade® no longer pays the penalty of the S3 protocol overhead. Our vLLM bench test showed Pure KVA S3 performance was 3.9X to 8.2X times faster than vLLM. This gives you the flexibility to choose the protocol that best suits your inference stack requirements.
Figure 6: Time to first token running Kimi-k3 on B300 – vLLM vs. Pure KVA (S3).
Conclusion
On-premises enterprise inference requires maximizing the GPU footprint you have. The best lever is to reuse KV cache to allow for more user turns per GPU hour. User sessions will churn from the GPU HBM, and recomputing a prior session is a waste of valuable GPU time. Reloading KV cache to eliminate the impact of session churn allows you to squeeze as much inference as possible out of your existing GPU footprint.
Contact your account team today to get faster inference and start saving GPU cycles.












