KV Cache
A buffer that stores previously computed key and value tensors during autoregressive generation, so each new token only requires computing one step of attention instead of replaying the whole sequence.
A KV cache stores the key and value tensors used by an autoregressive transformer's attention layers. During generation, the new token attends to earlier positions. Caching their keys and values avoids recomputing those tensors at each generation step.
For full attention, each new query still attends over the growing history. KV caching removes repeated computation of the prior keys and values; it does not make the cost of attending to a longer context disappear. Sliding-window and other attention architectures use different retention patterns.
For full attention, cache storage grows with retained sequence length and the number of active sequences. Its size also depends on the number of layers, KV heads, head dimension, and storage precision. These factors make KV-cache capacity an important part of sizing a deployment for its target concurrency and context length.
Cache-management techniques include paged allocation to reduce memory waste, prefix reuse across matching requests, and quantization to reduce storage per value. Ion's published Grace Hopper design also describes background mirroring of completed KV blocks to CPU memory. Placement across GPU and CPU memory adds transfer and access considerations to the capacity plan.
Related Terms
vLLM
An open-source LLM serving engine that introduced PagedAttention. The de facto baseline for high-throughput open-weight model serving.
SGLang
An open-source LLM serving engine that uses RadixAttention to share KV cache across requests with overlapping prefixes — especially strong for agentic and tool-use workloads.
Prompt Cache
A response cache keyed on the request — exact-match for identical requests, prefix for shared system prompts, and optionally semantic for paraphrased questions. Cuts input tokens dramatically on real traffic.
Ion
Cumulus' proprietary inference engine, combining model-specific execution and custom GPU kernels in dedicated deployments tuned around the model, hardware, and workload.
Inference Latency
The end-to-end time from request arrival to response delivery. For LLMs, decomposed into time-to-first-token (TTFT) and inter-token latency (ITL).