Memory & Efficiency
KV Cache
FundamentalThe KV cache is what makes token generation fast - and what makes inference memory-bound. This page builds from the core idea to how production systems tier it across HBM, DRAM, NVMe, and pooled storage.
Prerequisite: this page assumes you know what keys and values are. If not, read Attention Mechanisms (and Tokenization) first.
Why a cache at all?
Generating each new token requires attending to every previous token. Without a cache, the model recomputes the keys and values for the entire sequence at every single step - an O(n²) explosion. The KV cache saves those keys and values once and reuses them, turning generation from quadratic to linear. The price is memory: that cache grows with every token, every layer, every concurrent user. Below you can watch the difference, then size it for real models.
Models have no memory
The reason a cache is even possible starts here: a transformer is stateless. It keeps nothing between decode steps - feed it the same tokens and it does the identical work. To write token N it must attend to all N−1 prior tokens, so without saving anything it would recompute every token's keys and values from scratch, every step. The KV cache is the workaround.
A language model has no memory
It is a pure function: feed it the same tokens and it does the exact same work every time. Nothing carries over between steps - so to write the next token it must look at all the tokens before it, again.
model
f(tokens)
Recompute every step
1 tokRe-derives keys & values for the whole prefix on every token - work grows with the sequence, total cost is O(n²).
Remember K/V (the cache)
1 tokComputes only the new token's K/V (green); reads cached K/V for the rest (dim). Work is constant per step, total cost is O(n).
Because the model is stateless, the keys and values for past tokens are identical at every step - so there is no reason to recompute them. The KV cache simply stores each token's K/V once and reuses it. That is the entire trick, and the rest of this page is about the price it charges: memory.
What exactly gets cached
Self-attention is why keys and values are the things worth keeping. Each new token forms a query and scores it against every prior token's key, then blends their values. Crucially, a past token's K and V never change - so they can be computed once and reused forever.
Self-attention, in one picture
The newest token forms a Query, scores it against every prior token's Key (Q·Kᵀ → softmax), then mixes their Values by those weights.
Yellow bars = softmax attention weights from the blue Query to each past Key. The weighted blend of their Values becomes this token's context.
Query (Q)
from the new token only
Keys (Kᵀ)
of every prior token
Values (V)
blended by the weights
attn(Q, K, V) = softmax(Q·Kᵀ / √d) · V
A past token's K and V depend only on that token and the weights - never on what comes after. They are frozen the moment the token is processed, so they can be computed once and cached forever. That is precisely what the KV cache stores.
Quadratic work, or linear?
Skipping the recompute turns generation from O(n²) into O(n). Step through token generation and watch the work tally for each mode - by a thousand tokens the no-cache path is doing hundreds of times the attention work.
The payoff - quadratic vs linear, step by step
Watch each new token generate. Without a cache the model re-derives K/V for the whole prefix every step; with a cache it only computes the newest token and reads the rest.
No cache
O(n²)Recomputes K/V for all 1 tokens this step.
With cache
O(n)Computes 1 new token (green); reads 0 cached (cyan).
Cumulative attention work after 1 tokens
Total work, no cache
1
units (O(n²))
Total work, cached
1
units (O(n))
Cache speedup
1.0×
less attention work
The gap widens with every token. At a 1,000-token context the no-cache path does about 501× the attention work of the cached path - which is why no production decoder runs without a KV cache. The cost moves from compute to memory.
The KV cache doesn't fit - so it tiers
HBM is fast but tiny. At scale the KV cache dwarfs it, so serving stacks spill cold KV down a hierarchy - DRAM, local NVMe, then pooled network storage. Each step down is ~10–100× more capacity but ~10× slower. The art is keeping hot KV in HBM and demoting the rest rather than throwing it away and recomputing.
KV cache memory hierarchy - watch blocks cascade
New KV blocks land in fast HBM. As it fills, the coldest blocks are demoted down the tiers - faster & smaller at the top, vast & persistent at the bottom.
PagedAttention · vLLM · SGLang · TRT-LLM · NIM · Dynamo
vLLM CPU swap · LMCache · Dynamo KVBM
LMCache · Mooncake · AIBrix
VAST · Tensormesh · Mooncake 3FS · NIXL · BlueField CMX
The lower a block sits, the cheaper to store but slower to read back. Demotion (instead of eviction) means a returning user's context can be re-promoted rather than recomputed from scratch - the whole reason tiered KV systems exist. Turn off local NVMe to send cold blocks straight to the pooled network tier.
Size it: the tiering & capacity planner
Pick a real 2026 model, precision, context, and concurrency, then a GPU fleet. See how dramatically attention architecture (dense GQA vs MLA vs sparse/MoE) changes the per-token bill, then watch the planner place weights in HBM and fill the KV cache top-down through the hierarchy - showing exactly how much must spill to network storage.
KV tiering & capacity planner
Pick a model, precision, context and concurrency, then a GPU fleet. The planner fills the memory hierarchy top-down: HBM → DRAM → local NVMe → network storage. Whatever doesn't fit upstream must land on pooled storage - which is exactly why networked KV tiers exist.
MLA latent (512 + 64 RoPE) per layer - tiny cache for a 671B model.
Hopper baseline - 80GB HBM3 per GPU, 900 GB/s NVLink. × 4 nodes = 2.50 TB fleet HBM.
Model weights
671.0 GB
FP8 · resident in HBM
Total KV at peak
5.62 TB
1,000 users × full context
Spills to network
0 MB
after HBM/DRAM/NVMe fill
Where the KV cache lands
Tiers fill in order. Try DeepSeek R1 (MLA) vs Mistral Large (dense GQA) at the same context and users - MLA's tiny per-token cache keeps everything in HBM where dense models overflow to storage. That overflow is the bandwidth-vs-capacity wall VAST's pooled KV is built to break.
Software that stretches the cache
Before adding hardware, modern serving engines wring far more out of the HBM you already have. These four techniques are the backbone of every production LLM stack.
PagedAttention
2–4× throughputvLLM
Stores KV in fixed-size pages (like OS virtual memory) instead of one contiguous block per request. Eliminates the 60–80% fragmentation waste of pre-allocating max-length buffers, so far more requests fit in HBM at once.
RadixAttention
5–6× on shared prefixesSGLang
Organises cached prefixes in a radix tree so requests that share a system prompt or few-shot examples reuse the same KV blocks. Huge for chat and agent workloads where every request starts with the same instructions.
Continuous batching
Higher GPU utilisationOrca / vLLM
Instead of waiting for a whole batch to finish, it swaps completed sequences out and new ones in at every decode step. Keeps the GPU saturated when requests have wildly different output lengths.
Prefix / KV reuse
TTFT 11s → 1.5sLMCache · Mooncake
“Prefill once, reuse everywhere.” Cache the KV of a long document or shared context and serve it to many requests from a fast tier instead of recomputing prefill each time. VAST + LMCache cut time-to-first-token ~7× on 128K contexts.
Throughput figures are vendor/paper-reported and workload-dependent. See Inference & Serving for prefill/decode and disaggregated serving.
Quantizing the cache itself
The KV cache is just numbers - and like weights, it can be stored at lower precision. Dropping KV from BF16 (2 bytes) to FP8 (1 byte) halves the cache footprint and the bandwidth read per decode step, with minimal quality loss on most workloads. It composes with every attention variant and every tiering trick above.
Toggle KV precision in the planner above between BF16 and FP8 and watch the network spill collapse. The same model that overflowed to storage at BF16 can sit entirely in HBM at FP8.
See Quantization & Precision for how FP8/FP4 formats actually work.
GPU HBM at a glance
HBM capacity and bandwidth are the hard ceiling on how much KV stays hot. This is the fleet the planner sizes against.
| Node | HBM / GPU | GPUs | Total HBM | Bandwidth | NVLink |
|---|---|---|---|---|---|
| 8× H100 (DGX/HGX) | 80 GB | 8 | 640 GB | ~3.35 TB/s | 900 GB/s |
| 8× H200 | 141 GB | 8 | 1,128 GB | ~4.8 TB/s | 900 GB/s |
| 8× B200 | 180 GB | 8 | 1,440 GB | ~8 TB/s | 1800 GB/s |
| 8× B300 | 288 GB | 8 | 2,304 GB | ~8 TB/s | 1800 GB/s |
| GB200 NVL72 | 186 GB | 72 | 13,392 GB | ~8 TB/s | 1800 GB/s |
| GB300 NVL72 | 288 GB | 72 | 20,736 GB | ~8 TB/s | 1800 GB/s |
Specs are approximate and vendor-published. NVLink figures are per-GPU intra-domain bandwidth; NVL72 systems pool all 72 GPUs into a single domain.
The other way out: have no KV cache at all
Everything on this page tackles a KV cache that grows linearly with context. State-space models (Mamba) sidestep it entirely: instead of caching every past token's keys and values, they carry one fixed-size recurrent state that's updated token-by-token - so memory stays constant no matter how long the context grows. The trade-off is lossy recall, which is why frontier models go hybrid (mostly SSM layers, a few attention layers).
When the cache outgrows the cluster: pooled, persistent KV
At 100K concurrent users with 64K-token contexts and multi-day retention, the KV cache reaches tens of petabytes - far beyond any local NVMe. A pooled, networked KV tier (VAST, NVIDIA CMX, Mooncake) makes that practical: KV persists across requests and nodes, returning users skip prefill entirely, and data reduction shrinks the footprint further. It turns the KV cache from a per-GPU scratchpad into shared cluster infrastructure.