Foundations
Attention Mechanisms
AdvancedMHA, MQA, GQA, MLA, and sliding-window attention - what each variant is, why it exists, and exactly how each shapes the KV-cache memory bill.
Queries, keys, values, and the KV cache
At each transformer layer, every token projects into three vectors: query (Q), key (K), and value (V). Attention computes softmax(QKᵀ/√d) · V - the query vector “asks” which tokens are relevant; the key vectors surface those tokens; the value vectors carry their content. During inference, newly generated tokens must attend to every previously seen token, so the K and V tensors from those prior tokens are saved to GPU memory rather than recomputed - that is the . The size of that cache is what makes attention variant selection a first-class infrastructure decision, not just a model-quality tradeoff. See the KV cache deep-dive for the full picture.
KV-cache bytes per token - the formula, drawn
Every term is one dimension of the block a single token leaves in the cache. Pick a model or change the K/V heads and precision; hover or tap a term to light up its dimension.
bytes = 2 × num_layers × num_kv_heads × head_dim × bytes_per_element
Multiply by batch size to get total GPU memory consumption. Only num_kv_heads changes between MHA, GQA and MQA - the other terms are fixed by the model's depth, width and serving precision.
Multiply by sequence length to get total cache for one request; by batch size to get total GPU memory consumption. Model Sizing & Parallelism covers the full memory budget.
Multi-Head Attention (MHA)
The original Vaswani et al. formulation. The hidden dimension is split into H parallel heads, each learning independent Q, K, and V projections. This lets different heads specialize in different relational patterns simultaneously - some attend to syntax, others to coreference, others to positional proximity. Critically, every query head has its own K/V head, so num_kv_heads = H. That makes the most expressive variant and the one with the largest KV cache. At 32 heads, BF16, 32 layers, and head_dim = 128, each token costs 32 × 32 × 128 × 2 × 2 = 524 KB - and that cost accumulates linearly with sequence length.
MHA: one K/V head per query head, for every token
Each row is a query head wired to its own K/V head (8 of 32 drawn). Every token appends a K/V pair to every row, on every layer - pick a context length to watch the cache grow.
One orange cell = 512 tokens of K/V for that head · repeated on all 32 layers
Context length
524 KB/token × 8,192 tokens
4.3 GB
For one request, at BF16. Multiply by concurrent requests for the GPU-wide bill.
Strengths
- ▸Maximum representational capacity - the baseline every other variant is compared against
- ▸Well-studied; most hardware and kernel optimizations target MHA first
- ▸Flash Attention / Flash Attention 2 make prefill fast by tiling into SRAM
Limitations
- ▸Largest KV cache of all variants - scales as H × layers × seq_len
- ▸Memory bandwidth during decode is proportional to num_kv_heads; high H hurts throughput
- ▸Impractical for very long context without complementary techniques
Multi-Query (MQA) and Grouped-Query (GQA) attention
Both variants reduce num_kv_heads below H, which directly shrinks the KV cache and the memory-bandwidth cost of decoding.
Blue circles are query heads; squares are the Key/Value heads that get cached. Fewer K/V heads = smaller KV cache. MHA, GQA, and MQA sit on one continuum - they differ only in how many query heads share each K/V head.
MHA
8 KV heads1× cache (baseline)
GQA (G=2)
2 KV heads¼× cache
MQA
1 KV head⅛× cache
MHA gives every query head its own K/V head - maximum expressiveness, largest cache. MQA collapses all heads onto one shared K/V - smallest cache, some quality loss. GQA is the middle ground production models settled on: a handful of K/V heads keeps quality near MHA while shrinking the cache 4–8×.
Multi-Query Attention (MQA)
num_kv_heads = 1All H query heads share a single K/V head. The KV cache shrinks by roughly 1/H - an 8-head model uses 1/8 of the MHA footprint. The tradeoff is expressive capacity: all heads see the same key and value, so the diversity MHA relies on is eliminated. At moderate model sizes this leads to measurable quality regression; larger models tolerate it better. Noam Shazeer proposed MQA in 2019 precisely to accelerate autoregressive decode. Used by Falcon, early PaLM, and early Gemini variants.
Grouped-Query Attention (GQA)
num_kv_heads = GQuery heads are partitioned into G groups; all heads in a group share one K/V head (so each K/V head serves H/G query heads). At G = H you recover MHA; at G = 1 you have MQA. In practice G = H/4 or H/8 (e.g., 8 KV heads for 32 query heads) offers the sweet spot: near-MHA quality with a 4–8× smaller cache. Llama 2 and Llama 3 both use , as do Mistral and Gemma. Ainslie et al. (2023) showed GQA checkpoints can be obtained by mean-pooling MHA checkpoints, making retrofitting practical.
Decode bandwidth intuition: At each decode step the GPU streams K/V tensors from HBM. With GQA at G = 8 instead of H = 32, you stream 4× fewer bytes per step - which translates nearly linearly to 4× higher decode throughput when the bottleneck is HBM bandwidth, not compute.
K/V heads read from HBM per layer, per cached token (32 query heads)
Multi-head Latent Attention (MLA)
Introduced in DeepSeek-V2 (2024) and carried into DeepSeek-V3, takes a fundamentally different approach: instead of storing H separate K/V heads per token, it compresses the entire K/V representation into a single low-rank latent vector per token per layer. During decoding, the full K/V matrices are reconstructed from that latent on the fly.
What actually gets cached per token - to scale
~93% smaller per token (DeepSeek-reported, vs an equal-dimension MHA model).
Standard attention caches full K and V for every head (large). MLA caches only the small shared latent c and reconstructs K/V during attention. The up-projection matrices are permanent weights, so they cost memory once - not once per token.
MLA also handles positional encodings through a decoupled mechanism: separate positional K heads are cached in full to avoid the incompatibility between low-rank compression and position-dependent rotations. The full architecture is detailed in the DeepSeek-V2 technical report.
Why DeepSeek's MLA is different - and why it wins
~70 KB / token · 61 layersStep through the four ideas that make MLA work - each tab pairs a picture with the detail.
Solid = stored in the cache per token · dashed = reconstructed, never stored.
For each token, the model projects the hidden state down to a compressed latent vector c (where d_c << d_model). Only c is cached. At attention time, K and V are recovered via up-projection matrices W_K and W_V - these matrices are shared across all heads and all positions, so they live in parameters (permanent weights), not the per-token cache.
GQA and MQA save memory by throwing away K/V heads (sharing them across queries). MLA keeps full per-head expressiveness but caches a single learned low-rank latent (d_c = 512) instead of the heads themselves - reconstructing K and V from it at attention time. It's compression, not head-sharing.
DeepSeek reported the 93.3% figure and MHA-parity quality (measured against a hypothetical equal-dimension MHA model), and third-party analyses partly echo them. Numbers reflect DeepSeek-V2 (60 layers) / V3 (61 layers).
Sliding-window and sparse attention for long context
Even with GQA or MLA, KV-cache memory grows linearly with sequence length. A 128K-token context at BF16 with Llama 3 8B (GQA, 8 KV heads, 32 layers, head_dim 128) still consumes roughly 128 000 × 131 KB ≈ 16.8 GB. Sparse attention variants cap this growth by restricting which tokens each position can attend to.
Each row is a token deciding which earlier tokens it can attend to (a lit cell = allowed). Full causal attention lets every token see all of its history - the lit area, and the KV cache, grow without bound. Sliding-window caps each token to the last 4 positions, so the cache stays a fixed size no matter how long the sequence gets.
Full causal
cache O(seq_len)Sliding window (W=4)
cache O(W)Sparse attention patterns, side by side
Each grid is an attention mask: row = query position, column = key position, lit = allowed. Tap a pattern for details (12 positions, illustrative window and dropped keys).
Sliding window
Mistral, MixtralMost open long-context models mix the two: gpt-oss alternates 128-token sliding-window layers with full-attention layers, Gemma 3 and 4 run five sliding-window layers per global one, and Llama 4 confines three of every four layers to 8K-token chunks - so only the global layers grow with context.
Playground: see each variant side by side
Switch between the variants to watch two things change at once: which Key/Value heads get cached (left) and which tokens each position is allowed to attend to (right). GQA and sliding-window expose a slider so you can feel the memory-vs-quality tradeoff directly.
Attention playground - one cache, two ways to shrink it
The grid is the KV cache for one layer: each row is a cached K/V head, each column a token. Play to decode tokens and watch it grow; switch variants to see head sharing remove rows, MLA replace them with a latent, and a sliding window stop the columns. Toy model: 8 query heads, head dim 128 (illustrative).
G = 8 is MHA · G = 1 is MQA
GQA: t1 decoded, 512 values cached per layer (MHA: 2,048).
KV cache for one layer
1 tokens × 2 K/V heads × 2 × 128 = 512
Who t1 can attend to
Row t1 attends to all 1 keys so far, so every column must stay cached.
GQA - Grouped-Query Attention
Query heads share K/V heads in groups, so the grid loses rows. Fewer groups = smaller cache, slightly less expressive.
KV bytes per token across attention generations
Every attention generation since 2017 has largely been an argument about one number: how many bytes the model must keep for each token it has already seen. A 70B model with full multi-head attention caches about 2.5 MiB per token; grouped-query attention cuts that 8×, DeepSeek's latent attention (MLA) compresses it to about 69 KiB, and DeepSeek V4.1-Flash gets it down to about 890 bytes by compressing tokens, storing values and letting most layers reuse another layer's cache. Switch the metric to see what that means for a whole sequence at 128K or 1M tokens - and how sparse attention changes a different number: how many cached tokens each new token reads.
KV cache bytes per token (whole model, log scale)
The MLA cache plus a small indexer key per token (128 dims in FP8) - it stores slightly more than MLA and saves reads, not memory.
Tap a row for what it stores. Each step down the ladder changes what is cached, not just how it is read.
MHA through the hybrid row assume a BF16 cache (FP8 would halve them); the DeepSeek V4 rows use their FP8/FP4 layouts. Illustrative: MHA. Exact arithmetic from the model config: GQA-8, MLA, Kimi K3. Estimates from the config plus an assumed cache layout: DSA (FP8 indexer key), V4-Pro and V4-Flash (576 B per compressed entry, 64 B FP4 indexer key). Model-card figure: V4.1-Flash, 890 B, which the config reproduces (4 KV layers × 356 B at 2:1, 2:1, 2:1 and 1:1). Read counts are averaged over attention layers and leave out the indexer's own scan.
Variant comparison
The table below compares all five variants on the dimensions that matter for infrastructure sizing.
| Variant | KV heads (vs. MHA) | Relative cache size | Quality impact | Example models |
|---|---|---|---|---|
| MHA Multi-Head Attention | H (one per query head) | 1× (baseline) | Baseline - no compromise | GPT-2/GPT-3, Llama 1, BERT |
| MQA Multi-Query Attention | 1 (shared by all heads) | ≈ 1/H | Slight loss at large H; faster decode | Falcon, PaLM, Gemma 2B |
| GQA Grouped-Query Attention | G (1 < G < H) | G/H | Near-MHA; sweet spot at G=H/4 or H/8 | Llama 2/3, Mistral, Gemma |
| MLA Multi-head Latent Attention | Latent vector (low-rank) | Small - model-specific | Near-MHA quality at a fraction of the cache | DeepSeek-V3, Kimi K2, Mistral Large 3 |
| SWA Sliding-Window Attention | H, but bounded window W | W/seq_len (capped) | Degrades on tasks requiring full global attention | Mistral 7B, Gemma 3/4, gpt-oss |
Relative cache size assumes constant layers, head_dim, and precision. MHA = H heads; GQA relative size = G/H (varies by config). For SWA the bounded window W replaces seq_len - cache does not grow beyond W regardless of context length. Exact figures depend on model architecture. See the KV cache calculator and model sizing for your specific config.
Worked examples: bytes per token
Applying the formula to real and illustrative model configs at BF16 (2 bytes/element) shows how starkly the variants diverge. Bars are scaled to the largest row (524 KB).
| Model | Variant | Layers | KV heads | Head dim | KV bytes/token |
|---|---|---|---|---|---|
| Llama 3 8B | GQA | 32 | 8 | 128 | 32 × 8 × 128 × 2 × 2 = 131 KB |
| Llama 3 70B | GQA | 80 | 8 | 128 | 80 × 8 × 128 × 2 × 2 = 328 KB |
| GPT-style (illustrative MHA, 32 heads) | MHA | 32 | 32 | 128 | 32 × 32 × 128 × 2 × 2 = 524 KB |
| Llama 4 Scout (17B-16E) | GQA | 48 | 8 | 128 | 48 × 8 × 128 × 2 × 2 = 197 KB*hatched = dense upper bound |
| Qwen3-235B-A22B | GQA | 94 | 4 | 128 | 94 × 4 × 128 × 2 × 2 = 193 KB |
| Gemma 3 27B | GQA | 62 | 16 | 128 | 62 × 16 × 128 × 2 × 2 = 508 KB*hatched = dense upper bound |
| DeepSeek-V3 (MLA) | MLA | 61 | - | latent 512+64 | ≈ 61 × 576 × 2 = 70 KB |
| Kimi K2 (Moonshot, MLA) | MLA | 61 | - | latent 512+64 | ≈ 61 × 576 × 2 = 70 KB |
| DeepSeek-V2 (MLA) | MLA | 60 | - | latent 512+64 | ≈ 60 × 576 × 2 = 69 KB |
DeepSeek MLA figures are approximate - the cached latent is c = 512 per layer plus a decoupled RoPE key (≈ 64 dim), so V2/V3/Kimi K2 cache 576 per layer. *Gemma 3 and Llama 4 Scout are dense upper bounds: Gemma 3 interleaves sliding-window and global layers (5:1) and Llama 4 caps three of every four layers at 8K-token chunks, so their real long-context caches are far smaller than shown. KV bytes per token multiply by sequence length to get total cache for one request.
Try it: interactive comparator
Dial in a model config and context length to see how the per-token and total KV cache diverge across all four variants - and how the gap widens with longer context and more concurrent requests.
How many users fit? Same model, four attention variants
The only thing that changes between rows is how many K/V vectors each token caches per layer. Follow that one factor from a single token, to bytes per token, to how many long requests fit on a GPU node.
Model shape · 80 layers · 64 query heads · head dim 128 · GQA G = 8
1 · What one token caches in one layer
Why they differ: MHA caches a K and a V for every one of the 64 query heads. GQA lets 8 query heads share each K/V pair; MQA lets all 64 share one. MLA caches one learned latent that every head rebuilds its K and V from.
2 · KV bytes per token, all 80 layers (log scale, fixed axis)
3 · 32K-token requests that fit on one 8× H100 (DGX/HGX)
640 GB HBM − 140 GB weights = 500.0 GB left for KV
One square per request, each at full length. KV only - activations and fragmentation are ignored, so real numbers are lower.
Per request = bytes per token × request length. Same layers, head dim and precision: the attention variant alone moves this node from 5 to 372 concurrent 32K-token requests.
Decision guide: picking attention for a memory budget
These cards are not architecture choices - if you are serving an existing model, the variant is fixed. They are useful for evaluating new model choices or advising teams building from scratch.
Pick your constraint
Choose the scenario closest to yours; the strip lights the variant(s) that fit it.
Recommendation
GQA (G = H/8 or H/4)
Cuts KV cache by 8–4× versus MHA with minimal quality cost. The Llama 3 family proves this works at 70B+ scale.
Going deeper: the 2025-26 research frontier
Everything above is enough to size a KV cache for today's models. The newest designs go further: attention that reads only part of the cache, linear-attention layers that swap the cache for a fixed-size state, and small changes to sinks, gating and position encoding that decide how far a model can stretch its context.
Trainable sparse & linear attention
MHA/GQA/MQA/MLA all still compute full attention over the context. The big 2025–2026 shift is making attention itself sparse or linear - and doing it in a way the model trains with, so the savings hold up in production rather than just at inference.
Four frontier mechanisms, drawn
Pick one to see what it changes about attention. Block and score values are illustrative.
context (query = newest token →)
1 · compress: coarse block summaries
2 · select: top-k fine-grained blocks
3 · sliding window: recent tokens
three branches → combined into one attention output
Native Sparse Attention (NSA)
Speed + long contextDeepSeek · Feb 2025
Sparse attention that is trainable from scratch, not bolted on at inference. Each query runs three branches - compress past tokens into coarse summaries, select the top-k most relevant fine-grained blocks, and a local sliding window - then combines them, with a hardware-aligned access pattern so the sparsity turns into real wall-clock speedups. NSA is a research design: no DeepSeek model shipped it. Its ideas - learned selection and compression - fed the different designs that did ship: DeepSeek Sparse Attention (V3.2) and V4's compressed attention, shown below.
Sparse attention in shipped models (late 2025 - 2026)
Sparse attention doesn't make the model forget the context - it changes how much of the cache each new token reads. (V3.2, also in GLM-5.x) and MiniMax-M3's block-sparse attention keep the full KV cache and use a cheap scorer to pick which tokens or blocks to read. IndexShare (GLM-5.2 and 5.3) reuses one layer's selection across the next three. DeepSeek V4 goes further and compresses what is stored in the first place. Step through each design to compare what stays in memory with what gets read.
How sparse attention picks what to read
One long context, scaled down to 128 cached tokens, and the newest query at the end. Pick a design to see which tokens the query reads - and whether the cache itself gets smaller. Scores and picks are illustrative.
cached context
128 tokens (illustrative)
At real scale: each query reads the top 2,048 tokens - 1.6% of a 128K context. The cache is not smaller: the indexer keys add about 20% on top of the MLA cache (~48 KB per token with the FP8 layout). DSA saves reads and compute, not memory.
DeepSeek Sparse Attention (DeepSeek V3.2, 2025; also used in GLM-5): a small lightning indexer scans every token, then attention reads only the top-k tokens it picked - scattered individual tokens, not contiguous blocks.
Sparse and linear attention compose with the head-level variants above - e.g., DeepSeek-V3.2 layers sparse attention on top of MLA. See Model Architectures for how these pair with MoE and state-space designs.
Linear attention grows up: the delta rule and hybrids
Linear-attention layers replace the growing KV cache with a fixed-size state matrix that stores key-value associations. The update rule decides how well that memory behaves. Plain linear attention only adds, so a key written twice returns a blend of old and new values. The delta rule writes just the error between what the state predicts and the new value, which cleanly overwrites stale pairs. adds a decay gate per head and a decay per channel, so the model can also choose what to forget.
How a fixed-size memory learns
A 4×4 state matrix S stores key → value associations: each row is a key channel, each column a value channel, and a cell's shade is the stored number. Step through six events and switch the update rule. Keys are one-hot and β = 1 so the arithmetic stays readable; the numbers are illustrative.
S ← S + β k (v − Sᵀk)ᵀ
DeltaNet first reads what S already returns for k, then writes only the error. A rewritten key replaces its old value instead of stacking on it - but nothing is ever forgotten on purpose.
Write k1 → A.
State S - always 4 × 4
The grid never grows - the millionth token is written into the same 16 numbers.
This step
Recall - query key
The retrieved vector matches the latest value exactly.
The state is the same size for every sequence, whether it has read 8K tokens or 1M - it does not grow with context. The price: it holds a blended summary, so it cannot recall an arbitrary old token verbatim. That is why production hybrids such as Qwen3.5, Kimi K3 and Nemotron 3 still keep some full-attention layers.
A fixed state can't recall an arbitrary old token word for word, so production models keep some full-attention layers: 1 in every 4 for Qwen3.5 and 3.8 (Gated DeltaNet), Kimi Linear and Kimi K3 (KDA), and GLM-5.3-Flash; roughly 1 in 5 to 10 for NVIDIA's Nemotron 3 and IBM's Granite 4.0 (Mamba-2). That cuts KV by 70-80% at long context - see the real layouts on Model Architectures, and what the fixed state means for prefix caching on KV Cache.
Sinks, gates and position encoding: the small changes that matter
makes every row of attention weights add up to 1, so a token with nothing to look up still has to put its weight somewhere - models learn to park it on the first token, an . StreamingLLM (2023) worked around this by always keeping the first 4 tokens in the cache. gpt-oss (Aug 2025) instead adds a learned “nothing” logit per head, which makes 128-token sliding-window layers workable and halves its KV at 128K. Gated attention (NeurIPS 2025 Best Paper), used in Qwen3-Next, Qwen3.5 and Kimi K3, lets a head close a sigmoid gate on its output, so no sink forms at all - for under 2% extra latency and no KV cost.
Why softmax needs somewhere to put attention - sinks and output gates
Rows = query tokens, columns = key tokens (causal, 12 tokens, illustrative weights). Brighter = more attention weight. Pick a fix:
Softmax weights in each row must sum to 1, even when a token has nothing useful to look up. Models learn to park that spare weight on the first token - an attention sink. Token 0 carries little meaning; it is just always visible.
Avg. weight on token 0
KV impact
Sinks go with huge activation outliers on a few channels, which makes FP8 / FP4 quantization harder.
Position encoding in 2026
Most models mark word order by rotating pairs of query and key dimensions by an angle that grows with position (RoPE). The slow-turning pairs only ever sweep a small arc during training, so past the training length they point at angles the model has never seen. fixes this by slowing those pairs (gpt-oss stretches 4K to 128K with factor 32), partial RoPE rotates only part of each head (Qwen3-Next 25%, MiniMax-M2 half), and Llama 4's iRoPE caps positions with 8,192-token chunks plus position-free global layers. Kimi K3 and GLM-5.3-Flash drop rotation from their MLA layers entirely and let their linear-attention layers carry order, reaching 1M context without YaRN.
Position encoding in 2026
Each clock is one pair of dimensions in an attention head (illustrative 32-dim head, 16 pairs, pretrained at 4K tokens). The hand shows how far RoPE has rotated that pair at the chosen token position.
RoPE rotates each pair of query/key dimensions by an angle proportional to the token's position, each pair at its own speed, so attention scores depend on the distance between tokens. Fast pairs wrap around many times during training; the slowest only ever sweep a small arc (the shaded wedge). Past the training length they point at angles the model never saw - the red hands.
Hands outside trained range
Needs a context-extension trick?
KV effect