Foundations
Attention Mechanisms
AdvancedMHA, MQA, GQA, MLA, and sliding-window attention - what each variant is, why it exists, and exactly how each shapes the KV-cache memory bill.
Queries, keys, values, and the KV cache
At each transformer layer, every token projects into three vectors: query (Q), key (K), and value (V). Attention computes softmax(QKᵀ/√d) · V - the query vector “asks” which tokens are relevant; the key vectors surface those tokens; the value vectors carry their content. During inference, newly generated tokens must attend to every previously seen token, so the K and V tensors from those prior tokens are saved to GPU memory rather than recomputed - that is the KV cache. The size of that cache is what makes attention variant selection a first-class infrastructure decision, not just a model-quality tradeoff. See the KV cache deep-dive for the full picture.
KV-cache bytes per token (formula)
bytes = 2 × num_layers × num_kv_heads × head_dim × bytes_per_element
- 2 - one tensor for K, one for V
- num_layers - every transformer block caches independently
- num_kv_heads - the number that varies by attention variant (H for MHA, 1 for MQA, G for GQA)
- head_dim - hidden_dim / num_query_heads; typically 64–128
- bytes_per_element - 2 for BF16/FP16, 1 for FP8, 4 for FP32
Multiply by sequence length to get total cache for one request; by batch size to get total GPU memory consumption. Model Sizing & Parallelism covers the full memory budget.
Multi-Head Attention (MHA)
The original Vaswani et al. formulation. The hidden dimension is split into H parallel heads, each learning independent Q, K, and V projections. This lets different heads specialize in different relational patterns simultaneously - some attend to syntax, others to coreference, others to positional proximity. Critically, every query head has its own K/V head, so num_kv_heads = H. That makes MHA the most expressive variant and the one with the largest KV cache. At 32 heads, BF16, 32 layers, and head_dim = 128, each token costs 32 × 32 × 128 × 2 × 2 = 524 KB - and that cost accumulates linearly with sequence length.
Strengths
- ▸Maximum representational capacity - the baseline every other variant is compared against
- ▸Well-studied; most hardware and kernel optimizations target MHA first
- ▸Flash Attention / Flash Attention 2 make prefill fast by tiling into SRAM
Limitations
- ▸Largest KV cache of all variants - scales as H × layers × seq_len
- ▸Memory bandwidth during decode is proportional to num_kv_heads; high H hurts throughput
- ▸Impractical for very long context without complementary techniques
Multi-Query (MQA) and Grouped-Query (GQA) attention
Both variants reduce num_kv_heads below H, which directly shrinks the KV cache and the memory-bandwidth cost of decoding.
Blue circles are query heads; squares are the Key/Value heads that get cached. Fewer K/V heads = smaller KV cache. MHA, GQA, and MQA sit on one continuum - they differ only in how many query heads share each K/V head.
MHA
8 KV heads1× cache (baseline)
GQA (G=2)
2 KV heads¼× cache
MQA
1 KV head⅛× cache
MHA gives every query head its own K/V head - maximum expressiveness, largest cache. MQA collapses all heads onto one shared K/V - smallest cache, some quality loss. GQA is the middle ground production models settled on: a handful of K/V heads keeps quality near MHA while shrinking the cache 4–8×.
Multi-Query Attention (MQA)
num_kv_heads = 1All H query heads share a single K/V head. The KV cache shrinks by roughly 1/H - an 8-head model uses 1/8 of the MHA footprint. The tradeoff is expressive capacity: all heads see the same key and value, so the diversity MHA relies on is eliminated. At moderate model sizes this leads to measurable quality regression; larger models tolerate it better. Noam Shazeer proposed MQA in 2019 precisely to accelerate autoregressive decode. Used by Falcon, early PaLM, and early Gemini variants.
Grouped-Query Attention (GQA)
num_kv_heads = GQuery heads are partitioned into G groups; all heads in a group share one K/V head (so each K/V head serves H/G query heads). At G = H you recover MHA; at G = 1 you have MQA. In practice G = H/4 or H/8 (e.g., 8 KV heads for 32 query heads) offers the sweet spot: near-MHA quality with a 4–8× smaller cache. Llama 2 and Llama 3 both use GQA, as do Mistral and Gemma. Ainslie et al. (2023) showed GQA checkpoints can be obtained by mean-pooling MHA checkpoints, making retrofitting practical.
Decode bandwidth intuition: At each decode step the GPU streams K/V tensors from HBM. With GQA at G = 8 instead of H = 32, you stream 4× fewer bytes per step - which translates nearly linearly to 4× higher decode throughput when the bottleneck is HBM bandwidth, not compute.
Multi-head Latent Attention (MLA)
Introduced in DeepSeek-V2 (2024) and carried into DeepSeek-V3, MLA takes a fundamentally different approach: instead of storing H separate K/V heads per token, it compresses the entire K/V representation into a single low-rank latent vector per token per layer. During decoding, the full K/V matrices are reconstructed from that latent on the fly.
What actually gets cached per token - to scale
~93% smaller per token (DeepSeek-reported, vs an equal-dimension MHA model).
Standard attention caches full K and V for every head (large). MLA caches only the small shared latent c and reconstructs K/V during attention. The up-projection matrices are permanent weights, so they cost memory once - not once per token.
How it works
For each token, the model projects the hidden state down to a compressed latent vector c (where d_c << d_model). Only c is cached. At attention time, K and V are recovered via up-projection matrices W_K and W_V - these matrices are shared across all heads and all positions, so they live in parameters (permanent weights), not the per-token cache.
Cache size
The per-token cache is proportional to d_c (the compressed dimension), not to num_heads × head_dim. In DeepSeek-V2, d_c = 512 versus 128 heads × 128 head_dim = 16 384 for a hypothetical full MHA - roughly 32× smaller. DeepSeek-V3 cites a 93.3% reduction in KV cache compared to a standard MHA baseline.
Quality
DeepSeek reports MLA matches or exceeds MHA quality at the same model scale. This is possible because the compression is learned, not a post-hoc approximation - the model trains end-to-end with the latent bottleneck. The practical cost is implementation complexity: custom CUDA/Triton kernels are needed; standard MHA kernels cannot be reused directly.
MLA also handles RoPE positional encodings through a decoupled mechanism: separate positional K heads are cached in full to avoid the incompatibility between low-rank compression and position-dependent rotations. The full architecture is detailed in the DeepSeek-V2 technical report.
Why DeepSeek's MLA is different - and why it wins
~70 KB / token · 61 layersWhat's different
GQA and MQA save memory by throwing away K/V heads (sharing them across queries). MLA keeps full per-head expressiveness but caches a single learned low-rank latent (d_c = 512) instead of the heads themselves - reconstructing K and V from it at attention time. It's compression, not head-sharing.
The decoupled-RoPE trick
RoPE position rotations don't commute with the latent up-projection, so you can't compress and still apply positions cleanly. DeepSeek's fix: carry one small 64-dim positional key (RoPE applied) alongside the compressed content latent. 512 + 64 = 576 cached elements per layer.
Why it's better
The compression is learned end-to-end, not a lossy post-hoc approximation - so DeepSeek reports MLA matches or beats MHA quality while shrinking the cache ~93% versus an equivalent full-MHA model. Against a GQA model it's a smaller but still large ~2.7–4.7× reduction.
The cost: custom kernels
MLA needs bespoke kernels (FlashMLA, vLLM/SGLang MLA paths). At decode a matrix-absorption trick folds the K/V up-projections into the query and output projections, so attention runs directly against the cached latent without ever materialising full K/V.
The 93.3% figure and the MHA-parity quality claim are DeepSeek-stated (measured against a hypothetical equal-dimension MHA model); they are plausible and partly echoed by third-party analyses, but treat them as vendor-reported. Numbers reflect DeepSeek-V2 (60 layers) / V3 (61 layers).
Sliding-window and sparse attention for long context
Even with GQA or MLA, KV-cache memory grows linearly with sequence length. A 128K-token context at BF16 with Llama 3 8B (GQA, 8 KV heads, 32 layers, head_dim 128) still consumes roughly 128 000 × 131 KB ≈ 16.8 GB. Sparse attention variants cap this growth by restricting which tokens each position can attend to.
Each row is a token deciding which earlier tokens it can attend to (a lit cell = allowed). Full causal attention lets every token see all of its history - the lit area, and the KV cache, grow without bound. Sliding-window caps each token to the last 4 positions, so the cache stays a fixed size no matter how long the sequence gets.
Full causal
cache O(seq_len)Sliding window (W=4)
cache O(W)Sliding-Window Attention (SWA)
Mistral, MixtralEach token attends only to the W most recent tokens (the “window”). The KV cache per layer is capped at W entries regardless of sequence length, making memory consumption O(W) rather than O(seq_len). Mistral 7B uses W = 4096 on alternating layers while keeping a few full-context layers for global coherence - the “Mistral pattern.” Mixtral extends this with a sparse MoE stack. The limitation: tasks requiring attention across tokens further than W apart (e.g., recalling details from the beginning of a long document) degrade unless global layers compensate.
Other sparse patterns
- ▸Strided / dilated: attends to every k-th token in addition to the local window, preserving coarse long-range signal (Longformer).
- ▸Global tokens: a handful of designated positions (e.g., [CLS], task prompts) attend to and are attended to by all tokens - cheap global context.
- ▸Linear / sub-quadratic approximations: Linformer, Performer, Hyper-Attention; reduce the O(n²) attention complexity but require architectural compromises.
- ▸KV eviction / compression: runtime systems like H2O or SnapKV drop low-importance cache entries mid-sequence - orthogonal to the attention variant and compatible with any of them.
Most production long-context models (e.g., Gemini 1.5 Pro) use a combination: full attention with GQA or MLA for moderate lengths, with architectural tricks to extend further without unbounded cache growth.
Playground: see each variant side by side
Switch between the variants to watch two things change at once: which Key/Value heads get cached (left) and which tokens each position is allowed to attend to (right). GQA and sliding-window expose a slider so you can feel the memory-vs-quality tradeoff directly.
Attention playground
Pick a variant and watch how it changes what gets cached and which tokens each position can see. GQA and sliding-window have sliders to explore the tradeoff.
G=8 → identical to MHA · G=1 → identical to MQA
What gets cached
Attention mask (who sees whom)
Standard causal mask: each token attends to all positions before it.
GQA - Grouped-Query Attention
Query heads share K/V heads in groups. Drag the slider: fewer groups = smaller cache, slightly less expressive.
The frontier: trainable sparse & linear attention (2025–26)
MHA/GQA/MQA/MLA all still compute full attention over the context. The big 2025–2026 shift is making attention itself sparse or linear - and doing it in a way the model trains with, so the savings hold up in production rather than just at inference.
Native Sparse Attention (NSA)
Speed + long contextDeepSeek · Feb 2025
Sparse attention that is trainable from scratch, not bolted on at inference. Each query runs three branches - compress past tokens into coarse summaries, select the top-k most relevant fine-grained blocks, and a local sliding window - then combines them. The access pattern is hardware-aligned so the sparsity turns into real wall-clock speedups on long sequences. Shipped as DeepSeek Sparse Attention (DSA) in DeepSeek-V3.2, and carried into the DeepSeek-V4 preview (Apr 2026), which pairs token-wise compression with DSA on top of the same MLA latent lineage - an evolution of MLA, not the original V2/V3 version.
MoBA (Mixture of Block Attention)
Flexible long contextMoonshot AI / Kimi · Feb 2025
Applies the Mixture-of-Experts idea to attention: the context is split into blocks and a cheap router picks which blocks each query attends to. Unlike NSA's three fixed branches, it's a single learned block-routing scheme designed as a drop-in toggle - the same model can switch between full and sparse attention. Deployed in Kimi for long-context requests.
Lightning / linear attention
Extreme context lengthMiniMax-01 · Jan 2025
Replaces the quadratic softmax with a kernel formulation whose cost grows linearly with sequence length. Pure linear attention loses quality, so MiniMax-01 uses a 7:1 hybrid (7 linear layers per 1 softmax layer). The 456B-param model (45.9B active) trains at up to 1M tokens and extrapolates toward ~4M at inference.
FlashAttention-3
Faster, not differentTri Dao et al. · kernel
A kernel/IO optimisation, not an architecture change - it computes the same attention math far faster on Hopper GPUs via warp-specialised pipelines, matmul/softmax interleaving, and FP8. ~1.5–2× over FlashAttention-2 and ~75% H100 utilisation. Because it's a kernel, it composes with MHA, GQA, or MLA - you keep your architecture and just run it faster.
Speedup and quality figures here are vendor-reported. Sparse/linear attention composes with the head-level variants above - e.g., DeepSeek-V3.2 layers sparse attention on top of MLA. See Model Architectures for how these pair with MoE and state-space designs.
Variant comparison
The table below compares all five variants on the dimensions that matter for infrastructure sizing.
| Variant | KV heads (vs. MHA) | Relative cache size | Quality impact | Example models |
|---|---|---|---|---|
| MHA Multi-Head Attention | H (one per query head) | 1× (baseline) | Baseline - no compromise | GPT-4, early Llama, BERT |
| MQA Multi-Query Attention | 1 (shared by all heads) | ≈ 1/H | Slight loss at large H; faster decode | Falcon, PaLM, early Gemini |
| GQA Grouped-Query Attention | G (1 < G < H) | G/H | Near-MHA; sweet spot at G=H/4 or H/8 | Llama 2/3, Mistral, Gemma |
| MLA Multi-head Latent Attention | Latent vector (low-rank) | Small - model-specific | Near-MHA; DeepSeek claims parity | DeepSeek-V2, DeepSeek-V3 |
| SWA Sliding-Window Attention | H, but bounded window W | W/seq_len (capped) | Degrades on tasks requiring full global attention | Mistral 7B, Mixtral |
Relative cache size assumes constant layers, head_dim, and precision. MHA = H heads; GQA relative size = G/H (varies by config). For SWA the bounded window W replaces seq_len - cache does not grow beyond W regardless of context length. Exact figures depend on model architecture. See the KV cache calculator and model sizing for your specific config.
Worked examples: bytes per token
Applying the formula to real and illustrative model configs at BF16 (2 bytes/element) shows how starkly the variants diverge.
| Model | Variant | Layers | KV heads | Head dim | KV bytes/token |
|---|---|---|---|---|---|
| Llama 3 8B | GQA | 32 | 8 | 128 | 32 × 8 × 128 × 2 × 2 = 131 KB |
| Llama 3 70B | GQA | 80 | 8 | 128 | 80 × 8 × 128 × 2 × 2 = 328 KB |
| GPT-style (illustrative MHA, 32 heads) | MHA | 32 | 32 | 128 | 32 × 32 × 128 × 2 × 2 = 524 KB |
| Llama 4 Scout (17B-16E) | GQA | 48 | 8 | 128 | 48 × 8 × 128 × 2 × 2 = 197 KB |
| Qwen3-235B-A22B | GQA | 94 | 4 | 128 | 94 × 4 × 128 × 2 × 2 = 193 KB |
| Gemma 3 27B | GQA | 62 | 16 | 128 | 62 × 16 × 128 × 2 × 2 = 508 KB* |
| DeepSeek-V3 (MLA) | MLA | 61 | - | latent 512+64 | ≈ 61 × 576 × 2 = 70 KB |
| Kimi K2 (Moonshot, MLA) | MLA | 61 | - | latent 512+64 | ≈ 61 × 576 × 2 = 70 KB |
| DeepSeek-V2 (MLA, approx.) | MLA | 60 | - | latent 512 | ≈ 60 × 512 × 2 = 61 KB |
DeepSeek MLA figures are approximate - the cached latent is c = 512 per layer plus a decoupled RoPE key (≈ 64 dim), so V3/Kimi K2 cache 576 per layer. *Gemma 3 is a dense upper bound: it interleaves sliding-window and global layers (5:1), so its real long-context cache is far smaller than the figure shown. KV bytes per token multiply by sequence length to get total cache for one request.
Try it: interactive comparator
Dial in a model config and context length to see how the per-token and total KV cache diverge across all four variants - and how the gap widens with longer context and more concurrent requests.
KV cache comparator - same model, four attention variants
Set a model config; see how the per-token KV cache and the total memory diverge across variants. MLA uses DeepSeek's fixed latent (d_c=512 + 64 RoPE) so it ignores head count.
KV cache per token
MHA total
80.00 GB
baseline
GQA total
10.00 GB
88% smaller
MQA total
1.25 GB
98% smaller
MLA total
2.81 GB
96% smaller
Total = per-token × sequence length × batch. This is KV cache only - model weights are a separate, fixed cost. Notice how the gap explodes with context and concurrency: at long context, the attention variant is what decides how many users fit on a GPU.
Decision guide: picking attention for a memory budget
These cards are not architecture choices - if you are serving an existing model, the variant is fixed. They are useful for evaluating new model choices or advising teams building from scratch.
If: Tightest memory budget, serving many users
GQA (G = H/8 or H/4)
Cuts KV cache by 8–4× versus MHA with minimal quality cost. The Llama 3 family proves this works at 70B+ scale.
If: Very long context windows (100 K+ tokens)
SWA or sparse hybrids
Full-context KV cache grows O(seq_len) - sliding-window caps it at O(W). Some layers are global, others windowed (Mistral pattern).
If: Extreme scale, MoE or dense 100B+ models
MLA (if building from scratch)
DeepSeek-V2/V3 show MLA keeps cache small even at massive hidden dim, at the cost of a more complex kernel implementation.
If: Maximum quality, memory is not a constraint
MHA
Each query head gets its own K/V projection - the most expressive representation, and the baseline everything else is measured against.
If: Decode latency is the primary constraint
MQA or GQA
Decode is memory-bandwidth-bound. Fewer KV heads means less data to stream from HBM per token, directly improving tokens/sec.
Related pages
Attention variant choice is inseparable from the wider memory and infrastructure story.