Foundations

Model Architectures

Intermediate

From the decoder-only transformer through MoE, state-space models, and multimodal - and what each choice costs at inference time.

Prerequisite: this page references attention, KV cache, and head dimensions throughout. If those are new, read Attention Mechanisms first.

The decoder-only transformer: today's default

Almost every production LLM you encounter today is a decoder-only transformer - the architecture family that started with GPT and now covers Llama, Mistral, Gemma, Qwen, and hundreds of open and proprietary variants. It won out over encoder-decoder (BERT, T5) and pure-encoder designs for generative tasks because autoregressive generation maps cleanly onto it: each new token attends to all prior tokens and produces the next one.

A typical model stacks 32–96 identical transformer blocks. Each block has four logical components:

Self-attention

Each token attends to every token before it. Keys and Values are cached (the KV cache) so the model never recomputes prior context. Quadratic in sequence length during prefill.

Feed-forward / MLP

Two linear projections with a non-linearity between them. Typically 4× the model dimension. In a dense model this runs on every token; in MoE only a subset of experts fires.

Residual stream

A fixed-width vector (the hidden dimension d_model) flows through every layer. Each block reads from it and adds back. The stream is the model’s working memory for a token.

Layer normalisation

Stabilises training by normalising activations before attention and the MLP. Most modern models use RMSNorm (cheaper than full LayerNorm) applied pre-block.

Residualstream

d_model

Token vector enters ↓

RMSNorm

normalise activations

Self-attention (Q · K · V)KV cache

mixes information across tokens - Keys & Values cached

RMSNorm

normalise again

Feed-forward / MLPdense → all fire · MoE → few fire

per-token transform (≈ 4× d_model wide)

repeat × 32–96 layers

↓ next-token prediction

Each block reads from the residual stream and adds back (the “+” steps). That additive structure is why deep stacks train stably and why you can quantise or prune individual layers without collapse.

The “residual stream” is the key abstraction: a vector of dimension d_model (e.g., 4096 for Llama-3 8B, 8192 for 70B) that persists per token through all layers. Every attention head and every MLP reads from and writes back to it. This additive structure lets deep networks train reliably and is why you can prune or quantise individual layers without catastrophic collapse.

See Attention Mechanisms for a deep dive into multi-head, grouped-query, and multi-latent attention variants.

Dense vs sparse: capacity vs compute

The single most important distinction in modern model design is how many parameters fire per token. Dense models activate everything; sparse models route each token through a fraction of their capacity. The tradeoff changes the memory footprint, the compute cost, and the serving strategy.

Dense

100 % of parameters activate per token

Every weight in every MLP fires on every forward pass. Memory equals the full parameter count. Predictable compute; straightforward to serve.

Examples: GPT-4 (est.), Llama-3, Mistral 7B

Sparse MoE

~10–25 % of parameters activate per token

A router selects a small number of expert MLPs per token. You load all parameters into memory but only compute through the active ones - high capacity, lower FLOPs/token.

Examples: Mixtral 8x7B, DeepSeek-V3, Qwen3-MoE, Llama-4

The critical implication for infrastructure: a sparse model's device memory requirement is set by its total parameters, not its active ones. Mixtral 8x7B has ~47 B total parameters and must reside in memory as such - even though only ~13 B of those weights execute per token. You get the FLOPs of a 13 B model with the memory bill of a 47 B one.

Mixture-of-Experts (MoE)

MoE replaces each dense MLP layer with a set of expert sub-networks plus a lightweight router. The router decides which experts handle each token; the rest sit idle. This lets a model scale capacity (total parameters) far beyond what its compute budget (active parameters) would normally allow.

Routing: dense vs Mixture-of-Experts

Current tokenThe→ router picks experts

Expert 1

active

Expert 2

active

Expert 3

idle

Expert 4

idle

Expert 5

idle

Expert 6

idle

Expert 7

idle

Expert 8

idle

Loaded in memory

47B

all weights resident

Active this token

13B

2 of 8 experts fire

Compute saved

~72%

fewer FLOPs / token

This is why MoE is preferred at the frontier: you pay the memory bill of the full 47B parameters, but the compute bill of only the 13B that fire. Capacity scales with total params; cost scales with active params. Anchored on Mixtral 8×7B (8 experts, top-2); frontier MoEs like DeepSeek-V3 and Kimi K2 push this far further - hundreds of fine-grained experts with only a handful active per token.

Top-k gating

A lightweight linear router produces a score for each expert. The top-k experts (typically k=1 or k=2) are selected and their outputs are summed with learned weights. k=1 minimises compute; k=2 adds redundancy.

Expert capacity

Each expert has a token budget per batch. Tokens routed to an overloaded expert are dropped or rerouted. Capacity factor controls the budget headroom and the quality/throughput tradeoff.

Load balancing loss

Without regularisation, the router collapses - routing everything through one expert. An auxiliary load-balancing loss penalises uneven routing and keeps expert utilisation spread.

Expert parallelism

Experts are distributed across GPUs. A token may visit two experts that live on different devices, requiring all-to-all communication. This makes MoE harder to serve than an equivalent dense model.

Real-world MoE models

Model	Total params	Active params	Routing
DeepSeek-V3 / R1	~671 B	~37 B	256 routed + shared, top-8
Kimi K2 (Moonshot)	~1 T	~32 B	384 experts, top-8
Qwen3 235B-A22B	~235 B	~22 B	128 experts, top-8
Llama 4 Scout / Maverick	~109 B / ~400 B	~17 B	16 / 128 experts, top-1
Mixtral 8x7B	~47 B	~13 B	8 experts, top-2

DeepSeek-V3 / R1: Fine-grained experts + Multi-head Latent Attention (MLA) shrink the KV cache to ~70 KB/token. Auxiliary-loss-free balancing and FP8 training make it the efficiency reference point - see the callout below.

Kimi K2 (Moonshot): One of the largest open-weight MoEs (2025). Only ~3% of weights fire per token; the MuonClip optimiser kept a 1 T-param model stable across 15.5 T training tokens.

Qwen3 235B-A22B: Alibaba. Competitive with dense models several times its active-param count on reasoning and agentic tasks.

Llama 4 Scout / Maverick: Meta's first MoE Llama series (2025), natively multimodal. Scout targets very long context (up to ~10 M tokens); Maverick targets frontier quality. Both keep active params near ~17 B.

Mixtral 8x7B: Mistral AI (2023). The model that popularised open MoE - 13 B active params matched dense 70 B quality. Used as the anchor in the demo above.

Why is DeepSeek-V3 so much more efficient?

671 B total · 37 B active

DeepSeek-V3 isn't just “another MoE.” It stacks four optimisations that compound - which is why it matches frontier dense models at a fraction of the training and serving cost.

Multi-head Latent Attention (MLA)

Instead of caching full Keys and Values per head, MLA compresses them into a small shared low-rank latent vector and reconstructs on the fly. The KV cache drops to ~70 KB/token - roughly 4–7× smaller than a comparable GQA model - so the same GPU serves far longer contexts.

Auxiliary-loss-free load balancing

Most MoEs add a load-balancing penalty that quietly fights the main objective and hurts quality. DeepSeek-V3 balances experts with a dynamic per-expert bias term instead - no auxiliary loss, so capacity is spread evenly without taxing accuracy.

Fine-grained shared + routed experts

256 small routed experts (top-8) plus always-on shared experts. Smaller experts specialise more precisely, and the shared experts capture common patterns so routed ones don’t waste capacity relearning them - more effective capacity per active parameter.

FP8 training + Multi-Token Prediction

The first frontier model trained largely in 8-bit floating point, roughly halving memory and bandwidth versus BF16. Multi-Token Prediction trains the model to predict several tokens ahead, improving sample efficiency and enabling faster speculative decoding at inference.

The headline number: MLA compresses the KV cache to ~70 KB per token versus ~327–516 KB for comparable GQA models - a 4–7× reduction that lets one GPU serve far longer contexts. The numbers table below makes this concrete.

Parameter counts are approximate; architectures continue to evolve. See Model Sizing & Parallelism for the memory arithmetic.

State-space models: Mamba and beyond

Attention's quadratic cost - O(n²) compute and O(n) KV cache memory with sequence length - is a fundamental constraint. State-space models (SSMs) replace the attention mechanism with a learned linear recurrence: a compact hidden state that is updated token-by-token without ever materialising a full attention matrix.

Attention (Transformer)

O(n²) compute

Every new token attends to all previous tokens. The Keys & Values of past tokens are stored in the KV cache, which grows linearly with context length.

State-space model (Mamba)

O(1) memory

Each token updates one fixed-size hidden state and is discarded. No KV cache - memory is constant no matter how long the context grows, but the state must compress all history (weaker exact recall).

Linear recurrence

SSMs propagate a fixed-size hidden state forward in time. Each new token updates the state in O(1) - inference is a recurrence, not a full attention over history.

Selective state space (Mamba)

Classic SSMs have input-independent state matrices and struggle to selectively remember tokens. Mamba makes the state-space parameters (B, C, Δ) input-dependent, giving it content-aware filtering.

Mamba-2 & SSD

Structured State-Space Duality rewrites the Mamba scan so it can be computed as a block-decomposed matrix multiply - enabling hardware-efficient training on GPUs while preserving linear inference.

Long-context advantage

Attention cost grows as O(n²) in sequence length; SSM inference stays O(n). For very long contexts (100k+ tokens), this gap becomes decisive for both memory and latency.

How an SSM keeps its memory

An SSM doesn't store the history - it keeps a running, compressed summary of it in one fixed-size state vector h. Each token updates that state with a linear recurrence, then is discarded:

h_t = Ā · h_t-1 + B̄ · x_t // fold the new token into the state

y_t = C · h_t // read an output out of the state

The state is the memory

h is a lossy snapshot of everything seen so far - the same size at token 5 or 5,000,000. History is folded in, not stored alongside, so memory is O(1).

Ā = controlled forgetting

The transition matrix decides how much old state survives. Eigenvalues near 1 keep long memory; near 0 decay fast - a learned, per-dimension retention dial.

Δ = selective gate (Mamba)

Mamba makes B, C and the step size Δ depend on the token. Large Δ absorbs/refreshes on this token; small Δ preserves the state - content-aware memory.

The trade-off: the transformer keeps an exact transcript (lossless recall, but it grows forever); the SSM keeps fixed-size notes (constant memory, but it can't quote an arbitrary old token verbatim). That weakness is why frontier models go hybrid.

KV cache consumption, mathematically

The contrast is exact, not hand-wavy: the transformer's KV cache is linear in context length, the SSM's state is constant, and a hybrid pays only for its few attention layers. Slide the context and watch the gap open.

KV consumption, mathematically - Transformer vs SSM vs Hybrid

The transformer caches K & V for every token at every layer, so its memory is 2·L·H·d·b·n - linear in context n. An SSM keeps one fixed state per layer (L·d_inner·d_state·b) that doesn't depend on n at all. Slide the context length.

Context length (n)128K tokens

1K1M

Transformerfull attention · O(n)32.00 GB

2·L·H·d·b·n = 256 KB/tok × 128K

Hybrid8/64 attn layers · small O(n)4.01 GB

32 KB/tok × 128K + 14.0 MB state

SSMfixed state · O(1)16.0 MB

L·d_inner·d_state·b = 16.0 MB (constant)

At 128K context, the transformer needs ~8× the memory of the hybrid - and the SSM is flat at 16.0 MB.

Because only n moves the transformer term, the gap widens without bound as context grows. That linear-vs-constant split is the whole reason long-context systems lean on SSM and hybrid designs.

Illustrative ~7-70B-class config (BF16): L=64, KV heads=8, d_head=128, d_inner=8192, d_state=16; hybrid caches 8 of 64 layers. Bars use a log scale (the real linear gap is even larger) - the numbers are exact. Real models vary; try them in the KV tiering calculator.

Hybrid models (Transformer + SSM)

Pure SSMs sacrifice some quality on tasks that benefit from exact long-range recall. Hybrids interleave Transformer attention blocks with Mamba blocks - getting most of the quality from the attention layers and most of the memory efficiency from the SSM layers. Only the Transformer layers produce KV cache entries.

Jamba

AI21 Labs

Transformer + Mamba (7:1 ratio)

Interleaves Transformer and Mamba blocks. Gets near-Transformer quality with significantly lower KV cache memory at long context.

Zamba2

Zyphra

Shared-weight Mamba + sparse attention

Achieves strong performance at 2.7B and 7B scales. Shared attention layers reduce parameter count while the Mamba backbone handles most tokens.

Falcon Mamba

TII

Pure Mamba

One of the first production-scale pure-Mamba LLMs. Demonstrates that SSMs can compete with Transformers at the 7B scale.

SSMs eliminate the KV cache and replace it with a fixed-size recurrent state. See KV Cache for why that matters when context grows long.

Vision & multimodal: encoders bolted onto LLMs

Modern vision-language models (VLMs) don't retrain the LLM from scratch with images. The dominant pattern: take a pretrained image encoder, project its outputs into the LLM's token space, and concatenate them with the text sequence. The LLM's weights are largely frozen; only the projector (and optionally the encoder) is fine-tuned. The LLM then “reads” image patches as if they were text tokens.

Image

Raw pixels

split into patches

→↓

Vision encoder

ViT (CLIP / SigLIP)

each patch → a vector

→↓

Projector

MLP adapter

map into the LLM's token space

→↓

LLM backbone

Decoder transformer

imgimgimg“whatisthis?”

image tokens sit in the same sequence as text

The LLM treats image patches as ordinary tokens: they take sequence positions, consume KV cache, and are attended to like words. A single high-res image can become 500–2,500 tokens - often more context than the text prompt itself.

Vision encoder

A pretrained image model (e.g., CLIP ViT-L/14, SigLIP, InternViT) converts a raw image into a sequence of patch embeddings. Each patch becomes a fixed-size vector.

Projector / adapter

A small MLP or cross-attention module maps the vision encoder’s embedding space into the LLM’s token embedding space. This is the cheapest component to fine-tune.

LLM backbone

The decoder treats image tokens exactly like text tokens - they occupy sequence positions, consume KV cache, and are attended to by all subsequent tokens. An image patch grid at 336 px/224 px typically produces 256–576 tokens.

Audio / video (extended)

Audio is usually spectrogram-encoded (e.g., Whisper-style encoder); video is sampled into frame sequences. Token counts explode quickly - a 10-second clip at 1 fps easily consumes 2,500+ tokens.

Representative VLMs

Model	Modality	Vision encoder	Notes
Qwen3-VL	Image, video	ViT, dynamic resolution + mRoPE	Alibaba's current flagship VLM (2025). Dynamic resolution adjusts patch count to image size, and time-aware position encoding handles video - strong agentic and long-context vision.
Llama 4 Maverick	Image, text	Native multimodal (MoE backbone)	Meta (2025). Vision trained into a 400B/17B-active MoE rather than bolted on, with a 128K+ context for high-resolution images.
Gemini 2.5 Pro / Flash	Image, audio, video, text	Undisclosed (native)	Google DeepMind (2025). End-to-end multimodal with very long native context; video understanding is a primary use case.
InternVL3	Image, video	InternViT + Qwen2.5 backbone	OpenGVLab (2025). Native multimodal pretraining (not adapter-bolted) for strong cross-modal grounding; a leading open-weight VLM.
GPT-4o	Image, audio, video	Undisclosed (native)	OpenAI. Audio and vision trained end-to-end; popularised real-time multimodal interaction.
LLaVA-1.5	Image	CLIP ViT-L	The reference open VLM that introduced the connector/projector pattern most open models still follow.

Infrastructure note: serving a VLM at high resolution can double or triple effective sequence length per request. KV cache sizing must account for image token counts, not just text. See KV Cache.

Architecture drives the infrastructure bill

Every architecture choice above maps directly to a serving cost. The table below summarises the key dimensions: how memory scales, how sequence-length scaling behaves, and what that means when you size a cluster.

The numbers: weights vs KV cache, by model

Two separate memory bills dominate serving. Weights are a fixed, one-time cost (loaded once, shared across all requests). KV cache is a per-request, per-token cost that grows with context length and concurrency - and it is where architecture choices show up most dramatically.

Model	Architecture	Params	Weights (BF16)	KV / token	KV @ 128K ctx
Llama 3.1 8B	Dense · GQA	8 B	16 GB	~128 KB	~16 GB
Mixtral 8x7B	MoE · GQA	47 B / 13 B active	94 GB	~128 KB	~16 GB
Llama 3.1 70B	Dense · GQA	70 B	140 GB	~320 KB	~40 GB
Llama 3.1 405B	Dense · GQA	405 B	810 GB	~516 KB	~63 GB
DeepSeek-V3	MoE · MLA	671 B / 37 B active	1.34 TB (≈671 GB FP8)	~70 KB	~9 GB
Mamba-2 (SSM)	State-space	varies	scales with params	no KV cache	constant state

Read across the bottom three rows: a dense 405 B model needs ~63 GB of KV cache for a single 128K-token request, while DeepSeek-V3 - a far larger 671 B model - needs only ~9 GB thanks to MLA, and an SSM needs essentially none. KV cache, not weights, is what limits how many long-context users you can pack onto a GPU. Weights assume BF16 (2 bytes/param); KV-cache figures are per single request and scale linearly with batch size.

Qualitative comparison

Family	Total vs active params	Sequence scaling	Example models	Infra implication
Dense Transformer	Total = active (100 %)	O(n²) attention	Llama-3 70B, Mistral 7B	Memory proportional to full param count. Predictable but expensive at scale. KV cache grows linearly with batch × context.
Sparse MoE	~5–20 % active	O(n²) attention	Mixtral 8x7B, DeepSeek-V3	Must load all params onto device(s). Needs expert parallelism. FLOPs/token low but memory footprint is the total - not the active - parameter count.
SSM / Mamba	Total = active (100 %)	O(n) inference	Mamba-2, Falcon Mamba	No KV cache; fixed-size hidden state instead. Memory flat w.r.t. context length - transformative for very long sequences.
Hybrid (Transformer + SSM)	Total = active (100 %)	Mixed O(n²) / O(n)	Jamba, Zamba2	Partial KV cache (only for Transformer layers). Good balance of quality and memory at long context. Complexity increases deployment.
Multimodal VLM	Dense or MoE backbone	O(n²) - images inflate n	Qwen3-VL, Llama 4 Maverick, InternVL3	Image tokens occupy sequence budget. High-res inputs produce 500–2000 extra tokens per image - KV cache and prefill cost spike accordingly.

Parameter counts and active-param ratios are illustrative; exact figures vary by configuration and version. Confirm against model cards and provider documentation before sizing production clusters.

Dig deeper

Architecture determines memory layout, serving topology, and cost - explore these pages to put the numbers on the table.

Attention mechanisms →KV cache →Model sizing & parallelism →NVIDIA infrastructure →