Model Architectures

The decoder-only transformer: today's default

Almost every production LLM you encounter today is a decoder-only transformer - the architecture family that started with GPT and now covers Llama, Mistral, Gemma, Qwen, and hundreds of open and proprietary variants. It won out over encoder-decoder (BERT, T5) and pure-encoder designs for generative tasks because autoregressive generation maps cleanly onto it: each new token attends to all prior tokens and produces the next one.

A typical model stacks 32–96 identical transformer blocks. Each block has four logical components:

The “residual stream” is the key abstraction: a vector of dimension d_model (e.g., 4096 for Llama-3 8B, 8192 for 70B) that persists per token through all layers. Every attention head and every MLP reads from and writes back to it. This additive structure lets deep networks train reliably and is why you can prune or quantize individual layers without catastrophic collapse.

See Attention Mechanisms for a deep dive into multi-head, grouped-query, and multi-latent attention variants.

Dense vs sparse: capacity vs compute

The single most important distinction in modern model design is how many parameters fire per token. Dense models activate everything; sparse models route each token through a fraction of their capacity. The tradeoff changes the memory footprint, the compute cost, and the serving strategy.

The critical implication for infrastructure: a sparse model's device memory requirement is set by its total parameters, not its active ones. Mixtral 8x7B has ~47 B total parameters and must reside in memory as such - even though only ~13 B of those weights execute per token. You get the FLOPs of a 13 B model with the memory bill of a 47 B one.

Mixture-of-Experts (MoE)

MoE replaces each dense MLP layer with a set of expert sub-networks plus a lightweight router. The router decides which experts handle each token; the rest sit idle. This lets a model scale capacity (total parameters) far beyond what its compute budget (active parameters) would normally allow.

Real-world MoE models

Parameter counts are approximate; architectures continue to evolve. See Model Sizing & Parallelism for the memory arithmetic.

State-space models: Mamba and beyond

Attention's quadratic cost - O(n²) compute and O(n) KV cache memory with sequence length - is a fundamental constraint. State-space models (SSMs) replace the attention mechanism with a learned linear recurrence: a compact hidden state that is updated token-by-token without ever materialising a full attention matrix.

How an SSM keeps its memory

An SSM doesn't store the history - it keeps a running, compressed summary of it in one fixed-size state vector h. Each token updates that state with a linear recurrence, then is discarded:

The tradeoff: the transformer keeps an exact transcript (lossless recall, but it grows forever); the SSM keeps fixed-size notes (constant memory, but it can't quote an arbitrary old token verbatim). That weakness is why frontier models go hybrid.

KV cache consumption, mathematically

The contrast is exact, not hand-wavy: the transformer's KV cache is linear in context length, the SSM's state is constant, and a hybrid pays only for its few attention layers. Slide the context and watch the gap open.

Hybrid models (attention + linear / SSM layers)

Pure SSMs sacrifice some quality on tasks that benefit from exact long-range recall. Hybrids interleave full-attention layers with fixed-state layers - Mamba-2 in NVIDIA's Nemotron 3 and IBM's Granite 4.0, Gated DeltaNet in Qwen3.5, and Kimi Delta Attention in Kimi K3 and GLM-5.3-Flash - getting most of the quality from the attention layers and most of the memory efficiency from the rest. Only the attention layers produce KV cache entries; the others keep a fixed-size state per sequence.

SSMs eliminate the KV cache and replace it with a fixed-size recurrent state. See KV Cache for why that matters when context grows long.

Vision & multimodal: encoders bolted onto LLMs

Modern vision-language models (VLMs) don't retrain the LLM from scratch with images. The dominant pattern: take a pretrained image encoder, project its outputs into the LLM's token space, and concatenate them with the text sequence. The LLM's weights are largely frozen; only the projector (and optionally the encoder) is fine-tuned. The LLM then “reads” image patches as if they were text tokens.

Representative VLMs

How many tokens is an image, a video, a recording?

Current models cut images into 14-16 px patches, then merge each 2×2 block into one token - so one token covers a 28-32 px square, and the count grows with the image's area. Others cut the image into fixed tiles plus a thumbnail, or squeeze every image into a fixed budget. The same picture can cost ten times more tokens on one model than another.

Video is a stack of frames sampled at a fixed rate, and audio arrives at a steady token rate. Both grow with running time, which is where the numbers get large:

Why video blows up memory

Image, video and audio tokens sit in the KV cache exactly like text, at the same bytes per token. Qwen3-VL-235B caches about 188 KB per token in BF16. One minute of video at 2 frames per second is 43K tokens - about 8 GB of KV. An hour at Gemini's 1 frame per second is about a million tokens, or roughly 200 GB of KV for a single session - more than two H100s hold in total. A hundred users each sharing one minute of video need about 830 GB of KV on top of 470 GB of weights.

Serving adds one more stage: the vision encoder runs once per image or frame during prefill. Engines cache its output by image hash so a repeated image is not re-encoded, and vLLM, SGLang and NVIDIA Dynamo can run the encoder on separate GPUs from prefill and decode (encode-prefill-decode disaggregation). What passes between them is small - about 17 MB of embeddings for a 1080p screenshot - while the KV cache it turns into is about 0.4 GB.

Size it: open the KV planner with 100 users each sharing 10 minutes of video on Qwen3-VL-235B, then change the resolution, frame rate or clip length.

Architecture drives the infrastructure bill

Every architecture choice above maps directly to a serving cost. The views below summarize the key dimensions: how memory scales, how sequence-length scaling behaves, and what that means when you size a cluster.

The numbers: weights vs KV cache, by model

Two separate memory bills dominate serving. Weights are a fixed, one-time cost (loaded once, shared across all requests). KV cache is a per-request, per-token cost that grows with context length and concurrency - and it is where architecture choices show up most dramatically.

Read across the bottom three rows: a dense 405 B model needs ~63 GB of KV cache for a single 128K-token request, while DeepSeek-V3 - a far larger 671 B model - needs only ~9 GB thanks to MLA, and an SSM needs essentially none. KV cache, not weights, is what limits how many long-context users you can pack onto a GPU. Weights assume BF16 (2 bytes/param); KV-cache figures are per single request and scale linearly with batch size.

Qualitative comparison

Parameter counts and active-param ratios are illustrative; exact figures vary by configuration and version. Confirm against model cards and provider documentation before sizing production clusters.