Memory & Efficiency
Model Sizing & Parallelism
IntermediateA model's parameter count is only the start. This page breaks down where GPU memory actually goes, then shows how the six parallelism strategies split a model across hundreds or thousands of GPUs - with the math, the interconnect limits, and the real recipes frontier labs use.
Where the memory goes
Inference pays for weights and the KV cache. Training pays for far more: every weight needs a gradient and an optimizer state, and Adam's states alone are ~6× the weights. A 70B model that serves in 140 GB needs over 1.1 TB of state to train - which is the whole reason parallelism exists. Training & Fine-Tuning breaks this bill down interactively; here the point is what gets sharded and why.
Model weights
params × bytes/param70B × 2 B (BF16) = 140 GB
The parameters themselves. Fixed by model size and precision - the one term you pay for in both training and inference.
Gradients
≈ params × bytes/param70B × 2 B = 140 GB
One gradient per weight, same precision. Training only - inference never allocates this.
Optimizer states
≈ 12 bytes/param (Adam)70B × 12 B = 840 GB
Adam keeps FP32 momentum + variance + a master copy. The biggest single term in training - and what ZeRO/FSDP shards first.
Activations
∝ batch × seq × hidden × layersVaries; cut by recompute
Intermediate tensors kept for the backward pass. Grows with batch and context; activation checkpointing trades compute to shrink it.
Six ways to split a model
When a model - or its training state - is too big for one GPU, you split it across a node of 8 GPUs (and beyond). Each strategy cuts along a different axis and pays a different communication cost. Pick one to see exactly what lands on each of the 8 GPUs, how they talk to each other, and when you'd reach for it.
Parallelism explorer - what gets split across 8 GPUs
Each strategy cuts the workload along a different axis. Pick one to see what lands on each GPU, how they communicate, and when you'd reach for it.
8-GPU node
GPU 0
Weights 1/8
every layer, sliced
GPU 1
Weights 2/8
every layer, sliced
GPU 2
Weights 3/8
every layer, sliced
GPU 3
Weights 4/8
every layer, sliced
GPU 4
Weights 5/8
every layer, sliced
GPU 5
Weights 6/8
every layer, sliced
GPU 6
Weights 7/8
every layer, sliced
GPU 7
Weights 8/8
every layer, sliced
What gets split
Individual weight matrices. Each layer's matmul is sliced column/row-wise across GPUs and recombined.
Sync frequency
Twice per layer, every forward & backward
Collective
All-reduce (activations)
Interconnect needs
Bandwidth-hungry - must stay inside one NVLink domain. Degree is usually capped at 8 (one node).
When to use TP
When a single layer is too big for one GPU, or to cut latency. The innermost axis.
In the wild
Megatron-LM standard; Llama 3 405B used TP=8 within each node.
TP - Tensor Parallelism
Make it concrete - split real model weights
Before any clever strategy, parallelism's first job is simply holding the weights. Pick a real model and spread it across 8 GPUs (or 1, 2, 4, 16) to see how much HBM each device carries - and how many GPUs you need just to fit the parameters, before the KV cache and activations are added.
Weights-split playground - fit a real model across GPUs
Pick a model, a GPU count, and an HBM size. Tensor / pipeline / FSDP parallelism all do the same first job: divide the weights so each GPU holds only its share.
Model
GPUs
GPU model
Total weights
140 GB
BF16 · 70B params
Per GPU
17.5 GB
÷ 8 GPUs
Fits
12% of 141 GB
weights only - room for KV
8 × H200 (141 GB) - each holds 17.5 GB of weights
GPU 0
GPU 1
GPU 2
GPU 3
GPU 4
GPU 5
GPU 6
GPU 7
It fits. Splitting 140 GB across 8 GPUs leaves 17.5 GB per device - under the 141 GB budget, with headroom for activations and the KV cache. Spreading weights this way is exactly what tensor, pipeline, and FSDP parallelism do.
The bandwidth wall: why TP stays in one node
Tensor parallelism is the most communication-hungry strategy: it all-reduces activations twice per layer, on every forward and backward pass. That only works when GPUs share an NVLink domain. The moment you cross to InfiniBand or Ethernet - 18–144× slower - the GPUs stall waiting on each other. Watch the matmul split, then see the bandwidth cliff.
Split the matmul - tensor parallelism, visually
A layer's big weight matrix is sliced into column shards, one per GPU. Each GPU multiplies its slice, then an all-reduce stitches the partial outputs back together - every layer, every step.
Full matrix
8192 × 28,672
Cols / GPU
7,168
Params / GPU
59M
All-reduce / layer
2×
Why TP stays inside one node
TP's all-reduce runs twice per layer - for an 80-layer model that's 160 collectives per forward pass. At ~18–36× less bandwidth than NVLink, inter-node links would stall the GPUs waiting on each other. That's why TP degree is almost always capped at 8 (one NVLink node), while pipeline and data parallelism - which communicate far less - span nodes.
Real recipes combine them
Frontier runs never use one strategy - they nest several into a 3D (or 4D) mesh: TP innermost in the node, PP across nodes, DP as the outer loop, plus EP or CP when the model calls for it. Here's what that looks like in practice.
Hybrid parallelism - how the axes nest
Frontier runs never use one strategy. They nest several into a single mesh - from the GPUs inside a node out to the replicas across the cluster. Pick a real recipe to see how the axes stack.
1 data-parallel replica (of 128)
Pipeline stage 1
TP 1/8
TP 2/8
TP 3/8
TP 4/8
TP 5/8
TP 6/8
TP 7/8
TP 8/8
Pipeline stage 2
TP 1/8
TP 2/8
TP 3/8
TP 4/8
TP 5/8
TP 6/8
TP 7/8
TP 8/8
Pipeline stage 3
TP 1/8
TP 2/8
TP 3/8
TP 4/8
TP 5/8
TP 6/8
TP 7/8
TP 8/8
⋮ 16 pipeline stages total
Llama 3.1 405B · ~16,000 H100
The textbook 4D layout: TP=8 inside each NVLink node, PP=16 across nodes, CP=16 to reach 128K context, DP=128 as the outer loop. The axes nest from the GPU outward.
Diagrams show a representative slice - a single replica with its first few stages - not every GPU. Degrees are from published tech reports; they illustrate the nesting pattern, not exact reproductions.
| Model / job | Parallelism recipe | Fleet | Why |
|---|---|---|---|
| Llama 3.1 405B | TP=8 · PP=16 · CP=16 · DP=128 | ~16,000 H100 | TP inside each NVLink node, PP across nodes, CP for 128K context, DP as the outer loop. The textbook 4D layout. |
| DeepSeek-V3 / R1 671B | PP=16 · EP=64 · ZeRO-1 · DualPipe | ~2,048 H800 | MoE means expert parallelism dominates; DualPipe overlaps the pipeline bubble with computation to claw back utilisation. |
| 70B full fine-tune | FSDP / ZeRO-3 | 8–64 GPUs | Shard everything (params + grads + optimizer) so a model that needs ~1.1 TB of state fits a modest fleet. No TP complexity. |
| 7B–13B pretrain | DP only | 8–512 GPUs | Fits one GPU, so just replicate and shard the batch. The simplest case - no model-splitting needed at all. |
Fleet sizes and recipes are from published tech reports and papers; GPT-4's exact layout is unconfirmed. Numbers illustrate the scaling pattern, not exact reproductions.
Which ones you actually use
Not every strategy shows up in every run. Two are nearly universal, two are standard once you outgrow a single GPU, and two are reserved for specific model shapes.
Almost always present
Cheap to communicate (DP) or essential for memory (ZeRO). The default outer loop of nearly every training run.
Common at scale
Needed once a model no longer fits one GPU. TP within a node, PP across - the standard pairing for 70B+ dense models.
Situational
EP only matters for MoE models; CP only for very long context. Powerful but narrow - you reach for them when the workload demands it.
Memory is the bottleneck - everywhere
Parallelism splits the model across GPUs; quantization shrinks what each GPU has to hold; the KV cache is the memory that grows with every token served. They're three views of the same constraint - HBM is scarce and bandwidth is finite.