Systems & Infrastructure
Training & Fine-Tuning
IntermediateTraining is how a model is built; fine-tuning is how it's adapted. Both are bounded by the same four resources - data, compute, memory, and time. This page works from the next-token objective up through the C = 6ND scaling law, the memory bill, distributed training, and the fine-tuning recipes that put adaptation on a single GPU.
What training actually is
At its core, training is one objective repeated trillions of times: predict the next token. Feed the model a sequence, it produces a probability distribution over the next token (the forward pass); compare that to the real next token to get a loss; the backward pass computes how every weight should nudge to reduce that loss; the optimizer step applies the nudge. Repeat over trillions of tokens and broad capability emerges from that single, simple game.
One loop, repeated trillions of times
Every training step runs the same four stations. Play the loop or tap a station.
Every section below is really about feeding that loop. It has four consumers:
What the loop consumes
Four resources bound every run. Each tile shows where that cost comes from.
Data
Trillions of tokens of curated text, scored and deduplicated.
Compute
C = 6 × N × D FLOPs - the dominant, unavoidable cost.
Memory
~16 bytes/param of training state, plus activations.
Time
Wall-clock weeks, gated by fleet size, MFU, and failures.
The memory bill: where VRAM goes
Serving a model stores only its weights. Training stores far more. With Adam and mixed precision, each parameter costs roughly 16 bytes: 2 ( weights) + 2 (BF16 gradients) + 4 (FP32 master copy) + 4 (FP32 momentum) + 4 (FP32 variance). On top of that sit activations - the intermediate tensors the backward pass needs - which scale with batch, sequence length, and depth.
Where the VRAM goes during training
Each column is one data-parallel GPU. Inference only stores weights; training also keeps gradients, an FP32 master copy and the optimizer's momentum & variance - about 16 bytes/param before a single activation. Step through the ZeRO stages to watch that state get sliced across the GPUs.
Per-GPU total
106 GB per GPU · 1.3× an 80 GB GPU - shard harder, recompute, or add GPUs
Switch Adam → 8-bit Adam to halve the moment bytes, or SGD to drop variance entirely. Then step to ZeRO-3 and watch params, grads and optimizer divide by your GPU count - that is how a 70B trains on a node that could never hold its full 16 B/param state on one card. Activations are not sharded by ZeRO; recompute is what shrinks them. The activation estimate is a rough rule of thumb.
The architecture being trained
The thing under training is a stack of transformer decoder blocks - each a self-attention sublayer (relating tokens) followed by an (the bulk of the parameters). Gradients flow back through every block each step.
Dense vs. Mixture of Experts - what one token touches
Step through a sentence. The dense block uses all of its MLP for every token; the MoE block's router sends each token to a few experts.
Dense decoder block
MLP - every weight used
Attention + MLP, every parameter active for every token. Compute and memory both scale with the full parameter count.
Mixture of Experts (MoE)
Token The goes to experts 2 and 6.
Many expert MLPs; a router sends each token to a few. DeepSeek-V3 is 671B total but only 37B active per token - cutting FLOPs/token while adding expert-parallel routing.
DeepSeek-V3 parameters, one scale
Eight experts with top-2 routing is a simplification for the picture; real MoE models have many more experts.
The key asymmetry: compute scales with active params, but memory and checkpoint size scale with total params. An is cheap to run per token yet heavy to store.
The stages of building a model
A shipped model is the output of a pipeline, not a single run. Pretraining builds raw capability; later stages make it follow instructions, align with preferences, and reason. Each stage has its own dataset, GPU budget, and output artifact.
Five stages from raw text to a reasoning model
Tap a stage for its objective, budget and output. The charts below put every stage's GPU count and duration on the same log scales.
Pretraining
- Objective
- Next-token prediction on web-scale corpus
- Outcome
- Base model, broad knowledge
- Compute / GPUs
- 1000s–16K+ GPUs · 1e24–1e25 FLOPs
- Dataset size
- 1–15T+ tokens
- Output artifact
- Base weights (405B → ~810 GB BF16)
- Typical duration
- Weeks–months
Example: Llama 3 405B: 16K H100, 15.6T tok, ~3.8e25 FLOPs
GPUs per run
Wall-clock duration
Bars are an order-of-magnitude reading of each stage's range (e.g. “100s-1000s” of GPUs); the open dot is SFT on a single GPU with LoRA.
The C = 6ND formula & Chinchilla scaling
Training compute has a famously simple form: C = 6 × N × D FLOPs, where N is parameters and D is training tokens. The 6 is 2 FLOPs for the forward pass plus 4 for the backward pass per parameter per token.
Tokens per parameter - optimal vs. over-trained
One log ruler of training tokens per model parameter. Tap a marker to read it.
training tokens per parameter (log scale)
Read the gap as compute: with C = 6ND and N fixed, training cost grows with D - so ~1875 tokens/param is ~94× the training compute Chinchilla would pick for an 8B model, paid once to make every later token cheaper to serve.
Distributing across GPUs (the training angle)
No single GPU holds a frontier model's training state, so the run is split many ways at once. These nest into 3D/4D parallelism: TP stays intra-node on , PP spans nodes, DP scales the global batch, and shards the replicated state.
Six ways to cut one training job
Left: the work - a model (layers × width), a batch of sequences, and one long sequence of tokens. Right: what each of four GPUs holds after the cut, and the traffic that cut creates.
The work, cut by DP
What each GPU holds, and who talks to whom
Gradients all-reduced each step
Data Parallel (DP)
What is split: The batch. Every GPU holds a full copy of the model and its training state.
Replicate the model, split the batch. Scales throughput; gradients all-reduced each step. ZeRO/FSDP shards the replicated state.
Schematic: four GPUs and an 8 × 8 model grid stand in for real sizes. Real runs nest these cuts - TP inside a node, PP across nodes, DP over the global batch.
The interactive parallelism visualizer lives on the Model Sizing page - here the focus is the resource and throughput impact: every extra dimension buys capacity but spends interconnect bandwidth.
Model Sizing & Parallelism visualizer →How long does it take?
Put it all together. Compute is fixed by C = 6ND; wall-clock is that compute divided by what your fleet actually delivers (GPU count × peak FLOPS × MFU). The default uses the real Llama 3 405B configuration.
How long does a training run take?
Pick a model size, a GPU fleet and precision, then a token budget. The run obeys C = 6 × N × D FLOPs, divided by what your fleet actually delivers (peak × MFU). The default is the Llama 3 405B config - 405B params, 15.6T tokens, ~3.8e25 FLOPs on 16K H100s. The result is the ideal compute time; real runs take longer (see below).
1 · Compute: C = 6 × N × D
3.79e25 FLOPs
Dashed diagonals are equal-compute lines (FLOPs); the dot is your run, the box its N × D. Violet = Chinchilla-optimal D ≈ 20N.
2 · ÷ What the fleet delivers
6.5 EFLOP/s
16,384 × H100 BF16 989 TFLOPS = 16.2 EFLOP/s peak
3 · = Ideal wall-clock time
67.7 days
Log time axis. Faded tail: real runs land later (interruptions, restarts).
GPU-hours
26.62M
fleet-independent compute cost
Dataset
15.60T tok
≈ 62.40 TB on disk (~4 B/tok)
Training-state memory
6.48 TB
16 B/param: 2+2 wts/grads, 4 master, 4+4 Adam
Checkpoint size
6.48 TB
full state · 810.0 GB weights-only
Ideal vs. real: 54 days of the Llama 3 405B run
466 interruptions · about one every ~3 h
During a 54-day window of Meta's 405B pretraining the job was interrupted 466 times - yet kept >90% effective training time through fast checkpoint/restart. Budget real wall-clock above the ideal estimate, not below it. (Ticks evenly spaced for illustration.)
Try the GB200 preset at FP8 - the same run collapses from weeks to days because per-GPU throughput jumps ~5×. That throughput is only reachable if the storage and interconnect can keep the GPUs fed; a stalled GPU wastes its peak FLOPS.
Checkpoints & async checkpointing
A checkpoint snapshots the full training state - weights, gradients, optimizer moments - at ~16 bytes/param. That is ~1.12 TB for a 70B and ~6.5 TB for a 405B, written every few hundred steps. Naively, every GPU stalls while it writes.
Synchronous vs asynchronous checkpoints - same clock
Two checkpoints in the same window. Watch what the GPUs do while each one is written. Durations are illustrative units, not measurements.
At time 0: synchronous has stalled the GPUs for 0 units; asynchronous has paused them for 0.
Synchronous checkpoint
The naive approach: every GPU stalls while the full training state is written out, and the run only resumes once it has landed on storage.
Asynchronous checkpoint
Async / non-blocking checkpointing: copy state to CPU memory fast, then write to storage in the background while training continues - overlapping the I/O with compute so the GPUs barely pause. This demands enormous, sustained write bandwidth to parallel storage.
466 interruptions in 54 days - what each one costs
Llama 3's 405B run averaged a failure about every ~3 hours, and each one rolls the job back to its last checkpoint. Pick a checkpoint cadence to see how much work gets thrown away.
The whole run · 54 days
Zoom: one day of the run (illustrative)
Work thrown away over 54 days
~117 h
≈ 9.0% of the run's wall-clock, redone by every GPU in the job
Without frequent, fast checkpoints, each crash would discard hours of a multi-million-dollar run. Checkpointing often only pays if each write is cheap - which is what async checkpointing and fast parallel storage buy.
Rough estimate: each failure loses half a checkpoint interval on average (466 × interval ÷ 2); restart time and the checkpoint writes themselves are not counted.
That cadence is why training is a storage problem as much as a compute one - multi-TB writes overlapping a live run need high-throughput parallel storage feeding a fast streaming data pipeline, the kind VAST is built for.
See it in action: Feeding the GPUs & MFU →What a run costs in storage
The memory bill above is per-GPU VRAM during a step. The storage bill is everything the run parks on disk: the dataset it streams from, the weights it ships, and the rolling checkpoints it writes for safety. All three scale linearly with the parameter count, so as models climb from millions to trillions of parameters the footprint marches from gigabytes into petabytes. Drag the model size to see each stage.
Storage footprint across model scale
VRAM is the bottleneck during a step; storage is the bottleneck across the run. As parameters climb from millions to trillions, the dataset, the shipped weights, and the rolling checkpoints all grow linearly with the model - and the total quietly crosses into petabytes. All figures are bytes on disk, not GPU memory.
Storage across model scale (log-log)
drag on the chart to move the model size
At 7.0B params · share of the total
140.0B tokens × 4 B (uint32 shards)
7.0B params × 2 B - what you ship
7.0B params × 16 B (weights + grads + optimizer) - counted inside the retained set, not added again
rolling safety net against the next crash
A 7.0B-param run at 20× tokens needs 560.0 GB of training data and writes 112.0 GB every checkpoint. Keeping 5 of them alone is 560.0 GB.
Retained ≠ written. A run writes a checkpoint every few hundred steps - hundreds to thousands over a multi-week run - but garbage-collects old ones, keeping a rolling window plus a few permanent milestones. Capacity is set by what you retain; write bandwidth is set by how often you save (the async-checkpoint problem above).
The dataset dominates once you over-train. Chinchilla-optimal is ≈ 20 tokens/param, but Llama 3 8B saw ~15T tokens (≈ 1875×). For reference, HuggingFace's FineWeb is 15T tokens ≈ 44 TB on disk as text; the same count stored as uint32 ids is ~60 TB. Drag toward a trillion parameters and the total crosses into petabytes - which is why frontier training is a high-throughput parallel-storage decision, not just a GPU one.
Fine-tuning in depth
Fine-tuning adapts an existing base instead of training from scratch. A full fine-tune updates every weight (a 7B costs ~100–120 GB of state). freezes the base and trains a small low-rank adapter (~15–23 GB for a 7B, ~0.1–1% of params, recovering ~90–95% of full quality). 4-bit-quantizes the frozen base (<10 GB for a 7B) - enough to fine-tune a 65B on a single 48 GB GPU. Adapters can be swapped or merged per task.
Full fine-tune vs. LoRA vs. QLoRA
A full fine-tune carries the entire 16 B/param training state. LoRA freezes the base and trains a tiny low-rank adapter (~0.1–1% of params). QLoRA goes further - it 4-bit-quantizes the frozen base, so a 65B model fine-tunes on a single 48 GB GPU.
Does it fit on one GPU?
What LoRA actually trains
per module: 2 × d × r = 2 × 4,096 × 16 = 131K
× 4 modules × 32 layers = 16.8M trainable (0.24%)
Not to scale: r is drawn far wider than life (r = 16 vs d = 4,096) so the adapter stays visible.
| Method | Trainable | VRAM | Quality recovery | Runs on |
|---|---|---|---|---|
| Full FT | 100% | 104.3 GB | 100% (reference) | Multi-GPU (>80 GB) |
| LoRA | 0.24% | 13.3 GB | ~90–95% of full | Consumer 24 GB (RTX 4090) |
| QLoRA | 0.24% | 3.5 GB | ~90–95% of full | Consumer 24 GB (RTX 4090) |
LoRA trains 16.8M params (0.24% of the model) yet recovers ~90–95% of full-FT quality. QLoRA's 4-bit base is what drops a 70B from a multi-GPU job to a single 48 GB card - the same trick covered under Quantization & Precision. Columns count model state only; activations come on top.
Audit, logging & reproducibility
A training run that can't be reproduced or evaluated is a liability. The same discipline that catches a diverging loss curve also proves a model's quality and provenance.
Four practices, each guarding part of the run
The run flows top to bottom; each lane shows the stretch a practice covers. Tap a lane or a name to read it.
- Raw corpus
- Filters & dedup
- Training batches
- Training run
- Checkpoints
- Evaluation
- Quality claim
Data lineage
covers raw corpus → training batches
Track which corpus, filters, and dedup produced each batch. Provenance matters for licensing, contamination, and debugging.
The tooling stack
No one writes a training loop from scratch at this scale. A mature stack of frameworks handles parallelism, sharding, adapters, and RL rollouts - on top of the interconnect and storage that keep the GPUs busy.
The training stack, layer by layer
Adaptation tools sit on frameworks that train at scale, which sit on the infrastructure that keeps GPUs busy. Tap any block.
Adapt & align
adapters, preference tuning, RL rollouts
Train at scale
parallelism and sharding
Infrastructure
what keeps the GPUs fed
NVIDIA NeMo
End-to-end framework for pretraining and customizing LLMs at scale.
Training feeds everything
A training run is a sizing problem, a hardware problem, and a data problem at once. Size the fleet, pick the silicon, and keep the GPUs fed.
Vendor TFLOPS figures vary by sparsity (the dense peaks used here are ~2× lower than sparse marketing numbers) and by precision convention (FP4 vs FP8). Published "training cost" figures usually count only the final run, excluding R&D, failed runs, and prior experiments.