Systems & Infrastructure
Training & Fine-Tuning
IntermediateTraining is how a model is built; fine-tuning is how it's adapted. Both are bounded by the same four resources - data, compute, memory, and time. This page works from the next-token objective up through the C = 6ND scaling law, the memory bill, distributed training, and the fine-tuning recipes that put adaptation on a single GPU.
What training actually is
At its core, training is one objective repeated trillions of times: predict the next token. Feed the model a sequence, it produces a probability distribution over the next token (the forward pass); compare that to the real next token to get a loss; the backward pass computes how every weight should nudge to reduce that loss; the optimizer step applies the nudge. Repeat over trillions of tokens and broad capability emerges from that single, simple game.
01
Forward pass → logits
02
Loss vs. true token
03
Backward pass → grads
04
Optimizer step
Every section below is really about feeding that loop. It has four consumers:
Data
Trillions of tokens of curated text, scored and deduplicated.
Compute
C = 6 × N × D FLOPs - the dominant, unavoidable cost.
Memory
~16 bytes/param of training state, plus activations.
Time
Wall-clock weeks, gated by fleet size, MFU, and failures.
The memory bill: where VRAM goes
Serving a model stores only its weights. Training stores far more. With Adam and mixed precision, each parameter costs roughly 16 bytes: 2 (BF16 weights) + 2 (BF16 gradients) + 4 (FP32 master copy) + 4 (FP32 momentum) + 4 (FP32 variance). On top of that sit activations - the intermediate tensors the backward pass needs - which scale with batch, sequence length, and depth.
Where the VRAM goes during training
Inference only stores weights. Training also keeps gradients, an FP32 master copy, and the optimizer's momentum & variance - about 16 bytes/param before a single activation. ZeRO/FSDP shards those across the fleet to make it fit.
Per-device memory breakdown
Switch Adam → 8-bit Adam to halve the moment bytes, or SGD to drop variance entirely. Then crank ZeRO-3 and watch params/grads/optimizer divide by your GPU count - that is how a 70B trains on a node that could never hold its full 16 B/param state on one card.
The architecture being trained
The thing under training is a stack of transformer decoder blocks - each a self-attention sublayer (relating tokens) followed by an MLP (the bulk of the parameters). Gradients flow back through every block each step.
Dense decoder block
Attention + MLP, every parameter active for every token. Compute and memory both scale with the full parameter count.
Mixture of Experts (MoE)
Many expert MLPs; a router sends each token to a few. DeepSeek-V3 is 671B total but only 37B active per token - cutting FLOPs/token while adding expert-parallel routing.
The key asymmetry: compute scales with active params, but memory and checkpoint size scale with total params. An MoE is cheap to run per token yet heavy to store.
The stages of building a model
A shipped model is the output of a pipeline, not a single run. Pretraining builds raw capability; later stages make it follow instructions, align with preferences, and reason. Each stage has its own dataset, GPU budget, and output artifact.
| Stage | Objective | Outcome | Compute / GPUs | Dataset size | Output artifact | Typical duration | Example |
|---|---|---|---|---|---|---|---|
| Pretraining | Next-token prediction on web-scale corpus | Base model, broad knowledge | 1000s–16K+ GPUs · 1e24–1e25 FLOPs | 1–15T+ tokens | Base weights (405B → ~810 GB BF16) | Weeks–months | Llama 3 405B: 16K H100, 15.6T tok, 54 days |
| Continued / mid-training | Extend context, add domains, anneal high-quality data | Longer-context / domain-shifted base | 100s–1000s GPUs | 10s–100s B tokens | Updated base weights | Days–weeks | DeepSeek-V3 context-extension: 119K H800-hrs |
| SFT / instruction tuning | Imitate curated prompt → response demos | Helpful instruction-follower | 10s–100s GPUs (or 1 w/ LoRA) | 10K–1M+ examples | Instruct weights or LoRA adapter | Hours–days | Llama 3 post-train ~10M samples |
| Preference alignment (RLHF/DPO/RLAIF/GRPO) | Optimize toward human/AI preferences | Aligned, safer outputs | 10s–100s GPUs; RL needs rollout + train GPUs | 10K–1M preference pairs | Aligned weights | Hours–days | DeepSeek-V3 post-train total 5K H800-hrs |
| Reasoning RL (R1-style) | RL on verifiable rewards (math/code) for CoT | Reasoning model | 100s–1000s GPUs, heavy rollout | Verifiable problem sets | Reasoning weights | Days–weeks | DeepSeek-R1: SFT ⇄ GRPO loop |
The C = 6ND formula & Chinchilla scaling
Training compute has a famously simple form: C = 6 × N × D FLOPs, where N is parameters and D is training tokens. The 6 is 2 FLOPs for the forward pass plus 4 for the backward pass per parameter per token.
Compute-optimal (Chinchilla)
For a fixed compute budget, the best loss comes from balancing N and D - roughly D ≈ 20 × N tokens. That is the cheapest way to reach a given quality.
Deliberate over-training
Labs push far past optimal - Llama 3 8B trained on ~15T tokens (≈ 1875 tok/param). It costs more to train but shifts cost away from inference: a smaller model that's cheap to serve forever.
Distributing across GPUs (the training angle)
No single GPU holds a frontier model's training state, so the run is split many ways at once. These nest into 3D/4D parallelism: TP stays intra-node on NVLink, PP spans nodes, DP scales the global batch, and ZeRO/FSDP shards the replicated state.
Data Parallel (DP)
Replicate the model, split the batch. Scales throughput; gradients all-reduced each step. ZeRO/FSDP shards the replicated state.
Tensor Parallel (TP)
Split individual matmuls across GPUs. Chatty - keep it intra-node over NVLink.
Pipeline Parallel (PP)
Split layers into stages across nodes. Micro-batches keep the pipeline full and hide the bubble.
Expert Parallel (EP)
For MoE: place experts on different GPUs and route tokens to them. Adds all-to-all routing traffic.
Context Parallel (CP)
Split the sequence dimension for very long context, sharing attention across GPUs.
ZeRO / FSDP sharding
Shard params, gradients, and optimizer state across data-parallel ranks so no single GPU holds the full 16 B/param state.
The interactive parallelism visualizer lives on the Model Sizing page - here the focus is the resource and throughput impact: every extra dimension buys capacity but spends interconnect bandwidth.
Model Sizing & Parallelism visualizer →How long does it take?
Put it all together. Compute is fixed by C = 6ND; wall-clock is that compute divided by what your fleet actually delivers (GPU count × peak FLOPS × MFU). The default reproduces the real Llama 3 405B run.
How long does a training run take?
Pick a model size, a GPU fleet and precision, then a token budget. The run obeys C = 6 × N × D FLOPs, divided by what your fleet actually delivers (peak × MFU). The default is the Llama 3 405B config - 405B params, 15.6T tokens, ~3.8e25 FLOPs on 16K H100s. This is the ideal compute time; real runs take longer (see below).
Training FLOPs
37.91e24
C = 6 × 405B × 15.60T
Ideal compute time
67.7 days
at 40% MFU · before failures/restarts
GPU-hours
26.62M
fleet-independent compute cost
Dataset
15.60T tok
≈ 62.40 TB on disk (~4 B/tok)
Training-state memory
6.48 TB
16 B/param: 2+2 wts/grads, 4 master, 4+4 Adam
Checkpoint size
6.48 TB
full state · 810.0 GB weights-only
Compute vs. the 1e25 FLOP frontier
Ideal vs. real: this is pure compute time. Meta's actual 405B run took ~54 days because it hit 466 interruptions (one failure every ~3 hours) - yet still kept >90% effective training time through fast checkpoint/restart. Budget real wall-clock above the ideal estimate, not below it.
Try the GB200 preset at FP8 - the same run collapses from weeks to days because per-GPU throughput jumps ~5×. That throughput is only reachable if the storage and interconnect can keep the GPUs fed; a stalled GPU wastes its peak FLOPS.
Checkpoints & async checkpointing
A checkpoint snapshots the full training state - weights, gradients, optimizer moments - at ~16 bytes/param. That is ~1.12 TB for a 70B and ~6.5 TB for a 405B, written every few hundred steps. Naively, every GPU stalls while it writes.
Async / non-blocking checkpointing
Copy state to CPU memory fast, then write to storage in the background while training continues - overlapping the I/O with compute so the GPUs barely pause. This demands enormous, sustained write bandwidth to parallel storage.
Failure recovery is the point
Llama 3's 405B run hit 466 interruptions in 54 days - about one failure every ~3 hours. Without frequent, fast checkpoints, each crash would discard hours of a multi-million-dollar run.
That cadence is why training is a storage problem as much as a compute one - multi-TB writes overlapping a live run need high-throughput parallel storage feeding a fast streaming data pipeline, the kind VAST is built for.
See it in action: Feeding the GPUs & MFU →What a run costs in storage
The memory bill above is per-GPU VRAM during a step. The storage bill is everything the run parks on disk: the dataset it streams from, the weights it ships, and the rolling checkpoints it writes for safety. All three scale linearly with the parameter count, so as models climb from millions to trillions of parameters the footprint marches from gigabytes into petabytes. Drag the model size to see each stage.
Storage footprint across model scale
VRAM is the bottleneck during a step; storage is the bottleneck across the run. As parameters climb from millions to trillions, the dataset, the shipped weights, and the rolling checkpoints all grow linearly with the model - and the total quietly crosses into petabytes. All figures are bytes on disk, not GPU memory.
Storage by stage
140.0B tokens × 4 B (uint32 shards)
7.0B params × 2 B - what you ship
7.0B params × 16 B (weights + grads + optimizer)
rolling safety net against the next crash
A 7.0B-param run at 20× tokens needs 560.0 GB of training data and writes 112.0 GB every checkpoint. Keeping 5 of them alone is 560.0 GB.
Retained ≠ written. A run writes a checkpoint every few hundred steps - hundreds to thousands over a multi-week run - but garbage-collects old ones, keeping a rolling window plus a few permanent milestones. Capacity is set by what you retain; write bandwidth is set by how often you save (the async-checkpoint problem above).
The dataset dominates once you over-train. Chinchilla-optimal is ≈ 20 tokens/param, but Llama 3 8B saw ~15T tokens (≈ 1875×). For reference, HuggingFace's FineWeb is 15T tokens ≈ 44 TB on disk as text; the same count stored as uint32 ids is ~60 TB. Drag toward a trillion parameters and the total crosses into petabytes - which is why frontier training is a high-throughput parallel-storage decision, not just a GPU one.
Fine-tuning in depth
Fine-tuning adapts an existing base instead of training from scratch. A full fine-tune updates every weight (a 7B costs ~100–120 GB of state). LoRA freezes the base and trains a small low-rank adapter (~15–23 GB for a 7B, ~0.1–1% of params, recovering ~90–95% of full quality). QLoRA 4-bit-quantizes the frozen base (<10 GB for a 7B) - enough to fine-tune a 65B on a single 48 GB GPU. Adapters can be swapped or merged per task.
Full fine-tune vs. LoRA vs. QLoRA
A full fine-tune carries the entire 16 B/param training state. LoRA freezes the base and trains a tiny low-rank adapter (~0.1–1% of params). QLoRA goes further - it 4-bit-quantizes the frozen base, so a 65B model fine-tunes on a single 48 GB GPU.
Full fine-tune
Update every weight. Best ceiling, heaviest bill.
- Trainable params
- 100%
- Training VRAM
- 104.3 GB
- Quality recovery
- 100% (reference)
LoRA
Frozen BF16 base + low-rank adapter.
- Trainable params
- 0.24%
- Training VRAM
- 13.3 GB
- Quality recovery
- ~90–95% of full
QLoRA
4-bit NF4 frozen base + adapter.
- Trainable params
- 0.24%
- Training VRAM
- 3.5 GB
- Quality recovery
- ~90–95% of full
LoRA trains 16.8M params (0.24% of the model) yet recovers ~90–95% of full-FT quality. QLoRA's 4-bit base is what drops a 70B from a multi-GPU job to a single 48 GB card - the same trick covered under Quantization & Precision.
Audit, logging & reproducibility
A training run that can't be reproduced or evaluated is a liability. The same discipline that catches a diverging loss curve also proves a model's quality and provenance.
Experiment tracking
Weights & Biases, TensorBoard, MLflow log every loss curve, learning-rate schedule, and hyperparameter so a run is reproducible and comparable.
Eval harnesses
lm-eval-harness and friends score checkpoints on MMLU, GSM8K, HumanEval - turning a falling loss into a defensible quality claim.
Data lineage
Track which corpus, filters, and dedup produced each batch. Provenance matters for licensing, contamination, and debugging.
Determinism & seeds
Fixed seeds, pinned library versions, and deterministic kernels let you reproduce - or bisect - a divergent run.
The tooling stack
No one writes a training loop from scratch at this scale. A mature stack of frameworks handles parallelism, sharding, adapters, and RL rollouts - on top of the interconnect and storage that keep the GPUs busy.
NVIDIA NeMo
End-to-end framework for pretraining and customizing LLMs at scale.
Megatron-LM
NVIDIA's reference for 3D/4D parallelism (TP/PP/DP/EP).
DeepSpeed
ZeRO sharding, offload, and pipeline engine for massive models.
PyTorch FSDP
Native fully-sharded data parallel - the open default.
HF PEFT / TRL
LoRA/QLoRA adapters and RLHF/DPO trainers.
Axolotl
Config-driven fine-tuning wrapper over the HF stack.
vLLM
Fast inference engine - increasingly used for RL rollouts.
Infrastructure
High-throughput parallel storage, InfiniBand / NVLink, and a streaming data pipeline to keep GPUs fed.
Training feeds everything
A training run is a sizing problem, a hardware problem, and a data problem at once. Size the fleet, pick the silicon, and keep the GPUs fed.
Vendor TFLOPS figures vary by sparsity (the dense peaks used here are ~2× lower than sparse marketing numbers) and by precision convention (FP4 vs FP8). Published "training cost" figures usually count only the final run, excluding R&D, failed runs, and prior experiments.