NVIDIA Infrastructure

Many jobs, one fabric

Almost everything in AI infrastructure exists to keep expensive GPUs busy. But “keeping them busy” means different things for different workloads - and the right hardware choice follows the workload, not the other way around. These are the major classes you will be sizing for:

The hardware shown is a representative sweet spot, not a limit - any modern data-center GPU can serve inference; the right one depends on model size, context length, and your latency and cost budget.

The GPU lineup

Three generations are in play today: Hopper (H100, H200), Blackwell (B200, GB200) and Blackwell Ultra (B300, GB300), with Ada Lovelace (L40S) and the RTX PRO 6000 covering inference, graphics, and simulation. Each generation moves the same three levers: more memory, more bandwidth, and faster links between GPUs.

The roadmap ahead

NVIDIA now ships a new data-center architecture roughly every year, so sizing a multi-year build means knowing what's next. Three things define the next two years: memory moves to HBM4, the rack becomes the unit of compute, and inference begins to disaggregate its prefill and decode phases across the fleet. Click an era to see what changes.

Reading a spec sheet

Four numbers tell you most of what you need to know about a data-center GPU:

Compute throughput

Peak tensor-core throughput per GPU, by precision. Lower precision trades numerical range for speed: each step down (FP16 → FP8 → FP4) roughly doubles the rate, which is why inference has marched toward FP4 on Blackwell.

Representative dense peak figures (PF = PFLOPS, TF = TFLOPS); structured sparsity roughly doubles them. These are per-chip numbers - GB200 / GB300 rows are per Grace-Blackwell superchip (two GPUs), and their dies clock higher than the air-cooled HGX boards. A full NVL72 rack multiplies these across 72 GPUs (e.g. ~1.44 EF FP4 sparse for GB200 NVL72), so the per-rack totals in the rack diagram below are far larger and do not contradict these figures. Confirm against current NVIDIA datasheets before sizing.

Nodes and racks

GPUs ship in standardized building blocks. What matters for sizing is how many GPUs share one fast memory domain - and how much HBM that adds up to, because that pooled memory is what a single large model or KV cache can spread across.

How you actually buy it

The same GPUs reach customers through very different consumption models - from a bare baseboard an OEM builds a server around, to a turnkey validated cluster, to rented capacity in a neocloud. The choice is usually about who owns the integration risk and the CapEx, not the silicon - and a reference architecture is the thread that de-risks every path.

The NVL72 rack, up close

The flagship building block deserves a closer look. An NVL72 rack groups its 9 NVLink switch trays in the center, flanked by two banks of compute trays, with power shelves at top and bottom - a copper spine fuses all 72 GPUs into one all-to-all NVLink-5 domain. The GB200 and GB300 builds share this rack; GB300 raises HBM, FP4, and power.

The power wall - and why racks went liquid

A single NVL72 rack draws 120–142 kW - five to six times what an air-cooled rack can dissipate. That's why Blackwell racks ship direct-to-chip liquid-cooled by default, and why per-GPU power keeps climbing every generation. Toggle the views to see both walls - and where the roadmap (600 kW, then ~1 MW racks) is heading.

Power is the new constraint

Once power - not chips - caps how much you can deploy, the metric that matters shifts from FLOPS to tokens per watt and cost per token. On those, each generation is a step change: NVIDIA reports Blackwell delivering far more tokens per megawatt and a fraction of Hopper's cost per token. Drag the slider to see how that reshapes the bill at scale.

From one GPU to a SuperPOD

A frontier model never fits on one GPU. NVIDIA's answer is a hierarchy of ever-larger “single” computers, each tier glued together by a faster network than the last.

Scale up vs scale out

That hierarchy splits into two moves. Scaling up grows the NVLink domain into one bigger GPU; scaling out stitches racks together over InfiniBand or Ethernet. Zoom from a single node to a SuperPOD and watch where the fat NVLink links give way to the thin network - and where storage bypasses the CPU entirely.

Six paradigms driving the build-out

Above the technical workloads sits a market-level view: the kinds of AI demand pulling infrastructure into data centers. The first two are two faces of the same plant - an AI factory manufactures intelligence (raw materials: data + electricity; product: a model), and the token factory turns that model into tokens, where the currency is tokens per watt. (NVIDIA often uses “AI factory” for the whole operation, training and inference alike - the split below is about emphasis.)

The flagship hardware story across these: GB200 and GB300 NVL72 racks, where 72 GPUs act as one accelerator and FP4 maximizes tokens per watt for large-scale reasoning inference.

The real moat: CUDA and the full stack

A competitor can match a chip far sooner than it can match twenty years of CUDA. The durable lock-in is the software stack - the CUDA platform, the sprawling CUDA-X libraries (cuDNN, cuBLAS, cuDF, cuML, cuVS, cuGraph, NCCL and hundreds more), and the frameworks and microservices built on top. Explore the layers to see why the ecosystem, not the silicon, is the hard thing to leave.

The serving and training layers (NIM, NeMo, Dynamo, TensorRT-LLM) get their own treatment on the inference & serving and training & fine-tuning pages.

The interconnect

At scale, the network is the computer. How fast GPUs exchange gradients and activations decides whether a 10,000-GPU cluster behaves like 10,000 GPUs or a fraction of that.

DPUs and SuperNICs - keeping GPUs fed

Two specialized NICs sit in every Blackwell tray. The ConnectX-8 SuperNIC moves GPU-to-GPU traffic at up to 800 Gb/s; the BlueField-3 DPU offloads networking, storage, and security off the host CPU so its cores aren't stolen from the work that feeds the GPUs. Toggle the node to see the difference. The next generation, BlueField-4, doubles networking to 800 Gb/s via ConnectX-9 and adds a new headline role - offloading KV-cache “context memory” to a petabyte NVMe tier.

The bandwidth cliff

Those interconnects aren't interchangeable - they span more than two orders of magnitude in bandwidth. Plotting them on a log scale shows why the topology is built the way it is: stay on NVLink and you move terabytes per second; cross InfiniBand and you fall off a cliff.

NVIDIA networking: Spectrum-X

Spectrum-X is NVIDIA's Ethernet platform for AI - Spectrum-4 switch silicon paired with BlueField/ConnectX SuperNICs, adding adaptive routing and congestion control that standard Ethernet lacks. Two switches anchor the 400G and 800G tiers:

Try it yourself

Two calculators make the trade-offs concrete. The first asks whether a job fits one NVLink domain; the second shows how long a transfer takes across every tier - and why the storage and network path, not raw compute, usually decides utilization.

GPUDirect Storage - skipping the CPU

The last mile of keeping GPUs busy is the path from storage into HBM. GPUDirect Storage lets the NIC DMA data straight into GPU memory, bypassing the CPU bounce buffer - higher bandwidth, lower latency, and CPU cores freed for real work. Flip between the two paths to see why it matters for utilization.