Systems & Infrastructure

Inference & Serving

Intermediate

Serving a model is a different problem from training one. Inference splits into two phases with opposite bottlenecks, and the whole stack - batching, parallelism, KV-cache management, routing, and disaggregation - exists to squeeze the most goodput out of expensive GPUs while holding latency SLOs. This page builds from the two phases up to datacenter-scale disaggregated serving.

What happens at inference time

LLMs generate autoregressively - one token at a time, each conditioned on everything before it. That work splits into two phases with opposite bottlenecks. Prefill processes the entire prompt in one parallel pass, populating the KV cache; it's compute-bound and governs time-to-first-token (TTFT). Decode then emits one token at a time, each step reading the whole KV cache back from HBM; it's memory-bandwidth-bound and governs time-per-output-token (TPOT / inter-token latency).

Prefill vs decode - two phases, two bottlenecks

Both phases run the same weights, but they sit on opposite sides of the GPU's roofline. Prefill crunches all P·B prompt tokens in one big matmul - lots of math per weight read → usually compute-bound (sets TTFT). Decode makes one token per sequence per step, re-reading the whole model + a growing KV cache for almost no math → memory-bound, leaving Tensor Cores idle (sets TPOT). The two cards below show which resource is the wall in each phase.

Prompt length Ptokens processed in prefill2,048

Generation length Gtokens produced in decode512

Batch Bconcurrent sequences8

← memory-boundarithmetic intensity · FLOP/byte (log)compute-bound →

ridge 295

decode

prefill

Prefill sits at 16.1k FLOP/byte; decode at 7.8. Raise the batch and watch decode crawl toward the ridge - that is exactly why serving engines batch decode aggressively, and why a fat KV cache (which adds bytes per step) drags it back left.

Prefill

compute-bound

Tensor Cores (compute)100% · saturated

HBM bandwidth (memory)2% · idle

One read of the weights drives 16,384 prompt tokens of math (16.1k FLOP/byte, well past the 295 ridge). Compute is the wall - this is what sets time-to-first-token.

Decode

memory-bound

Tensor Cores (compute)3% · idle

HBM bandwidth (memory)100% · saturated

Each step re-reads the model + a 2,304-token KV cache to make just 8 tokens - so Tensor Cores sit 97% idle and memory bandwidth is the wall. That is what sets time-per-output-token.

TTFT ≈ prefill

2.32 s

time to first token

TPOT ≈ decode step

42.7 ms

time per output token

End-to-end latency

24.17 s

prefill + 512 decode steps

With a long generation, decode dominates end-to-end time - every token re-streams the model and a growing KV cache from HBM. This is why decode is memory-bandwidth-bound and why KV-cache size directly throttles throughput. Formulas are illustrative first-order estimates (70B-class model on an H100-class GPU), not a production latency model.

The KV cache, and why it dominates memory

To avoid recomputing attention over the whole sequence each step, the model caches the key and value vectors for every token, at every layer. That cache grows linearly with context and batch:

KV bytes = 2 × layers × kv_heads × head_dim × bytes × seq_len × batch

At long context and high concurrency the KV cache rivals or exceeds the weights, and since decode re-reads it every step it directly throttles throughput. The levers that shrink it: GQA/MLA cut kv_heads, and FP8 KV halves bytes. See KV Cache for the full treatment and Quantization for FP8/INT4.

Serving across GPUs & nodes

A single request on one GPU is the easy case. Real deployments fan many concurrent requests across multiple GPUs and nodes, and the parallelism strategy you pick is dictated by how much they have to talk to each other - which interconnect can carry that traffic.

Tensor Parallel (TP)

Splits every layer's matrices across GPUs. Each forward pass needs frequent all-reduce - keep it intra-node over NVLink (~900 GB/s on H100, ~1.8 TB/s on Blackwell).

Pipeline Parallel (PP)

Splits the model by layers into stages. Only activations cross stage boundaries, so bandwidth demand is low - it tolerates inter-node InfiniBand.

Expert Parallel (EP)

For MoE models: experts are spread across GPUs and tokens are routed to them via all-to-all communication. Scales sparse models past a single node.

Rule of thumb: TP stays inside a node on NVLink; PP spans nodes over InfiniBand; EP appears once you serve MoE. They compose.

Batching keeps the GPU busy

Because decode is bandwidth-bound, the way to use a GPU well is to run many sequences at once. Batching strategy is what turns idle silicon into throughput - and it has evolved a long way past the naive loop.

Static batching

Wait to collect a fixed batch, run it to completion, then start the next. Simple but the whole batch stalls on its slowest sequence - GPU idles.

↓

Continuous / in-flight batching

Schedule at the iteration level: finished sequences leave and new ones join every step, keeping the GPU saturated. 3–5× throughput over a naive loop.

↓

Chunked prefill

Break a long prompt's prefill into chunks and interleave them with ongoing decode steps, so a big prompt doesn't freeze everyone else's tokens - smooths TTFT spikes.

Serving engines

A handful of inference engines dominate production. They trade off ease of use, peak performance, hardware breadth, and how aggressively they manage the KV cache.

Engine	Maker	License	Best at	Throughput	Latency/TTFT	Ease	HW	KV features	Quant	Multi-node	Disagg-PD
vLLM	vLLM project / community	Apache 2.0	general-purpose, widest models, easy start	3–5× naive	good	high (no compile)	NVIDIA/AMD/others	PagedAttention, prefix cache	FP8/INT4/AWQ/GPTQ	yes (TP/PP)	yes (w/ LMCache/Dynamo)
TensorRT-LLM	NVIDIA	Apache 2.0	max perf on NVIDIA	highest on NVIDIA	best (compiled)	low (engine build)	NVIDIA only	paged KV, in-flight	FP8/INT4/FP4	yes (TP/PP/EP)	yes
SGLang	LMSYS / SGLang	Apache 2.0	shared-prefix / RAG / agents	up to 6.4× on shared prefixes	best on cached prefixes	medium	NVIDIA/AMD	RadixAttention prefix tree	FP8/INT4	yes	yes
NVIDIA Dynamo	NVIDIA	Apache 2.0	datacenter-scale orchestration (above vLLM/TRT-LLM/SGLang)	7×/GPU (reported)	2× TTFT via KV-routing (reported)	medium	NVIDIA-centric	KVBM offload, KV-aware routing	delegates to engine	yes (core)	yes (native)
HF TGI (legacy)	Hugging Face	Apache 2.0	maintenance mode (Mar 2026)	lower; deprecated	moderate	was high	NVIDIA/others	basic	limited	limited	no
LMDeploy	Shanghai AI Lab	Apache 2.0	quantized + long-context on NVIDIA	high	good	medium	NVIDIA	paged KV, prefix cache	strong (AWQ/INT4/FP8)	yes	partial

Vendor/benchmark multipliers are reported on specific hardware and workloads, not universal. HF TGI is in archived/maintenance mode as of March 2026 - shown here as legacy.

Signature techniques per engine

Each engine is known for one or two core ideas. Knowing them tells you which workload each is built for.

vLLM

PagedAttention + continuous batching

Stores KV in fixed-size blocks like OS pages, eliminating fragmentation and enabling near-100% memory use; continuous batching keeps the GPU busy every iteration.

SGLang

RadixAttention

Keeps a radix tree of cached prefixes so any shared prompt prefix is reused across requests - up to 6.4× on workloads with heavy prefix sharing (RAG, agents, few-shot).

TensorRT-LLM

in-flight batching + compiled kernels

Builds a fused, hardware-specific engine ahead of time and schedules requests in-flight, squeezing the most out of NVIDIA Tensor Cores at the cost of a compile step.

NVIDIA Dynamo

disaggregated serving + KV-aware routing + KVBM offload

Sits above the engines: separates prefill/decode pools, routes by KV locality, and offloads KV down a memory hierarchy (KVBM) to serve at datacenter scale.

How engines manage KV cache

Since the KV cache is the memory bottleneck, the engines compete on how cleverly they store and reuse it. Three ideas show up everywhere.

PagedAttention

Treats KV cache as non-contiguous fixed-size blocks, allocated on demand like OS virtual memory pages. Kills fragmentation, so far more sequences fit in HBM.

Prefix / prompt caching

Persist the KV for a common prompt prefix (a system prompt, a document) and reuse it across requests instead of re-prefilling it every time.

RadixAttention prefix sharing

Generalizes prompt caching with a radix tree of prefixes, so partially-overlapping prompts share whatever KV they have in common automatically.

KV cache acceleration projects

Beyond the engines, a research-driven layer of projects pushes KV reuse and offload further - especially for long context and RAG.

LMCache

Adds: Multi-tier KV layer: offloads KV to CPU DRAM / disk and shares it across instances.

Helps when: Long contexts and repeated prefixes that overflow HBM - 3–10× latency cut with vLLM.

Mooncake

Adds: Kimi/Moonshot's KVCache-centric disaggregated architecture (separate KV pool).

Helps when: High-QPS production serving - up to 525% throughput in simulation under SLOs, +75% real requests.

CacheBlend

Adds: Reuses NON-prefix KV chunks for RAG by recomputing only a small subset of tokens.

Helps when: RAG where reused chunks aren't at the prompt start - TTFT −2.2–3.3×, throughput +2.8–5× (EuroSys'25 best paper).

Dynamo KVBM

Adds: KV block manager that tiers GPU → CPU → SSD → remote, moving blocks over NIXL.

Helps when: Serving more concurrent/long sessions than HBM alone allows, at datacenter scale.

Reported multipliers come from specific hardware/workloads, not universal.

Disaggregated prefill–decode

Prefill and decode have opposite bottlenecks, so co-locating them on the same GPUs forces a TTFT-vs-TPOT tradeoff and leaves resources idle - a burst of prefill stalls everyone's decode. Disaggregation puts them in separate pools connected by a fast KV-transfer path, so each pool can be scaled and tuned independently for higher goodput under SLOs.

Disaggregated prefill–decode serving

Requests hit a KV-aware router, get their prompt processed in a compute-heavy prefill pool, then the KV cache is shipped over a fast transfer layer to a bandwidth-heavy decode pool that streams tokens. Prefill pegs the Tensor Cores; decode pegs memory bandwidth - splitting them lets each pool run flat-out and scale independently.

Requests

req A

req B

req C

→

KV-aware Router

routes by prefix locality & pool load; assigns prefill then decode

→

Prefill pool

compute-bound · bursty

Compute~95%

HBM bandwidth~10%

KV transfer (NIXL)

~tens of GB/s of KV cache shipped per step

Decode pool

bandwidth-bound · steady

Compute~10%

HBM bandwidth~95%

→

Tokens out

Themodelstreamsonetoken…

Compute (Tensor Core) utilizationHBM-bandwidth utilizationNote the inversion: prefill maxes compute & barely touches memory; decode does the opposite.

Co-located (interference)

prefill bursts stall decode → TTFT-vs-TPOT tug-of-war, idle resources

Disaggregated

DistServe up to 7.4× more requests · Splitwise 2.35× at equal cost

Utilization percentages are illustrative, not measured. Vendor/benchmark multipliers are reported on specific hardware and workloads, not universal.

The reported wins are large: DistServe up to 7.4× more requests / 12.6× tighter SLO (OSDI'24); Splitwise 2.35× at equal cost; Mooncake +75% real requests. But it isn't free: the KV cache must cross the network between pools. Disaggregation pays off at high QPS with multiple replicas, and is overkill at low QPS / single replica, where the transfer overhead dominates the gain.

Disaggregated goodput calculator

Goodput = requests/sec that meet both their TTFT and TPOT SLOs - not just requests served. Both modes own the same GPUs and the same raw capacity (22/s here); the difference is how much of it stays SLO-compliant. The headline is sustainable goodput: the highest load each mode can carry without breaking an SLO.

Offered load λrequests/sec (drives the latency readout)9/s

Total GPUssame hardware budget for both modes12

Prompt lengthdrives prefill cost / TTFT3,072

Output lengthdrives decode steps / TPOT256

TTFT SLOtime-to-first-token, ms1000 ms

TPOT SLOtime-per-output-token, ms80 ms

Co-located · sustainable

6.8 req/s

prefill & decode share every GPU · interference caps SLO-safe load

Disaggregated · sustainable

12.1 req/s

9.4 prefill / 2.6 decode GPUs · each pool meets its own SLO

SLO-safe throughput gain

1.8×

more SLO-compliant requests on the same GPUs

Co-located latency under load

TTFT1140 ms ✗ (≤ 1000)

TPOT94 ms ✗ (≤ 80)

prefill bursts delay first token → TTFT 1140 ms > 1000 ms SLO

Disaggregated latency under load

TTFT747 ms ✓ (≤ 1000)

TPOT63 ms ✓ (≤ 80)

Each pool is batched & queued for its own SLO, so prefill bursts never stall decode tokens.

Goodput vs offered load

- disaggregated- co-located

Co-located goodput peaks then collapses as interference violates SLOs; disaggregated holds up far longer on the same GPUs.

Real-world anchors: DistServe reports up to ~7.4× more requests (≈2× under tight SLOs), and Splitwise ~2.35× at equal cost - by giving prefill and decode their own pools.

Illustrative first-order model (70B-class model, H100-class GPU): prefill time ∝ prompt length, decode step ∝ context, with a load-dependent interference tax on the co-located mode. The interference model is illustrative and the cited multipliers (DistServe ~7.4×, Splitwise ~2.35×) are hardware- and workload-specific.

Inference routing

Once you have a pool of workers, where you send each request matters. KV-cache-aware routing sends a request to the worker that already holds its prefix, avoiding a redundant prefill. The router also handles prefill/decode assignment and load-aware balancing to keep pools even.

Inference routing - send work where the prefix already lives

Requests share a system prompt (group S); some are unique (U). KV-aware routing sends a request to a worker that already cached its prefix, so the shared prompt isn't re-prefilled. Watch the cache hits glow and the redundant-prefill counter drop.

Routing

Shared-prefix ratiohow much of the prompt is the shared system prefix70%

Workers4

Workers (cache hits glow )

worker 1

5 reqs · 4 hits

S1S2 ✓S4 ✓S9 ✓S16 ✓

worker 2

5 reqs · no hits

U3U10U14U17U20

worker 3

5 reqs · 4 hits

S5S6 ✓S8 ✓S11 ✓S18 ✓

worker 4

5 reqs · 3 hits

U7S12S13 ✓S15 ✓S19 ✓

Prefix cache hits

11/14

shared-prompt reqs reusing KV

Redundant prefill avoided

6,160 tok

shared-prefix tokens not recomputed

TTFT saved vs no-cache

−123 ms

prefill skipped on cache hits

All three policies on this exact request stream

Round-robin

10/14 hits · −112 ms · max load 5

KV-aware

11/14 hits · −123 ms · max load 5

Load-aware

10/14 hits · −112 ms · max load 5

Round-robin and load-aware land on near-identical prefix reuse - both ignore where the KV lives (load-aware only balances request count). That near-tie is the lesson, not a bug: only KV-aware routing turns the shared prompt into cache hits. It does so with cost-based routing, not blind pinning - a cached worker absorbs extra load until a fresh worker is cheap enough to be worth a re-prefill, so the prefix replicates across workers. Add workers and watch shared work spread across several cache replicas (lower max load); raise the shared-prefix ratio and it concentrates onto fewer workers to maximize reuse. That hit-rate-vs-load balance is exactly what NVIDIA Dynamo and vLLM tune.

KV-aware routing pins shared-prefix requests to the worker holding their KV, turning repeat system prompts into near-free cache hits - the bigger the shared-prefix ratio, the larger the TTFT win. Illustrative token/latency model.

Metrics & SLOs

You can't tune what you don't measure - but not every metric matters for every workload. Each one is set by a specific phase, and the right SLO depends on what the workload looks like.

TTFT

Prefill

Time to first token

How long before anything appears. Sets perceived responsiveness; grows with prompt length.

TPOT / ITL

Decode

Time per output token / inter-token latency

How fast text streams once it starts. Memory-bandwidth-bound; must beat reading speed.

E2E latency

Both

End-to-end latency

TTFT + (output tokens × TPOT). The full wall-clock a request takes - tail (p99) often matters more than mean.

Throughput

Serving

Tokens/s or requests/s

Raw work the cluster does. Drives cost-per-token, but says nothing about whether any single request felt fast.

Goodput

Serving

Requests/s meeting SLOs

Throughput counted only for requests that hit ALL latency SLOs. The one number that reflects user-visible quality.

Which metric matters for which workload

The same five metrics, ranked by how much each workload profile actually cares. One accent per profile.

Metric	Interactive chat	Agentic	RAG
TTFT	Critical	Critical	Critical
TPOT / ITL	Critical	Critical	Minor
E2E latency	Matters	Critical	Matters
Throughput	Matters	Critical	Matters
Goodput	Matters	Critical	Matters

Interactive chat

Short prompt, streamed reply, one human waiting.

TTFTCritical
TPOT / ITLCritical
E2E latencyMatters
ThroughputMatters
GoodputMatters

TTFT must feel instant and TPOT must beat human reading speed (a few tokens/s perceived). Balanced - both latencies are user-facing.

Agentic

Many short LLM turns chained with tool calls.

TTFTCritical
TPOT / ITLCritical
E2E latencyCritical
ThroughputCritical
GoodputCritical

Per-call TTFT and TPOT compound across every step, so low latency per step plus high concurrency dominate. Tail latency (p99) is critical - one slow step stalls the whole chain.

RAG

Very long retrieved context, short answer.

TTFTCritical
TPOT / ITLMinor
E2E latencyMatters
ThroughputMatters
GoodputMatters

Prefill-heavy: TTFT is dominated by processing the long prompt, so prefix / KV caching of shared context is the big lever. Decode is short, so TPOT barely matters.

Interactive chat

Short prompt, streamed reply, one human waiting.

TTFT must feel instant and TPOT must beat human reading speed (a few tokens/s perceived). Balanced - both latencies are user-facing.

Agentic

Many short LLM turns chained with tool calls.

Per-call TTFT and TPOT compound across every step, so low latency per step plus high concurrency dominate. Tail latency (p99) is critical - one slow step stalls the whole chain.

RAG

Very long retrieved context, short answer.

Prefill-heavy: TTFT is dominated by processing the long prompt, so prefix / KV caching of shared context is the big lever. Decode is short, so TPOT barely matters.

SLOs drive capacity: you provision enough prefill and decode workers so that, at your peak request rate, goodput stays above target - not just raw throughput. That's exactly what the calculator above estimates.

Going further

The frontier of serving optimization, mapped by what each technique buys you. Every lever trades against the others - push latency and you spend memory or compute; the art is holding goodput while you do.

Optimize Latency

Speculative decoding

Draft model proposes, big model verifies in parallel - 2–3× faster, lossless.

Prefix / KV caching

Reuse KV for shared prompt prefixes to skip redundant prefill - slashes TTFT.

Disaggregated prefill–decode

Separate pools so a prefill burst can't stall everyone's decode.

Optimize Throughput

Continuous batching

Sequences join/leave every iteration, keeping the GPU saturated.

SLO-driven autoscaling

Scale prefill/decode pools to hold goodput as load shifts.

MoE serving (EP)

All-to-all expert routing spreads sparse experts across GPUs.

Optimize Memory

Quantized / FP8 KV

Store KV at FP8 to halve cache footprint and decode memory traffic.

KV offloading / tiering

Spill KV down GPU → CPU → SSD (LMCache, KVBM) for more concurrent sessions.

Structured decoding

Constrain output to a grammar/JSON schema at near-zero overhead.

Many techniques span more than one lever (e.g. prefix caching cuts both latency and memory pressure) - they're placed under their primary win.

Serving ties the whole stack together

Every serving decision traces back to model size, the KV cache, and the GPUs underneath. Size the model, understand the cache, then map it onto the infrastructure.

Model Sizing & Parallelism →NVIDIA Infrastructure →KV Cache →