Systems & Infrastructure
NVIDIA STX & Context Memory
AdvancedAt GTC 2026 NVIDIA introduced BlueField-4 STX - a reference architecture for AI-native storage that inserts a dedicated context-memory tier between GPUs and storage. It is the rack-scale sequel to the KV cache story: caching context not inside one GPU, but across a whole pod.
When the KV cache outgrows the GPU
Long-context and agentic inference explode the size of the KV cache - the per-token attention state a model must keep. Once it overflows GPU HBM, the choice is grim: stall, evict and recompute, or fall back to slow storage. STX's bet is to add a dedicated, networked context-memory tier close to the GPUs, accelerated by BlueField-4, so previously computed context is persisted and reused across requests and GPUs instead of recomputed.
STX is a reference architecture, not a box NVIDIA sells directly - a blueprint its storage partners (VAST, DDN, Dell, HPE, IBM, NetApp and more) build around. CMX (Context Memory Storage) is its first rack-scale implementation, with partner platforms expected in the second half of 2026.
BlueField-4
the processor at the center
Vera CPU + ConnectX-9 SuperNIC with line-rate encryption and data protection - the engine that drives the context tier.
CMX
Context Memory Storage
STX's first rack-scale implementation: petabytes of shared, Ethernet-attached flash that holds KV cache close to the GPUs.
Spectrum-X
the fabric
RDMA / RoCE Ethernet with adaptive routing and congestion control, moving context between flash and GPU at line rate.
DOCA Memos
the software
A new DOCA framework for KV communication and storage that orchestrates how context blocks are staged and reused.
Vera Rubin
the GPU platform
The next-gen compute platform STX is built to feed - so the GPUs spend cycles generating tokens, not waiting on data.
New to the silicon itself? The NVIDIA Infrastructure page covers the BlueField-4 DPU and Spectrum-X switches in depth.
A new tier in the hierarchy: G3.5
NVIDIA frames the memory and storage stack as numbered tiers - G1 (GPU HBM), G2 (Vera/CPU DRAM), G3 (local node SSD), and G4 (durable enterprise storage). STX inserts a new one between them: G3.5, Ethernet-attached flash optimized for KV cache. It is bigger and more shared than any single node's SSD, yet close enough to feed GPUs without falling to the slow durable floor. Drag the working set and toggle the tier to feel why it exists.
The storage hierarchy STX redraws - meet tier G3.5
NVIDIA names the memory/storage tiers G1–G4 and inserts a brand-new one - G3.5: Ethernet-attached flash (CMX) tuned for KV cache. Set how big the pooled KV cache grows, then toggle the CMX tier on and off to see where the cache must land.
Live attention state during decode
First spill - keeps recent context warm
Per-node prefix reuse, session resume
Shared pod-wide KV cache tier
System of record - or recompute
Lands in Local node SSD (~90 µs)
Comfortably held in a fast tier. Push the working set higher to reach the band where CMX earns its place.
Capacities and latencies are illustrative, pod-scale figures for teaching the tiering - not a datasheet. *G4 latency stands in for “fetch from durable storage or recompute the prefill.”
VAST's role: the CNode runs inside the DPU
Most partners attach storage over the network. VAST goes further: it ports its CNode - the stateless compute layer of its DASE architecture - directly onto the BlueField-4 DPU. That lets KV cache move zero-copy from remote flash into GPU memory, bypassing the host CPU and local SSD entirely, while VAST layers data reduction, security, and lifecycle management onto that shared context tier.
How VAST moves KV cache to the GPU - two paths
VAST ports its CNode onto the BlueField-4 DPU, so context flows straight from shared flash into GPU memory. Compare it to the classic storage path the host CPU has to babysit.
VAST D-Node
shared flash (G3.5)
Spectrum-X
RDMA / RoCE
BlueField-4
runs VAST CNode · virtio-fs
GPU HBM
zero-copy DMA
Network hops
3
Memory copies
0
Host CPU on the path?
No - bypassed
Running the CNode on the DPU means the GPU server's CPU never touches the data path - VAST adds data reduction, security, and lifecycle management to that shared context tier, which is why NVIDIA frames VAST's contribution as “context reuse optimization at scale.”
Simplified for teaching. The zero-copy / virtio-fs detail is from analyst reporting (Blocks & Files) on the STX design, consistent with VAST's own context-reuse material.
Why it matters
The payoff of reusing context instead of recomputing it is more useful work from the same GPUs. NVIDIA's headline figures for STX over traditional CPU-based storage are below - bold claims worth understanding, and worth treating as vendor numbers until independent benchmarks land.
Context reuse vs recompute - the headline impact
When a returning request's context is fetched from CMX instead of recomputed on the GPUs, the same hardware serves far more tokens. Flip STX on to see NVIDIA's claimed gains over traditional CPU-based storage.
Vendor claims. Up to 5× token throughput, 4× energy efficiency, and 2× faster ingest are NVIDIA's figures for STX vs traditional CPU-based storage architectures - announced at GTC 2026, not yet independently benchmarked. Treat as directional.
Where this connects
STX is the infrastructure-scale version of everything the KV cache pages teach, and the data services on that tier are exactly what VAST's InsightEngine builds on for real-time RAG. Follow the thread: