VAST Data
InsightEngine with NVIDIA
IntermediateVAST + NVIDIA's full-stack system for real-time RAG - feeding GPUs directly and keeping them serving, not idling.
What InsightEngine is
Announced at GTC on March 18, 2025, VAST InsightEngine is described as the “industry's first secure full-stack system for real-time data inferencing.” It is pre-integrated with NVIDIA AI Enterprise and ships DGX-validated, expanding to NVIDIA-Certified Systems - turning the VAST platform into a real-time RAG engine rather than a stack of parts you wire together yourself.
Secure, full-stack, real-time
Storage, networking, vector search, and inference microservices come pre-integrated under one security model, so retrieval is grounded in fresh, governed data the moment it is written.
GA on DGX from March 2025
The system reached general availability on DGX systems starting March 2025, implementing NVIDIA's AI Data Platform reference design and expanding to NVIDIA-Certified Systems over time.
See the site's RAG & Vector Search, Inference & Serving, KV Cache, and NVIDIA Infrastructure pages for the building blocks InsightEngine assembles into one product.
“Certified” wording varies across announcements; presented as DGX-validated and expanding to NVIDIA-Certified Systems.
The NVIDIA hardware & software stack
InsightEngine is a layered system spanning NVIDIA silicon and software: BlueField-3 DPUs power the NVMe enclosures and run storage services as offloaded containers, Spectrum-X carries the RDMA fabric, cuVS on CNode-X does GPU-accelerated vector search, and NIM microservices plus the AI-Q and VSS blueprints sit on top. Click a layer to see what it does.
Click a layer
Implements NVIDIA's AI Data Platform reference design, pre-integrated with NVIDIA AI Enterprise.
AI-Q + VSS blueprints
Agents & video intelligence
- What it does
- NVIDIA blueprints running on the platform: the AI-Q Blueprint for agentic retrieval workflows and the VSS blueprint for video search and summarization.
- Why it matters for AI
- Ship working agent and video-RAG patterns out of the box instead of assembling them from scratch.
GPUDirect Storage: feeding GPUs without the CPU detour
On the legacy path, every read is copied into host system memory before a second copy lands it in GPU memory. GPUDirect Storage DMAs data straight from NVMe into GPU memory over RDMA, bypassing the CPU/system-memory copy entirely via NFS-over-RDMA and NVIDIA Magnum IO. Toggle the two paths to see the difference.
NVMe flash
VAST all-flash pool
RDMA / GPUDirect
NFS-over-RDMA + Magnum IO
GPU memory
direct DMA, no CPU copy
Data is DMA'd straight from NVMe into GPU memory over RDMA, bypassing the CPU and the system-memory copy entirely. NVIDIA Magnum IO GPUDirect Storage over NFS-over-RDMA.
~33 GB/s
CPU-mediated read
DGX-2, 1 MB IO
94+ GB/s
GPUDirect read
DGX-2, peaks ~98 GB/s
162 GiB/s
GPUDirect read
DGX A100, 8x HDR 200Gb NICs
VAST reports ~33 GB/s on the legacy CPU-mediated path vs 94+ GB/s sustained (peaks ~98 GB/s) at 1 MB IO with GPUDirect on a DGX-2, at ~15% CPU utilization; and 162 GiB/s read on a DGX A100 with 8× HDR 200Gb NICs. An earlier figure of 92.6 GB/s to a DGX-2 also circulates. All figures are VAST-reported and vary by platform and configuration.
Real-time RAG pipeline: embed-on-write → index → retrieve
Instead of a nightly batch job, InsightEngine turns every write into a retrieval-ready record: NIM embedding agents fire on write to create vectors and graph relationships, the index lives next to the source data, and GPU-accelerated cuVS search serves it back at inference time.
Embed on write
When data lands, the platform triggers NIM embedding agents that create vectors and graph relationships - no separate batch indexing job to schedule.
Index in place
Vectors and graph relationships are stored natively in the platform next to the source records, so the index never drifts from the data it describes.
Retrieve on the GPU
GPU-accelerated vector search via cuVS on CNode-X serves retrieval at inference time, feeding fresh, grounded context straight into the model.
KV-cache offload to fast storage
An emerging capability (with NVIDIA, ~Feb 2026) uses S3-over-RDMA as the KV-cache data plane for disaggregated inference at 100k+ token contexts. Inactive KV cache is evacuated from scarce GPU memory through tiers - GPU device memory → host RAM → local NVMe → external VAST pool - and paged back in across turns to cut prefill recompute. Step through a conversation to watch the KV state move.
GPU device memory
HBM - scarce, fastest
tens of GB
TB/s
Host RAM
system DRAM
hundreds of GB
~100s GB/s
Local NVMe
node-local flash
TBs
~GB/s
External VAST pool
S3-over-RDMA data plane
PB+
RDMA, networked
Prefill runs, KV cache for the conversation sits hot in GPU HBM while the model decodes the reply.
“S3 is the API, RDMA is the plane.”
VAST + NVIDIA use S3-over-RDMA as the KV-cache data plane, targeting disaggregated inference at 100k+ token contexts - keeping KV state in the serving loop across turns and paging in/out to cut prefill recompute.
Forward-looking capability (with NVIDIA, ~Feb 2026). No public latency numbers yet - presented as emerging.
Forward-looking capability; no public latency numbers yet.
Why this matters for AI factories
The throughput of an AI factory is increasingly set by the data layer, not the GPU. InsightEngine attacks all three failure modes at once: stale retrieval, idle GPUs, and repeated prefill.
Real-time RAG out of the box
Embed-on-write plus native vector search means retrieval is grounded in the latest data the moment it arrives - no nightly re-index, no stale answers.
GPUs stay fed, not idle
GPUDirect Storage feeds GPUs straight from NVMe over RDMA, so the most expensive hardware in the factory spends its time serving tokens, not waiting on the CPU.
KV-cache tiering cuts prefill cost
Keeping KV state in the serving loop across turns avoids recomputing prefill, which matters most at the 100k+ token contexts agentic workloads now demand.
Together these keep the most expensive hardware in the building serving tokens instead of waiting - the same throughput logic that runs through the NVIDIA Infrastructure, Inference & Serving, KV Cache, and RAG & Vector Search pages, applied end to end.