VAST Architecture: DASE

The scaling wall: shared-nothing & shared-disk

For two decades, scale-out storage came in two flavors, and both hit a wall. Shared-nothing systems give each server its own direct-attached drives, so capacity and compute are welded together and nodes must forward east-west traffic for data they do not own. Shared-disk systems let many nodes mount the same storage, but distributed locking and cache coherence become the bottleneck. Flip between them below to see the tradeoffs.

DASE: disaggregated, shared-everything

VAST introduced DASE in 2016. It splits compute logic from persistent capacity: compute runs as stateless containers, while all data, metadata and state live in shared NVMe enclosures that every compute node reaches directly over NVMe-oF. Any node can serve any request, so there is no inter-node chatter and no cache-coherence software. What makes this practical is networking roughly 100× faster than the 2000s - fast enough that remote NVMe behaves like local storage.

Shared-everything, not just shared disk

The “shared-everything” in DASE is the part people underestimate. Older “shared-disk” clusters share the blocks, but each node still keeps its own private cache and coordinates ownership through a distributed lock manager - so they spend their time on cache-coherence chatter, and the lock manager becomes the ceiling. VAST shares four things, not one:

Sharing data and metadata is what makes every CNode a perfectly parallel, interchangeable worker. Any CNode can serve any client request for any object at any time, because none of them depend on local state. Throughput scales almost linearly as you add nodes - there is no shared coordinator to bottleneck on. The same property makes a component failure a non-event - see Fail-in-place below.

CNodes, DNodes & the NVMe-oF fabric

CNodes and DNodes are logical constructs: both are containers, not fixed hardware roles, and on many deployments they run on the same physical boxes. CNodes (VAST Servers) are the stateless compute containers: they terminate the protocols, run the backend data services, and carry all of the system's compute. DNodes are the lightweight containers that own the NVMe devices, the SCM SSDs and QLC flash inside the enclosures (DBoxes). State lives only on those devices. The fabric between them is NVMe-over-Fabrics running on Ethernet or InfiniBand.

Stateless compute = elastic, no-rebuild scaling

Statelessness is the property that pays for itself. Because CNodes hold no data, the cluster scales linearly, upgrades fully online, and grows or shrinks on demand. Failure handling is under Fail-in-place below.

The write path: SCM → reduce → QLC

Every write lands in Storage Class Memory (SCM) first and is acked from there. Only later, off the critical path, is the data reduced and flushed to dense QLC flash in very large stripes. That deferral is what lets VAST absorb writes on low-endurance QLC without write amplification or garbage collection. Step through the three stages below.

Buffering in SCM, deferring data reduction, and flushing to QLC in very large stripes yields a ~10-year QLC lifespan and HDD-like cost on all-flash.

Locally-decodable erasure coding

Classic Reed-Solomon forces a tradeoff: wide stripes are capacity-efficient but expensive to rebuild, because reconstructing one lost strip means reading the whole stripe. VAST's locally-decodable scheme computes parity strips from different data-strip sets spanning more than one stripe, so with P parity strips you rebuild a lost strip by reading only ~1/P of the surviving data strips plus the P parity strips - about 41 of 149 for 146+4. Compare schemes below.

Fail-in-place: failures become a non-event

A failed CNode held no data and no exclusive locks, and any CNode reaches any SSD, so the survivors keep serving the same requests with no rebuild, no ownership failover, and no job restart. Each DBox is fronted by two DNodes, so a DNode failure is a non-event too: its partner keeps the drives reachable - which is also what lets the rolling upgrade above stay online. Fail a CNode or a DNode below.

Drives fail in place, because rebuilds are cheap. A lost strip is reconstructed by reading only ~1/P of the data strips plus the parity, and the 146+4 scheme tolerates several concurrent losses with an MTTDL measured in tens of millions of years, so VAST does not treat a dead SSD as an emergency. Failed media is simply left in place: the system routes around it, keeps serving at full speed, and you swap hardware in planned batches during a maintenance window, or run an enclosure to end of life without ever replacing individual drives. There is no urgent single-drive rebuild race and no degraded-performance window to sweat. Fail drives below and watch the system stay healthy.

Data reduction: deduplication, compression & similarity

Three techniques stack here, and they are not interchangeable. Deduplication removes blocks that are byte-for-byte identical to one already stored - powerful when whole copies repeat, but coarse: a single changed byte makes two blocks different, so near-duplicates slip straight through. Compression shrinks the bytes within a block using an encoder; it only ever sees one block at a time, a purely local view, so it cannot touch the redundancy that lives between similar blocks.

VAST's differentiator is the third technique: global similarity reduction. A similarity hash returns the same value for blocks that are merely similar, not identical. The first block of a hash becomes a compressed reference holding the full dictionary; every later similar block is stored as a small delta against it (ZSTD, without the dictionary overhead), capturing the mid-size repetitions that dedup and compression alone leave on the table. It runs on fine, dynamic chunks (on the order of 32 KB) rather than one fixed block, giving the hash far more chances to match. Crucially the hash table is global across the whole cluster and lives in SCM - so the all-flash DASE design lets the similarity search span the entire namespace as one reduction realm, a vastly larger scope than legacy systems whose dedup tables must fit in controller DRAM. A bigger search space finds more redundancy, which is exactly why “all data reduction is not created equal”. Toggle the techniques to see what each catches.

The Element Store & the V-Tree

The Element Store maps physical flash into user-facing “Elements” - files, S3 objects, tables and block volumes - in one namespace. The V-Tree is the metadata structure behind it: a wide, shallow B-tree variant whose nodes live on SCM and point to bytes, not blocks. Each level fans out roughly 512×, so a handful of hops can address on the order of 100 trillion elements, and because pointers reference exact byte ranges, copy-on-write snapshots and clones cost almost nothing. Take a snapshot below to see the new root reuse existing subtrees.

One namespace, every protocol

Because every protocol resolves to the same Element, NFS, SMB, S3 and GPUDirect Storage clients all reach the same bytes simultaneously, with no copies and no gateway translation layer. That matters for AI because a pipeline rarely speaks one protocol: data is ingested over S3, curated over file, streamed to GPUs over GPUDirect, and served back to inference and RAG, all against a single copy. Step through the stages to see which protocol each uses and why a unified namespace removes an entire class of copy-and-sync work.

Why this matters for AI / GPU workloads

AI infrastructure has an asymmetric appetite: training (dataset reads and bursty checkpoint writes) and inference want enormous, steady throughput to keep GPUs fed, while datasets grow into the petabytes on a different curve. DASE matches that shape.

Key takeaways