VAST DataBase

A columnar table format built into storage

The lakehouse pattern stores tables as Parquet files on object storage, then layers a table format (Iceberg, Delta, Hudi) plus a separate metastore on top to add schema, ACID, and time-travel.

VAST DataBase is a columnar table format too - analytical, schema-aware, ACID - but it is native to the platform: the table format, its metadata, and the data all live in DASE storage as one thing. The three properties below are what that buys; the “Beyond the lakehouse” section further down compares it with Iceberg and Delta point by point.

The write path: Persistent Write Buffer → Low-Cost Flash

VAST DataBase unifies transactional and analytical workloads in one table format. It writes in rows, perfect for transactions, and stores in columns, optimized for analytics - and it is fully ACID compliant. Writes land row-by-row into a Persistent Write Buffer (Storage-Class Memory), a low-latency tier that removes write hotspots, then background processes reshape those records into small ~32 KB columnar chunks on low-cost QLC flash. The result is low-latency ingest and fast columnar scans from the same table, without the read-versus-write tradeoff that forces most shops to run a separate OLTP database and analytical warehouse.

Compute is stateless over NVMe-oF: any node serves any query, with no sharding or partition owners, and ACID is enforced through decentralized object- and file-level locks at exabyte / trillion-row scale. Because the columnarization runs off the critical write path, ingest never pays the column-store tax.

Crucially, tables, files and objects all live in one namespace with atomic, unified permissions - so a table row, a Parquet object and a raw file are governed and transacted together rather than scattered across systems.

Inside the column: a 32 KB chunk that prunes itself

Each column lands on flash as a ~32 KB chunk with a footer of metadata - sorted projections, customer-defined sort keys, and per-chunk statistics (min/max and count) - held in SCM, with no separate metadata manager. A chunk is roughly 1/4000th the size of a Parquet row group, so min/max pruning skips almost everything for a selective query: the engine reads one chunk instead of scanning a whole row group. That fine granularity is also why table updates stay simple - there is no partitioning to design, no pruning or vacuuming to run, and cross-table change data capture works at scale.

Beyond the lakehouse

Iceberg, Delta Lake and Snowflake-style lakehouses pair Parquet files with a table format and a separate metastore - three loosely-coupled pieces to keep consistent. VAST DB is itself a native table format: metadata lives with the data, so there is no metastore bottleneck. Streaming and CDC into Iceberg / Delta spawn countless tiny Parquet files that constantly need compaction; VAST columnarizes off the critical write path into fixed 32 KB chunks on flash, which sidesteps the small-file problem entirely. And instead of append-only writes with snapshot rewrites, VAST does real-time mutable inserts, updates and deletes with query-in-place and atomic multi-table transactions - models and agents scan columnar tables directly on the source of truth, with no ETL copy into a separate warehouse.

Query engines: native, federated & pushdown

VAST has its own native query engine that runs in-place on the CNodes, executing SQL and vector search directly against the columnar tables, with heavy aggregations GPU-accelerated by Sirius - an open-source GPU SQL engine built on NVIDIA cuDF - on NVIDIA GPUs, in technical preview. Open engines such as Trino and Spark can also run natively on VAST serverless compute, or attach externally via push-down plugins that ship predicates and projections down to storage. BI tools reach the data through those SQL engines or via Arrow Flight SQL, and the Python SDK gives programmatic access - all of them meet at the same tables, shown in the diagram below.

Every engine, one table

Whatever tool a team already uses, it points at the same tables. SQL engines, the Python SDK, streaming events, and bulk imports all read and write one copy of the data over NVMe-over-Fabrics, under one governance model. Pick an engine or an ingest path in the diagram to see how it connects and where its data goes.

GPU-accelerated SQL with Sirius

Sirius is an open-source GPU SQL engine led by the University of Wisconsin-Madison with NVIDIA support - it accelerates DuckDB by plugging in through the Substrait query-plan format and running relational operators on NVIDIA cuDF, with no query rewrites. VAST integrates it with the DataBase so heavy aggregations execute on GPUs at the compute layer; the integration has been in technical preview since August 2026. VAST's columnar layout and predicate / projection pushdown cut how much data the GPU has to touch in the first place; Sirius speeds up the work that remains.

Two Sirius figures come from two different benchmarks. In early benchmarks, VAST DataBase with Sirius took up to 44% less query time and up to 80% less query cost - the race below. Separately, Sirius ran up to 20× faster than DuckDB on an NVIDIA RTX PRO 6000 GPU. They measure different setups, so read them side by side rather than as one number.

Sirius is one piece of VAST's end-to-end accelerated stack with NVIDIA:

Apache Arrow & Arrow Flight: zero-copy at wire speed

Apache Arrow is a standardized columnar in-memory format. Arrow Flight transports Arrow record batches over gRPC with parallel streaming and zero-copy, avoiding the ODBC / JDBC serialization overhead commonly cited at 60–90%. VAST's SDK is Arrow-native: queries return a streaming pyarrow.RecordBatchReader, so data flows from flash to your dataframe without a row-by-row reserialization tax.

The Python SDK in practice

The vastdb package (“vast-py”) installs with pip install vastdb. You connect with an endpoint plus access / secret keys, then run operations inside a session.transaction() block. The hierarchy is bucket → schema → table (PyArrow schemas): table.insert(pyarrow_table) writes rows, and table.select(...) returns a streaming reader with predicate and projection pushdown expressed via Ibis (e.g. (_.c2 > 2) & _.c3.isnull()).

import pyarrow as paimport vastdbfrom ibis import _ 1# 1. Connect: endpoint + access / secret keyssession = vastdb.connect(    endpoint="http://vip-pool.vast.example.com",    access="ACCESS_KEY",    secret="SECRET_KEY",) 2# 2. Everything runs inside an ACID transactionwith session.transaction() as tx:    # bucket -> schema -> table    schema = tx.bucket("ml").schema("features")    table = schema.table("user_events") 3    # 3. Insert a PyArrow table (row ingest -> SCM buffer)    batch = pa.table({        "user_id": [101, 102, 103],        "c2": [5, 1, 9],          # int column        "c3": ["a", None, "c"],   # nullable string    })    table.insert(batch) 4    # 4. Streaming read with predicate + projection pushdown (Ibis)    reader = table.select(        columns=["user_id", "c2"],        predicate=(_.c2 > 2) & _.c3.isnull(),    ) 5    # 5. reader is a pyarrow.RecordBatchReader -> pandas    df = reader.read_all().to_pandas()    print(df)

Beyond the basics: semi-sorted projections for fast secondary lookups, S3 Parquet import without a client-side copy, the VAST Catalog (query the filesystem itself as a table), and snapshots for point-in-time reads.

Serving features & metadata to AI pipelines

Because the table format holds fresh, mutable data and large-scale analytics together, AI pipelines read current, consistent data with no ETL hops. Features and metadata are served in real time from the source of truth: an agent can update a record and immediately query it, training jobs read the same tables that production writes, and the Event Broker exposes streams as queryable tables - closing the loop between operational and analytical data.

Key takeaways