VAST DataBase

A columnar table format built into storage

The lakehouse pattern stores tables as Parquet files on object storage, then layers a table format (Iceberg, Delta, Hudi) plus a separate metastore on top to add schema, ACID, and time-travel. It works, but the format, the files, and the catalog are three loosely-coupled pieces to keep consistent - and streaming ingest buries them under millions of tiny files that constantly need compaction.

VAST DataBase is a columnar table format too - analytical, schema-aware, ACID - but it is native to the platform: the table format, its metadata, and the data all live in DASE storage as one thing. There is no Parquet-files-plus-metastore split, and no small-file problem to compact away. Models and agents scan columnar tables directly on the source of truth, with no ETL copy into a separate warehouse.

The write path: Persistent Write Buffer → Low-Cost Flash

VAST DataBase unifies transactional and analytical workloads in one table format. It writes in rows, perfect for transactions, and stores in columns, optimized for analytics - and it is fully ACID compliant. Writes land row-by-row into a Persistent Write Buffer (Storage-Class Memory), a low-latency tier that removes write hotspots, then background processes reshape those records into small ~32 KB columnar chunks on low-cost QLC flash. The result is low-latency ingest and fast columnar scans from the same table, without the read-versus-write trade-off that forces most shops to run a separate OLTP database and analytical warehouse.

Compute is stateless over NVMe-oF: any node serves any query, with no sharding or partition owners, and ACID is enforced through decentralized object- and file-level locks at exabyte / trillion-row scale. Because the columnarization runs off the critical write path, ingest never pays the column-store tax, and there is no small-file compaction debt to chase.

Crucially, tables, files and objects all live in one namespace with atomic, unified permissions - so a table row, a Parquet object and a raw file are governed and transacted together rather than scattered across systems.

Inside the column: a 32 KB chunk that prunes itself

Each column lands on flash as a ~32 KB chunk with a footer of metadata - sorted projections, customer-defined sort keys, and per-chunk statistics (min/max and count) - held in SCM, with no separate metadata manager. A chunk is roughly 1/4000th the size of a Parquet row group, so min/max pruning skips almost everything for a selective query: the engine reads one chunk instead of scanning a whole row group. That fine granularity is also why table updates stay simple - there is no partitioning to design, no pruning or vacuuming to run, and cross-table change data capture works at scale.

Beyond the lakehouse

Iceberg, Delta Lake and Snowflake-style lakehouses pair Parquet files with a table format and a separate metastore. VAST DB is itself a native table format - metadata lives with the data, so there is no metastore bottleneck. Streaming and CDC into Iceberg / Delta spawn countless tiny Parquet files that need compaction; VAST's 32 KB chunking on flash sidesteps the small-file problem entirely. And instead of append-only writes with snapshot rewrites, VAST does real-time mutable inserts, updates and deletes with query-in-place and atomic multi-table transactions - no copy into a warehouse.

Query engines: native, federated & pushdown

VAST has its own native query engine that runs in-place on the CNodes, executing SQL and vector search directly against the columnar tables, with heavy aggregations GPU-accelerated by Sirius (VAST's open-source engine built on NVIDIA cuDF). Open engines such as Trino and Spark can also run natively on VAST serverless compute, or attach externally via push-down plugins that ship predicates and projections down to storage. BI tools reach the data through those SQL engines or via Arrow Flight SQL, and the Python SDK gives programmatic access. Pick an engine to see how it connects and where it fits.

GPU-accelerated SQL with NVIDIA Sirius

Sirius is an open-source GPU SQL engine - it accelerates DuckDB by plugging in through the Substrait query-plan format and running relational operators on NVIDIA cuDF, with no query rewrites. VAST embeds it inside the DataBase so heavy aggregations execute on GPUs at the compute layer, running on CNode-X (NVIDIA-Certified GPU servers). VAST's strengths - intelligent columnar layout and predicate / projection pushdown - cut how much data the GPU has to touch in the first place; Sirius makes the work that remains fly.

Sirius is one piece of VAST's end-to-end accelerated stack with NVIDIA:

A VAST number of ways to work with VAST DB

Whatever tool a team already uses, it points at the same tables. SQL engines, the Python SDK, streaming events, and bulk imports all read and write one copy of the data over NVMe-over-Fabrics - so there is no copying into a separate warehouse and one governance model covers everything. Tap a method to see how it connects.

Apache Arrow & Arrow Flight: zero-copy at wire speed

Apache Arrow is a standardized columnar in-memory format. Arrow Flight transports Arrow record batches over gRPC with parallel streaming and zero-copy, avoiding the ODBC / JDBC serialization overhead commonly cited at 60–90%. VAST's SDK is Arrow-native: queries return a streaming pyarrow.RecordBatchReader, so data flows from flash to your dataframe without a row-by-row reserialization tax.

The Python SDK in practice

The vastdb package (“vast-py”) installs with pip install vastdb. You connect with an endpoint plus access / secret keys, then run operations inside a session.transaction() block. The hierarchy is bucket → schema → table (PyArrow schemas): table.insert(pyarrow_table) writes rows, and table.select(...) returns a streaming reader with predicate and projection pushdown expressed via Ibis (e.g. (_.c2 > 2) & _.c3.isnull()).

import pyarrow as pa
import vastdb
from ibis import _

# 1. Connect: endpoint + access / secret keys
session = vastdb.connect(
    endpoint="http://vip-pool.vast.example.com",
    access="ACCESS_KEY",
    secret="SECRET_KEY",
)

# 2. Everything runs inside an ACID transaction
with session.transaction() as tx:
    # bucket -> schema -> table
    schema = tx.bucket("ml").schema("features")
    table = schema.table("user_events")

    # 3. Insert a PyArrow table (row ingest -> SCM buffer)
    batch = pa.table({
        "user_id": [101, 102, 103],
        "c2": [5, 1, 9],          # int column
        "c3": ["a", None, "c"],   # nullable string
    })
    table.insert(batch)

    # 4. Streaming read with predicate + projection pushdown (Ibis)
    reader = table.select(
        columns=["user_id", "c2"],
        predicate=(_.c2 > 2) & _.c3.isnull(),
    )

    # 5. reader is a pyarrow.RecordBatchReader -> pandas
    df = reader.read_all().to_pandas()
    print(df)

Beyond the basics: semi-sorted projections for fast secondary lookups, S3 Parquet import without a client-side copy, the VAST Catalog (query the filesystem itself as a table), and snapshots for point-in-time reads.

Serving features & metadata to AI pipelines

Because the table format holds fresh, mutable data and large-scale analytics together, AI pipelines read current, consistent data with no ETL hops. Features and metadata are served in real time from the source of truth: an agent can update a record and immediately query it, training jobs read the same tables that production writes, and the Event Broker exposes streams as queryable tables - closing the loop between operational and analytical data.