DataEngine & Event Broker

Bring compute to the data

The default pattern for processing data is backwards: data lands in storage, then a separate pipeline copies it out to a compute cluster, transforms it, and writes results back somewhere else. Every hop costs money, adds latency, and creates a stale copy. For AI this is fatal - embeddings and enrichment lag behind ingest, so the context your models retrieve is always a batch window out of date.

DataEngine inverts the model: run the function where the data already is. No ETL job to schedule, no second cluster to feed, no extra copy to keep consistent. The compute is event-driven and stateless, so it only runs when new data actually arrives.

DataEngine is an event-driven serverless compute fabric that runs functions on data where it lands. Serverless functions & triggers went GA on Nov 11, 2025. It has three execution modes that compose into one in-situ pipeline.

A function's lifecycle: code → container → trigger → in-situ run

You ship a function the same way you ship any container - the only new steps are registering it in VAST and binding it to a data event. Step through the lifecycle below.

def handle(event):
    obj = read(event.path)
    vec = embed(obj)   # NIM, OSS model…
    return {"vector": vec}

The Event Broker is a real-time event streaming engine built into the platform - it removes the need for a separate Kafka cluster and unifies streaming, storage, and analytics. Announced Feb 19, 2025 and available Mar 2025.

It is Kafka API-compatible, so existing producers and consumers work unchanged - but its brokers are stateless. There is no per-broker persistent state; durability and resilience come from VAST's global storage. Streams flow directly into VAST DataBase tables for instant correlation across structured and unstructured data.

Streams as tables: unifying streaming + analytics

Because brokers are stateless and storage is the source of truth, a stream is not a separate artifact you later ETL into a warehouse - it is a queryable table the moment it lands. The same record can be consumed as a Kafka message and joined against historical structured data in one query, with no copy in between.

Why this matters for AI

Together, DataEngine and the Event Broker make data active. Embeddings and enrichment happen automatically as data arrives, so the context your RAG and agent systems retrieve stays fresh in real time - not as of the last batch run.