DataEngine & Event Broker

How stale is the data your model sees?

The same objects land over time. A pipeline or batch job only touches them at its next run, so each one waits - hatched - before it is usable. Running compute per event closes the gap.

Takeaway: moving data to compute costs N hops, N copies and N failure points; a schedule leaves context one window stale. Functions that run in place on each event are fresh by construction.

Schematic: arrival times, the sweep period and job lengths are illustrative.

Three execution modes, three lifetimes

The same stream of data events, seen by each mode. Pick one to see what it does - together they compose into one in-situ pipeline.

Takeaway: triggers turn each data event into an invocation, functions exist only while there is an event to handle, and engines stay up for heavier work - all of it next to the data.

Illustrative: event times and run lengths are made up to show the shapes.

Drop a file, watch the platform react

Each write to a view becomes an event on the broker. Stateless function pods scale out to consume the events, write their result into a table next to the source path, and scale back to zero when the queue is empty.

Use case: Auto-embedding on ingest

Takeaway: compute runs where the data lands and only while there is work - per object, as it arrives, with no copy out to a separate cluster and no scheduled batch job.

Illustrative: tick length, pod start and run times, the 6-pod cap, file names and output values are made up to show the shape of the flow.

From handler.py to an in-situ run

The same function changes shape at every step - watch the artifact strip as you step through.

Step 1 of 5: Write Python locally

def handle(event):
    obj = read(event.path)
    vec = embed(obj)   # NIM, OSS model…
    return {"vector": vec}

Code, names and the registry path are illustrative. Steps 3 and 4 show what you do, not exact command syntax.

Add a broker, lose a broker

Run the same event on both designs. Stateful Kafka brokers have to copy partition data around; stateless VAST brokers just change who serves what, because the data already sits in shared storage.

Takeaway: when state lives in shared storage, adding or losing a broker changes only who serves a partition - there is no partition data to move, and the stream is already a queryable table.

Illustrative: six partitions, replication factor 2, placements and transfer times are chosen to show the mechanism, not measured.

One landing zone, two access patterns

Events written through the Kafka API land as rows of a table. A consumer streams them in order, and a query joins the newest event against months of history - the same rows, with no connector and no copy in between.

Months of prior data already sit in the table. Press play to stream new events in.

Takeaway: no connector to maintain and no schema drift between the stream and the warehouse - correlating live and historical data is bounded by query time, not an ingest pipeline window.

Illustrative: keys, dates and events are made up.

From “data written” to “data usable”

Follow one new document through the platform. The Event Broker carries the trigger; DataEngine turns it into fresh context. Tap any stage.

Takeaway: the broker turns new data into an immediate trigger, and DataEngine turns that trigger into fresh embeddings - so retrieval never waits for a batch window.