VAST SyncEngine

The last-mile data problem

Most of the data an AI project could use already exists. It sits in older file servers, object buckets and collaboration apps, invisible to AI pipelines until someone brings it to where those pipelines run. That last step hurts in four ways:

• You can't use what you can't see. Nobody can say how many PDFs there are, where, how old, or who owns them - so projects start with weeks of discovery.
• Moving it is a project, not a command. Hand-written scripts and separate tools for listing, copying and checking; a job that dies halfway starts over; nobody is sure the copy is complete.
• File count, not just size, sets the clock. Hundreds of millions of small files take far longer to move than bandwidth math suggests.
• Copies go stale. A one-time copy is out of date the next morning, and a RAG assistant answers from last quarter's documents.

See first, then move

SyncEngine pairs two jobs. The first is seeing: a metadata indexing job walks a POSIX file system and writes one row per file - path, size, owner, timestamps - into a VAST DataBase table. That table is a data catalog of the file system's metadata: no file contents move, and you query it with SQL engines or the vastdb SDK to size the job and decide what is worth bringing. The second is moving: scale-out workers copy the chosen buckets, folders, spaces and paths onto VAST, retry what fails, and keep the copy current.

Think of it as moving house: survey the rooms first, then hire the movers - and keep forwarding the mail until moving day. Step through one illustrative estate below.

What it connects to

Each migration reads one source - a bucket, a space, a folder or a path - and writes to one destination. Object stores and collaboration apps land in VAST S3; NFS exports land in VAST NFS. For collaboration apps, SyncEngine also converts content into formats AI pipelines read easily and turns sharing settings into S3 ACLs.

Inside a migration: control plane and workers

SyncEngine runs on hosts you provide, next to the platform rather than inside it. A control plane handles scheduling, coordination and monitoring; the workers, spread across several hosts, do the copying. Each worker runs many parallel transfer threads (by default twice its CPU count), and a worker label ties a worker to specific migrations. The software is included with VAST; the hosts are yours.

Jobs are built for long runs: they can be paused and resumed, failed files can be replayed without re-copying the rest, rate limits protect busy sources and SaaS APIs, and progress, logs and metrics flow to Grafana dashboards.

Why file count matters as much as bytes

Every file costs a fixed amount of work before and after its bytes move - listing, opening, creating, checking, closing. With large files that overhead disappears into the transfer; with millions of small files it dominates, and the job runs far below what the network could carry. Parallel threads hide the per-file wait, which is why the tuning guidance depends on file size: 80 to 128 threads per worker for many small files, 16 to 32 for large files. Past a point, more threads stop helping because the source, the network or the destination is the limit.

Planning a move: full copy, resync, cutover

A migration is not one copy. The first pass moves everything selected, which can take days while users keep working on the old system. Each incremental sync after that carries only what changed since the last pass, so the passes get short. The cutover is then a short pause: stop writes on the old system, run one final incremental pass, and point users and applications at VAST. SyncEngine shortens the cutover window; it does not remove it.

A skip-existing option avoids re-copying data that is already at the destination, and on S3 resyncs, objects with the same Last-Modified time on both sides are skipped. Values in the planner are illustrative.

Staying current after the move

Every connector can run in two modes. Manual runs a migration on demand. Automatic starts the first sync right away, then resyncs on a schedule set in hours - every 24 hours by default. For AWS S3 sources there is a third option: event-driven sync that reads the bucket's change events from an Amazon SQS queue, so new objects follow soon after they are written rather than at the next scheduled pass; any event that is missed is caught by the next full scan.

Deletions are handled deliberately. By default, a file deleted at the source stays on VAST. Mirroring deletions is an opt-in setting that makes the source the single source of truth - and files it removes at the destination do not come back if the setting is turned off later. For a RAG assistant, the resync cadence sets how stale its sources can get before the pipeline even sees a change.

How SyncEngine fits the AI OS

SyncEngine is the on-ramp. Its job ends when data lands in a VAST S3 bucket or NFS view; what happens next is the rest of the platform. A DataEngine trigger on the bucket starts a pipeline, InsightEngine chunks and embeds the documents, and the vectors and metadata are stored in VAST DataBase for retrieval by RAG applications and agents. SyncEngine itself does not chunk or embed. The vendor-neutral version of this flow is on the data pipeline page, and the InsightEngine page covers the embedding side.

Among other ways to move data, SyncEngine is the tool VAST includes for bringing data onto its own platform. Vendor-neutral migration tools focus on analyzing and moving data between any platforms, and cloud transfer services do the same job for their own clouds.

Key takeaways