Systems & Infrastructure
Data Pipeline for AI
FundamentalModels get the headlines, but a model is only as good as the data fed into it - and only as fast as the storage feeding the GPUs. This page starts with the plumbing in plain English, then goes deep on how AI data pipelines, lakes, and lakehouses actually work, and why high-performance storage is the hidden lever on cost and speed.
Start with the plumbing
A data pipeline is plumbing for data: pipes that move it from where it's created (apps, sensors, the web) to where it's used (dashboards, models), cleaning and reshaping it along the way. Where the water is stored matters too - and there are three common reservoirs.
Data warehouse
A tidy library
Structured, schema-on-write tables. Data is cleaned and modeled before it lands.
Strong: Fast, reliable SQL & BI dashboards.
Watch out: Rigid; struggles with raw text, images, and exploratory AI work.
Data lake
A giant storage unit
Raw files of any type (text, images, video, logs) dumped cheaply into object storage.
Strong: Holds everything; cheap; perfect for ML training corpora.
Watch out: No guarantees - easily becomes a “data swamp” with no order.
Lakehouse
A renovated warehouse on the lake
A lake plus an open table format (Iceberg/Delta/Hudi) that adds database guarantees.
Strong: Warehouse reliability over lake-scale raw data - one system for BI and AI.
Watch out: Younger ecosystem; needs governance discipline to stay clean.
ETL - Extract, Transform, Load
Clean and reshape data before loading it. Classic warehouse approach - you only store the polished result.
Great when the schema is known and storage is precious.
ELT - Extract, Load, Transform
Load the raw data cheaply first, transform it later - many times. The modern AI default, because you re-curate the same raw corpus again and again as ideas change.
Object storage is cheap, so keep the raw and transform on demand.
A normal pipeline vs. an AI pipeline
A traditional analytics pipeline ends at a clean table a human reads. An AI pipeline keeps going: the data has to become something a model can learn from or retrieve. That adds a whole new set of stages.
Classic analytics pipeline
Ingest
pull from apps & databases
↓
Clean & join
fix types, merge sources
↓
Model & aggregate
build SQL tables
↓
Dashboard
a human reads a chart
AI-centric pipeline (the extra stages)
Curation
select & weight sources by quality, not just volume
Deduplication
remove near-duplicate docs that waste compute & memorize
Tokenization
convert text to integer tokens, pack & shard for the GPUs
Embedding generation
encode chunks into vectors for retrieval (RAG)
Vector indexing
build ANN indexes so similar vectors are findable fast
Feature store
compute & serve the same features offline and online
Labeling / RLHF
human & AI feedback to align and rank model outputs
Same raw data, but it forks into a training corpus, a retrieval index, and a feature store - each with its own quality bar.
The data refinery: training data is manufactured, not found
Common Crawl is 9.5+ PB of raw web, ~250 TB and 2B+ pages in a single monthly snapshot - and over 80% of GPT-3's training tokens came from it. But you can't train on raw web; most of it is junk, boilerplate, and duplicates. Hugging Face's FineWeb distilled ~15 trillion high-quality tokens from 96+ snapshots - and beat bigger datasets by filtering and deduplicating harder, not crawling more. Watch the volume shrink at each stage.
The data refinery
Raw web in, training-ready tokens out. Each stage throws most of the data away - quality beats quantity. Toggle filter stages off to see how much more survives, and how the final token count balloons with junk.
Raw web crawl
Common Crawl: ~250 TB / month
250 TB
Language & quality filter
drop non-target langs, junk, boilerplate
45 TB
Deduplicate
MinHash / exact: kill repeats
16 TB
Curate & classify
FineWeb-Edu style quality scoring
8.7 TB
Tokenize
text -> tokens, pack & shard
8.7 TB
Survives the pipeline
3.46%
of the raw 250 TB snapshot
Training-ready output
15.00 T tokens
all filters on → FineWeb-scale ~15 T quality tokens
Turn filters off and the surviving volume balloons - but those extra “tokens” are duplicates and junk that hurt the model. FineWeb beat bigger datasets by filtering harder, not crawling more. Illustrative model; survival fractions approximate the real Common Crawl → FineWeb funnel.
Feeding the GPUs: compute is only as fast as the data
A training run is a race to keep thousands of GPUs busy. If storage can't stream training shards fast enough, GPUs sit idle and Model FLOPs Utilization (MFU) - the fraction of theoretical compute you actually use - collapses. Idle GPUs at $3/hr each are pure burned cash. This is why high-performance storage is an AI-infrastructure decision, not a commodity one.
Keeping the GPUs fed
A GPU at $3/hr earns nothing while it waits on data. If storage can't supply bytes as fast as the fleet consumes them, utilization (MFU) drops and idle GPU-time burns cash. Pick a storage tier and watch the fleet.
fast but small, manual staging, no sharing
GPU fleet (15/64 fed)
23% MFU
Burn rate while starved
$147/hr
49 idle GPUs at $3/hr each
Wasted since you arrived
$0
ticking in real time
Switch to the parallel all-flash filesystem and the ceiling jumps to ~1,000 GB/s - the fleet stays green even at full scale. Compute is only as fast as the data feeding it: high-performance storage is what turns expensive GPUs into useful FLOPs (the VAST value prop). Illustrative model; per-GPU demand and tier ceilings are approximate.
The other heavy I/O: checkpoints
Training also writes checkpoints - full snapshots so a crashed run can resume instead of starting over. With Adam mixed precision a checkpoint is roughly 16 bytes/param, so a 70B model is ~1 TB+. Frontier runs checkpoint frequently for fault tolerance, and the whole fleet stalls while writing - so fast write and fast read (for recovery) directly protect expensive GPU-hours.
Synchronous checkpoint
The classic approach: training pauses, copies the full model + optimizer state out of GPU memory, and waits for it to land on storage before resuming. Every GPU in the job sits idle for the whole write. On a large run that's thousands of GPUs burning money to do nothing - checkpoint a 70B every 30 minutes and the stall tax adds up fast.
Asynchronous checkpoint
The modern default (PyTorch DCP, NeMo, DeepSpeed). The state is snapshotted to fast CPU/GPU memory in a fraction of a second, then flushed to storage in the background while training keeps running. The GPU stall shrinks from minutes to near-zero, so teams can checkpoint more often (better fault tolerance) for less wasted compute. It leans even harder on high-bandwidth storage to absorb the burst.
Inference has a pipeline too - and a flywheel
Serving a model isn't a single forward pass. A modern request often retrieves context with RAG - embed the query, search a vector index, stitch results into the prompt - and looks up online features from a feature store. And every interaction feeds the most valuable asset in AI: the data flywheel.
The data flywheel
Users hit the model
real prompts, real edge cases
↓
Log interactions
capture inputs, outputs & feedback
↓
Curate & label
filter, dedup, score, annotate
↓
Retrain / fine-tune
fold the new data into the model
↓
Ship a better model
which attracts more usage…
… and the loop closes: more usage → more data → a better model → more usage. Whoever spins this fastest wins, and it all runs on the data pipeline.
The technology landscape
These are the names you'll meet in any AI-data conversation. Open table formats (Iceberg/Delta/Hudi) deserve a special call-out: they add ACID transactions, schema evolution, and time-travel on plain files in object storage - “database guarantees without a database” - turning a messy lake into a reliable lakehouse.
Apache Spark
Distributed engine for large-scale ETL/ELT across a cluster.
Apache Kafka
Streaming backbone - moves events in real time between systems.
Iceberg / Delta / Hudi
Open table formats: ACID, schema evolution & time-travel on files.
Apache Parquet
Columnar file format - compact, fast to scan, the lake default.
Object storage (S3)
Cheap, infinite, durable bucket storage - the floor of every lake.
Airflow / Dagster
Orchestrators - schedule, sequence & monitor pipeline jobs.
dbt
SQL transformation & testing layer - version-controlled models.
Feast / Tecton
Feature stores - kill training/serving skew with shared logic.
Ray
Python-native distributed compute for data prep & ML at scale.
Databricks / Snowflake
Managed lakehouse & warehouse platforms tying the stack together.
High-performance storage
NVMe & parallel filesystems that keep GPUs fed (VAST-class).
NVIDIA RAPIDS / cuDF
GPU-accelerated dataframes & Spark - up to ~40× faster ETL.
The medallion architecture
A common way to organize a lakehouse: Bronze (raw, untouched) → Silver (cleaned & conformed) → Gold (curated, business- & model-ready). Same ELT idea: keep the raw, refine in layers.
What “good” looks like
A healthy AI data platform is judged on these six properties. They are what separate a reliable model factory from a data swamp.
Freshness
data reflects the real world now, not last quarter
Quality
filtered, deduped, validated - garbage in, garbage out
Lineage
you can trace every output back to its sources
Reproducibility
rerun the same pipeline, get the same dataset
Throughput
fast enough to keep the GPUs and users fed
Governance
access, privacy & retention are enforced, not hoped for
The pipeline feeds everything else
Curated data trains the models, fast storage keeps the GPUs busy, and the flywheel turns usage back into better data. The data pipeline is the substrate the rest of the stack runs on.