Systems & Infrastructure
Simulation & Digital Twins
AdvancedEveryone knows GPUs train and serve AI. The quieter third job is to simulate - to build photorealistic virtual worlds where robots, cars, and factories learn before they ever touch reality. This is the engine behind digital twins, synthetic data, and the world models now reshaping physical AI.
The third GPU workload class
We usually split AI into two jobs: training (teach the model) and inference (run it). Simulation is a distinct third class - and it leans on a different kind of GPU work: real-time physics and ray-traced rendering, not just the matmul that powers the other two. NVIDIA frames this as a three-computer model: one computer trains, one simulates, one infers on the machine itself.
Train
DGXThe supercomputer that learns the model. Dense matmul on Tensor Cores, scaled across thousands of GPUs.
In the data center
Simulate
Omniverse / CosmosThe virtual proving ground. Real-time physics plus ray-traced rendering on RTX PRO / OVX systems generate worlds and data.
In the data center or workstation
Infer
Jetson AGX ThorThe brain on the robot or vehicle. Runs the trained policy in real time, on-device, at the edge.
On the machine
Three words, in plain English
Before the stack, the vocabulary. These three terms come up in every conversation about physical AI - here is what they actually mean.
Digital twin
A living video-game copy of a real thing - a factory, a car, a city block - kept in sync with reality through sensor feeds. Change the twin, predict what happens to the real thing.
World model
An AI that has learned how the physical world behaves and can predict what happens next. Show it a scene and an action; it imagines the consequence, like a flight simulator the AI built for itself.
Synthetic data
Computer-generated training examples that come pre-labeled, because the simulator already knows exactly what is in every frame. No humans drawing boxes, no guesswork.
Why simulate at all
Real-world data is slow to collect, dangerous to gather (you cannot crash ten thousand real cars), and expensive to label by hand. A simulator runs faster than real time, spins up thousands of parallel instances, and labels every frame for free - because it generated the frame and already knows what is in it. The economics are not a little better; they are orders of magnitude better.
Real world vs simulation
Collecting and labeling data in the physical world is slow, dangerous, and expensive. A simulator runs faster than real time, in thousands of parallel instances, and labels itself. Drag the slider and watch the two worlds diverge.
~$0.22/sample · human collection + 25 labelers
~$0.0009/sample · GPU render + auto-label · 2,000 parallel streams
244×
cheaper
12K×
faster
Illustrative model - unit rates are rough placeholders chosen to show the order-of-magnitude gap, not a vendor quote. Real economics vary by task, sensor suite, and GPU fleet.
Domain randomization: one scene, infinite data
A model trained on one perfect studio shot overfits to that studio. The fix is to deliberately randomize everything that does not matter - lighting, textures, camera angle, object pose - so the model is forced to learn the thing that does. Crank the variety high enough and the real world looks like just one more random variant. Drag the controls and watch a single base scene multiply into labeled training data.
One scene, infinite labeled training data
Start from a single base scene, then let the simulator randomize lighting, backgrounds, pose, and camera. Every variant comes with a perfectly accurate bounding box and class label for free - the model learns the object, not the studio it was shot in.
Base scene
Generated & auto-labeled
1 scene → 12 labeled samplesThis is domain randomization: by training across wild variety, the model becomes robust to the real world it has never seen. OpenAI's robotic hand learned dexterous manipulation entirely in simulation this way, then transferred to real hardware.
The sim-to-real loop
Simulation is never perfect - there is always a gap between the virtual and the real, the sim-to-real gap. The trick is to make that gap shrink over time: deploy, watch where reality breaks the model, recreate those failures in the simulator, and retrain. The single most valuable output of the real world is its failures. The loop, not any one model, is the product.
The sim-to-real loop
Failures in reality are not setbacks - they are the highest-value training data there is. Capture them, replay them in simulation, and the model gets better every cycle. Step through it, or let it run.
Train in sim
Millions of randomized episodes, faster than real time.
Deploy to real
Ship the policy to the robot or vehicle in the field.
Capture failure
A real-world edge case the sim never showed - logged in full.
Feed back to sim
Recreate the edge case as new scenarios. The simulator learns.
Each completed loop closes the sim-to-real gap with sharply diminishing returns - the last few points are the hardest, which is why the loop never really stops.
Illustrative model - accuracy gains are a synthetic curve to show the shape of the loop, not measured results.
NVIDIA Omniverse & the stack
Omniverse is the platform that ties all of this together. At its base sits OpenUSD - the shared 3D format that lets every tool agree on one scene. On top, the Kit engine renders and runs it; on top of that, domain apps for robotics, AVs, and synthetic data. Adoption is real: as of August 2025, 300,000+ downloads and 252+ enterprise deployments, with users including BMW, Toyota, Lucid, and TSMC.
OpenUSD
the shared 3D source of truthUniversal Scene Description, the open standard from Pixar. The HTML of 3D: one format where CAD, physics, robot paths, IoT signals, and AI labels all coexist and interoperate across Maya, Blender, Revit, and CATIA.
↑ built on ↑
Omniverse Kit
the engine & SDKThe development platform that loads, composes, and renders USD scenes - the runtime every Omniverse application is built on.
↑ built on ↑
Apps & frameworks
Isaac · DRIVE · ReplicatorPurpose-built on top of Kit for each domain.
Isaac Sim / Isaac Lab
Robotics simulation and reinforcement-learning gym. Train manipulation and locomotion policies across thousands of parallel virtual robots.
DRIVE Sim
Autonomous-vehicle simulation: sensor-accurate scenarios, rare edge cases, and full driving stacks tested before a wheel turns.
Replicator
The synthetic-data generator. Programmatically randomizes scenes and emits photorealistic, perfectly-labeled images at scale.
PhysX
The physics engine: rigid bodies, contacts, fluids, and deformables, so the virtual world pushes back the way the real one does.
RTX / ray tracing
Photoreal rendering that closes the sim-to-real gap - light, shadow, and material so convincing that models trained on it transfer to reality.
Three-computer reach
Sim outputs flow straight into training (DGX) and onto the device (Jetson Thor), making the stack one continuous pipeline.
BMW simulates its entire 31-factory network as digital twins in Omniverse - reporting up to a 30% reduction in production-planning costs. A Replicator + Isaac Sim pipeline generates thousands of photorealistic, labeled training images "with one click," and BMW piloted an entire virtual factory 2+ years before series production began.
World models - the frontier
The next leap is not hand-building scenes but generating them. World models learn the dynamics of reality from video and then imagine new, interactive, physically-plausible worlds on demand - turning the simulator itself into a neural network. This is where simulation, generative AI, and robotics converge.
NVIDIA Cosmos
World foundation models for physical AI, unveiled at CES 2025. Models from 4B to 14B params (Nano → Ultra) under an open model license, trained on 20 million hours of video. NeMo Curator processed, curated, and labeled those 20M hours in 14 days on Blackwell - a job that would take 3+ years on CPU. Early adopters span humanoid robotics (1X, Agility, Figure AI) and AVs (XPENG, Uber with Waabi).
Google DeepMind Genie 3
Announced August 2025: generates real-time, interactive worlds from a text prompt at 24 fps, with memory of what it has already shown. Waymo reportedly adopted Genie 3 to build the Waymo World Model for AV development.
Wayve
Builds end-to-end, embodied world models - an "AI driver" that learns to drive from raw experience rather than hand-coded rules, with simulation central to how it scales.
How much data does this actually take?
Simulation and training are, underneath the renders and the physics, a storage and data-movement problem. Follow the data for autonomous driving and it climbs by roughly three orders of magnitude at every step - from one car, to a fleet, to what a program keeps, to the simulated worlds that dwarf reality, to the corpus that trains the model.
One vehicle, one day
A production car streams ~4 TB/day from its cameras, LiDAR, and radar; a heavily-instrumented test rig hits 11–152 TB/day. A single hour of driving is 1–5 TB. - Intel / Tuxera ↗
A test fleet, one day
A ~200-vehicle fleet generates 2.2–30.4 PB of sensor data per day. The self-driving industry collectively ingests well over a petabyte every day. - Tuxera ↗
What one program keeps
Mobileye stores 200+ petabytes of driving footage - roughly 16 million one-minute clips, about 25 years of continuous driving - kept live for training and replay. - Mobileye / Intel, CES 2022 ↗
Then multiply by simulation
Waymo has driven ~200 million real autonomous miles - and 20+ billion miles in simulation. Roughly a hundred virtual miles for every real one, each one more data to generate, store, and replay. - Waymo ↗
The training corpus
NVIDIA Cosmos was trained on 20 million hours of video - about 9,000 trillion tokens, an estimated ~45 PB raw and ~2 PB after curation. Curating it took 14 days on Blackwell instead of 3+ years on CPU. - NVIDIA, CES 2025 ↗
Per-vehicle and fleet figures are widely-cited industry estimates; the Cosmos PB sizes are derived from NVIDIA's stated 20M-hour / 9,000T-token corpus. Storage totals (Mobileye) and simulated-mile counts (Waymo) are vendor-reported.
Robots learn from the same data pyramid
Embodied AI runs the identical loop - real demonstrations at the top, synthetic data generated in simulation underneath. A single released robot dataset is already the size of a small data lake, and simulation multiplies it further.
Open X-Embodiment
1M+ trajectories · ~32 TB
Over a million real-robot demonstrations pooled from 60 datasets across 22 robot types - the ImageNet moment for robot learning.
Google DeepMind et al. ↗AgiBot World
~1M trajectories · ~43.8 TB
Roughly 3,000 hours of humanoid manipulation from 100 robots - one released dataset that already weighs as much as a small data lake.
AgiBot ↗NVIDIA Isaac GR00T
780K sim trajectories in 11 hrs
Simulation generated 780,000 synthetic trajectories - the equivalent of 6,500 hours (nine months) of human teleoperation - in just 11 hours. This is why synthetic data wins.
NVIDIA, GTC 2025 ↗And none of it is cold storage
Every petabyte here has to stay live - read back constantly to curate, replay edge cases, generate synthetic variations, and retrain. The data isn't a byproduct of simulation; at this scale it is the product, and feeding it to the GPUs fast enough is the real bottleneck.
Simulation is a data factory
Every loop here produces enormous volumes of rendered frames, sensor logs, and labeled samples that have to be stored, curated, and fed back into training. A single autonomous-vehicle or robotics program routinely generates petabytes of sensor and rendered data per day - and all of it must stay live for curation, replay, and retraining. Simulation does not stand alone - it sits on the same GPU fleet and data platform as everything else, which is exactly the scale problem VAST is built for.