Simulation & Digital Twins

The third GPU workload class

We usually split AI into two jobs: training (teach the model) and inference (run it). Simulation is a distinct third class - and it leans on a different kind of GPU work: real-time physics and ray-traced rendering, not just the matmul that powers the other two. NVIDIA frames this as a three-computer model: one computer trains, one simulates, one infers on the machine itself.

Three words, in plain English

Before the stack, the vocabulary. These three terms come up in every conversation about physical AI - here is what they actually mean.

Why simulate at all

Real-world data is slow to collect, dangerous to gather (you cannot crash ten thousand real cars), and expensive to label by hand. A simulator runs faster than real time, spins up thousands of parallel instances, and labels every frame for free - because it generated the frame and already knows what is in it. The economics are not a little better; they are orders of magnitude better.

Domain randomization: one scene, infinite data

A model trained on one perfect studio shot overfits to that studio. The fix is to deliberately randomize everything that does not matter - lighting, textures, camera angle, object pose - so the model is forced to learn the thing that does. Crank the variety high enough and the real world looks like just one more random variant. Drag the controls and watch a single base scene multiply into labeled training data.

{
  "image": "variant_001.png",
  "class": "bottle",
  "bbox_xywh": [0.34, 0.32, 0.27, 0.29],
  "rotation_deg": -37,
  "lighting_hue": 223
}

The sim-to-real loop

Simulation is never perfect - there is always a gap between the virtual and the real, the sim-to-real gap. The trick is to make that gap shrink over time: deploy, watch where reality breaks the model, recreate those failures in the simulator, and retrain. The single most valuable output of the real world is its failures. The loop, not any one model, is the product.

NVIDIA Omniverse & the stack

Omniverse is the platform that ties all of this together. At its base sits OpenUSD - the shared 3D format that lets every tool agree on one scene. On top, the Kit engine renders and runs it; on top of that, domain apps for robotics, AVs, and synthetic data. Adoption is real: as of August 2025, 300,000+ downloads and 252+ enterprise deployments, with users including BMW, Toyota, Lucid, and TSMC.

World models - the frontier

The next leap is not hand-building scenes but generating them. World models learn the dynamics of reality from video and then imagine new, interactive, physically-plausible worlds on demand - turning the simulator itself into a neural network. This is where simulation, generative AI, and robotics converge.

How much data does this actually take?

Simulation and training are, underneath the renders and the physics, a storage and data-movement problem. Follow the data for autonomous driving and it climbs by roughly three orders of magnitude at every step - from one car, to a fleet, to what a program keeps, to the simulated worlds that dwarf reality, to the corpus that trains the model.

Per-vehicle and fleet figures are widely-cited industry estimates; the Cosmos PB sizes are derived from NVIDIA's stated 20M-hour / 9,000T-token corpus. Mobileye reported the storage totals and Waymo the simulated-mile counts.

Robots learn from the same data pyramid

Embodied AI runs the identical loop - real demonstrations at the top, synthetic data generated in simulation underneath. A single released robot dataset is already the size of a small data lake, and simulation multiplies it further.