In our experiments, scaling general video from 1,000 to one million hours cut the teleoperation needed to reach well-calibrated held-out action loss from about 5,900 hours to 28.
Isaac is a 36-billion-parameter sparse model. It reads images, video, language instructions, robot state, and previous actions. It can answer questions about a video, point to objects and parts, track them over time, report task progress, and generate robot actions. Teams can fine-tune it as a policy or use its visual outputs inside a planner, controller, or data engine.
We trained Isaac on data from more than 35 robot systems, 100K hours of robot experience, 1M hours of general video, and 3T multimodal tokens. We co-trained video understanding, spatial grounding, task progress, and robot action from the beginning.
Control-recipe comparison
MODEL
ROBOT TRAINING SCOPE
OBS. STEPS
RTC- TRAINED
PREV. ACTIONS
MISTAKE MODELING
NON-ROBOT VIDEO
FLOW EXPERT
OPEN SOURCE
Isaac 0.5
35+ embodiments
1–3
π0.7
multiple robots
≤6 / camera
π0.5
~7 robots
Qwen-VLA
~10 robots
LingBot-VLA
9 robots
MolmoAct2
~5 emb.
SmolVLA
1 emb.
Octo
25 datasets
OpenVLA
970k traj.
We are releasing Isaac as a complete system: checkpoints, training code, and inference code via LeRobot. The technical report contains the full training and evaluation details.
A scaling law for video and robot action data
Recent work has established both volume and diversity of pretraining as essential for reliable control. Physical Intelligence has shown compositional generalization can emerge from diverse multimodal data and conditioning. Generalist has proven task-specific mastery can be achieved by scaling high-fidelity physical-interaction data. Dyna demonstrated that scaling egocentric data is one viable path to human-level control.
Both video (i.e., general, egocentric) and robot experience (i.e., UMI, teleop) are necessary, but the optimal ratio across them has not been well studied in the open-source, if at all.
Holding a fixed 80:30:30 ratio of general-video:egocentic:UMI, we scaled Isaac’s pretraining and measured action-prediction loss on held out robot trajectories. The resulting law shows that we can trade cheap video data for expensive teleop.
Adding more video to pretraining lowers the teleop requirement. With one hour of teleoperation, increasing video tenfold lowers action loss by about 0.006. Beyond roughly 100 hours of teleoperation, the same increase lowers loss by about 0.21.
This interaction produces the 210× result. To reach action loss 2.50, a model trained on 1,000 hours of video needs about 5,900 hours of teleoperation. A model trained on one million hours of video needs 28.
AFixed-loss contours2.252.502.75210× less teleoperationat loss 2.5028 h≈5,900 h2101001k10k1k10k100k1MTeleoperation hoursGeneral-video hours
BEmpirical loss change from 10× more video1101001k10k−0.20−0.100.00Teleoperation hoursLoss change per 10× video−0.006 at 1 habout −0.21 from 100 h onward
Teleoperation is shown on the horizontal axis and general video on the vertical axis. Each contour connects training mixtures that reach the same held-out action loss.
General video covers many objects, activities, and environments. Egocentric video adds close-up, task-relevant views of hands, motion, and contact. UMI connects visual knowledge to manipulation. Teleop ties all of these learnings to a complete embodiment and action space. We train on these sources together, so all four shape the representation Isaac uses for action.
The scaling law gives teams a practical way to plan data collection, given a fixed budget.
ACost of a fixed composition$100k$10M$1M1101001k10k1k10k100k1M1571.57k15.7k157kTeleoperation hoursGeneral-video hoursH100 h · one epoch
BLoss versus budget$1M · 2.24$100k$1M$10M1.92.12.32.52.72.9Modeled budgetHeld-out action loss
Beyond data mixtures, Isaac 0.5 offers insight into the impact of training objective on scaling.
Semantic world modeling for robotics
Robot models can learn to predict different kinds of futures, and the prediction target shapes what transfers to control. Diffusion and pixel-reconstruction models generate future frames. Joint-embedding models predict learned visual features. Direct action models predict actions.
Isaac 0.5 adds semantic world modeling as a cotraining objective, learning from video by predicting future percepts. A percept is a task-relevant state or change visible in an observation: an object state, a spatial relation, an affordance, a task phase, a visible contact change, or a likely near-future task state.
Percepts sit between pixels and actions. Training does not supervise percepts directly. It supervises semantic descriptions of them, constructed automatically from future observations, including general video carrying no human-annotated action labels.
Given observations through time t, Isaac predicts the percept at a later time t + Δ:
L_percept = E[ℓ(gθ(o≤t), zt+Δ)]
Here, o≤t is the observation history and zt+Δ is the future percept. This trains the same backbone Isaac uses to produce actions.
While the objective is proprietary, it is not the only one we use. We cotrain video perception, embodied visual reasoning, semantic future-percept prediction, and robot control. Every training objective updates the shared backbone. The discrete (FAST) and continuous (Flow) decoders remain distinct, but both read from the same shared representation.
One model, several useful interfaces
Isaac 0.5 unifies images, video, time, robot state, and past actions into a single sequence jointly modeling across all modalities. Unifying this multimodal interface into a single model requires 3 challenges (1) resolving concurrent streams into a single typed format (2) given PB scale datasets a high throughput multimodal data stack and (3) novel architectures that allow dynamic compute and joint perception and action.
SOURCE FAMILIESTYPED COMPILERPACKEDSHARED BACKBONEINTERFACESmHarmonyIsaac 0.5 backboneGeneral videoframes, time, scene textEgocentric + UMIhands, contact, motionTeleoperationcameras, state, actionsLanguage + spatialinstructions, pointsschema validationembodiment loweringclock alignmentTensorStream eventstextvisiontimestateFASTvision encoder30 × GDN10 × full attentionnull-routed MLPsshared expert + MTPPerceptiontext, coordinatesEmbodied reasoningstate, progressDiscrete controlFAST tokensFlow expert36-block DiTContinuous controlaction chunksevery source keepsprovenance, time, andembodimenttyped modality, time,state, coordinates,actionsFigure 4. From heterogeneous experience to one model. mHarmony validates and lowers source schemas into typed TensorStream events before packing.
Effectively modeling multimodal inputs across, image, video, and control trajectories requires serializing to a consistent interface. We develop and release mHarmony, a typed compiler based on OpenAI’s harmony that extends to modeling streams of multimodal events. mHarmony ensures that across our data, pre-training, inference, and RL pipelines data remains in a consistent and optimized format. For model training we convert mHarmony structures into TensorStream - a packed multimodal tensor format.
Scaling up to 1M+ hours of video gets beyond the limits of NFS across datacenters. Doing an epoch over our data is no longer feasible locally and requires a cloud streaming solution however network bandwidth, storage limits, and CPU heavy processing can hinder MFU. Our training data stack builds a topology aware predictive plan ensuring we can efficiently yield batches in training while planning ahead and giving us granular dataset level mixture control.
a Process tiersREMOTETRAINER1/RANKWORKERN/RANKDECODER1/WORKERsliceshard downloadbatch + metadatabytesframesobject storeS3 / GCS shardsDataStreamsmix + transformstraining batchStatefulDataLoaderwork plangenerate_work()node-coherent, panel bshard cacheschedule + fetchRust schedulerDocumentlower + transformidle spans compactedrender→ TensorStreamtyped eventscollatepack + precomputepanel csupervisorforks + restartscodec childnative codecb Node-coherent planrelabeling offw04719w125838 shards for this nodenode-coherentw03388w183832 shardsc Sequence packingrendered documentsbest-fitpadone packed sequence, 16,384 tokens
The current state of control models has been built on smaller dense models due to the real-time requirements, preventing model scaling as an effective solution for control. Mixture of Experts give us a dimension of scaling total parameter counts while controlling inference through the active parameter count. Isaac 0.5 is built on an extension to Mixture of Experts called Null Experts - a new paradigm where a token is capable of selecting a varying number of experts, this gives our model the ability to scale up and down compute depending on task complexity. Visualizing compute maps we see strong saliency patterns for our architectures.
Real routes remain concentrated on recognizable visual structure
patch null fraction0% · 8 real→100% · 0 real
null 62.7%2.98 real
null 72.0%2.24 real
null 42.0%4.64 real
null 64.6%2.83 real
null 34.8%5.22 real
null 68.2%2.55 real
null 67.9%2.57 real
null 56.7%3.46 real
null 72.8%2.17 real
null 46.6%4.27 real
null 45.8%4.33 real
null 30.7%5.55 real
null 44.7%4.42 real
null 24.3%6.06 real
null 67.9%2.57 real
null 61.0%3.12 real
Compute-allocation maps, not salience maps · 16 of 142 candidate renderings, selected for localized contrast rather than task balance or prevalence
Isaac 0.5 supports both discrete and continuous control from the same shared backbone. The discrete path uses FAST to compress action trajectories into tokens, letting the model predict robot actions with the same autoregressive machinery it uses for language and grounding. In parallel, a dedicated Flow expert generates continuous action chunks by iteratively transforming noise into a trajectory conditioned on the backbone’s representation of the scene. Supporting both gives the model two complementary action interfaces: FAST provides a simple, unified token-based representation that integrates naturally with the VLM, while Flow preserves the precision and multimodality of continuous robot control. Crucially, both read from the same perception and reasoning backbone, so what the model learns from images, video, and embodied reasoning can directly shape its actions.
Evaluation across perception and control
We evaluate the same Isaac checkpoints on multimodal video understanding and spatial grounding, action prediction on held-out trajectories, and closed-loop robot control. Together, these evaluations connect what the model sees and predicts to how it behaves in a task.
On grounding and counting, Isaac 0.5 reaches 62.6 on ScreenSpot-Pro, 32.8 on LVIS Count, and 19.1 on CARPK, against 54.8, 28.7 and 6.0 for the strongest Qwen3-VL run measured under the same harness. Across all five task families we evaluate — physical and temporal reasoning, spatial and embodied reasoning, structured visual understanding, general visual intelligence, and grounding and counting — no open comparator matches it, and it leads each at 8.5× lower inference cost than the strongest of them: 26.9 TFLOP for one three-image request against 228.4.
Shared weights show up in how fast Isaac picks up a new task. On a physical chess-manipulation benchmark, we finetune each policy for one epoch on a single expert episode and test on a held-out episode: a perturbed piece pose, a different move given in chess notation, or a different branch of the same opening line. Isaac 0.5 records the largest loss reduction at all three levels, 10.5x on the fixed move, 9.5x under notation conditioning, and 7.0x on the defensive line, with π0.5 closest at 3.1x, 2.6x, and 2.3x. Individual margins sit close to the seed-to-seed standard error, so the result to read is the ordering, which holds at every level. These are the same weights that produce the perception scores above.
Aone epoch on one episode0.40.20.10.050.025held-out action lossSmolVLAGR00T N1.7MolmoAct2π0.5Isaac 0.5fixed movenotationdefensive lineeach pair: zero-shot → after one epoch · log scale, one tick = 2×Badaptation gain from one demonstration8×4×2×1×× lower error3.1×10.5×2.6×9.5×2.3×7.0×fixed movenotationdefensive lineIsaac in orange, baselines in grey · higher is betterCwhere in the chunk the gain lands (notation)not scoredmean-action predictor100140047007100010120012msH0.10.20.30.40.5held-out action losszero-shotafter one epochSmolVLA50GR00T N1.716MolmoAct2n/dπ0.5n/dIsaac 0.550prediction offset (action row, 10 Hz grid) · scored to Heval = 10, RTC delay trained to D = 12
Building with Isaac in the Open
We see open-source models as essential drivers of robotics progress. This is why we continue to open source our work.
Isaac 0.5 is available today as a starting point for embodied systems. You can fine-tune it on your own demonstrations, use its visual outputs inside a larger system, and run it yourself via LeRobot.
If you’d like to learn more about deploying Isaac on your infrastructure, please reach out to support@perceptron.inc.
