KINO-2: Production AI for Real World Manipulation

1 From robot learning to real-world workflows

Bear has more than 16,000 robots in commercial service across many different environments, from hospitality to commercial facilities and warehouses. Running at that scale has shaped how we build navigation and coordinate fleets, and it gave us the data and learning infrastructure we now use to improve robot perception.

KINO-1 established our in-house approach to learning manipulation from demonstrations. KINO-2 extends that work for commercial deployment in two ways: a common framework for training and deploying different model architectures, and an accelerated runtime that runs more capable models onboard. This lets us scale our own models and bring in post-trained foundation models such as NVIDIA Isaac GR00T as customer workflow demands grow.

A customer workflow rarely ends when a robot reaches its destination. Goods may need to be picked, packed or replenished using equipment already on site. KINO-2 teaches these skills through demonstrations and simulated practice, cutting the custom programming each new workflow would otherwise need.

Take a warehouse packing workflow. Learned manipulation picks bottles and fills crates, while Bear’s mobility and workflow software handles crate transport and the sequence of work. Customers need the complete task done, not a single skill, so the complete task is what we build and measure.

Figure 3. Configuration of Bear’s in-house manipulation model

The illustrated in-house configuration predicts 40 action steps at 10 Hz, representing a four-second horizon. KR1 executes one 0.1-second step before replanning from updated observations. This lets subsequent predictions respond to changes such as a bottle shifting during contact. The observation history and execution horizon are specific to this model configuration.

Supporting larger model architectures

KINO-2 supports both single-task and multi-task models within the same learning and deployment infrastructure, including, our in-house models and post-trained foundation models. We match the model to the workflow’s required behavior and onboard compute, introducing larger architectures as customers need broader capabilities.

Supporting different models requires more than replacing a checkpoint. Each retains its own encoders and action-generation architecture. We adapt its data representation and robot interfaces, keeping the meaning of observations and actions consistent from training through execution.

That flexibility depends on two foundations: task data that captures real-world variation, and an onboard runtime that turns predictions into timely robot actions. The next two sections cover each.

3 Scaling robot learning through data and simulation

A skill that only works on its recorded example is not useful to a customer. We start with demonstrations of the customer’s actual task, then broaden that experience through visual augmentation and simulated interaction. Each method plays a distinct role within the same learning pipeline.

Starting with real task demonstrations

Demonstrations capture how the work is done. Through teleoperation and other supervised workflows, we record camera observations alongside robot state and operator actions. Synchronizing these records preserves the relationship between what the robot sees and how it moves, including the timing of grasp and release. The recordings supply training and post-training examples for the selected model.

Broadening visual variation with synthetic data

A task can look different under new lighting or against a different background. Our augmentation pipeline uses NVIDIA Cosmos open world models to restyle demonstration videos while retaining their associated action, state and timing data. The purpose is to vary appearance while preserving the demonstrated motion so that the action labels remain usable. This paper describes the data-generation capability; it does not quantify a task-performance gain from Cosmos augmentation.

Parallel training and testing in simulation

Simulation lets a model learn through interaction, including attempts that do not succeed. Parallel environments make it practical to vary task setups, repeat difficult cases and develop recovery behavior under controlled conditions.

We train KR1 policies in NVIDIA Isaac Lab, an open training and evaluation framework, running 1,024 simulated KR1 instances in parallel in the pick-task configuration. That gives the learning process over a thousand copies of the task at once, without requiring a thousand physical robots.

In one simulated pick-task training run, time per pick halved, from 7.4 to 3.7 seconds, as training progressed. The model used simulator-provided states as input, so the result shows improvement within that training environment; physical-robot tests add camera observations, contact and real execution timing.

These data sources feed a one development loop.. Simulation provides controlled practice at scale, while hardware tests and customer pilots surface the real-world conditions that the next training cycle needs to address.

4 Accelerating KINO-2 with NVIDIA Isaac OS

A larger model is useful on a robot only if the complete system can keep up. On KR1, learned manipulation shares an NVIDIA Jetson AGX Thor with the robot’s existing autonomy software, so every millisecond counts. NVIDIA Isaac OS, an open-source, deterministic runtime, forms the backbone of our perception runtime. Bear’s engineering around it produced the most significant results in this paper: 4.2× faster inference, Isaac GR00T open VLA model running on the robot and substantially smoother motion, making efficient use of onboard compute from camera input through model inference.

An accelerated perception and inference pipeline

Isaac OS is built on NVIDIA Holoscan SDK (HSDK) 5.0. We use HSDK to connect camera ingestion to observation processing and inference in an accelerated GPU pipeline. Frames remain in GPU memory through preprocessing and model execution, while NVIDIA TensorRT optimizes execution of the model graphs.

Integrating the runtime with KR1

We custom-designed KR1’s Thor carrier board around this pipeline, replacing an earlier FPGA-based camera-board approach with an integrated design. Owning both the hardware and software lets us optimize the sensor-to-model path as one system. Early access to NVIDIA technologies and direct engagement with its engineering teams supported rapid iteration.

NVIDIA Isaac Bridge connects the Isaac OS/HSDK pipeline to our existing robotics stack, so the accelerated runtime works with the mobility and workflow KR1 already uses. Proven robot capabilities stay in place while allowing new models and their predicted actions to plug into the same task and control interfaces.

Accelerating the in-house model

Moving our in-house model into the HSDK 5.0 pipeline reduced reported inference time from 72.3 to 17.2 milliseconds—a 4.2× speedup over PyTorch eager execution. That headroom is what lets KR1 run larger post-trained foundation models on the same onboard hardware.