Introducing Perceptron Egocentric API

Perceptron Egocentric turns raw robot and egocentric video into structured, policy-trainable supervision: temporal segmentation into atomic manipulation events, self-contained subtask labels, and dense per-hand grounding signals.

Today we are opening early access for robotics labs, researchers, and data providers building large-scale annotation pipelines.

Why it matters for robotics labs

Policy training is bottlenecked on supervision. Video is cheap to collect and expensive to label: human annotation runs roughly $50 per video-hour, and automated pipelines built on general-purpose VLMs produce flat text labels with weak temporal boundaries and no grounding in what the hands did. Perceptron Egocentric closes that gap. It outperforms Gemini Robotics ER-1.6 and Gemini 3.5 Flash, the previous state of the art for this domain, and has been tested extensively on our internal use-cases. Given an episode, it segments the demonstration into atomic manipulation events and labels each with an actionable, self-contained description, with or without the task instruction. Its capabilities include:

• Dense hand-annotation. Perceptron Mk1 hand detection and pose-estimation sidecar tracks both hands through the video, producing per-frame bounding boxes and full 21-keypoint skeletons (wrist plus four joints per finger) with left/right identity:Manipulation boundaries follow hands, not pixels, so the segmenter gets a direct signal for where contact, grasp, and release actually happen.Each hand is captioned independently into a pre-defined action manipulation taxonomy (reaching, grasping/pinching, lifting, holding, placing/inserting, pushing/pulling, rotating, opening/closing, releasing, etc) plus visibility state.

Dense hand-annotation.

• Perceptron Mk1 hand detection and pose-estimation sidecar tracks both hands through the video, producing per-frame bounding boxes and full 21-keypoint skeletons (wrist plus four joints per finger) with left/right identity:

Perceptron Mk1 hand detection and pose-estimation sidecar tracks both hands through the video, producing per-frame bounding boxes and full 21-keypoint skeletons (wrist plus four joints per finger) with left/right identity:

• Manipulation boundaries follow hands, not pixels, so the segmenter gets a direct signal for where contact, grasp, and release actually happen.

Manipulation boundaries follow hands, not pixels, so the segmenter gets a direct signal for where contact, grasp, and release actually happen.

• Each hand is captioned independently into a pre-defined action manipulation taxonomy (reaching, grasping/pinching, lifting, holding, placing/inserting, pushing/pulling, rotating, opening/closing, releasing, etc) plus visibility state.

Each hand is captioned independently into a pre-defined action manipulation taxonomy (reaching, grasping/pinching, lifting, holding, placing/inserting, pushing/pulling, rotating, opening/closing, releasing, etc) plus visibility state.

• Granular sub-task annotation.Atomic actions taken by either a robot or human within videos are split up, captioned and verified producing significantly higher quality annotations.

Granular sub-task annotation.

• Atomic actions taken by either a robot or human within videos are split up, captioned and verified producing significantly higher quality annotations.

Atomic actions taken by either a robot or human within videos are split up, captioned and verified producing significantly higher quality annotations.

• Instruction-optional operation. The system runs in two profiles: with the episode's task instruction as context (serving a set of needs where the task is known ahead of time), or fully blind with no instruction at all.

Instruction-optional operation. The system runs in two profiles: with the episode's task instruction as context (serving a set of needs where the task is known ahead of time), or fully blind with no instruction at all.

Core results

WGO-Bench ("What's Going On") is Macrodata's benchmark for automated subtask annotation: episodes spanning HomER (egocentric), DROID (external robot camera), and Galaxea (robot head camera), with 743 manually annotated segments across 62 task instructions over 71.5 minutes of video.

Evaluation against WGO-Bench delivered two core results:

• Our with-instruction profile is 10-15x cheaper than human annotation. Macrodata benchmarks human annotation at roughly $50 per video-hour.

Our with-instruction profile is 10-15x cheaper than human annotation. Macrodata benchmarks human annotation at roughly $50 per video-hour.

• With instructions, Perceptron Egocentric delivers +77% end-to-end F1 while costing less than Macrodata's full pipeline.

With instructions, Perceptron Egocentric delivers +77% end-to-end F1 while costing less than Macrodata's full pipeline.

Cost vs quality on WGO-Bench (semantic end-to-end F1, standard list pricing). Left — with the task instruction: the Pareto frontier (dashed) runs from WGO's one-pass at the low-cost end to Perceptron Egocentric at the high-quality end; WGO's seeded-relabeling pipeline sits inside it. Right — no instruction: WGO's seeded variant holds the low end; both Perceptron profiles deliver strictly higher quality, up to 0.225 end-to-end F1.

Head-to-head on WGO-Bench

WGO rows are from Macrodata's final end-to-end evaluation table (Gemini 3.5 Flash, instruction provided). Their one-pass variant labels segments in the same pass that produces them, which is the like-for-like comparison with our single-pass pipeline. Their seeded relabeling variant adds a second, per-segment Gemini labeling pass on top.

Headline deltas, Perceptron Egocentric (with instruction) vs WGO one-pass:

• Semantic end-to-end F1: 0.280 vs 0.158 (+77% relative). This is the metric that measures the finished product: correctly bounded and correctly labeled subtasks. We produce 181 fully correct segments out of 743 gold versus ~101 (recall 0.136) for WGO.

Semantic end-to-end F1: 0.280 vs 0.158 (+77% relative). This is the metric that measures the finished product: correctly bounded and correctly labeled subtasks. We produce 181 fully correct segments out of 743 gold versus ~101 (recall 0.136) for WGO.

• Segment F1: 0.370 vs 0.302 (+23% relative), with segment precision 0.435 and recall 0.322 (239 of 743 gold segments matched at IoU ≥ 0.75 from 550 predictions).

Segment F1: 0.370 vs 0.302 (+23% relative), with segment precision 0.435 and recall 0.322 (239 of 743 gold segments matched at IoU ≥ 0.75 from 550 predictions).

• Semantic precision 0.330 vs 0.190 and semantic recall 0.244 vs 0.136. More of what we emit is right, and we recover ~80% more of the gold subtasks end-to-end.

Semantic precision 0.330 vs 0.190 and semantic recall 0.244 vs 0.136. More of what we emit is right, and we recover ~80% more of the gold subtasks end-to-end.

• Without the instruction, the best-accuracy profile still beats the previous SOTA pipeline on segmentation and end-to-end. The system recovers what's going on from the video alone.

Without the instruction, the best-accuracy profile still beats the previous SOTA pipeline on segmentation and end-to-end. The system recovers what's going on from the video alone.

• The fast-inferred no-instruction profile is the cost/latency option, not the benchmark-max profile: it trims no-instruction cost by ~30% and lands at 0.182 end-to-end F1, which is still ahead of WGO's no-instruction seeded variant (0.138) on quality.

The fast-inferred no-instruction profile is the cost/latency option, not the benchmark-max profile: it trims no-instruction cost by ~30% and lands at 0.182 end-to-end F1, which is still ahead of WGO's no-instruction seeded variant (0.138) on quality.

The seeded-relabeling variant buys 78.1% label accuracy at the cost of an additional Gemini pass over every segment, and it still lands at 0.168 end-to-end — 40% below even our no-relabeling, with-instruction result (0.280). Conditional label accuracy is also easier at low recall: it is scored only on the 0.144-recall slice of segments their pipeline manages to match.

Why it wins

Perceptron Egocentric is built on Mk1, our frontier perceptive model. Mk1 is trained to perceive the physical world directly: hands, contact, object state, and spatial relationships are native concepts in its representation, learned at pre-training scale.

None of the annotation features were included in the training. Mk1 was never fine-tuned on the harness, the taxonomy, or WGO-style episodes. The pipeline is an inference-time framework built around a general embodied model. The results shown above, including the ability to recover task structure from video alone without any instruction, are natural outcomes of advanced embodied reasoning.

General-purpose VLM pipelines sample frames and guess what happened; Mk1 perceives the manipulation as it happens. That gap is the model, not the pipeline, which is why it holds without task-specific training and will widen as Mk1 does.

Notes on evaluation and costing

Scoring protocol (identical for all rows below): a predicted segment matches a gold segment when temporal IoU ≥ 0.75; label correctness is judged by the benchmark's prescribed LLM judge (the same judge model as Macrodata's published scoring); semantic end-to-end requires a prediction to pass both the boundary and the label test. All results are on the full benchmark with the task instruction provided (except where marked "no instruction").

Not included: WGO-Bench judge scoring (an evaluation cost, not an annotation cost; both systems are scored the same way), and the hand-pose estimation stream, which runs on our own GPUs rather than a metered API. Enabling it adds roughly ~$1 per video-hour. Figures are estimates from logged usage, not a billing export.

Sources

• WGO numbers: Macrodata's public blog post, including final end-to-end evaluation table (one-pass and seeded-relabeling variants), per-dataset segmentation breakdown, and per-stage cost breakdown. Their pipeline receives the episode instruction.

WGO numbers: Macrodata's public blog post, including final end-to-end evaluation table (one-pass and seeded-relabeling variants), per-dataset segmentation breakdown, and per-stage cost breakdown. Their pipeline receives the episode instruction.

• Perceptron numbers: full WGO-Bench runs (July 2026), all episodes completed, scored under the benchmark's published protocol (IoU ≥ 0.75 temporal matching and the benchmark's prescribed LLM label judge).

Perceptron numbers: full WGO-Bench runs (July 2026), all episodes completed, scored under the benchmark's published protocol (IoU ≥ 0.75 temporal matching and the benchmark's prescribed LLM label judge).

• Costs: computed from logged model usage at standard provider list rates as of July 2026; batch discounts excluded for both systems. Evaluation and judging costs excluded for both systems.

Costs: computed from logged model usage at standard provider list rates as of July 2026; batch discounts excluded for both systems. Evaluation and judging costs excluded for both systems.

We are partnering with robotics labs, researchers, and data providers on large-scale data annotation.