As part of our announcement of Perceptron Egocentric 1.0, we are releasing and open-sourcing EgoHome, a new benchmark of densely human-annotated egocentric video: 137 clips of people performing household tasks, 2 hours in total.
Washing dishesScrubbing the drain mat and basin with a sponge, then rinsing the mat under the tap.
The annotations span four different categories:
- Hand location bounding boxes
- Subtask annotations
- Left and right hand actions (from Ego4D’s closed-vocab schema)
- Left and right hand captions.
We release the raw annotations as well as code for evaluation metrics, which we believe better capture the inherent subjectivity of open vocabulary video description.
A new standard for Egocentric annotations
Recent work has established that high volumes of diverse pretraining data are essential for reliable control: Physical Intelligence has shown that compositional generalization can emerge from diverse multimodal data and conditioning [1], Generalist has shown that task-specific mastery can be achieved by scaling high-fidelity physical-interaction data [2], and Dyna that scaling egocentric data is one viable path to human-level control [3]. Rich egocentric annotation, in particular, is of significant interest in the robotics world due to its scalability [3].
Despite material investment in egocentric data, it is still not well-understood which set of annotations serve as the best pretraining objectives or how best to measure the quality of synthetic labeling for these tasks.
The most widely adopted egocentric benchmark for subtask segmentation, WGO-Bench [4], was a remarkably useful contribution to the ecosystem and has since become a de facto standard for evaluating the quality of a model or harness on describing the events in an egocentric video. Despite this, WGO-Bench has several gaps in coverage.
Perceptron’s EgoHome benchmark addresses gaps in the previous state of egocentric benchmarking in three ways:
- Four annotation types: We introduce benchmarking for (1) hand box location, (2) subtask timelines, (3) hand action classification (from the Ego4D closed vocab), and (4) hand action freeform captions.
- Sub-second annotations: We provide a large number of annotations at the sub-second scale. Specifically, we include 3,371 segments under 1 s and 773 under 0.5 s.
- Permissible subsegment Jaccard measurement: We introduce a new flexible metric we call permissible subsegment Jaccard: an LLM-as-judge based technique which accommodates the fact that there are many permissible ways to break down the granularity of a task.
Benchmark overview
Our benchmark is the first of its kind to combine all four annotation types on every video. This has revealed the spikiness of model capabilities at the frontier, with subtask timeline generation being much stronger than individual hand description for the very best models. We also offer very temporally fine-grained annotations allowing us to analyze how model capabilities break down on much faster action sequences.
Four annotation types
Subtask annotations
Similar to WGO-Bench, we have open vocab descriptions of the high level goal of what task the subject is performing e.g. “scrubbing the stovetop”. The average video has 14.0 segments.
Hand box locations
We annotate every hand box with human annotators at 10 fps then apply proprietary smoothing algorithms to extrapolate up to 30 fps.
Hand actions
We have annotators choose an accurate hand action label from the closed vocabulary Ego4D action schema [5]; there are 118 total actions to choose from (47 occur in the release).
Hand captions
We have annotators write free-form hand captions describing what the hands are doing during the video.
Evaluation metrics
Hand box metric: centroid hit fraction
To measure hand box prediction accuracy, we simply ask each model to produce timestamped bounding boxes for the hand locations throughout the video. For every frame with a human-annotated hand, we check whether the centroid of the model’s box for that hand falls inside the human box, then divide by the number of annotated hand-frames, plus any boxes the model predicts outside the annotated range.
Models that sample fewer frames per second are still scored on every annotated frame: between two of a model’s boxes, we linearly interpolate its prediction. A model sampled at 1 fps therefore only loses credit where the hand moves off the straight-line path between its samples.
Subtask, action, and hand caption metric: permissible subsegment Jaccard
We developed our subtask correctness metric while attempting to find a fair way to calibrate our annotators. We initially used a metric similar to WGO-Bench’s semantic F1, which performs bipartite matching between the target and prediction and then calculating F1 between the predicted and target sets.
Bipartite matching, which is common in object segmentation benchmarks, allows each ground truth segment to be paired with exactly one predicted segment. In other words, during scoring, the mapping between predicted and ground truth is one-to-one.
We’ve found, however, that there are many valid ways to divide up the timeline in a video. Even if you show the same annotator the same video, in one sitting they may break washing the dishes down into a single task or they may break it down into many smaller segments.
Rather than encouraging models to overfit to one particular choice of temporal granularity, we opted to allow the ground truth and predicted segments to match to each other in a many-to-many mapping, and we’ve tailored our LLM as judge prompt to permit predicted tasks to be subtasks of the ground truth or vice versa.
We call our metric permissible subsegment Jaccard.
Worked example: the stairs
Let’s say the ground truth labeled the time span 0 to 5s as one segment, “sweeping the stairs”.
The table below demonstrates how various predictions would be graded under our many-to-many permissible subsegment Jaccard, vs the one-to-one WGO-Bench semantic F1.
Ground truth (human)
Model
Ground truth (human)
Model
Ground truth (human)
Model
Ground truth (human)
Model
While calibrating annotators, we frequently ran into Case B, as illustrated in Table 2: a valid, more detailed breakdown gets a score of zero under one-to-one matching and full credit under ours. Case C shows we still give partial credit for the part the model got right, where a fixed IoU cutoff gives nothing. (WGO-Bench’s semantic F1 pairs segments one-to-one, counts a pair only when its temporal IoU is at least 0.75 and the judge accepts the labels).
We found that this metric adhered much more closely to subjective measures of fairness when evaluating human annotators’ performance during calibration, thus we opted to release it as our official metric for the benchmark.
The formal definitions of both metrics are in the appendix, and the scoring code is in our GitHub repository.
How different models stack up
We compare Perceptron Egocentric, at its default 4 fps and an 8 fps option, with GPT-6 Astra, Gemini 3.8 Flash and Elorian’s Glimpse, each at 4 fps and 1 fps, and with Pantheon’s Argus and NVIDIA’s FLORA. Subtasks, hand actions and hand captions are scored with permissible subsegment Jaccard, and hand boxes with centroid hit fraction; higher is better for both.
The Perceptron Egocentric harness matches or beats every other system on subtask labeling and is close to GPT-6 Astra on hand boxes, but trails it on hand actions and captions. Generally speaking, most systems struggle with identifying individual hand activities, with the 3 hand-related portions of this benchmark quite far from being saturated.
EgoHome results
Fine-grained results by gold segment length
Our benchmark is the only one with sufficient scale of annotations necessary to offer performance breakdowns at varying timescales. This reveals that specialist systems like NVIDIA's FLORA hold up much better at shorter timescales than some of the other models which fare well on the overall metric.
Subtask
Hand action
Hand caption
How we built EgoHome
Annotation methodology
We begin by synthetically labeling videos then having humans edit the initial annotations. Next, we had a member of our research team review every clip by hand. Due to the quality of our initial annotations and our proprietary review pipeline, we were able to get the overall average time to re-annotate a video down to 116 minutes per minute of video across all four annotation types.
Ensuring annotator quality
Every annotator undergoes a calibration where they have to achieve high agreement with a clip labeled by a member of our research staff. They are then allowed to re-attempt the calibration clip up to 3 times, with feedback after each failed attempt. They are allowed to see what parts of their answers were incorrect, but they are not allowed to see the ground truth.
After their initial calibration, crowdsourced annotators then go through a 10-clip calibration against each other, where they are graded by how high their interannotator agreement is compared to their agreement with the original labels. Via this process we chose 5 annotators (from a pool of 50) to serve as the annotators on our evaluation set, keeping lower performing members of our annotation pool to work on other tasks.
Open-source release
The data can be found in the Hugging Face link at the page header and the prompts and metrics can be found in the GitHub repository.
License. The EgoHome dataset is released under the Creative Commons Attribution-NonCommercial 4.0 license (CC BY-NC 4.0). As an additional permission, anyone, including commercial organizations, may use EgoHome to evaluate models and publish the results. Other commercial uses, such as selling the data or training commercial models on it, are not permitted.
Appendix
Metric formulas
Centroid hit fraction
Mathematically this can be expressed as
CHF=∑(f,h)∈G1[Centroid(p^f,h)∈gf,h]∣G∣+∣Pout∣(1)\operatorname{CHF} = \frac{\displaystyle\sum_{(f,h) \in G} \mathbb{1}\left[\operatorname{Centroid}(\hat{p}_{f,h}) \in g_{f,h}\right]}{|G| + |P_{\text{out}}|} \tag{1}
Where GG is the set of human-annotated frames at 30 fps and gf,hg_{f,h} is the ground truth box for hand hh on frame ff. p^f,h\hat{p}_{f,h} is the model’s box for hand hh at frame ff: either a box directly emitted for that frame, or the linear interpolation between its two nearest boxes; frames outside the span the model covered count as misses. PoutP_{\text{out}} is the set of frames for which the model has a box but the ground truth doesn’t. The indicator in the numerator is 1 when the centroid of p^f,h\hat{p}_{f,h} falls inside gf,hg_{f,h}.
Permissible subsegment Jaccard
It can be expressed mathematically as follows:
PSJ(P,T)=∑p∈Pm(p,T)+∑t∈Tm(t,P)∑p∈P∣I(p)∣+∑t∈T∣I(t)∣(2)\operatorname{PSJ}(P, T) = \frac{\displaystyle\sum_{p \in P} m(p, T) + \sum_{t \in T} m(t, P)}{\displaystyle\sum_{p \in P} |I(p)| + \sum_{t \in T} |I(t)|} \tag{2}
m(s,S)=∣ I(s)∩⋃s′∈SJ(s,s′)=1I(s′) ∣(3)m(s, S) = \Bigl|\, I(s) \cap \bigcup_{\substack{s' \in S \\ J(s, s') = 1}} I(s') \,\Bigr| \tag{3}
Let PP and TT be the predicted and ground-truth segments. Each segment ss has a time interval I(s)I(s) and a label, and ∣⋅∣|\cdot| is duration. J(s,s′)∈{0,1}J(s, s') \in \{0, 1\} is the LLM judge’s verdict on whether two labels are permissible matches: the same action, or one a subtask of the other. m(s,S)m(s, S) is the portion of ss’s duration that overlaps some segment in SS that the judge accepts.
Judge prompt
For subtasks and hand captions, an LLM judge (Gemini 3.5 Flash) decides whether two overlapping labels describe the same activity; hand actions are matched on their Ego4D verb. This is the judge’s prompt for subtasks. For hand captions, “subtask descriptions” reads “hand activity captions”. The prompts each model receives are in our GitHub repository.
Degenerate baselines
In order to ensure the metric isn’t trivially gameable, we chose a few degenerate baselines and determined how they'd be rated by the metric. We chose three types of generic baseline:
- Choosing a generic label like "doing chores" and applying it to the whole timeline
- Choosing a generic label and applying at fixed 2.5s windows
- Taking the most common/longest true clip from the ground truth and filling the whole video timeline with it as a single task.
Generic labels earn nothing on subtasks and captions. A constant “hold” earns 0.175 on hand actions, the share of time hands really are holding. These degenerate baselines represent a floor of how good a trivial baseline is at the task. The table below shows which models exceed the trivial baseline on the three tasks.
Annotation statistics
Subtasks
The distribution of durations is below:
Hand boxes
Here is the percentage of time the right and left hand are on screen in the corpus:
Hand actions
Hand captions
Citation
If you reference this work, please cite it as:
Acknowledgements
The raw footage for this data was collected by HumanSignal, a human data provider specializing in high-quality data collection and annotation for AI teams.
References
- Physical Intelligence. A Steerable Model with Emergent Capabilities. April 2026. pi.website/blog/pi07
Physical Intelligence. A Steerable Model with Emergent Capabilities. April 2026.
- Generalist. GEN-1: Scaling Embodied Foundation Models to Mastery. April 2026. generalistai.com/blog/gen-1
Generalist. GEN-1: Scaling Embodied Foundation Models to Mastery. April 2026.
- Dyna Robotics. Dyna-2: A 1-Million-Hour Scaling Law for World-Action Models. August 2026. dyna.co/dyna-2
Dyna Robotics. Dyna-2: A 1-Million-Hour Scaling Law for World-Action Models. August 2026.
- Macrodata Labs. Segmenting Robot Video into Actionable Subtasks. June 2026. macrodata.co/blog/annotating-robot-video-subtasks
Macrodata Labs. Segmenting Robot Video into Actionable Subtasks. June 2026.
- K. Grauman, A. Westbury, E. Byrne, et al. Ego4D: Around the World in 3,000 Hours of Egocentric Video. CVPR 2022. arXiv:2110.07058
K. Grauman, A. Westbury, E. Byrne, et al. Ego4D: Around the World in 3,000 Hours of Egocentric Video. CVPR 2022.
