Egocentric manipulation data is video and sensor data recorded from the point of view of the person performing a manipulation task. In our studios that means a head-mounted camera that sees approximately what the demonstrator sees, wrist-mounted cameras on one or both arms, and hand pose tracking, all time-aligned into a single episode. The demonstrator folds a shirt, threads a buckle, pours from a carafe, and the recording preserves the task the way an embodied agent would experience it.

The term gets used loosely, so it is worth separating it from its neighbours. Third-person data is filmed from tripods or a fixed array looking at the scene. Teleoperation data is recorded through a robot that a human is driving. Egocentric human demonstration sits between them: real human dexterity, at real human speed, captured from the actor's own frame of reference.

Why the first-person view matters for robot learning

A deployed robot does not observe the world from a tripod across the room. It observes from cameras on its own head and wrists: close range, hands permanently in frame, targets partially occluded by its own arms. Policies learn best from observations that resemble the observations they will act on. Train on third-person footage and you introduce a viewpoint gap the model has to bridge on its own; train on egocentric footage and the distribution already looks like deployment.

There is a second, less discussed reason. In manipulation, the information that matters is concentrated in the last few centimetres of a reach: the alignment of fingers with a button, the angle of a container lip as a pour begins. From across a room those moments occupy a handful of pixels. An egocentric rig puts the sensor where the information density is.

What the head view carries

The head camera records context and intent. It shows which object the demonstrator addresses next, where the free hand is staged, and how the task decomposes into phases. It also captures a subtle but valuable signal: people fixate the target of a reach before the hand arrives, so the head view telegraphs upcoming actions a beat before they happen. When our annotation team segments an episode into phases, they can usually do it from the head view alone.

What the head view cannot reliably tell you is whether a grasp was clean. By the time contact happens, the demonstrator's own forearm is frequently in the way.

What the wrist view carries

Wrist cameras exist for the moment of contact. They record close-range geometry as fingers meet the object: seam alignment on a garment, the exact purchase of a pinch grasp, whether a fastener seated or skidded. Crucially, they keep recording through the occlusions that blind the head view, which on cloth work is most of the time. Working with deformable objects is where wrist views stop being optional; we cover why in our note on deformable manipulation data.

We treat the two views as different sensors answering different questions, and we feel it immediately when one is missing. Head-only capture gives confident phase labels and guesswork grasp-quality labels. Wrist-only capture gives the opposite.

Egocentric does not mean egocentric only

Every station on our capture floor pairs the worn rig with a calibrated overhead multi-view array. The overhead cameras provide what egocentric video structurally cannot: a full-scene reference for verifying state, grounding annotations, and reconstructing 3D geometry when a client's pipeline needs it. The egocentric streams provide deployment-like observations. They ship together, aligned to one clock, and each makes the other more useful.

Why the industry converged on this

The bottleneck for manipulation policies is not architecture, it is embodied data. Language models trained on the accumulated text of the internet; there is no equivalent internet of skilled hands. The field's response has been large egocentric corpora, with EgoVerse-scale collections now treated as reference points, and the scaling evidence has followed: NVIDIA researchers have reported log-linear scaling between demonstration volume and policy success, meaning each multiplication of usable data buys a steady increment of capability. Reported results like these are why labs that once treated data collection as an afterthought now procure it deliberately.

The operative word is usable. Scaling curves are drawn through data that was captured, synchronised, and labelled well enough to train on, which is where studios earn their keep.

How this becomes an RLDS file

RLDS, short for Reinforcement Learning Datasets, is the episode format most manipulation labs now expect. An episode is an ordered sequence of steps. Each step holds an observation dictionary containing every camera stream and tracked pose for that instant, an action field, and per-step flags. The episode carries its own metadata: the language instruction for the task, the outcome grade, the rig configuration, the session identifiers that make an episode traceable back to the hour it was captured.

Labs ask for RLDS for an unromantic reason: their loaders already speak it. The format moves cleanly through TFDS tooling, it is the convention large cross-institution manipulation datasets standardised on, and it reduces onboarding a new data source to mapping a schema rather than writing a parser. We deliver in RLDS by default and treat the schema document as part of the deliverable.

What we sweat on the capture floor

The definition above is the easy part. The value of an egocentric dataset is decided by operational details:

  • Synchronisation. Every stream on a shared clock, with alignment verified against a common event at the start and end of each session.
  • Exposure discipline. Auto-exposure pumping mid-reach ruins the frames that matter most, so we lock exposure per station and light for it.
  • Rigid mounting. A wrist camera that flexes under motion corrupts pose estimation quietly. Mounts are checked every session.
  • Per-session calibration. Intrinsics per lens configuration, extrinsics re-swept whenever anything on the station moves.
  • Start-state protocol. Episodes begin and end from defined neutral positions so boundaries are consistent across thousands of episodes.

None of this is exotic. All of it is the difference between a dataset that trains and a hard drive of footage.