When people picture training data collection they picture cameras. In practice, cameras are the easiest decision we make. A capture programme is a pipeline with roughly nine stages, and the quality of the final dataset is set by the least disciplined stage, not the most expensive lens. Here is how ours runs, in our purpose-built studios in Asia Pacific, in the order the work actually happens.
It starts with task specification, not equipment
Every programme begins with task cards: written specifications that define the task family, the objects involved, the variation axes we will sweep, and, critically, what counts as success before a single frame is recorded. A card for a pouring task fixes the containers, the fill levels, the target vessels, the distractor objects, and the grading rubric. Ambiguity here becomes ambiguity in the labels, and label ambiguity is unfixable downstream. We would rather spend a week arguing about a rubric than a month re-annotating.
The task card also fixes episode counts per variation cell, which is how coverage gets designed rather than hoped for.
The rig
A standard station carries three classes of sensing. A head-mounted egocentric camera records approximately what the demonstrator sees. Wrist cameras on both arms record close-range contact detail through the occlusions that blind every other view. An overhead multi-view array around the station records full scene state for verification, annotation grounding, and reconstruction. We explain why the first-person streams matter so much in our note on egocentric manipulation data; the short version is that head and wrist views are the observations a deployed robot will actually have.
Lighting is fixed per station and varied only on schedule, as its own variation axis, never by accident.
Synchronisation, the part that decides everything
Multi-camera data is only as good as its time alignment. Where the hardware supports it we trigger cameras from a shared signal; where it does not, every stream is timestamped against a shared clock and residual drift is measured, not assumed. Each session opens and closes with a common visible event across all cameras, which gives us a hard check that alignment held for the whole session, not just at power-on.
The tolerance we hold is sub-frame, and the reason is simple arithmetic: at 30 frames per second, one frame of misalignment is 33 milliseconds, and a moving hand covers real distance in 33 milliseconds. At the moment of contact, that error is the difference between the wrist view confirming a grasp and contradicting it.
Calibration
Intrinsics are calibrated per lens configuration and re-done whenever a lens or focus setting changes. Extrinsics, the geometric relationships between cameras, are swept with calibration targets at the start of every session, because they silently invalidate themselves: a nudged mount or a re-rigged overhead arm changes the geometry without changing anything visible in the footage. Our rule is that any physical contact with a mount triggers a re-sweep, no debate. Calibration artefacts ship with the dataset so a client can verify rather than trust.
Running a session
Sessions are run to a variation schedule: which layouts, which object instances, which lighting states, which start conditions, in which order. Demonstrators are briefed on the task card and, just as importantly, rested on a fixed cadence. A tired demonstrator does not fail loudly; they drift into shortcut strategies, and a corpus quietly fills with degenerate technique. Session leads watch for this the way a factory watches tolerance drift.
Episode segmentation
An episode is one attempt at one task, and its boundaries are a convention we enforce physically: every episode begins and ends with hands at a defined neutral position. Cuts happen at task boundaries, never at convenient moments in the footage. This sounds pedantic until you train on the alternative. Ragged, inconsistent episode boundaries teach a policy that tasks begin and end arbitrarily, and the damage is spread so evenly through the data that it is nearly impossible to diagnose afterwards.
Quality control before labelling
Raw sessions pass through automated checks first: dropped-frame detection, blur metrics, sync drift verification, duration outliers, missing streams. What survives goes to human review against the task card, where protocol violations that software cannot see get caught: a crew member's sleeve in frame, a skipped variation cell, a reset done sloppily. Episodes fail QC and are cut; a real rejection rate is part of an honest pipeline, and we regard vendors who claim otherwise with suspicion. What separates a usable episode from noise is a long enough topic that we keep a separate note on data quality.
Labelling
Labelling happens against the taxonomy fixed in the task card, in three layers. Episode-level metadata: task instruction, object identifiers, rig configuration, session and demonstrator identifiers. Phase segmentation: the episode divided at observable events, reach, grasp, transport, place, using a controlled vocabulary rather than freehand descriptions. Outcome annotation: graded success against the rubric, and for failures, a failure class, because a labelled failure is training signal while an unlabelled one is contamination.
Packaging and delivery
Everything ships as RLDS episodes: per-step observation dictionaries carrying all camera streams and tracked poses, action fields, and the metadata layers above, with a schema document as part of the deliverable. Programmes deliver a sample batch first, sized for the client to run their own sync checks and training probes before the main volume flows. Acceptance criteria are agreed in writing at scoping, which keeps everyone honest, including us.
That is the pipeline. None of the stages is difficult to describe and every one of them is easy to do badly, which is, in the end, why purpose-built studios exist.