Data hyperscaler for physical AI
Billion Hours records first-person video of people doing real work, on capture headsets we build ourselves, and annotates it in-house. Take the footage raw, take it annotated, or take both.
Every clip moves through the same pipeline. You choose where you take delivery: raw video straight after QA, the same video with annotation, or the annotation as data on its own.
First-person footage of real tasks with hands in frame. Nothing drawn on it and nothing inferred from it.
Used forPretraining, world models, your own labelling
The same clips with annotation aligned to every frame: objects, hand pose, action segments and narration.
Used forPolicy and VLA training, video-language models
{ "schema": "billionhours.track/1",
"clip": "clip-7", "task": "Cleaning",
"setting": "Car detailing bay",
"duration": 6.0, "grid": 1000,
"entities": [
{ "id": "cloth", "kind": "box",
"label": "microfiber cloth",
"path": [
{ "t": 0.0, "box": [680,500,980,950] },
{ "t": 0.1, "box": [659,248,958,696] },
{ "t": 0.2, "box": [564,260,848,686] },
… 61 keyframes, 10 per second
] },
{ "id": "panel", "kind": "box",
"label": "vehicle body panel",
"path": [ { "t": 0.0, "box": [0,180,1000,1000] }, … ],
"states": [
{ "t": 0, "value": "unwiped" },
{ "t": 2.6, "value": "hood wiped" }
] },
{ "id": "rh1", "kind": "hand", "side": "right",
"path": [
{ "t": 0.0, "pts": [
[818,978],[769,930],[746,841],[739,774], wrist, thumb…
[740,732],[774,728],[762,640],[752,595],
[742,562],[800,727],[784,636],[772,589],
[760,552],[826,742],[813,657],[802,609],
[793,573],[851,768],[850,709],[847,673],
[843,644]
] }, … 58 keyframes, 21 joints each
] }
],
"groups": [ { "id": "hand-right", "members": ["rh1"] } ],
"segments": [
{ "from": 0, "to": 1.2, "verb": "wipe", "noun": "hood" },
{ "from": 1.2, "to": 2.6, "verb": "wipe", "noun": "hood edge" },
{ "from": 2.6, "to": 3.4, "verb": "move", "noun": "cloth" },
{ "from": 3.4, "to": 6.0, "verb": "wipe", "noun": "corner panel" }
],
"narration": [
{ "from": 0, "to": 1.2,
"text": "The [right hand](@hand-right) wipes the [hood](@panel) with a folded [microfiber cloth](@cloth)." },
{ "from": 1.2, "to": 2.6,
"text": "The [cloth](@cloth) works up toward the base of the windshield." },
{ "from": 2.6, "to": 3.4,
"text": "The view pans to the front corner of the car." },
{ "from": 3.4, "to": 6.0,
"text": "Flat-palm strokes cover the corner [panel](@panel) with the [cloth](@cloth)." }
],
"reasoning": [
{ "from": 0, "to": 3.0,
"q": "Which hand holds the [cloth](@cloth)?",
"a": "The [right hand](@hand-right), palm flat on the [cloth](@cloth) to spread pressure evenly." },
{ "from": 3.0, "to": 6.0,
"q": "What surface is being cleaned?",
"a": "A wrapped [vehicle body panel](@panel): first the hood, then the front corner." }
] }Each annotation layer as structured data, packaged in the format your loader already reads.
Used forFine-tuning, evaluation sets, grounding and VQA
Joint positions for both hands on every frame, in 2D and lifted to 3D.
Objects, tools and hands localised, outlined and tracked through the clip.
Start and end timestamps for each step of a task, with its verb and object.
Plain-language descriptions of what is happening, tied to the frames they describe.
Question and answer pairs that test what a model understood about the scene.
The moment an object changes: creased to pressed, loose to fastened, soapy to rinsed.
IMU and depth aligned to the video clock, where captured.
Each wearer is contracted directly, and consent for every clip can be verified by both the wearer and you. DPDP and GDPR compliant.
How consent worksDepth, motion and hand tracking recorded alongside the video.
Specs and formatsWe are a data hyperscaler: our own capture headsets and in-house annotation let us record to your spec and deliver faster as volumes grow.
The pipelineSkilled manual tasks recorded where they actually happen: homes, shops, kitchens and factories.
Task catalogLeft of the divider is the footage as recorded. Right of it is what annotation adds. Switch to Raw to see the video on its own, or to Annotated for every layer.
footage loadedfootage coming, placeholder frame
This is the raw deliverable: the clip as recorded, with nothing drawn on it. It ships with its metadata and, where captured, synced IMU and depth.
Point at an underlined phrase to find it in the frame. Point at something in the frame to find it in the text.
The narration and reasoning above are labels written for each clip. We are building a model layer that produces the same thing directly from egocentric video: it detects, points, describes and answers questions about a task as it happens. The labelled clips are its training target.
People wear our headsets while they do their normal work. Each wearer is contracted directly and consents clip by clip.
OutputRaw video, sensor logs
Clips are screened for framing, hands in view and usable motion, then reviewed before they move on.
OutputQA-passed hours
Hands, objects, actions, language and state changes are labelled against one shared clock.
OutputAnnotation layers
Grounded narration and reasoning turn a labelled clip into footage a model can be asked about.
OutputVQA and reasoning sets
Raw, annotated or annotation only, packaged for the stack you train on.
OutputLeRobot, MCAP, HDF5, RLDS, MP4 + JSON
dataset/ # v3 layout meta/ info.json stats.json tasks.parquet episodes/chunk-000/file-000.parquet data/chunk-000/file-000.parquet videos/observation.images.head/ chunk-000/file-000.mp4 dataset/ # v2 layout meta/ info.json episodes.jsonl tasks.jsonl data/chunk-000/episode_000000.parquet videos/chunk-000/observation.images.head/ episode_000000.mp4
clip-000123.mcap /camera/head/image/compressed sensor_msgs/CompressedImage /camera/head/camera_info sensor_msgs/CameraInfo /camera/depth/image sensor_msgs/Image, where captured /imu/data sensor_msgs/Imu, where captured /annotation/entities boxes, points, polygons, hands /annotation/segments action segments /annotation/language narration, reasoning
episode_000123.hdf5 /observations/images/head uint8 [T, H, W, 3] /observations/depth uint16 [T, H, W], where captured /observations/imu float32 [T, 6], where captured /annotation/hand_pose/left float32 [T, 21, 3] /annotation/hand_pose/right float32 [T, 21, 3] /annotation/boxes id, label, [T, 4] /annotation/segments from, to, verb, noun /language/narration string
episode
steps[]
observation
image (H, W, 3) uint8
depth (H, W) uint16, where captured
imu (6,) float32, where captured
hand_pose (2, 21, 3) float32
language_instruction string
is_first / is_last / is_terminal
episode_metadata
task, setting, consent_recordclip-000123.mp4 raw footage clip-000123.json annotation track, all layers clip-000123.meta.json task, setting, duration, sensors manifest.json clip list # clip-000123.json, abridged { "entities": [ { "id": "iron", "kind": "box", "path": [...] } ], "segments": [ { "from": 0, "to": 3.0, "label": "press shirt front" } ], "narration": [ { "text": "The [iron](@iron) crosses the [shirt](@shirt)." } ] }
Example layouts. The final structure follows your spec.
We record where the work already happens: skilled manual tasks done every day, with tools, materials and workplaces that existing egocentric datasets barely cover.
Each task as it is really done, with the tools and materials people actually use.
Homes, shops, kitchens and factories, recorded at volume.
Because we build the capture hardware ourselves, we can record to any format or spec you need, turn orders around faster, and keep things simple on your side.
Tell us what you are training and which deliverable you want to see. We will reply with sample clips and a scope.
We have your sample request. We will reply to your email with sample clips and a scope.