Video data can help train robots, including humanoids, to understand and perform physical tasks. This covers VLA models, which connect visual observations and language instructions to actions, and World models, which learn how an environment changes over time. The public web holds millions of videos of people doing that work, and finding a specific action inside them is hard, because a title describes a video’s general topic while leaving the actions in the footage unnamed. Descriptions and tags are usually no better.
For this study we used Bright Data’s Video Search API. Video search api is a tool that enables you to visually search moments inside web videos.
You hand it a natural-language description of an action, it searches the visual content of public videos rather than the text attached to them, and it returns candidate videos together with timestamps for where the action may appear. Conventional keyword search runs the other way around, matching your words against a video’s title, description, and tags, so the two approaches end up reaching very different footage.
We sent it 141 physical actions and got back 10,828 moments across 7,549 videos. For each retained moment, we checked whether any word of the action’s name appeared in its video’s title, description, or tags. In 90.8% of cases, none did.
If you collect training footage by keyword, that 90.8% is the data you miss, and it makes the API’s harvest 10× larger than what text search could reach. Those extra moments show the same actions you asked for, inside videos whose titles talk about something else, and they stay invisible to a text pipeline no matter how many queries you run against these fields.
We matched literal substrings against title, description, and tags, and we counted inside the corpus the API itself returned, so the ratio holds under those two conditions. Semantic search, multilingual search, transcript search, and YouTube’s own ranking sit outside what we tested.
The result in three numbers
| 10.9× | more moments than literal match | content search returned 10,828; 992 carry any word of the action’s name in their metadata |
| 90.8% | named nowhere | 9,836 of 10,828 moments carry no mention of the action in their title, description or tags |
| 56.1% | no clue in the title at all | a weaker test than search. We asked google/gemini-2.5-flash whether anything in the title relates to the action at all, however faintly, including clues far too vague to search on. It found nothing in 6,071 of 10,828 |
Where these three numbers come from
The first two are the same division: 9,836 of 10,828 moments carry no word of the action in any field, and the 992 that do are what a literal matcher reaches, 10,828 ÷ 992 = 10.9. The four buckets (title, description, tags, none) are mutually exclusive and add up to the checked total. Both are counted per match row; collapse them to the 8,849 distinct video-seconds instead and the ratio comes out at 8.9×.
The third counts the titles the model rated unrelated: 6,071 of 10,828, one row per moment across 7,549 distinct videos.
The timestamps carry a second consequence. Allow each moment at most 10 seconds of action and the 5,066.5 h of source video these matches sit inside becomes 24.58 h of clip, which cuts what has to be moved, stored and reviewed by 206.1×.
What retrieval actually returned. Retrieval being confident is not the same as retrieval being right, so we pulled a real frame from almost every match and had a vision model grade all of them. Section 3.5 has the rates. This audit surfaces candidate footage for discovery and curation, and no policy, world model or robot was trained on it.
Sample of six videos and their titles on YouTube
What a vision model saw, shown the image without the title, against the video’s own title, translated into English where a source language is named. An AI judge read every title in the corpus for any hint of its action, and it rated these six unrelated: a bare date, a Quran verse, an economy news bulletin. We confirmed each frame by eye and each video is live. Click a frame to play the video at that second.
| The moment | What the frame shows | What the video is called |
|---|---|---|
 mix_cement |
Person shoveling cement in mixing tub | Tawbah surah verse 26. “But (even so) where are you going?” (from tr) |
 refuel |
Hand inserting fuel nozzle into car filler | Bangladesh economy reform challenges and IMF message | Bangladesh Economy | IMF (from bn) |
 fold |
Two hands folding green paper in half | December 24, 2023 (from zh) |
 cut_knife |
Hand slicing avocado on chopping board | Food Grown in Moon Soil… Scientists Got Scared? (from hi) |
 emergency_stop |
Hand striking red mushroom emergency stop button | (Cheongju City/Main Line) Woojin Sanjeon Apollo 1100 Electric Ultra-Low-Floor Model PEV Standard 2024 June Edition 2027 Dongil Transportation 712 (from ko) |
 chisel_carve |
Hand guiding chisel into wood carving | adapt and overcome 🌱 |
90.8% of retrieved moments are named nowhere in their metadata
That share is the gap between what content retrieval reached on this corpus and what literal metadata matching would have reached.
The obvious objection is language, since we test English action words against titles that are often not English. Restricting to model-confirmed English titles still leaves the figure at 85.8%; Section 3.3 shows every slice.
Where the action is named, over all 10,828 retrieved moments
Table format
| Bucket | Moments | Share | What it means for a text pipeline |
|---|---|---|---|
| named in the title | 386 | 3.6% | literal matcher finds it |
| description only | 531 | 4.9% | only if you read descriptions |
| tags only | 75 | 0.7% | only if you read tags |
| named nowhere | 9,836 | 90.8% | literal matcher does not find it |
One real moment from each bucket, because “531 moments are named only in the description” is a claim and an actual title with the matched word marked is evidence:
| Bucket | Action | Video title (opens at the matched second) | Match |
|---|---|---|---|
| named in the title | pour |
Teaching a beginner how to pour latte art patterns (Expert Barista Guide) | matched on “pour” |
| description only | take |
GENTLE SOY SAUCE SOUP RAMEN (SHOYU RAMEN) | A Real Japanese Dad Cooks | Japanese cooking | matched on “take” |
| tags only | fold |
ورقة واحدة .. طيّات قليلة .. بدون لاصق.. وعالمٌ هرميٌ كامل! | matched on “fold” |
| named nowhere | take |
# viral video in social media # viral koriyaan # sad story of koriyan romance | no word of “take” appears anywhere in its metadata |
Invisible share, by domain
Numbers in table format
| Domain | Named nowhere | Moments checked |
|---|---|---|
| Tabletop (DROID) | 98.1% | 1,836 |
| Outdoor | 95.9% | 217 |
| Whole body | 95.4% | 866 |
| Household | 92.0% | 402 |
| Construction | 91.1% | 987 |
| Industrial | 89.6% | 4,612 |
| Automotive | 83.8% | 1,908 |
Every matched video in every domain carried scraped metadata, so each n is that domain’s full retrieved set. Nobody titles a video “putting a block in a bowl”; repair videos do announce what they fix, which is why automotive is the one domain a literal matcher partly reaches.
1 What a timestamp removes from the pipeline
Say you want ten seconds of someone soldering a joint. The usual route takes five steps, and every one of them runs on a whole video file: find a video by its title, extraction all of it, search your own copy for the moment, cut that moment out, then label what it shows.
Step one misses 90.8% of these moments outright. Searching the footage instead hands back the video and the second inside it, and one of the five steps stops existing. There is nothing to search within the extracted video, since you already know where the moment is. Cutting still has to happen, since Media Data API delivers whole videos, audio, captions and storyboards and takes no timestamp, but you cut against a known offset instead of hunting for one. Five steps become four.
Offline collection pipelines commonly split that work across several systems: moving video into cloud storage, annotating it frame by frame either manually or semi-automatically, and converting between robotics formats such as ROS or MCAP and general-purpose video formats. Each handoff is a place where throughput is lost and where the timing alignment between camera streams and motor commands can drift. We measured none of that here. It is the context that makes a returned timestamp worth having.
Conventional route (five steps, paid on whole videos):
- Find videos · keyword search over titles
- Extract · 5,066.5 h of source video (the expensive step, measured from the scraped duration of every matched video)
- Run a search · a second tool, over your own copy
- Cut · trim each clip to the moment
- Label · annotate what you cut
With moment retrieval (four steps, paid on seconds):
- Find scenes · content search returns the second
- Retrieve · ~24.58 h around the returned offsets (est: assumes each moment is at most 10 seconds of action)
- Cut · trim locally, against a known offset
- Label · one frame per moment
2 Method
We wrote one sentence describing each of the 141 actions and sent them to Video Search API. The soldering one read:
one of the 141 sentences we sent
close-up of a hand holding a soldering iron to a circuit board joint while feeding solder wire with the other hand, bench lamp lighting
We then scraped the title, description, and tags of every video that came back. Bright Data documents the index as matching on sampled frames rather than on title, description or tags, so the fields we graded should not be what found the footage. We did not test that independently.
We drew 22 action names from DROID‘s task strings, which we read for verb frequency, and added the remaining 119 as hand-authored actions to broaden the audit beyond DROID’s coverage.
3 Results
3.1 What each route costs
Everything here is in hours over this corpus, with durations summed from the scraper, so the source figure is a count rather than an estimate. We extracted no video, so we claim no byte count.
Two counts sit behind the 206.1×. The 5,066.5 h is a real sum over 7,549 videos. The clip figure counts each (video, second) once, over 8,849 distinct seconds, and multiplies by the 10-second window.
The ratio rests on two choices, how long a clip is and whether the long tail counts, and the long tail is the bigger lever of the two: 483 videos of three hours or more are ~6% of the set and carry ~60% of the hours. The full sensitivity table:
Numbers in table format
| Drop videos longer than | Videos | Corpus hours | Distinct seconds | Clip hours (10 s) | Reduction |
|---|---|---|---|---|---|
| 1 h | 6,554 | 1,113.5 | 7,605 | 21.12 | 52.7× |
| 2 h | 6,873 | 1,563.8 | 8,017 | 22.27 | 70.2× |
| 3 h | 7,066 | 2,031.3 | 8,247 | 22.91 | 88.7× |
| 6 h | 7,364 | 3,272.8 | 8,609 | 23.91 | 136.9× |
| 12 h | 7,546 | 5,012.3 | 8,846 | 24.57 | 204.0× |
| keep all (published) | 7,549 | 5,066.5 | 8,849 | 24.58 | 206.1× |
The same corpus, obtained either way: 10,828 match rows falling on 8,849 distinct video-seconds. Counting every match instead of each distinct second would give 30.08 clip hours and a 168.4× reduction. The same ratio applies to review time only if the alternative reviews every source video in full, review time scales linearly with duration, and stepping through 8,849 separate clips carries no per-clip overhead.
3.2 The matching rule, and what tightening it does
For each retained candidate moment, we checked the video’s title, description, and tags, the main metadata fields available in this audit.
A moment counts as named if any word of the action’s name longer than two characters appears as a case-insensitive substring, so “change” alone marks change_tire_wrench and “car” is found inside “carpet”. That is the most permissive rule we could have picked, which makes the headline a floor. Tighten it and it climbs:
Every rule stricter than the published one moves the figure up, which is what makes 90.8% a floor rather than a best case: 395 of the 992 moments the published rule counts as named owe that to a fragment inside a longer word. Every value is measured, four real passes over the corpus (supply_gap.py _variants()).
3.3 Much of it is titled in another language
We read a title language for every one of the 10,828 matched moments, so this is the whole corpus rather than a sample. Of them, 4,863 (~45%) sit in a video titled in one of 77 named non-English languages, plus one title the model could not place. The same English action descriptions returned 423 moments from Hindi-titled videos and 509 from Portuguese-titled ones. No geographic field was collected, so this says nothing about where the videos were made.
Title language of every matched moment, across all 79 languages found.
That mix is why we sliced the corpus toward English before trusting the headline number. Restricting to titles we know are English moves the figure by a few points and leaves most of it standing, so part of the gap is a language mismatch and the rest is a description failure that survives after the mismatch is removed:
The same numbers as a table
| Slice | Named nowhere | n |
|---|---|---|
| all moments | 90.8% | 10,828 |
| plain-ASCII titles | 87.6% | 5,560 |
| model-confirmed English titles | 85.8% | 5,965 |
| confirmed English audio | 95.0% | 317 (too few to lean on) |
3.4 Per action
Sorted by the share with no literal action term in the available metadata. 17 actions have no retained moment whose metadata names the target action. Across all 141 actions with at least 8 checked moments the share runs 50.0% to 100.0%, median 94.1%. The 12 least hidden:
Numbers in table format
| Action | Domain | Found | Says so in its text | Named nowhere |
|---|---|---|---|---|
wash_car |
Automotive | 72 | 36 | 50.0% |
wash_car_panel |
Automotive | 78 | 36 | 53.8% |
car_door_open |
Automotive | 92 | 38 | 58.7% |
car_door_close |
Automotive | 91 | 36 | 60.4% |
car_door |
Automotive | 93 | 36 | 61.3% |
pack_box |
Industrial | 73 | 28 | 61.6% |
grinder_use |
Industrial | 69 | 23 | 66.7% |
hand_saw_cut |
Industrial | 77 | 25 | 67.5% |
jack_up_car |
Automotive | 78 | 24 | 69.2% |
tape_box |
Industrial | 75 | 23 | 69.3% |
sit_stand |
Whole body | 28 | 8 | 71.4% |
impact_driver |
Industrial | 70 | 19 | 72.9% |
All 141 actions · found / says-so / named-nowhere, plus the index-wide extrapolation per action
| Action | Domain | Found | Says so | Named nowhere | Est. moments index-wide | Usable rate | Rate basis |
|---|---|---|---|---|---|---|---|
climb_ladder |
Whole body | 93 | 0 | 100.0% | 188,790 | 26.1% | action |
crane_hook |
Industrial | 84 | 0 | 100.0% | 170,436 | 9.5% | action |
flip |
Tabletop (DROID) | 87 | 0 | 100.0% | 176,523 | 26.7% | action |
lift |
Tabletop (DROID) | 89 | 0 | 100.0% | 180,581 | 24.7% | action |
load_cart |
Industrial | 82 | 0 | 100.0% | 166,460 | 9.8% | action |
pick_up |
Tabletop (DROID) | 72 | 0 | 100.0% | 146,088 | 47.9% | action |
push |
Tabletop (DROID) | 91 | 0 | 100.0% | 184,639 | 23.3% | action |
push_heavy_cart |
Whole body | 70 | 0 | 100.0% | 142,100 | 24.3% | action |
push_trolley |
Industrial | 65 | 0 | 100.0% | 131,950 | 20.0% | action |
refuel |
Automotive | 91 | 0 | 100.0% | 184,639 | 26.4% | action |
refuel_nozzle |
Automotive | 88 | 0 | 100.0% | 178,552 | 27.6% | action |
seatbelt_buckle |
Automotive | 87 | 0 | 100.0% | 176,523 | 29.9% | action |
slide |
Tabletop (DROID) | 82 | 0 | 100.0% | 166,378 | 32.1% | action |
sort_parcel |
Industrial | 82 | 0 | 100.0% | 166,460 | 37.8% | action |
stack |
Tabletop (DROID) | 65 | 0 | 100.0% | 136,958 | 21.9% | action |
unfold |
Tabletop (DROID) | 79 | 0 | 100.0% | 160,291 | 57.0% | action |
ziptie_bundle |
Industrial | 82 | 0 | 100.0% | 166,460 | 48.8% | action |
carry_upstairs |
Whole body | 87 | 1 | 98.9% | 176,610 | 25.6% | action |
hang |
Tabletop (DROID) | 92 | 1 | 98.9% | 181,596 | 12.1% | action |
pull |
Tabletop (DROID) | 87 | 1 | 98.9% | 176,523 | 18.4% | action |
pull_trolley |
Whole body | 92 | 1 | 98.9% | 186,760 | 22.8% | action |
close |
Tabletop (DROID) | 83 | 1 | 98.8% | 168,407 | 41.5% | action |
emergency_stop |
Industrial | 85 | 1 | 98.8% | 172,465 | 22.6% | action |
handbrake_pull |
Automotive | 82 | 1 | 98.8% | 166,378 | 31.7% | action |
place |
Tabletop (DROID) | 84 | 1 | 98.8% | 170,436 | 39.3% | action |
remove |
Tabletop (DROID) | 86 | 1 | 98.8% | 174,494 | 48.8% | action |
turn_on |
Tabletop (DROID) | 85 | 1 | 98.8% | 172,465 | 29.4% | action |
count_stock |
Industrial | 76 | 1 | 98.7% | 154,280 | 32.4% | action |
reach_overhead |
Whole body | 79 | 1 | 98.7% | 160,370 | 46.8% | action |
wipe |
Tabletop (DROID) | 77 | 1 | 98.7% | 180,581 | 28.0% | action |
crouch_down |
Whole body | 65 | 1 | 98.5% | 131,950 | 34.9% | action |
seat_chip |
Industrial | 65 | 1 | 98.5% | 131,950 | 46.9% | action |
plaster_wall |
Construction | 88 | 2 | 97.7% | 178,552 | 25.3% | action |
plug_in |
Household | 87 | 2 | 97.7% | 176,523 | 36.5% | action |
turn |
Tabletop (DROID) | 86 | 2 | 97.7% | 174,494 | 27.9% | action |
unplug_cable |
Industrial | 88 | 2 | 97.7% | 178,640 | 42.5% | action |
label_parcel |
Industrial | 85 | 2 | 97.6% | 172,550 | 45.9% | action |
move |
Tabletop (DROID) | 82 | 2 | 97.6% | 166,378 | 34.2% | action |
press |
Tabletop (DROID) | 83 | 2 | 97.6% | 168,407 | 42.2% | action |
spirit_level |
Industrial | 83 | 2 | 97.6% | 168,407 | 12.5% | action |
conveyor_pick |
Industrial | 79 | 2 | 97.5% | 160,291 | 35.4% | action |
grout_joint |
Construction | 81 | 2 | 97.5% | 164,349 | 28.4% | action |
open_hood |
Automotive | 80 | 2 | 97.5% | 162,320 | 53.8% | action |
trowel_mortar |
Construction | 79 | 2 | 97.5% | 160,291 | 28.2% | action |
balance_load |
Whole body | 77 | 2 | 97.4% | 156,310 | 21.1% | action |
shovel_load |
Outdoor | 74 | 2 | 97.3% | 150,220 | 31.5% | action |
scan_barcode |
Industrial | 71 | 2 | 97.2% | 144,059 | 27.1% | action |
desolder_remove |
Industrial | 63 | 2 | 96.8% | 127,890 | 51.6% | action |
pull_wire |
Construction | 61 | 2 | 96.7% | 123,769 | 30.0% | action |
take |
Tabletop (DROID) | 90 | 3 | 96.7% | 182,610 | 37.8% | action |
forklift_controls |
Industrial | 59 | 2 | 96.6% | 119,711 | 27.1% | action |
open_gate |
Whole body | 88 | 3 | 96.6% | 178,640 | 21.6% | action |
pour |
Tabletop (DROID) | 89 | 3 | 96.6% | 183,624 | 56.2% | action |
plug_in_socket |
Industrial | 85 | 3 | 96.5% | 172,550 | 36.5% | action |
shrink_wrap |
Industrial | 81 | 3 | 96.3% | 164,430 | 22.5% | action |
wipe_windshield |
Automotive | 82 | 3 | 96.3% | 166,378 | 34.6% | action |
control_dial |
Industrial | 72 | 3 | 95.8% | 146,088 | 31.9% | action |
steering |
Automotive | 72 | 3 | 95.8% | 146,088 | 59.7% | action |
open_carton |
Industrial | 69 | 3 | 95.7% | 140,070 | 31.9% | action |
put |
Tabletop (DROID) | 91 | 4 | 95.6% | 184,639 | 41.8% | action |
plant_seed |
Outdoor | 67 | 3 | 95.5% | 136,010 | 28.8% | action |
steering_turn |
Automotive | 43 | 2 | 95.3% | 87,247 | 60.5% | action |
pipe_fitting |
Industrial | 83 | 4 | 95.2% | 168,407 | 44.6% | action |
open |
Tabletop (DROID) | 79 | 4 | 94.9% | 173,987 | 44.2% | action |
dig_soil |
Outdoor | 76 | 4 | 94.7% | 154,280 | 44.6% | action |
sweep |
Household | 75 | 4 | 94.7% | 152,175 | 29.3% | action |
route_cable |
Industrial | 74 | 4 | 94.6% | 150,220 | 43.8% | action |
gear_shift |
Automotive | 73 | 4 | 94.5% | 148,117 | 37.0% | action |
stack_pallet |
Industrial | 70 | 4 | 94.3% | 142,100 | 12.9% | action |
thermal_paste |
Industrial | 69 | 4 | 94.2% | 140,070 | 19.1% | action |
ev_charge_plug |
Automotive | 85 | 5 | 94.1% | 172,465 | 26.2% | action |
carry_box |
Household | 84 | 5 | 94.0% | 170,436 | 25.0% | action |
open_trunk |
Automotive | 84 | 5 | 94.0% | 170,436 | 28.6% | action |
socket_ratchet |
Industrial | 82 | 5 | 93.9% | 166,378 | 31.7% | action |
top_up_fluid |
Automotive | 82 | 5 | 93.9% | 166,378 | 37.0% | action |
press_machine_button |
Industrial | 79 | 5 | 93.7% | 160,291 | 31.6% | action |
valve_turn |
Industrial | 79 | 5 | 93.7% | 160,291 | 19.0% | action |
insert_usb |
Industrial | 78 | 5 | 93.6% | 158,340 | 46.0% | action |
caulk_gun |
Industrial | 86 | 6 | 93.0% | 174,494 | 34.1% | action |
adjust_mirror |
Automotive | 85 | 6 | 92.9% | 172,465 | 20.0% | action |
sand_surface |
Industrial | 85 | 6 | 92.9% | 172,465 | 55.3% | action |
cut_pipe |
Construction | 68 | 5 | 92.6% | 137,972 | 41.8% | action |
machine_unload |
Industrial | 68 | 5 | 92.6% | 137,972 | 30.9% | action |
unload_truck |
Industrial | 68 | 5 | 92.6% | 138,040 | 35.3% | action |
machine_lever |
Industrial | 80 | 6 | 92.5% | 162,320 | 21.2% | action |
assemble_snap_fit |
Industrial | 77 | 6 | 92.2% | 156,310 | 51.3% | action |
measure_tape |
Industrial | 76 | 6 | 92.1% | 154,204 | 42.1% | action |
chisel_carve |
Industrial | 49 | 4 | 91.8% | 99,421 | 42.9% | action |
wrench_bolt |
Industrial | 85 | 7 | 91.8% | 172,465 | 34.1% | action |
hammer_nail |
Industrial | 84 | 7 | 91.7% | 170,436 | 20.2% | action |
solder_pipe |
Construction | 70 | 6 | 91.4% | 142,030 | 37.1% | action |
step_over |
Whole body | 70 | 6 | 91.4% | 142,100 | 28.6% | action |
gauge_read |
Industrial | 78 | 7 | 91.0% | 158,262 | 35.9% | action |
fold |
Tabletop (DROID) | 77 | 7 | 90.9% | 158,769 | 57.1% | action |
tile_place |
Construction | 83 | 8 | 90.4% | 168,407 | 30.1% | action |
solder |
Industrial | 55 | 6 | 89.1% | 111,595 | 45.5% | action |
screwdriver |
Industrial | 72 | 8 | 88.9% | 146,088 | 43.1% | action |
bend_lift |
Whole body | 62 | 7 | 88.7% | 125,860 | 32.3% | action |
power_drill_hole |
Industrial | 70 | 8 | 88.6% | 142,030 | 40.9% | action |
solder_joint |
Industrial | 61 | 7 | 88.5% | 123,830 | 55.7% | action |
mix_cement |
Construction | 68 | 8 | 88.2% | 137,972 | 43.3% | action |
nail_gun |
Construction | 68 | 8 | 88.2% | 137,972 | 29.8% | action |
drywall_screw |
Construction | 72 | 9 | 87.5% | 146,088 | 28.2% | action |
lay_brick |
Construction | 80 | 10 | 87.5% | 162,320 | 27.5% | action |
clamp_workpiece |
Industrial | 62 | 8 | 87.1% | 125,798 | 35.5% | action |
machine_load |
Industrial | 62 | 8 | 87.1% | 125,798 | 19.4% | action |
cut_vegetable |
Household | 83 | 11 | 86.7% | 168,407 | 42.2% | action |
torque_wrench |
Industrial | 83 | 11 | 86.7% | 168,407 | 31.3% | action |
cut_knife |
Household | 73 | 10 | 86.3% | 148,117 | 34.2% | action |
tire_pressure |
Automotive | 73 | 10 | 86.3% | 148,117 | 47.9% | action |
weld_seam |
Industrial | 87 | 12 | 86.2% | 176,523 | 31.4% | action |
pliers_grip |
Industrial | 72 | 10 | 86.1% | 146,088 | 50.7% | action |
paint_brush |
Construction | 90 | 13 | 85.6% | 182,610 | 22.2% | action |
wire_cut |
Industrial | 76 | 11 | 85.5% | 154,204 | 34.2% | action |
paint_roller |
Construction | 79 | 13 | 83.5% | 160,291 | 15.2% | action |
breadboard_wire |
Industrial | 65 | 11 | 83.1% | 131,950 | 44.6% | action |
change_tire |
Automotive | 69 | 12 | 82.6% | 140,001 | 50.0% | action |
kneel_work |
Whole body | 55 | 10 | 81.8% | 111,650 | 45.5% | action |
change_tire_wrench |
Automotive | 71 | 13 | 81.7% | 144,059 | 46.4% | action |
assemble_part |
Industrial | 70 | 13 | 81.4% | 142,030 | 64.3% | action |
check_oil_dipstick |
Automotive | 80 | 15 | 81.2% | 162,320 | 43.8% | action |
screw_panel |
Industrial | 81 | 16 | 80.2% | 164,430 | 37.5% | action |
lathe_operate |
Industrial | 51 | 11 | 78.4% | 103,479 | 31.4% | action |
engine_bay |
Automotive | 77 | 17 | 77.9% | 156,233 | 56.6% | action |
probe_test |
Industrial | 63 | 14 | 77.8% | 127,890 | 54.8% | action |
machine_control |
Industrial | 78 | 18 | 76.9% | 158,262 | 36.4% | action |
screwdriver_drive |
Industrial | 78 | 20 | 74.4% | 158,262 | 39.5% | action |
weld |
Industrial | 86 | 22 | 74.4% | 174,494 | 31.4% | action |
power_drill |
Industrial | 70 | 18 | 74.3% | 142,030 | 48.6% | action |
impact_driver |
Industrial | 70 | 19 | 72.9% | 142,030 | 35.3% | action |
sit_stand |
Whole body | 28 | 8 | 71.4% | 56,840 | 35.7% | action |
tape_box |
Industrial | 75 | 23 | 69.3% | 152,250 | 26.4% | action |
jack_up_car |
Automotive | 78 | 24 | 69.2% | 158,262 | 27.3% | action |
hand_saw_cut |
Industrial | 77 | 25 | 67.5% | 156,233 | 32.9% | action |
grinder_use |
Industrial | 69 | 23 | 66.7% | 140,001 | 42.6% | action |
pack_box |
Industrial | 73 | 28 | 61.6% | 148,117 | 50.0% | action |
car_door |
Automotive | 93 | 36 | 61.3% | 188,697 | 20.4% | action |
car_door_close |
Automotive | 91 | 36 | 60.4% | 184,639 | 26.4% | action |
car_door_open |
Automotive | 92 | 38 | 58.7% | 186,668 | 22.8% | action |
wash_car_panel |
Automotive | 78 | 36 | 53.8% | 158,262 | 41.6% | action |
wash_car |
Automotive | 72 | 36 | 50.0% | 146,088 | 45.1% | action |
Per-action findability. “Says so” counts moments whose own title, description or tags carry any word of the action’s name under the published substring rule. The extrapolation columns multiply each action’s measured per-shard yield by the index’s shard count and divide by the measured cross-action overlap; “rate basis” says whether the usable rate was measured on that action’s own frames, inherited from its domain, or inherited from the corpus-wide rate.
3.5 Getting the frames
To check that retrieval was right at all, we pulled a real frame from each matched video and looked at it. What YouTube exposes without extracting the video is the storyboard, a sprite sheet of thumbnails whose sampling interval widens with runtime: we measured 1 s on videos up to 109 s, 2 s from 175 s to 275 s, 5 s from 454 s to 892 s, and 10 s from 1,366 s upward. The step boundaries between those bands were not resolved by the 12 videos we probed. The image we judged is the nearest sample at or before the matched second, so it can be up to one interval early, 10 s in the worst case and 5 s or 10 s for a little over half the frames. Measured by storyboard_probe.py; the per-video specs are in data/storyboard_spec.json.
A vision model judged all 10,750 of 10,828 frames we obtained, so every rate here is measured on 99.3% of the corpus rather than on a sample. 93.8% are real footage rather than animation, a game, or a screen recording; 25.7% caught the action visibly underway and 57.4% showed it underway or set up; 34.5% scored 3 or better on a usability scale of 0 to 5. That last figure is a judgement about a downstream training process the model cannot observe, and it behaves like one: on a 60-frame sample, four runs of the identical configuration spanned 3.4 points, and re-wording the rubric moved it 20. The action-visible share moved 1.7 across that same rubric change, which makes it the figure to compare across runs.
Frame acquisition:
Numbers in table format
| Outcome | Moments | Share |
|---|---|---|
| Frames obtained | 10,750 / 10,828 | 99% |
| No parseable storyboard | 75 | 0.7% |
| Sprite fetch failed | 3 | 0.03% |
Frame acquisition. We fetched storyboards directly and hit rate limiting on 863 of the 10,828 moments, 856 of which a later pass recovered, so none of the 78 lost moments ended on a 429; Media Data API is the supported route for this step and returns storyboards directly. This audit made no call to it. Those moments sit on 7,549 videos, and a video costs one watch-page fetch plus at least one sprite sheet, cached per video.
4 The moments, playable
Matches shown against the nearest storyboard frame at or before the second they were matched. Ranking is not curated. 10,750 of the 10,828 moments have a frame of their own, and the full gallery holds 10,761 rows because 11 more share an image with another action matched at the same second. Those rows rest on 8,792 distinct images, because the filename names a video and a second rather than an action.
Below are the top-scoring matched moments, one per action, straight off the ranking. Click any frame to play the source video at the matched second.
 |
 |
 |
measure_tape · 0.391 · useful 4/5Hand using tape measure on wooden board Shootout review and a work story! |
cut_pipe · 0.388 · useful 2/5Hands holding copper pipe cutter tool How Raw Copper Turns into a Masterpiece with Artistic Hammer Strikes⚒️🔥🇮🇷 |
machine_load · 0.386 · useful 2/5Pink gloved hands arranging white metal small parts THK rail | Installation guide | THK rail | Rail removal and installation | 0363. |
 |
 |
 |
forklift_controls · 0.383 · useful 5/5Hands on forklift controls and steering wheel JUST DRIVE : Forklift #38 |
solder · 0.382 · useful 3/5Hands near circuit board, unclear soldering setup ⚡ Simple DIY Repair Old Bike Battery And Install Separate USB Charging Port For |
steering_turn · 0.378 · useful 5/5Both hands gripping steering wheel, turning 1995 Volvo 850 T5-R Wagon – POV Driving Impressions |
 |
 |
 |
tape_box · 0.376 · useful 1/5Hand near tape rolls and packing supplies Cell 32140 high discharge line – battery pack accessories 0824457979 |
wipe · 0.374 · useful 3/5Hands manipulating white fabric on surface Washed Notebook ASMR 📘✨ Radiator‑Dried Pages With Intense Crinkles |
machine_unload · 0.373 · useful 1/5Gloved hands near industrial machine setup Transmission Assembly – 1929 Ford Model A |
 |
 |
 |
nail_gun · 0.371 · useful 4/5Hands positioning nail gun on timber frame Building a deck for garden raised bed – Part 1 |
sit_stand · 0.371 · useful 5/5Person rising from chair using armrests Flexibility and Mobility Senior Chair Exercise |
count_stock · 0.371 · useful 2/5Hand writing in notebook near clipboard [26.4.16.Thu.] Study with me | Studying together | Thursday meet-up! | Firefight |
 |
 |
 |
probe_test · 0.371 · useful 5/5Two hands holding multimeter probes on circuit How to Measure Electronic Components with a Multimeter (Complete Guide) |
open_carton · 0.369 · useful 2/5Hands holding bottles near box opening, no cutting Oddly Satisfying Video-How to Cutting New Wooden Fruits and Vegetables ASMR – Cu |
kneel_work · 0.369 · useful 3/5Person prone on floor in work position Lupimorphie, animal-performance by Régis Moulu, Capsule 114, dance |
 |
|
 |
push_heavy_cart · 0.367 · useful 4/5Person pushing heavy cart with both hands 2321FI”2-Step Multi-Functional Aluminum Alloy Step Trolley: Comprehensive Usage |
chisel_carve · 0.366 · useful 5/5Hand guiding chisel into wood carving adapt and overcome 🌱 |
lathe_operate · 0.362 · useful 5/5Hand turning lathe cross-slide handwheel with spinning part Unbelievable Damage! Caterpillar Cylindrical Roller Bearing Destroyed 😱 |
Each cell: the action it was matched for and its relevance score, then what the vision model saw in the frame (shown the image without the title), then the video’s own title in English. Timestamps open two seconds early so the moment plays into view.
5 Related work
Retrieving a moment by description is an established task: temporal grounding on Charades-STA, highlight detection on QVHighlights, natural language queries in Ego4D. We introduce no new method here.
The construct has precedent. TVR (2020) labelled its 108,965 queries by whether video, subtitles or both answered them, reporting 74.2% video-only. What differs is the corpus and the purpose: open web video rather than six TV shows, physical actions, a verified subsample, and a cost ratio. Release format follows AVA and HowTo100M: identifier, timestamp and label, with no video redistributed. Retrieval throughout is Video Search API. Frames here were obtained by parsing the YouTube storyboard off the watch page; no clip was extracted at any point. For a production run the supported route is Media Data API.
What the audit establishes is narrower than a replacement for robot demonstrations. The web holds a large and poorly indexed layer of physical-action footage, and timestamped visual search makes that layer considerably easier to reach. Whether footage curated this way measurably improves a robotics or world-model system is the next question, and answering it needs a training experiment this audit did not run.
6 Citation
@misc{webmoments2026,
title = {Finding the Web's Hidden Robot Training Moments in
Public Web Video Data},
author = {Bright Data},
year = {2026},
howpublished = {Technical report},
note = {141 physical actions, 10,828 retrieved match rows over
8,849 distinct video-second moments, and
10,750 frames reviewed by a vision model}
}
