From web video to robot control: Lessons from two robotics meetups

OpenMind forward-deployed engineer and head of developer relations Prachi Sethi described a familiar deployment problem: Her team ran robots for days in the office, then encountered new failures at a customer’s site.

Sethi attributed those failures to differences between the development and deployment environments. Researchers investigating such failures need to examine the training data alongside the hardware and software.

We discussed these problems at two events at Bright Data’s Web Data Loft in San Francisco. On Nov. 4, we hosted Robot Learning #02: World Models & Dexterous Manipulation with HackerSquad founder Adam Chan and 434 VC general partner Shilpi Nayak. On Nov. 7, we joined Saturday Robotics co-founder Junfan Zhu for Reading Club 37: From Data to Action.

Researchers at AI labs pretrain models on large collections of video, images and text to develop visual and semantic representations. For robotics, they need to connect those representations to physical actions.

Researchers use vision-language-action models, or VLAs, to generate robot actions from visual observations and language instructions. With action-conditioned world models, they predict how the environment may change after an action. Those objectives require different choices about training data and supervision.

In our previous event recap, we explored the role of web-scale pretraining in robot learning. These sessions examined how researchers select the data, represent motion and evaluate transfer to a robot.

Match the data source to the capability

Toloka physical AI and data collection specialist Irina Mira described the disagreement over teleoperation and egocentric video. One research team gets better results with teleoperation. Another favors human video. Without comparable training methods and evaluations, researchers cannot determine why the results differ.

Markov Robotics co-founder Greg Tarr argued for architectures that can learn from both sources, along with simulation.

Researchers can use robot logs to connect observations with recorded actions and robot state. With human video, they can study activities and environments without operating a robot for each recording. They then have to determine which parts of that experience transfer and what supervision they can extract.

For AI labs, this distinction affects decisions about pretraining and adaptation. Researchers can use web video to broaden a pretraining corpus and robot trajectories to provide action supervision for a particular embodiment. They should test the contribution of each source through ablations and downstream task success.

Researchers training an imitation policy need to distinguish demonstrations they want to reproduce from behavior they want to avoid. Researchers training a world model to predict action outcomes need examples of failures as well as successes.

The panelists also questioned the value of collecting thousands of hours at one station under the same conditions. A team can increase its hour count without covering new situations.

Tarr argued that researchers need to improve how models reuse prior knowledge. Collecting demonstrations for each variation of a task would require a growing data budget.

Before planning another collection run, identify the missing capability. Recognizing unfamiliar objects and controlling grip pressure require different evidence.

Align actions in physical time

PsiBot director Kyla Zhang explained how researchers can create conflicting training targets by combining demonstrations recorded at different control frequencies.

Many robot policies generate chunks of actions indexed by discrete steps. But the same step index can refer to different elapsed times. Ten control intervals span two seconds at 5 hertz and half a second at 20 hertz.

If researchers combine those sequences without accounting for physical time, they may assign incompatible targets to the same motion.

Zhang presented Frequency-Aware Flow Matching, or FAFM. The researchers transform action trajectories into discrete cosine transform coefficients and train a flow-matching policy in that representation. They reconstruct a continuous trajectory over physical time and add a derivative regularizer to encourage smoother motion.

They make those changes without adding network parameters.

Before attributing poor results to model capacity, inspect the control rates, timestamps and action conventions in the corpus. Two datasets can use the same file format while describing motion on different terms.

Researchers using FAFM need to account for differences in embodiment and action semantics when combining robot datasets. Aligning time addresses one part of the problem.

Validate hand motion before using it as supervision

Memories.ai head of research engineering Dylan Lin focused on converting egocentric recordings into action supervision.

A researcher can watch a person open a drawer and understand the task. Extracting a training target requires more precision: hand positions in physical units, a consistent coordinate frame and a trajectory that separates hand movement from camera movement.

Lin discussed errors in hand reconstruction, including incorrect depth and jitter. Researchers who use those estimates as action targets risk training a policy to reproduce errors from the labeling process.

He described captioning failures such as confusing which hand acted, reversing the direction of movement and describing motion that did not occur. A researcher reviewing the caption “the person pulls the drawer open” must check whether the person opened or closed it.

Lin previewed work that Memories.ai had not released at the time of the session. Researchers will need public models and evaluations to assess its performance claims. They can apply the validation criteria to existing pipelines: inspect metric accuracy, check temporal consistency and compare action boundaries with the recording.

Lin and Zhang addressed related problems. Researchers need accurate motion labels and a training representation that preserves their physical meaning.

Include contact sensing and hardware in the analysis

At the Wednesday panel, researchers discussed the limits of learning manipulation from video alone.

You can observe someone holding a cup without measuring their grip force. You may see the cup slip but miss the tactile signal that preceded it.

Tarr discussed adding tactile and other sensor streams to pretrained models while preserving their existing capabilities. He favored a unified architecture that could learn across modalities. He presented that as his research direction, not an established answer.

BionIT Labs founder and CEO Giovanni Zappatore approached the control problem through hardware design.

His team uses a differential mechanism to distribute torque across the fingers of its hands. As the fingers contact an object, the mechanism accommodates its shape. The team can control the grasp without computing each finger movement.

Zappatore calls this “mechanical intelligence.” Researchers evaluating a manipulation system need to account for how its mechanical design changes the control problem and the data required to train it.

Before commissioning another dataset, determine whether the system lacks a measurement, a training example or a suitable control interface.

Turn deployment failures into test cases

Sethi described an OpenMind robot continuing to talk after the person it was addressing had left.

A researcher investigating that failure needs to test how the system responds to changes in an interaction, including when to stop. Producing a valid sentence does not establish that the robot responds to the person’s behavior.

Tarr shared an example from Markov Robotics. During a conveyor task, a robot used two fingers to pull an object into reach before picking it up. He said the team had not trained that recovery behavior and had used about 30 hours of robot data for the policy.

Researchers should treat that as a reported observation rather than a controlled measure of generalization. They can test the underlying question by evaluating recovery from states outside the successful demonstration sequence.

At Saturday’s session, Zhang described how PsiBot uses robot rollouts, human correction and reinforcement learning. She also discussed verifying and relabeling world-model predictions before using them as training examples.

Researchers investigating deployment failures need to preserve the observations, commanded actions, timing and outcome. With that record, they can test whether they need better sensing, different supervision or more coverage.

Keep the failure as a regression test. After retraining, rerun it under the same conditions and evaluate on held-out conditions to check whether the improvement transfers.

Find relevant interactions before retrieving more footage

Bright Data solutions engineer Dhruv Parikh focused on finding video for pretraining and targeted data experiments.

A researcher studying fabric manipulation needs a more specific collection brief than “more household video.” Useful examples might include people folding towels, separating layers of clothing or repositioning a garment after a failed grasp.

Those interactions may occupy a few seconds of a long recording. The uploader may never mention them in the title.

Researchers curating pretraining corpora can search visual content with Bright Data’s Video Search API, which we offer in beta. They submit a text description or visual example and receive ranked results with source URLs and timestamps. They can use the Media Data API to retrieve the footage.

Researchers can inspect candidate moments before committing to a larger retrieval and processing job. They must check relevance, remove duplicates and validate any labels they generate. They need to establish the rights for their intended use; finding a public video does not settle that question.

At Bright Data, we help AI labs source pretraining data that matches their research questions. Researchers define the tasks and interactions they need to study, then use targeted discovery to assemble candidate footage for evaluation.

For your next data experiment, start with one failure category. Define the examples you need, inspect the supervision and measure performance on held-out conditions. Track the change in task success alongside the size of the dataset.

Thanks to HackerSquad, Shilpi Nayak, Junfan Zhu, Saturday Robotics and the speakers who shared their research and deployment experience.

Watch the Wednesday panel or read Junfan Zhu’s technical recap of Saturday’s talks.

If you are sourcing video for a pretraining corpus, request Video Search API beta access.

Join the next robotics discussion

Meet researchers working on world models, VLAs and physical AI data. Check the calendar for upcoming technical sessions at the Web Data Loft and online.

Explore upcoming robotics and AI sessions →

Speak to the robotics community

We’re looking for research talks on world models, VLAs and physical AI data. Share an experiment or deployment with enough detail for other researchers to assess your methods and results.

Propose a robotics talk →