How much teleoperation data do you actually need to train a robot policy?

Every teleoperation programme eventually runs into the same meeting. Someone has to put a number in a spreadsheet — two operators, six weeks, how many episodes? — and say whether the policy will actually work at the end of it. The number gets guessed, the budget gets approved, and nobody finds out whether the guess was right until the evaluation runs.

That guess is avoidable. You cannot know the exact demonstration count up front, but you can size the collection defensibly, measure the return on every additional batch, and spot the point where you are paying for data that no longer moves the policy.

The short answer

There is no universal number, and anyone who quotes one without asking about your task has dodged the question. What does generalise is the shape of the problem.

For a single, tightly scoped manipulation task on a fixed rig — same table, same object, same lighting — published imitation-learning results routinely train from demonstration counts in the tens to low hundreds per task. The ALOHA/ACT work, for example, trained from roughly 50 demonstrations per task, and success rates flatten quickly beyond that. Past the flattening point, another hundred episodes in the same scene buy far less than the same operator-hours spent on new scenes, new object instances and new starting positions.

So the quantity to budget is not “how many demos” but “how many distinct conditions, multiplied by enough demonstrations per condition for the task to be learnable”. A policy that has to survive varied objects, poses, clutter and lighting lands in the high hundreds to low thousands of episodes — not because the model is hungrier, but because the condition count is higher.

Treat 50–100 clean demonstrations per condition as a probe, not a target. Collect them, train, evaluate, and let the measured curve — not a vendor’s rule of thumb — decide the size of the next batch.

Three variables that move the number

Task horizon and precision. Behaviour cloning suffers from compounding error: a small action error pushes the robot into states that never appear in the training data, where its next action is worse still. The classic analysis of imitation learning (Ross and Bagnell, 2010) shows the cost of those errors growing quadratically with episode length rather than linearly. A three-second pick therefore needs far less data than a ninety-second assembly ending in a sub-millimetre insertion, because the long task gives the robot many more chances to drift off the data manifold. Action chunking — predicting a short sequence of actions instead of one step at a time — is the standard mitigation, and it works by shortening the horizon over which those errors compound.

Scene and object diversity. A policy trained on one table, one lighting condition and one object instance will learn that table, that light and that object. Every axis of variation the policy has to survive in production is an axis you pay for in data.

Embodiment and action space. Joint-position targets, end-effector deltas and Cartesian velocity commands are not interchangeable training signals. Change the action space, the control rate or the gripper halfway through collection and the dataset fragments into incompatible subsets; the effective demonstration count silently drops to the size of the largest consistent subset.

Coverage is the real unit of account

Budget in cells, not episodes. A cell is one combination of the things that vary: object instance, start-pose bucket, clutter level, lighting and — where relevant — the physical workcell itself. Multiply them out and the arithmetic turns honest fast. Eight object instances across five clutter-and-lighting conditions is forty cells; at twenty demonstrations each, that is eight hundred episodes before anyone has touched a stopwatch.

This framing matches where the research has been heading. Work on data scaling laws in imitation learning for robotic manipulation (Lin et al., 2024) reports power-law relationships between how well a policy generalises and the number of training environments and objects, with sharply diminishing returns from piling more demonstrations onto an environment that is already covered. Check the paper for the exact exponents before quoting them; fitted coefficients belong to the task and setup they were measured on, and they do not transfer to yours unchecked.

The operational rule that falls out of this is blunt: when a fixed budget forces a choice, spread it. Twenty demonstrations in each of forty cells generally beats eight hundred in one.

Quality is the multiplier nobody budgets for

The RoboMimic study (Mandlekar et al., 2021) produced a finding that survives contact with real programmes: datasets pooled from several human operators of mixed proficiency are substantially harder to learn from than datasets from one proficient operator, even at equal size. People solve the same task in different ways, and a policy trained on the mixture has to model the mixture as well as the task.

In practice, these are the things that quietly halve the value of a dataset:

• Operator drift. Grasp strategy, approach angle and speed all shift over a six-week collection. Freeze a written protocol before episode one, then re-check it weekly against recorded video.
• Idle time inside episodes. Long pauses while the operator thinks teach the policy to pause. Trim them, or gate recording on motion.
• Inconsistent success criteria. If “done” means one thing on Tuesday and another on Thursday, your evaluation is measuring noise.
• Timing and sync defects. Dropped frames, unlogged latency between camera timestamp and action timestamp, a control rate that drifts. These corrupt the state–action correspondence the policy is trying to learn, and they stay invisible until a scaling curve refuses to rise.
• Discarded failures. Failed episodes and recoveries are not waste. Label them and keep them in a separate split; recovery behaviour is exactly what a diet of pure successes lacks.

Where a marginal hour of budget should go

Source	What one unit costs	What it buys	Main failure mode
More teleop demos in a covered cell	Operator time plus reset time	Marginal robustness	The fastest-saturating spend on the list
Teleop demos in a new cell	Operator time plus rig reconfiguration	Generalisation across the axes you varied	Reconfiguration time is routinely underestimated
Human-gated interventions (DAgger-style)	An operator supervising a running policy	Data exactly where the policy fails	Needs a working policy first, and is harder to schedule
Automated augmentation (MimicGen-style generation from a seed set)	Compute plus engineering	A large synthetic expansion from a modest set of human demos	Inherits the seed set’s blind spots
Simulation	Engineering plus compute	Cheap coverage of poses and layouts	The transfer gap is a separate engineering project
Public corpora for pretraining (Open X-Embodiment, DROID)	Download plus training compute	A better initialisation, not your task	Embodiment and action-space mismatch

Open X-Embodiment pools more than a million real robot trajectories across 22 embodiments contributed by over twenty institutions; DROID adds a large in-the-wild Franka manipulation set spanning hundreds of scenes. Both are worth evaluating as pretraining substrate, and both ship dataset cards worth reading before you assume the action space matches yours. Neither removes the need for in-domain demonstrations — they change the slope of your curve, not the need to plot it.

Run your own scaling curve

This is the step that turns the question from opinion into measurement, and it costs roughly one afternoon of compute.

1. Freeze an evaluation set first. A fixed list of trial conditions — object, start pose, clutter — that never appears in training, with a written pass/fail criterion.
2. Subsample the dataset you already have at roughly 10%, 25%, 50% and 100%, stratified so every cell keeps its share.
3. Train identical policies on each subset: same architecture, same hyperparameters, several seeds per point. Seed variance in imitation learning is not small, and a single run will mislead you.
4. Run enough evaluation trials to mean something. A success rate is a binomial estimate: twenty trials cannot reliably separate a 70% policy from an 85% one, because the confidence intervals overlap heavily. Compute the interval before you believe a bump.
5. Plot success against demonstration count on a log axis and find where the slope collapses.
6. Apply a stop rule written in advance. For example: stop when doubling the dataset moves success by less than the width of your evaluation confidence interval, and put that budget into new cells instead.

The curve also answers something no single number could — whether you are data-limited at all.

When more data is the wrong fix

If the curve is flat from the first subsample onward, the bottleneck sits upstream of data volume. The usual culprits: observability (the state the task depends on never reaches the observation — contact forces, occluded geometry, an in-hand pose no camera sees), hardware repeatability (fixture or gripper variance wider than the tolerance the task demands), action-space mismatch, or an evaluation protocol too noisy to detect an improvement. In every one of those cases the next operator hour buys nothing. Fix the sensing, the fixture or the evaluation first, then resume collecting.

Budget in hours, not episodes

Episode counts do not book operator time; cycle time does. Measure your own throughput on a real collection day rather than assuming a figure: time the demonstration and the scene reset, track how long labelling takes and what share of episodes you throw away, then work out usable episodes per operator-hour. Reset time frequently exceeds demonstration time, which is why reset design — fixtures, jigs, object trays, a scripted return-to-home — is one of the highest-leverage engineering investments in a data programme. Put curation on the plan as its own line item; a dataset nobody has reviewed is a dataset whose quality you cannot state.

Log so the question is answerable next time

Use an established dataset format — the LeRobot format or RLDS, both common across open robot-learning stacks — rather than an ad-hoc folder layout, and attach per-episode metadata: operator ID, cell and scene ID, object instance, start-pose bucket, outcome, intervention flags, controller version, and synchronised timestamps for every stream.

That metadata is what makes stratified subsampling, per-operator quality analysis and honest coverage reporting possible. Without it you can count episodes but you cannot say what they cover, and “how much data do we need” stays a guess for the next programme as well.

Two more items belong in the plan. The first is safety: a teleoperation rig is still an industrial robot installation, and ISO 10218 — together with ISO/TS 15066 where the operation is collaborative — governs the safety case for a person standing beside a moving arm with a teleoperation interface in their hands. The second is provenance: record where footage came from and what consent covers it whenever it captures people or third-party premises. A dataset you cannot lawfully use is worse than one you never collected.