Hmm… If you haven’t already, my first thought is that this might be a good one to bring to the LeRobot Discord…:
The questions in your post seem especially well suited to people who are actively collecting and training on LeRobot datasets: which metrics are useful in practice, which diagnostics are missing, and whether something like this would actually be used before training. The LeRobot documentation points to the Discord as its community discussion space, so you may find more people there who can answer with concrete examples from their own recording and training workflows.
If you have already posted it there, linking that discussion from this thread might also make it easier for people to find and build on the same conversation.
One distinction that could be interesting to discuss there is the difference between a useful warning and a reason to remove data.
For example, a finding might mean:
- there is a structural problem that probably needs fixing;
- an episode is worth opening in a visualizer;
- the recording may need to be repeated;
- training should probably wait until the issue is understood;
- or the episode is unusual but still valuable because it contains a difficult case, correction, recovery, or alternative strategy.
Those cases can look similar in summary statistics, but lead to very different decisions.
A high-jerk or unusually long episode may be a poor demonstration, for example, but it could also contain contact, hesitation around a difficult state, or a useful recovery. In that sense, Calibra may already be useful even when a metric only helps someone decide what to inspect, rather than deciding automatically what should be deleted.
There is a nearby proposal in LeRobot issue #3760 that follows this kind of workflow:
compute episode-level diagnostics
-> inspect a short list in the visualizer
-> explicitly decide whether to edit the dataset
The proposed analyzer is intentionally read-only and is described as a way to turn “watch 200 episodes” into “watch these 8”. That seems close enough to be a useful comparison point, although Calibra covers a much broader area than that proposal.
A little more context on the quality-versus-diversity question
Another nearby discussion is LeRobot PR #2936, which explores dead-frame, smoothness, consistency, coverage, and temporal metrics together with an overall dataset score.
One concern raised in that discussion is that some consistency or temporal measures could penalize useful diversity. The possible response considered there is not necessarily to discard those metrics, but to present some of them separately or make their use optional rather than treating every unusual episode as evidence of low quality.
That seems closely related to the kind of feedback Calibra could receive from real users:
- Which findings reliably indicate recording or data-integrity problems?
- Which are mainly useful for ranking episodes for review?
- Which unusual behaviors should remain because they cover difficult or rare situations?
- At what point would a user feel comfortable turning a warning into an automatic filtering rule?
Calibra’s own benchmark notes also suggest that selection behavior can depend on the retention budget. In the reported PushT comparison, the relative outcome changes across different retention levels. The documentation also notes that the detailed evaluation is centered on one dataset and uses held-out trajectory-prediction error rather than closed-loop task success.
I do not read that as a problem with doing the experiment. It mainly suggests that “this episode looks unusual”, “this episode should be reviewed”, and “removing this episode improves policy training” are three different claims.
People with real LeRobot collection examples may be able to help connect those layers. A particularly useful example would be an episode that a metric flags, but that the user knows should remain because it contains a valid correction or recovery.
So rather than asking the Discord community to judge the entire toolkit, it may be enough simply to show the Space and a few representative findings and see what kinds of decisions users associate with them.
That could lead naturally to feedback on metrics, false positives, missing diagnostics, and possible integration with existing LeRobot visualization or dataset-editing workflows—without requiring the project to settle all of those questions in advance.