For now, from a quick Colab test:
I like the decision to put Dataset Integrity before the broader quality and coverage analysis. It makes the first question much easier to understand.
I ran a small CPU-only probe against six public LeRobot datasets using Calibra v0.7.1, including matched ALOHA human/scripted pairs. The motion checks did appear to detect real structure in the data—especially the difference between human teleoperation and scripted waypoint trajectories.
The part I was less sure about was not the usefulness of the metrics themselves, but the action attached to a CRITICAL result.
In this small probe:
- all six datasets produced at least one motion-related
CRITICAL finding and returned exit code 1;
- five of the six still had an overall summary status of
Warning, rather than Critical;
- the ALOHA scripted datasets had much higher jerk-spike rates than their human equivalents;
- the camera checks were not evaluated for these LeRobot v2/v3 video-backed datasets.
So one possible distinction might be:
| Dimension |
Examples |
| Finding severity |
OK, WARNING, CRITICAL |
| Suggested action |
informational, inspect, block |
| Evaluation coverage |
evaluated, not evaluated, analyzer error |
| CI policy |
which actions should cause a nonzero exit |
A motion finding could remain statistically CRITICAL while still being primarily an inspection signal, rather than an automatic statement that the dataset should not be used for training.
For example:
CRITICAL finding
|
|-- Objective acquisition / format / synchronization failure
| `-- plausible default CI blocker
|
`-- Context-dependent motion anomaly
`-- plausible default review signal
`-- optionally block under a stricter CI policy
This would preserve the information in LDLJ, jerk, and velocity-discontinuity measurements without requiring every unusual motion signature to have the same operational consequence as dropped timestamps or synchronization failure.
It may also make the command output easier to interpret if the summary and the CI decision are shown separately—for example:
Overall summary: Warning
CI result: Failed
Reason: 1 CRITICAL motion-review finding
Likewise, an analyzer that could not run may be clearer as NOT EVALUATED rather than being absent from the public result. The internal pipeline already records skipped analyzers, but the current integrity JSON exposes only critical, warnings, passed, integrity_score, and status.
What the small probe observed
I pinned the probe to Calibra v0.7.1 commit:
1440843897071220c0693bb2f10871e207d7b5ad
It used the actual calibra integrity --json command and recorded its exit code. I downloaded metadata and scalar Parquet data only, not the videos, so this was not a semantic visual review of the episodes.
| Dataset |
CRITICAL findings |
Summary |
Exit |
lerobot/aloha_sim_insertion_human |
LDLJ |
Warning — 86 |
1 |
lerobot/aloha_sim_insertion_scripted |
LDLJ, jerk spike |
Warning — 71 |
1 |
lerobot/aloha_sim_transfer_cube_human |
LDLJ |
Warning — 79 |
1 |
lerobot/aloha_sim_transfer_cube_scripted |
LDLJ, jerk spike |
Warning — 71 |
1 |
lerobot/pusht |
LDLJ, jerk spike, velocity discontinuity |
Critical — 57 |
1 |
lerobot/xarm_lift_medium |
velocity discontinuity |
Warning — 79 |
1 |
The human/scripted comparison was particularly interesting:
| Task |
Human jerk-spike rate |
Scripted jerk-spike rate |
| ALOHA insertion |
0.64% |
24.9% |
| ALOHA transfer cube |
0.94% |
21.9% |
So the jerk metric was not merely producing random noise. It clearly distinguished the collection signatures.
The smoothness analyzer also contains logic recognizing this pattern: high jerk-spike rate combined with low velocity-discontinuity rate can indicate a scripted waypoint planner, and that signature is described as non-pathological in the action/state-divergence interpretation.
That makes the current behavior interesting:
- the analyzer can recognize that the pattern may be a normal consequence of scripted collection;
jerk_spike_rate can still remain an Integrity-layer CRITICAL;- any Integrity-layer
CRITICAL makes calibra integrity return exit code 1.
This seems less like a problem with detecting the signature and more like a question about what the severity should cause downstream.
There is also a limited but useful counterexample to interpreting every motion CRITICAL as “do not train”.
The human ALOHA transfer-cube dataset produced an LDLJ CRITICAL in this probe. However, the LeRobot ACT model card trained on that dataset reports an 83% success rate across 500 evaluation episodes.
That does not establish that the dataset is flawless or that the LDLJ finding is wrong. It only shows that:
motion CRITICAL
and:
dataset cannot support useful policy training
are not necessarily equivalent claims.
A possible result flow
A slightly more explicit result flow might look like this:
Did the check run?
|
|-- No
| `-- NOT EVALUATED
| `-- include the reason
|
`-- Yes
|
|-- Objective integrity failure?
| |-- timestamp dropout
| |-- malformed or missing data
| |-- decoding failure
| `-- action/observation synchronization failure
| `-- BLOCK candidate
|
`-- Context-dependent anomaly?
|-- unusually short episode
|-- LDLJ
|-- jerk spike
`-- velocity discontinuity
`-- REVIEW candidate
|-- confirmed recording defect -> escalate
|-- valid scripted/corrective/recovery motion -> retain
`-- unclear -> inspect or compare downstream
The CI policy could then be a separate choice:
Default CI policy
`-- fail BLOCK findings
Strict CI policy
`-- optionally fail both BLOCK and REVIEW findings
This would not require weakening a statistically severe finding. It would only separate:
How unusual is this?
from:
What should an automated system do about it?
There is a nearby LeRobot proposal in issue #3760 that uses duration, jerk, velocity, static-frame fraction, and final-pose statistics to rank episodes for inspection.
Its intended flow is:
compute diagnostics
-> inspect the short list in the visualizer
-> explicitly decide whether to edit the dataset
The proposed tool is read-only and is described as turning “watch 200 episodes” into “watch these 8”. It is not the same project or necessarily a standard Calibra should follow, but it is a useful nearby example of attaching motion statistics to a review workflow rather than directly to an automatic deletion decision.
Two smaller boundaries that may be worth exposing
1. Overall status versus CI result
The current implementation calculates the two differently:
status comes from the average credit across Integrity findings;- exit code
1 is returned if any Integrity finding is CRITICAL.
That is why five datasets in the probe displayed Warning while still failing the command.
Both outputs can be meaningful, but they appear to answer different questions:
| Output |
Possible question |
| Overall summary |
How does the dataset look across all evaluated checks? |
| CI result |
Did any finding match the configured blocking policy? |
Representing them as separate fields might make the contract clearer:
{
"summary_status": "Warning",
"ci_status": "failed",
"exit_reasons": [
{
"metric": "ldlj",
"severity": "CRITICAL",
"suggested_action": "review"
}
]
}
The exact schema is only an example; the useful part is making the two meanings visible.
2. Passed versus not evaluated
The pipeline already records analyzers that were skipped because the required capability was unavailable.
In this probe, the duplicate-frame, camera-freeze, and blur analyzers were skipped for all six datasets. That matches the current roadmap: image-based checks for LeRobot v2/v3 video-backed datasets are not supported yet.
However, calibra integrity --json does not currently include skipped_analyzers, so a downstream user cannot distinguish:
camera checks passed
from:
camera checks did not run
using the Integrity JSON alone.
Something like this could make coverage explicit:
{
"not_evaluated": [
{
"check": "camera_freeze",
"reason": "video-backed LeRobot v2/v3 decoding is not supported in this release"
}
]
}
Again, this is not a criticism of an unfinished roadmap item. It is mainly about making an incomplete evaluation safe to consume programmatically.
Action representation may also matter for motion checks
This was not the main result of the real-dataset probe, but it appeared clearly in a controlled synthetic check.
The smoothness analyzer supports:
positionvelocityacceleration
and changes the number of derivatives accordingly. The Integrity command currently constructs ControlSmoothnessAnalyzer() with its default action_type="position".
In a synthetic velocity-command dataset, the same values produced:
interpreted as position -> CRITICAL
interpreted as velocity -> OK
That makes sense mathematically: differentiating a position signal three times is not equivalent to differentiating an already-velocity signal twice.
LeRobot now also distinguishes joint versus end-effector space and absolute, relative, and delta action representations. Therefore the meaning of a discontinuity in the stored action array may depend on the representation and processing pipeline.
This may be enough reason to expose the assumed action semantics somewhere in the report, even if automatic inference is not always possible.
Possible behaviors could include:
- infer the representation from dataset metadata when available;
- accept an explicit
--action-type;
- report the assumption used;
- or keep motion findings review-only when the action semantics are unknown.
I would not use this probe to conclude that any specific public dataset was analyzed with the wrong semantics, but it seems like a useful boundary to keep visible when interpreting the metric.
As for an additional integrity check before training, leader/follower calibration drift looks particularly useful.
LeRobot issue #3758 describes a failure mode where the recorded leader action and follower state contain a stable joint offset. Training loss can still look normal because the policy learns the biased mapping, while deployment consistently undershoots or overshoots.
The concrete example there was approximately a 17-degree offset on one joint producing about a 6 cm Cartesian gripper error. The proposed check uses stable frames and reports per-motor statistics for:
action - observation.state
It is post-hoc, read-only, and uses information already present in many teleoperation datasets, so it seems well matched to an Integrity stage. It may also complement temporal and smoothness checks because the problem can remain invisible in ordinary training metrics.
Overall, the motion checks looked useful in this probe. The main possible refinement I saw was to separate:
severity
action
coverage
CI policy
so that the same diagnostic information can support both conservative automated gating and human review without forcing every unusual motion pattern into the same operational decision.
This was only a six-dataset probe, without video review, so I would not generalize it to the entire LeRobot Hub or label the findings false positives. But the result-to-action boundary reproduced consistently enough that it seemed worth reporting.
LeRobot (v1 image datasets with 
Image-based integrity for LeRobot v2/v3 video-backed datasets is next on the roadmap.