agi.diy serves robot policies over an API. The flywheel turns the runs back into a dataset: one call after a session ends, and its recording is one LeRobot v3.0 episode in a Hugging Face dataset — the same format as the teleop data the policy trained on, with a success label. Train on it, and the next checkpoint has seen its own mistakes.
curl -s -X POST https://api.agi.diy/v1/sessions/ses_…/export -H "authorization: Bearer $AGI_API_KEY" \
-H 'content-type: application/json' -d '{"label":"success","task":"pick up the x piece and place it on the center cell"}'
# 201 {"session":"ses_…","dataset":"cagataydev/so101-ttt-flywheel-sim","episode_index":17,"episodes":18,"frames":283,"seconds":9.43,
# "fps":30,"label":"success","source":"sim","commit":"5e4698aa…","url":"https://huggingface.co/datasets/cagataydev/so101-ttt-flywheel-sim"} /play does this after every judged attempt: the move log reads episode N saved with the dataset link.
The format — one contract, tested#
The reference is cagataydev/, the 210-episode teleop dataset recorded on the real SO-101 (30 fps, lerobot
0.6.2). Every flywheel episode is written by lerobot's own LeRobotDataset at the same commit, with the same features:
| feature | dtype | shape | units |
|---|---|---|---|
observation.state, action | float32 | [6] — shoulder_pan, shoulder_lift, elbow_flex, wrist_flex, wrist_roll, gripper | degrees; gripper 0–100 % |
observation.images.top, observation.images.wrist | video | 480×640×3 | AV1 (libsvtav1, crf 30, g 2, yuv420p), 30 fps |
timestamp, frame_index, episode_index, index, task_index | float32 / int64 | [1] | — |
The test that says so is agi flywheel check <reference root> <flywheel root> (agi-diy ≥ 0.5.11, or python -m agi.flywheel check): it compares
meta/info.json feature by feature, the parquet column schema and the codec parameters of one video per camera, and exits
non-zero on any difference. Frame content differs by construction — see the two sources below.
The wire speaks so101-normalized-v1 (arm −100…100, gripper 0…100); the dataset is degrees, converted with the per-joint
scale of the calibration the reference was recorded with (blue_follower: pan 1.0927, lift 0.9685, elbow 1.0336, wrist 0.9913,
roll 0.5556 normalized units per degree; gripper unchanged). A dataset in one unit, always.
Two sources, said on every episode#
meta/flywheel.jsonl carries one row per episode: session, model, checkpoint, label, judge, source, and three
fields that say how the data was obtained — read them before you train.
source: sim — POST …/export of a traced session | source: real — agi rollout … --record | |
|---|---|---|
| actions | the executed trajectory: the real-time-chunking splice replayed from the recording (chunk k covers rows [cursor_k, cursor_k + n), the latest chunk that had arrived rules each row; holds are not rows), then the policy's 15 Hz upsampled to 30 fps by linear interpolation. cursor_source: recorded when the client sent its cursor, estimated (matched to the measured state) on older traces | what the arm was commanded, read by LeRobot at 30 Hz |
observation.state | state: reconstructed — the target one tick earlier, corrected to the measured state at every observation (exact there) | state: measured — the arm's joints at 30 Hz |
| frames | frames: held — the frames the policy saw, one per call (~1 a second), each repeated until the next call | frames: measured — both cameras at 30 fps |
| label | the caller's: /play passes its physics judge's verdict (judge: ttt-sim-judge-v1/browser) | yours: --success, --failure, or the y/n prompt at the end |
A sim episode is a rendered simulation, not a measurement of the real arm; its in-between frames are the last frame the
policy saw. Say so wherever a number from it appears.
POST /v1/sessions/:id/export#
Owner cookie or the key that leased the session; the session must have ended (409 session_live otherwise — the episode
is the whole recording). Body, every field optional:
| field | default | meaning |
|---|---|---|
label | null | "success", "failure" or null (unlabeled: kept, never trains) |
task | the trace's | the task string of the episode (one per episode; /play passes the cell it asked for) |
source | "sim" | "sim" or "real" — picks the default dataset and is stamped on the episode |
dataset | cagataydev/ / -real | any <org>/<repo>; your own dataset needs your hf_token (write access; forwarded to the runner once, never stored) |
judge | — | who labelled it, ≤ 64 chars |
private | true | the dataset's visibility when it is created |
Answer 201 {session, model, dataset, episode_index, episodes, frames, seconds, fps, task, label, source, commit, url}.
Errors: 404 no_trace (nothing recorded, record:false, or older than 24 h — export within a day), 409 export_running,
502 export_failed (the runner's one-line reason, e.g. the trace carries no task string), 504 export_timeout.
Exports into one dataset run in series on the runner; a dataset written from elsewhere since the last export is re-read from
the Hub first, never overwritten. One export is 10–20 s (AV1 encoding of two 30 fps videos, then the push).
Your own arm, your own dataset#
agi rollout agi-wfm/so101-ttt --record my-org/so101-ttt-rollouts --dry-run -- \
--robot.type=so101_follower --robot.port=/dev/tty.usbmodem… --robot.cameras= '{"front":{…},"wrist":{…}}' --task="pick up the x piece and place it on the center cell"--dry-run prints the composed command and leases nothing; remove it to run. --record <org>/<repo> runs LeRobot's episodic
rollout strategy under the lease: one episode per run, both cameras and the joint reads at the true 30 Hz, the task string, then
the push to your dataset (private). Label it with agi flywheel label <org>/<repo> --episode N --success|--failure, which writes
the same provenance row (source: real, frames: measured, state: measured). The writer is the same lerobot commit the runner
uses, so agi flywheel check passes against the reference by construction.
Train on it#
A flywheel dataset loads like any LeRobot dataset — LeRobotDataset("cagataydev/ — and lerobot-train
reads it unchanged. Train on label == success rows of meta/flywheel.jsonl only; failures are kept for a critic, never for
imitation. Our own retraining loop (what is mixed, how a round is scored and promoted) is agi-wfm's docs/FLYWHEEL.md.