Traces.

Every session is recorded on the runner as it runs: each observation the policy saw (joint state, task, camera frames) and each action chunk it answered with. The trace is served read-only through api.agi.diy, to the user who owns the session. API access to traces lasts 24 h from session creation. The runner periodically deletes inactive files older than its retention window; this is not an exact-time deletion guarantee. record: false on POST /v1/sessions turns recording off for that session.

Frames may contain private spaces. They are stored in session recordings, not publicly listed. The owner cookie, leasing key and authorized operator cookies can read them; operator reads are logged. Session/usage accounting is separate from trace retention.

Routes (Worker, public, same auth as GET /v1/sessions/:id)#

Auth = the dashboard cookie of the session's owner or Authorization: Bearer <key> of the key that leased it. Authorized operator cookies may also read traces, with administrative access logged. Any other caller → 404 not_found (same as GET /v1/sessions/:id). 404 no_trace when the session has no recording (record:false, older than 24 h, or the runner reports no trace server). 503 runner_offline when the runner's trace server does not answer.

Trace = { session: "ses_…", model: "agi-act/so101", state: "leased"|"active"|"ended",
          type: "live" | "replay", source_session_id: null | "ses_…",      // replay: which trace it was fed from
          action_names: ["shoulder_pan.pos","shoulder_lift.pos","elbow_flex.pos","wrist_flex.pos","wrist_roll.pos","gripper.pos"],
          fps: 30, action_horizon: 100, cameras: ["top","wrist"],
          frames_base: "/v1/sessions/ses_…/trace/frames/",                    // frames_base + cam + "/" + request_id + ".jpg"
          events: [ Event … ],                                                // chronological, obs then its act — only obs/act kinds
          ended: false }                                                      // the runner wrote its `end` line: nothing more will arrive

GET …/trace?from=<line> skips the first <line> recorded lines (the same cursor …/trace/live takes), so a reader that already has the head of a long trace can ask only for the rest. The answer is private, max-age=60 once the session has ended and private, no-store while it is live.

action_names is the policy's own feature names (from the server descriptor); state[i] and chunk[*][i] are in the same order and the same units as the wire (current checkpoints: normalized arm −100…100, gripper 0…100). The deployed so101-degrees-gripper-percent-v1 label is misleading; values must not be read as degrees. The trace shows requested targets, not measured proof that those targets were reached.

Events (JSON lines, ~/agi-diy/traces/<session_id>/events.jsonl on the runner)#

{ "t": 1761330000.123,                 // runner wall clock, unix seconds, when the envelope was seen
  "kind": "obs",
  "request_id": "a1b2…",               // lerobot request_id — pairs the obs with its act
  "seq": 7,                            // client-side request sequence (1-based, per session)
  "state": [12.1, -45.0, 30.2, 0.0, 5.5, 10.0],   // observation.state, N floats (6 for so101)
  "task": "pick up the cube",
  "capture_t": 1761329999.980,         // client's capture_time as sent (its own monotonic clock — relative only)
  "cams": { "top": "frames/top/a1b2….jpg", "wrist": "frames/wrist/a1b2….jpg" },   // relative to frames_base
  "cursor": 46,                        // the client's playback cursor (rows committed when it took this frame, lerobot `observation_cursor`); null on old clients — FLYWHEEL reads it (docs/FLYWHEEL.md)
  "bytes": 48213 }                     // wire payload size
{ "t": 1761330000.251,
  "kind": "act",
  "request_id": "a1b2…",
  "chunk": [[…6 floats…], …],          // canonical_actions, n rows (action_horizon)
  "n": 100,
  "rtt_ms": 128,                        // runner-side: act.t − obs.t for the same request_id (client RTT is larger by the network)
  "server_ms": { "policy": 19.8, "queue": 0.1, "worker": 20.4 },   // lerobot server_durations ×1000 (keys as the server sends them)
  "tokens": 100 }                       // = n, what this call was billed

Other kinds may appear and must be ignored by readers: "session" (first line: {kind:"session", session, model, deployment, action_names, fps, action_horizon, cameras, semantics, type, source_session_id}) and "end" (last line: {kind:"end", session, reason, events, frames} — counts of events and frames written).

Frames#

frames/<cam>/<request_id>.jpg — the JPEG bytes exactly as they crossed the wire when the client used encoding=jpeg (the default handed out in Session.zenoh.encoding); a raw-RGB observation is encoded on the runner at quality 80. <cam> is the camera short name (observation.images.top → top). Missing frame → 404.

Live#

A live session has nothing recorded until its first call, and that is reported as emptiness, not absence: …/trace answers 200 with events: [] and ended: false, and …/trace/live answers 200 text/event-stream with a retry: 2000 hint and closes, so a browser EventSource reconnects by itself until the first chunk lands (W2-103 — the runner itself still has no directory, the Worker fills in the empty shape). 404 no_trace is kept for the cases where a recording really will never exist: record:false, a session older than 24 h, no trace server on the runner, or an ended session that never produced an observation.

GET /v1/sessions/:id/trace/live streams data: <Event JSON>\n\n as lines are appended, starting from the current end of the file (pass ?from=0 to get the whole history first). A : keepalive comment every 15 s. The stream ends when the session is ended (the runner writes an end event) or after 10 min.

Public sessions#

A session is private. Its owner (dashboard cookie, or the key that leased it) may make an ended one public:

PATCH /v1/sessions/ses_…   {"public": true}      → 200 Session (public: true, public_at)
PATCH /v1/sessions/ses_…   {"public": false}     → 200 Session (public: false)

At public:true the Worker copies the recording — the Trace JSON and every frame the policy saw — into its own object store, once. From then on GET /v1/sessions/:id, GET …/trace and GET …/trace/frames/<cam>/<request_id>.jpg answer for anyone: no cookie, no key, access-control-allow-origin: *, cache-control: public, max-age=86400, and they keep answering after the runner's 24 h retention has deleted the original. The public Session carries zenoh: null (there is nothing to connect to) and public: true; the public Trace carries public: true. What a stranger can never get: …/wire (a public session is a recording, not a robot to drive), …/trace/live, …/replay, the owner's identity, keys. The dashboard shows a Public / Private toggle on each ended session, and /s/<ses_id> is the page that replays a public one.

GET /v1/sessions/public?model=<id>&limit=<n> lists the newest public sessions (anyone; {id, model, type, created_at, ended_at, tokens_used, public_at}, newest public_at first, max 100).

Refusals: 409 session_live (end it first), 404 no_trace (the runner no longer holds the recording — publish within 24 h of the session), 409 too_large (more than 2,000 frames), 404 not_found (not yours). public:false deletes the snapshot; a later public:true copies again from the runner, if it still has the recording.

Run inference again (replay)#

Viewing/scrubbing a saved trace is not new inference. Replay below does not move a robot or restore the physical scene; it is a fresh, billed model evaluation on saved inputs. Whether identical inputs give identical outputs depends on the row, and it changed on 2026-10-25. agi-act/so101 is deterministic by construction and always reproduced its chunks exactly. For agi-wfm/so101 the served deployment now fixes the sampler seed: the same observation sent twice came back byte-identical on the served path (max_abs_diff 0.0, measured 2026-10-25), so replaying a recording through it returns the same chunks. agi-smolvla/so101 samples and its served deployment sets no seed, so treat it as non-reproducible until measured. Two cautions that survive the fix: the published held-out figures were produced by unseeded runs, so the chunk_spread in eval is a real property of those numbers and a figure-level comparison still has to clear it; and GET /v1/models does not yet publish a serving-side seed, so a client cannot check reproducibility programmatically — ask for it rather than inferring it from eval, where seed means the harness's own seed.

POST /v1/sessions/:id/replay { model?: "agi-act/so101" } → 201 Session with type: "replay" and source_session_id: ":id" (404 no_trace when nothing is recorded, 409 incompatible_features when the target model's cameras (matched by short name, no renaming) or state shape differ from the recording — e.g. agi-act/so101 top/wrist → agi-smolvla/so101 front/wrist is refused, 409 no_observations for an empty recording). Which check answers first. The target's live slot is resolved before the recording's features are compared, so the order you meet is: nobody serves that model → 404 unknown_model; somebody does but no slot is free right now → 409 no_slots with retry_after_s (retry, it is not a mismatch); a slot is held and the shapes differ → 409 incompatible_features, naming both sides (source_cameras, target_cameras). Measured: an agi-act/so101 recording aimed at agi-smolvla/so101 answered no_slots while that pool was busy, and incompatible_features (["front"] vs ["top","wrist"]) once a slot was free.

A custom checkpoint cannot be replayed by default. model defaults to the source session's model, and for an hf:<org>/<repo> session that is the pinned hf:<org>/<repo>@<sha> — a deployment that no longer exists once the session ended, so the replay answers 404 unknown_model naming it. Pass model explicitly: a catalog model whose camera short names and state shape match the recording, or lease the checkpoint again (POST /v1/sessions with the hf: id) and replay into that session's model while it is live.

The runner that holds the recording leases a slot for model (default: the source's model) and feeds the recorded observations to it at the recorded cadence (runtime/tracing/replay.py, lerobot's RemoteClient over the local router). The new session is recorded like any other, and billed normally — one token per action in each returned chunk. When the feeder finishes, the runner heartbeats replay_done (or replay_failed: <why>) and the session ends client_done (or replay_failed). GET …/trace of the replay carries source_session_id, so a client can overlay the two chunks for the same seq. Feeder start-up (torch import on the runner) is ~10 s before the first observation lands; a replay of a 10-call ACT trace ran 14 s wall-clock end to end and produced bit-identical chunks (deterministic policy, same input).

Runner side (reference implementation: runtime/tracing/ (the stdlib owns the name trace))#

• The broker's zenoh subscriber on …/sessions/*/obs and …/sessions/*/act hands every envelope to a queue; a writer thread decodes (lerobot codec, JPEG bytes taken as-is) and appends. The zenoh callback never blocks.
• Unknown deployment → session binding (the lease may land one heartbeat later): events are buffered by deployment and flushed to the session directory when the binding arrives (same window FIX-METER closes for metering).
• Trace server runtime/tracing/server.py on 127.0.0.1:7460, published through a cloudflared quick tunnel. Every request carries ?sig=<hex hmac_sha256(INTERNAL_KEY, "<session_id>:<exp>")>&exp=<unix seconds>; the Worker signs with exp = now + 60 s. No sig, bad sig or exp in the past → 403.
  • GET /t/<session>/events → JSON array of events (+ ?from=<line>), GET /t/<session>/meta → the session line
  • GET /t/<session>/frames/<cam>/<rid>.jpg → image/jpeg
  • GET /t/<session>/live → SSE
• The runner heartbeat gains trace_url: "https://<quick-tunnel>.trycloudflare.com" (omitted = no trace server → 404 no_trace). The Worker always uses the latest value, so a tunnel restart heals within one heartbeat.
• Retention: a sweeper deletes traces/<session> directories older than 24 h. Disk guard: frames are skipped (events still recorded, cams: {}) while ~/agi-diy/traces exceeds 20 GB.