The wire.

A page cannot speak the inference wire. LeRobot's remote inference rides zenoh, and zenoh's own ws/ link cannot traverse a CDN tunnel (it connects to ws://<ip>:<port>/ with Host: <ip>, and has no TLS — D11). So the runner fronts a leased policy with a plain WebSocket: observations in, action chunks out, one socket per session.

Nothing about the metered path changes. On the runner the gateway is just another LeRobot RemoteClient on the local zenoh router, so the same broker subscribers that watch a robot's traffic do the billing (1 token per action) and the tracing, and the same liveliness token flips the session leased → active (D13).

  page on agi.diy ──wss──▶ cloudflared ──▶ gateway :7461 ──zenoh (loopback)──▶ lerobot-policy-server
         │                                      │                                      │
         └─ POST /v1/sessions, GET …/wire ──▶ Worker ◀── heartbeat wire_url, meter ── broker

1. Three calls#

// lease a slot (CORS: https://agi.diy, https://www.agi.diy, *.vercel.app — bearer key only, never cookies; the dashboard on api.agi.diy is same-origin and uses its cookie — that is how its `probe` works)
const ses = await (await fetch("https://api.agi.diy/v1/sessions", {
  method: "POST", headers: { authorization: `Bearer ${key}`, "content-type": "application/json" },
  body: JSON.stringify({ model: "agi-act/so101" }),
})).json();

// mint a socket URL (valid 120 s to OPEN the socket; the socket then lives as long as the session)
const { url } = await (await fetch(`https://api.agi.diy/v1/sessions/${ses.id}/wire`,
  { headers: { authorization: `Bearer ${key}` } })).json();

const ws = new WebSocket(url);           // wss://<tunnel>/w/ses_…?exp=…&sig=…
ws.binaryType = "arraybuffer";

npm i agi-diy@^0.3.0 does all three (0.3.0 is the first release with the wire client; npm latest 0.2.1 is REST only):

// in a page: apiKey (the visitor's own, kept in memory), state (6 numbers) and jpegBytes come from your app
import { createClient, connectBrowser } from "agi-diy";
const client = createClient({ apiKey });
const session = await client.session("agi-act/so101");
const wire = await connectBrowser(session);                 // or: await session.connect()
const chunk = await wire.predict({ state, images: { top: jpegBytes, wrist: jpegBytes }, task: "pick up the cube" });
chunk.row(0);            // Float32Array(6) — first action, in wire.actionNames order
wire.close(); await session.end();

2. GET /v1/sessions/:id/wire#

Auth = the owner's dashboard cookie or the key that leased the session (any other caller → 404 not_found, as with GET /v1/sessions/:id). An ADMIN_LOGINS cookie is not enough here, unlike session and trace reads: minting this URL is not a read — a socket on it sends observations, gets chunks back and meters every action to the owner's credits — so an operator gets 404 not_found and the attempt is logged {admin_denied: "wire", login, session} (W2-201).

200 { url: "wss://<runner tunnel>/w/ses_…?exp=<unix>&sig=<hex>",
      exp: 1761336000, session: "ses_…", model: "agi-act/so101",
      expires_at: "2026-10-24T21:00:00.000Z", protocol: "agi-wire/1", encoding: "jpeg" }
409 session_ended   this session is over — lease a new one
503 no_wire         the runner holding it advertises no gateway

sig = hmac_sha256(INTERNAL_KEY, "<session_id>:<exp>"), the same construction the trace server uses. The URL is single-session: a signature minted for session A does not open session B, because the gateway reads the session → deployment binding from the broker's own status, never from the client. Never cache the URL — the host changes whenever the runner's tunnel restarts; mint it per socket.

3. Frames#

One format in both directions:

[u32 little-endian header_len][UTF-8 JSON header][blob][blob]…

The header is padded with spaces to a multiple of 4, so every blob starts 4-byte aligned and an action chunk maps straight into a Float32Array with no copy. Control messages are plain JSON text frames.

hello (text, first message) — the policy's real shape. Use these names, don't hardcode:

{ "type": "hello", "session": "ses_…", "model": "agi-act/so101", "semantics": "so101-degrees-gripper-percent-v1",
  "action_horizon": 100, "action_dim": 6, "fps": 30, "cameras": ["top", "wrist"], "state_dim": 6,
  "action_names": ["shoulder_pan.pos", "…"], "encoding": "jpeg", "connect_ms": 56.7,
  "frame": "u32le header_len + json header + blobs" }

hello does not repeat the units either: semantics is the legacy label (wrong for both live checkpoints — the data is LeRobot normalized, arm −100…100 and gripper 0…100). A browser client that reads only hello should take the units from GET /v1/models (units, units_note) before it moves anything. Tracked as W2-206.

observation (binary, client → gateway) — blobs follow in exactly the order images lists them:

{ "type": "obs", "request_id": "c7", "task": "pick up the cube", "state": [0, 2, 4, 6, 8, 10],
  "images": [{ "cam": "top", "bytes": 16972 }, { "cam": "wrist", "bytes": 16972 }] }

cam must be the short name hello.cameras lists (front, top, wrist — a custom checkpoint names its own, often camera1…), not the feature key observation.images.<cam> that a Session's features shows. A name we cannot place is refused before the policy sees anything — bad_observation, "camera 'x' is not one of this model's cameras: …" — so a typo costs nothing. Omitting a camera entirely is still a choice, not an error: it reaches the policy as black, you get a normal chunk and you are billed for it, so send every camera hello.cameras names unless you mean to blind one. A camera you do not have belongs OUT of the lease, not black in the frame: for a custom checkpoint pass cameras on POST /v1/sessions (BYO-RTC) and hello.cameras lists only those — a SmolVLA checkpoint masks the ones you left out, which is what it saw in training. (Before this, a misnamed camera behaved like an omitted one: measured on a live custom checkpoint, observation.images.front and no image at all returned a byte-identical chunk — sha256 7c0e3f17… — while the correctly named camera returned b65a51b2….) Any JPEG size is fine; the gateway resizes to the camera's shape. Limits: a WebSocket message may be at most 8 MB (two 640 × 480 JPEGs are ~60 KB); the JSON header at most 64 KB. An oversized frame is not answered with an error — the gateway closes the socket with 1009 and the byte counts in the reason (measured: frame with 9437312 bytes exceeds limit of 8388608 bytes). The session itself stays leased, so mint a fresh wire ticket and carry on with smaller frames. The gateway pings every 20 s and drops a peer that does not answer within 20 s; task is echoed in the chunk, truncated to 512 characters.

chunk (binary, gateway → client) — header, then one blob of n × dims float32 little-endian, row-major:

{ "type": "chunk", "request_id": "c7", "seq": 7, "n": 100, "dims": 6, "dtype": "float32", "tokens": 100,
  "gateway_ms": 71.3, "server_ms": { "preprocessing": 4.8, "policy": 50.5, "postprocessing": 18.8, "queue": 0.1, "worker": 75.3 },
  "task": "pick up the cube", "cursor": 0, "merge": "aligned" }

Real-time chunking (cursor, next runner update). Add "cursor": <int> to an obs — the number of actions you have committed to the arm so far, counted when you took the frame. The chunk echoes it (cursor, with merge: "aligned", the negotiation the gateway now makes with the policy server). To never let the arm wait: send the next observation while about 0.75 s of motion is still queued (the same threshold agi rollout uses), and when the chunk arrives drop its first committed_now − chunk.cursor rows — the steps that played during the round trip — and replace your unplayed tail with the rest. Measured over tls/zenoh.api.agi.diy with LeRobot's client doing exactly this: act 0 waits in 60 s (sequential: the arm idles ~2.5 s per minute), smolvla 0 waits (sequential: ~11 s per minute). Omit cursor and nothing changes: 0 comes back and the chunk is a plain prediction from that frame. A non-integer or negative cursor is refused (bad_observation) before the policy runs. Until the runner update lands the chunk carries no cursor field — treat its absence as sequential. /play does exactly this (PLAY-RTC): its observation carries the rows the simulated arm has committed, the next one goes up while max(0.75 s, 1.25 × the measured round trip) of motion is still queued, and the chunk is spliced where the arm is; a wire that echoes no cursor gets the old append loop. Measured in the browser sim with agi-wfm/so101-ttt (32 rows at 15 Hz, a fixed 1.0 s round trip, identical seeds): the sequential loop held the arm ~0.5 s per call, the spliced loop 0 s. Note what this is: the policy predicts from the observation alone (the gateway sends delay: 0 and no action prefix), so it is a client-side splice, not prefix-inpainting RTC — the first rows of a fresh chunk can disagree with the rows they replace by the policy's own variance.

const hlen = new DataView(buf).getUint32(0, true);
const head = JSON.parse(new TextDecoder().decode(new Uint8Array(buf, 4, hlen)));
const chunk = new Float32Array(buf, 4 + hlen, head.n * head.dims);   // row i = chunk.subarray(i*dims, (i+1)*dims)

control (text): {"type":"ping"} → {"type":"pong",t} · {"type":"bye"} closes cleanly.

errors (text): {"type":"error", "code": …, "message": …, "request_id"?, "request_id_origin"?: "caller" | "gateway"}. Live today: a refusal raised after your observation header was read echoes the request_id you sent (bad_observation, an unreadable JPEG, policy_error) — measured, "LIVE-OBS" came back — while the refusals that happen before there is a header (not_leased, idle, a text frame that is not JSON, an unknown control type, a binary frame whose type is not obs) carry no id, so correlate those by order. The change that gives every refusal an id plus request_id_origin (caller when it is yours, gateway when we minted it) is merged and arrives with the next runner update: the browser gateway runs on the runner, not in the Worker, so it does not ship with a web deploy. Three things close the socket: not_leased, idle, and a frame over the 8 MB cap (close 1009, no error frame). After any other error, send the next observation.

4. Rules#

• One socket per session. A second open on the same session is refused at the handshake (HTTP 409).
• One observation in flight. The gateway answers in order; send the next one while the current chunk still has ≥ 0.75 s of motion left (see cursor above) — or after the chunk arrives if you play chunks back to back (sequential: the arm idles one round trip per chunk).
• tokens on each chunk is exactly what is billed: 20 calls on agi-act/so101 (horizon 100) = 2000 tokens.
• JPEG only. Keep an observation under ~1 MB; the hard cap is 8 MB per frame and breaking it closes the socket (1009).
• DELETE /v1/sessions/:id when you stop. A page that just closes bills the same, but the session ends runner_lost a few seconds later instead of client_done.
• Handshake refusals (nothing opens, no zenoh traffic exists): 403 wrong Origin · 403 forged, missing or expired signature, or a signature for another session · 409 already connected · 404 unknown path. GET /healthz on the gateway is unsigned and says nothing about any session.
• A page may read its own session's trace while it runs: GET /v1/sessions/:id/trace, …/trace/live, …/trace/frames/<cam>/<request_id>.jpg (docs/TRACE.md) carry CORS for the same origins.

5. Runner side#

runtime/gateway/server.py     WebSocket gateway, 127.0.0.1:7461 (stdlib + `websockets`), one RemoteClient per socket
runtime/start.sh wire         gateway + `cloudflared --config runtime/cloudflared/quick.yml --url http://127.0.0.1:7461`
                              → ~/agi-diy/run/wire_url (pidfiles run/wire.pid, run/wire-tunnel.pid)
runtime/stop.sh wire          stops it; stopping wire-tunnel removes run/wire_url

The broker re-reads that file every heartbeat and sends it as wire_url (https host only — the Worker rejects anything else and answers 503 no_wire). ~/.cloudflared/config.yml must not be used: its ingress ends in a 404, which is why quick.yml exists.

Repro, both from the repo:

AGI_KEY=agi_sk_live_… node runtime/gateway/wire_client.mjs --model agi-act/so101 --calls 20   # terminal client
AGI_KEY=agi_sk_live_… runtime/gateway/wire_refusals.sh                                   # every refusal above
AGI_KEY=agi_sk_live_… node sdk/js/scripts/browser-wire-proof.mjs agi-act/so101 20            # a real Chrome on agi.diy

Measured 2026-10-24 from a Mac in New Jersey to Thor, through Cloudflare, 20 chunks each: agi-act/so101 60–143 ms (median 66–137), agi-smolvla/so101 364–635 ms (median 420–489) — the policy itself is ~50 ms for ACT and ~400 ms for SmolVLA; the wire adds ~15 ms.

Checkpoint units and task limits#

The stock-* SO-101 descriptors still advertise so101-degrees-gripper-percent-v1 (the agi-* ones advertise so101-normalized-v1), and under either label the numeric arm state/action values are LeRobot normalized −100…100, gripper 0…100. Do not treat them as degrees; keep the handshake identifier until the deployment contract changes and validate hardware calibration/transforms separately. Both policies require six-value state and two 640×480 RGB views: ACT top+wrist, SmolVLA front+wrist. ACT is not language-conditioned; SmolVLA is a sim-trained orange-to-plate checkpoint, not proven arbitrary instruction following.

A successful chunk is evidence of inference, not successful or safe manipulation. Session recording includes frames by default; see TRACE.md for access, retention and opt-out.