The long form.

The long form of the short page: every example, every rule behind a number, in full. Start on /docs; come here when you need the why.

Start here#

Send camera images and joint state. Get the next chunk of actions back. agi.diy runs the policy. Your client runs the control loop.

Pick the path that matches what you have:

• No robot: Open the browser simulation. A simulated SO-101 driven by the live policy. No install.
• Your own SO-101: Prepare your robot. Calibration, cameras and units before anything moves.
• An existing loop: Call from Python, JavaScript or MCP. Keep your code, replace the local inference step.

Everything served today is for the SO-101: our own ACT and SmolVLA pick-and-place checkpoints (the default tier, trained on real teleoperation), our world-foundation-model preview agi-wfm/so101, the third-party originals under their own ids, and any LeRobot checkpoint you bring with hf:. Each was trained for one task in one scene — the table below is the current list, with what each one costs and how it scored.

Browser simulation#

1. Open the simulation. Choose a world and model. Moving the view or dragging an object is local; inference starts only when you run the policy.
2. Sign in when prompted. GitHub login returns to the dashboard; open play again to continue. If the page requests a key, create one in the dashboard and paste it there.
3. Check the displayed cost, then start. Observations travel to the hosted model; returned actions drive the simulated arm. Stop when you have seen enough.
4. Open the session from your sessions. Compare camera inputs, joint state, returned actions and tokens used.
5. Deep link a scene: /play?world=ttt-board&policy=hf:cagataydev/ttt-act-so101&task=pick+up+the+x+piece opens a world with a policy and a task preselected; a world may list the policies it was built for, and a private hf: repo takes your Hugging Face read token for that lease only. The tic-tac-toe world's measured success board is under the page's details and at /boards/ttt-board.json.

The room and robot are simulated. The policy inference and token charges are real. The policy sees rendered geometry cameras, not the decorative Gaussian-splat room view. Moving an object or changing the instruction does not guarantee the policy will complete the task.

Your SO-101#

For an SO-101 with two cameras and a LeRobot-style control loop. A returned chunk is not proof it is safe to execute on your arm.

Check compatibility first#

The two stock-* rows exist so the third-party capacity is nameable rather than hidden: ask for one directly and you get those weights; ask for ours and you get ours or nothing — when every slot of ours is busy the lease answers 409 no_slots (retry_after_s: 15), it never quietly hands you someone else's checkpoint. Either way the session names the weights that answered (pool, checkpoint). Each model has its own page with live availability, inputs, cost per call and a copy-paste first call: agi-act/so101 · agi-smolvla/so101 · all models.

Both cameras are RGB, 640 × 480 pixels (uint8, shape [480, 640, 3]). State is float32, shape [6], ordered shoulder_pan, shoulder_lift, elbow_flex, wrist_flex, wrist_roll, gripper. Camera position, orientation and scene appearance matter as well as names and resolution. Name your cameras like the model's (top/front and wrist, or names containing them such as wrist_cam) or pass --rename_map yourself; agi-diy 0.4.9 and later match names like wrist_cam, refuse to guess between two unrecognised names, and refuse a leftover camera whose name says it is another view (front for ACT's wrist slot; every case); earlier versions can swap views silently. Automatic name mapping does not align your cameras with training views. See where each camera sits and the exact images SmolVLA received (simulation renders matched to its training rig; for ACT, compare your top view with its training data on the Hub).

agi-wfm/so101 is a preview, and the table says so for a reason. It is our own world-foundation-model checkpoint (post-trained from nvidia/Cosmos3-Edge, licence OpenMDW-1.1), its weights are private, and GET /v1/models carries its status (post-training in progress) plus an eval block saying what it is measured against — held-out windows of the same real-teleoperation dataset (lerobot/svla_so101_pickplace, episodes 45–49), scored against our trained-ACT and hold-the-action baselines, which is a prediction error and not a task-success rate.

The flagship's held-out error is withheld, and that is deliberate. Two harnesses measure this checkpoint between 14.74 and 16.35 over the same 2.13 s with every other cause eliminated, so gates.ours.chunk is null, measured_range carries what each one measured and disputed says no comparison may rest on it — treat any single number you see for this row as a bug. What is not in dispute, over that same span: repeating the last action scores 33.33, our ACT 15.76. And note what cannot be done yet: the checkpoint's weights are private, so no one outside can rerun the scoring — two of our own harnesses already differ on it by about 6%. These are our measurements, not verified ones.

Compare over equal TIME, not equal steps. A chunk is horizon actions played at fps, so the three cover different futures: 100 at 30 Hz is 3.33 s, 50 at 30 Hz is 1.67 s, 32 at 15 Hz is 2.13 s. An RMSE over a chunk is therefore an average over that many seconds of motion, and the same horizon is not the same time when the rates differ — which is exactly how the inverted ranking happened. Two rules survive it: compare figures only when their eval.span_s matches, and otherwise use the first action (first_step), the one number no span can distort. For a comparison between models, read eval.common_span — the same policies scored over one span every family can produce, which makes those the only rows you can subtract.

Three more rules, because a number here is easy to misread. (1) Reproducible is not the same as reconciled, and a gap smaller than the spread is not a result. Our ACT is deterministic and reproduces exactly. The flagship and our SmolVLA sample, so the same checkpoint can score differently twice; fixing the sampler's seed makes a run repeat — on the flagship's served path the same observation now returns byte-identical chunks — but that is reproducibility within one harness, not agreement between two. Scoring the same seeded checkpoint through a different loader has landed close to a point away in our own runs, which is why eval carries its spread fields (chunk_spread, first_step_spread, and an across-seed spread where it is known) and why a figure here names the harness and configuration that produced it. Treat any ordering inside the spread as undecided — including an ordering between two of our own models. (2) A figure belongs to a configuration, not just a model. How many denoise steps the sampler runs changes both the error and the latency, so a figure is only comparable with another measured the same way — the hosted API serves one fixed configuration and the record names it where it can (denoise_steps); a number you measure yourself at a different setting, or with a different window stride, is a different measurement rather than a second opinion. (3) None of this is task success. eval.zero_shot is published because it is the unflattering one: scored on two public SO-101 datasets neither checkpoint trained on, both of ours come out far worse than repeating the last action and indistinguishable from each other — no zero-shot transfer has been measured, so do not plan on a scene or task the checkpoint has not seen. Judge a policy on your own trials (Did it work?), not on a third decimal place. Treat it as something to try, not something to trust with hardware you mind: shorter chunk (32 actions at 15 Hz, so about 2.1 s of motion) and fewer slots than the default tier. It is also the slowest of the three to answer — measured on 2026-10-25 over five calls from two machines, a chunk took 1.4–2.1 s (the policy itself 1.5–1.8 s of that) against 372 ms for agi-act/so101. Its chunk is 2.13 s of motion, so the call consumes roughly two thirds to all of the time it buys you: a loop has almost no margin, and a dropped frame means an empty queue. Build in a second request ahead of the one you are playing, or stay on the default tier for anything timed. The price is the same $0.0001 per action, and the tier name buys you nothing but capacity and our attention.

The previous ids still work. act/so101 and smolvla/so101 are aliases of agi-act/so101 and agi-smolvla/so101: a lease accepts either, and answers with the canonical id — so existing code keeps running, and GET /v1/models lists each model's aliases if you want to check. Nothing about your request changes.

The third-party originals are their own ids now. stock-act/so101 and stock-smolvla/so101 lease the checkpoints our own were trained to replace — public weights, apache-2.0, served as their authors published them — so you can measure ours against the original yourself instead of taking our word for it. Same price, same shapes; GET /v1/models carries each one's eval and licence.

The lease tells you whose weights answered. pool is "ours" (one of agi.diy's own checkpoints), "third-party" (a stock checkpoint served as its author published it) or "custom" (your own hf: repo), and checkpoint names the repository that actually answered — with license and provider beside them. A lease of one of our ids never falls back to a third-party checkpoint — ours busy is 409 no_slots, not a different policy on the same id — so pool tells you what you asked for rather than warning you of a substitution. So you never have to infer the weights from a runner id. The API is identical whichever it is: same inputs, same chunk shape, same $0.0001 per action, same normalized units.

Licences, for the public checkpoints, as the Hub cards declare them: both checkpoints are apache-2.0; SmolVLA's training data (LightwheelAI/leisaac-pick-orange) is apache-2.0; ACT's training data (vineetr196/so101_pickplace_v1) declares no licence. agi.diy serves these third-party weights unchanged — check the terms that apply to your own use. Policies and their sources.

30 Hz is the rate your robot plays a chunk back (fps), not a request rate: one ACT call returns 100 rows ≈ 3.3 s of motion, so a loop calls the policy about once per chunk it plays (every few seconds), not 30 times a second.

Check live availability or agi models --json before a run. The table describes the current checkpoints; the session's features, action_horizon and fps describe the model you actually leased.

Action units — read before moving hardware#

A lease of an agi-* model reports semantics: so101-normalized-v1; a stock-* row still reports the legacy so101-degrees-gripper-percent-v1. Read units, not that label. Under either label the arm state/actions are LeRobot normalized values (−100 to 100), with gripper 0 to 100, not joint angles in degrees. Do not send raw returned values to a degree-based motor controller. Do not change the handshake string yourself: it is still the deployed protocol identifier.

LeRobot's default is degrees. The pinned LeRobot defaults so101_follower to use_degrees=true: the arm then reports degrees as state and executes returned values as degrees. These checkpoints need --robot.use_degrees=false. agi-diy 0.4.6 and later add it for so101_follower / so100_follower and refuse --robot.use_degrees=true; 0.4.5 and earlier do not — upgrade with pip install -U agi-diy. Every command in these docs still passes it explicitly; keep it in yours. If you drive LeRobot yourself, set it yourself.

Verify units before motion (one call, nothing executed):

1. Read your calibrated arm in normalized units, in the environment where agi rollout … --dry-run installed LeRobot's Feetech extra. Connecting energizes the motors and holds the current pose; it commands no motion.

from lerobot.robots.so_follower import SO101Follower, SO101FollowerConfig

robot = SO101Follower(SO101FollowerConfig(port="/dev/tty.usbmodemYOUR_ARM", id="YOUR_ARM_ID", use_degrees=False))
robot.connect()  # uses the calibration saved by lerobot-calibrate
state = {k: round(v, 1) for k, v in robot.get_observation().items() if k.endswith(".pos")}
robot.disconnect()
print(state)  # arm joints within -100..100, gripper within 0..100; near a mid pose most arm values sit near 0

2. Send that state once and look at the range of what comes back, without executing it: agi run agi-act/so101 --probe -n 1 checks the wire; for your own state use session.predict() as in the Python example below with observation.state set to these six values, then compare each column's min/max to the same -100..100 / 0..100 ranges.
3. Stop if any value is far outside those ranges, or if step 1 printed values like 90 or -45 at a pose that should be near the middle: your arm is reporting degrees or is calibrated differently.

Confirm your calibrated LeRobot input and output transforms, joint order and limits against the checkpoint before enabling motion. The browser simulator contains its own conversion; that does not calibrate a physical arm. This known metadata mismatch means the hardware path requires verification, not a blind copy-and-run.

ACT is not language-conditioned. SmolVLA takes task text but agi-smolvla/so101 was fine-tuned on one real-teleoperation task (an orange onto a plate, lerobot/svla_so101_pickplace); passing different task text is not evidence of general instruction following. (The third-party stock-smolvla/so101 row is the simulation-trained checkpoint it replaced.) The deployment setting language.enabled: false disables text-generation queries, not task conditioning for action inference. SmolVLA receives task text, but this checkpoint is not validated for arbitrary tasks.

Install in a dedicated environment#

Use Python 3.12. LeRobot, installed on the first inference call, requires Python 3.12 or later, and on 3.14 agi rollout fails inside LeRobot's own flag parser — the CLI says so before installing or leasing. (agi models, agi usage and the REST API work from 3.10.) Run these commands in a terminal, not at the Python >>> prompt:

python3.12 -m venv .venv      # or: uv venv -p 3.12
source .venv/bin/activate
python -m pip install agi-diy
agi login

Create a key in the dashboard and paste it at the login prompt. It is shown once. agi login stores it in ~/.agi/credentials with mode 0600. Alternatively set AGI_API_KEY in your process environment. Never commit a key or put one in a URL.

The first inference call installs LeRobot's remote client from source in the same interpreter (including ML dependencies), so allow time and disk space. AGI_NO_INSTALL=1 disables this and prints the required install command. A dedicated environment avoids changing another application's dependencies. Already use LeRobot? Check the pinned client and dependency requirements in the Python package description.

Verify inference without motion#

agi models
agi run agi-act/so101 --probe -n 1
agi usage

This is one synthetic test: black camera images and a slightly varied synthetic joint state. No hardware connects or moves. For the current ACT checkpoint it costs 100 tokens ($0.01 in usage credit). The CLI prints the chunk's shape, measured latency, bill and session trace link. (100, 6) means 100 future action rows with six joint values each, not 100 separate requests. --probe without -n 1 makes three calls.

Prepare a rollout without starting it#

Calibrate the arm with LeRobot's SO-101 setup, identify its serial port and both cameras, and verify the normalized-unit pipeline above. This complete example is a dry run: replace the port and camera indices for your setup. It prints configuration; it does not lease, install dependencies or move hardware.

agi rollout agi-act/so101 --dry-run -- \
  --robot.type=so101_follower --robot.use_degrees=false --robot.port=/dev/tty.usbmodemYOUR_ARM \
  --robot.cameras='{"top":{"type":"opencv","index_or_path":0,"width":640,"height":480,"fps":30},"wrist":{"type":"opencv","index_or_path":1,"width":640,"height":480,"fps":30}}' \
  --interactive=true --return_to_initial_position=false

Only after the checks pass, removing --dry-run starts the hardware process. --interactive=true waits for /start before policy playback; hardware connection may still energize motors. Keep the workspace clear and a physical stop within reach. --return_to_initial_position=false avoids a commanded return-to-start motion on exit. Ctrl-C is not a safety-rated emergency stop. A network timeout or successful handshake is not a safety guarantee.

If the link drops mid-run — what LeRobot's remote rollout does with the values agi rollout sets (LeRobot @d3793b7e, the commit the runners serve):

1. It finishes the queued chunk, open loop. The arm keeps playing the actions it already has, with no new camera image or joint reading reaching the policy: at most one chunk: 100 rows, about 3.3 s for act/so101; 50 rows, 1.67 s for smolvla/so101. Usually less — agi rollout asks for the next chunk when 0.75 s of motion is left (--inference.refill_seconds; LeRobot's own default is 0.5 s).
2. The moment the queue is empty, it holds. LeRobot re-sends the last position targets, so the motors stay energized at the last commanded pose. This is a position hold, not a brake: the arm can still settle toward that target. A chunk that arrives now resumes the run from a fresh camera frame.
3. After the grace, the run ends. If no chunk arrives within the 3 s grace agi rollout sets (starvation grace, --inference.action_starvation_grace_s; LeRobot's own default is 1 s), the run faults with "action starvation grace expired".
4. On the way out it moves once more, then lets go. When the run ends LeRobot first drives the arm back to its initial position, the pose it had when the run started (a smooth 3 s move; --return_to_initial_position=false skips it), then disconnects with the follower's default disable_torque_on_disconnect=true: torque is released and the arm can fall under gravity. Support the arm, or pass --robot.disable_torque_on_disconnect=false to keep the motors powered after exit.

The same number is your latency budget, and it is not the chunk length. agi rollout asks for the next chunk when 0.75 s of motion is left, so a round trip has about 750 ms before the queue empties and step 2 begins — whatever the horizon. A chunk twice as long does not buy twice the time; it only means you pay for a call half as often. A turnaround over that threshold means a hold on every chunk; one over the 3 s grace ends the run at step 3.

Plan the workspace for the open-loop part and for the move home: whatever the last chunk was doing, the arm completes it, holds, travels back to where it started (unless you turned that off), and then lets go. A hosted policy cannot protect a physical arm through a network outage; your local stop can.

agi rollout fills in remote flags, 30 Hz and JPEG encoding; it installs the required LeRobot extras (including Feetech for SO-101), and closes its lease on exit. Do not switch ACT to SmolVLA without changing top to the correct front camera and rechecking the scene and task. --dry-run prints the exact lerobot-rollout command it would run, every injected flag included.

Camera names#

Name your cameras like the model's and nothing is renamed. Otherwise agi rollout (0.4.9 and later) matches by name, never by order, and refuses rather than guess; a refusal prints the exact --rename_map to pass once you have checked which physical camera is which. For agi-act/so101 (top, wrist) — every row is run against the SDK on each build:

Existing integration#

A session reserves one running policy. Connect within connect_by (60 seconds), send observations, then close it. Default lifetime is 600 seconds; ttl_s may be 60–3600. A lease alone does not run inference. The client remains responsible for observation capture, action scheduling, local safety and ending the session.

Python#

After the terminal setup above, run the one-call probe once to install and verify the wire dependencies. Save this as probe.py, then run python probe.py. It sends synthetic inputs, does not control a robot, and costs one ACT call:

import agi
import numpy as np

observation = {
    "observation.state": np.zeros(6, dtype=np.float32),
    "observation.images.top": np.zeros((480, 640, 3), dtype=np.uint8),
    "observation.images.wrist": np.zeros((480, 640, 3), dtype=np.uint8),
}
with agi.Policy("agi-act/so101") as session:
    actions = session.predict(observation)
    print(actions.shape, actions.dtype)
    print("trace:", "https://api.agi.diy/dashboard/sessions/" + session.id)

Expected structure for today's ACT checkpoint: (100, 6) float32. Values and latency depend on the model and input; this is not a task-success test.

For your real loop, replace all three arrays with synchronized state and RGB camera observations in the checkpoint's units and views. session.features() lists the negotiated names, shapes and dtypes. session.connect() returns LeRobot's admitted RemoteClient; session.config() returns its RemoteInferenceConfig. session.predict(obs, timeout_s=30) returns one chunk; it does not execute it. Use a context manager so exceptions close the lease.

JavaScript and the browser wire#

Nothing to install: three REST calls and one WebSocket are the whole protocol (agi-wire/1 — the frame shapes are in the example below, and hello names the rest). Node 22+ for the global WebSocket; in a browser the page must be served from an allowed origin — api.agi.diy, agi.diy, www.agi.diy or a Vercel preview. Set AGI_API_KEY privately in the environment, save two 640 × 480 RGB JPEGs as top.jpg and wrist.jpg, then node probe.mjs. The zero state makes this an input-format test, not a motion plan — do not execute its actions on hardware.

import { readFile } from "node:fs/promises";

const API = process.env.AGI_API_URL ?? "https://api.agi.diy";
const H = { authorization: `Bearer ${process.env.AGI_API_KEY}`, "content-type": "application/json" };
const api = async (path, init) => {
  const r = await fetch(API + path, { ...init, headers: { ...H, ...(init?.headers ?? {}) } });
  if (!r.ok) throw new Error(`${path} → ${r.status} ${await r.text()}`);
  return r.status === 204 ? null : r.json();
};

const session = await api("/v1/sessions", { method: "POST", body: JSON.stringify({ model: "agi-act/so101" }) });
try {
  const { url } = await api(`/v1/sessions/${session.id}/wire`);          // signed, 120 s to open
  const ws = new WebSocket(url);
  ws.binaryType = "arraybuffer";
  const hello = await new Promise((ok, no) => { ws.onopen = () => {}; ws.onerror = no; ws.onmessage = (e) => ok(JSON.parse(e.data)); });
  const cams = hello.cameras;                                            // ask the policy, never hardcode
  const blobs = await Promise.all(cams.map((c) => readFile(`${c}.jpg`)));
  const head = JSON.stringify({ type: "obs", request_id: "c1", state: Array(hello.state_dim).fill(0),
                                images: cams.map((cam, i) => ({ cam, bytes: blobs[i].length })) });
  const pad = " ".repeat((4 - (Buffer.byteLength(head) % 4)) % 4);       // blobs stay 4-byte aligned
  const len = Buffer.alloc(4); len.writeUInt32LE(Buffer.byteLength(head + pad), 0);
  const chunk = await new Promise((ok, no) => { ws.onmessage = (e) => ok(e.data); ws.onerror = no;
    ws.send(Buffer.concat([len, Buffer.from(head + pad), ...blobs])); });
  const hlen = new DataView(chunk).getUint32(0, true);
  const info = JSON.parse(new TextDecoder().decode(new Uint8Array(chunk, 4, hlen)));
  const rows = new Float32Array(chunk, 4 + hlen, info.n * info.dims);    // row i = rows.subarray(i*dims, (i+1)*dims)
  console.log(info.n, info.dims, info.tokens, [...rows.subarray(0, info.dims)]);
  ws.close();
} finally {
  await api(`/v1/sessions/${session.id}`, { method: "DELETE" });          // the lease, and the bill, stop here
}

hello carries the shapes to use (cameras, state_dim, action_horizon, action_dim) — read them instead of hardcoding, because they differ per model. tokens in the chunk header is what the call cost: one token per returned action.

The same thing with the npm client#

npm i agi-diy@^0.3.0

The browser wire (session.connect(), connectBrowser) ships in agi-diy 0.3.0. Until that version is on npm, its latest tag (0.2.1) has the REST client only — so this line pins ^0.3.0 and fails loudly (notarget) instead of installing a client that cannot run the example below. The frames above need no package and work today.

import { readFile } from "node:fs/promises";
import { createClient } from "agi-diy";

const images = {
  top: new Uint8Array(await readFile("top.jpg")),
  wrist: new Uint8Array(await readFile("wrist.jpg")),
};
const agi = createClient(); // AGI_API_KEY from the environment
const session = await agi.session("agi-act/so101");
let wire;
try {
  wire = await session.connect();
  const chunk = await wire.predict({ state: [0, 0, 0, 0, 0, 0], images });
  console.log(chunk.n, chunk.row(0), chunk.tokens, chunk.rttMs);
} finally {
  wire?.close();
  await session.end();
}

Use both camera views plus matching normalized state for a real observation. A task string travels with the observation for SmolVLA, and the wire gateway passes on its first 512 characters only — a longer instruction is cut without an error, so keep it short and read the echo back. In a browser, JPEG values may also be Blob or ArrayBuffer. The visitor's key must remain private and in memory; never ship your own key in a bundle. The same-origin product can use its signed-in cookie. One socket and one outstanding observation per session; obtain a fresh wire ticket on reconnect, never cache the signed URL.

Existing LeRobot or strands-robots#

LeRobot can use the session's zenoh.endpoint, deployment, encoding=jpeg and zenoh_mode=client directly. Add --robot.use_degrees=false to your own lerobot-rollout command (see action units). Read them from the lease, and read zenoh.endpoints (plural) as an ordered list: the first entry is our TLS router, tls/zenoh.api.agi.diy:7447 — one stable hostname with a public certificate, the host to allow out through a firewall — and the public relay follows as a fallback whose host and port do change. LeRobot takes a single endpoint, so commands.lerobot_rollout carries the first one; a client you write yourself should try them in order, because the router is dropped from the list whenever its bridge to the runner is down. Browser WebSocket transport is agi-wire/1, not a browser Zenoh client.

For strands-robots dev/1.0, session.policy() creates an adapter with the lease's encoding and client mode. PyPI 0.5.x uses a different API. Install the dev package into a dedicated environment and resolve its Zenoh constraint against the LeRobot pin before use:

python -m pip install "strands-robots[feetech,lerobot] @ git+https://github.com/strands-labs/robots@dev"

Inside your already configured application, session.policy() is the adapter passed to arm.run(...). That method moves hardware: review calibration, cameras, units and safety first. A bare strands_robots.Policy(session.url) does not carry the JPEG/client settings. No zero-setup or dependency-compatibility guarantee is implied by the adapter.

Connect a coding agent with MCP#

Read docs and live model availability without a key:

claude mcp add --transport http agi https://api.agi.diy/mcp

For account tools, configure the same server with a private Authorization: Bearer … header. Hosted MCP can create/end sessions, inspect usage/traces and run a recorded session through a model again (billed). It is not a hosted motor-control service. Local MCP adds synthetic probe and rollout_dry_run; after agi login:

claude mcp add agi-local -- agi mcp

Suggested first instruction: “Read the agi.diy docs and inspect model compatibility. Prepare a dry run. Do not start paid inference or move hardware without asking me.”

Bring your own checkpoint#

Any LeRobot checkpoint on the Hugging Face Hub whose config.json says type: act, smolvla or diffusion, weighs at most 3 GB and declares observation.state plus at least one camera can be served on the same wire as the catalog models — the runner loads it for your session only:

agi run hf:<org>/<repo> --probe -n 1         # one synthetic call; a revision pin works too: hf:<org>/<repo>@<sha-or-tag>

import agi
with agi.Policy("hf:<org>/<repo>", hf_token=None) as session:   # hf_token: Hub read token for a private repo, used for this lease only
    print(session.hub)                                           # {"repo", "revision", "policy_type", "license", "size_bytes"}
    actions = session.predict(obs, task="pick up the cube")      # camera names are the checkpoint's own input features

Cameras: bring the ones you have. A checkpoint's config.json often declares more cameras than it was trained with — SmolVLA checkpoints carry camera3 or empty_camera_0 placeholders that were masked in training, and declare 256×256 while the policy resizes whatever arrives. Feeding black frames into those slots is not harmless: on a real tic-tac-toe SmolVLA it made the first action 3× worse than holding still (first-step RMSE 19.5 vs 0.44 with only the two trained cameras at their native 480×640). So the wire carries the cameras you name, by lerobot's own rules: an ACT or diffusion checkpoint needs every declared camera at the declared size; a SmolVLA checkpoint accepts any subset that covers the cameras it was trained with and masks the rest — never a black frame — at any size when resize_imgs_with_padding is set. Default = the trained cameras (read from the checkpoint's train_config.json → its dataset's meta/info.json) at their trained size; when that metadata is unreachable nothing is guessed and every declared camera is required.

curl https://api.agi.diy/v1/models/hf:<org>/<repo>      # required / optional / resizable cameras — nothing leased, nothing billed
agi rollout hf:<org>/<repo> --dry-run -- --robot.type=so101_follower --robot.port=/dev/ttyACM0 --robot.use_degrees=false \
  --robot.cameras='{"camera1": {"type":"opencv","index_or_path":0,"width":640,"height":480,"fps":30}, "camera2": {...}}'   # the plan: which cameras lease, which are left out
agi rollout hf:<org>/<repo> -- --robot.type=so101_follower --robot.port=/dev/ttyACM0 --robot.use_degrees=false \
  --robot.cameras='{"camera1": {"type":"opencv","index_or_path":0,"width":640,"height":480,"fps":30}, "camera2": {...}}'

plan = agi.Policy.inspect("hf:<org>/<repo>")              # {"cameras": {"rule": "subset", "required": [...], "optional": [...], "resizable": true, ...}}
with agi.Policy("hf:<org>/<repo>", cameras={"camera1": (480, 640), "camera2": None}) as session:   # your frame size; the policy resizes
    ...

agi rollout derives cameras from your --robot.cameras (names mapped the usual way, your own width×height when the policy resizes) and tells you which optional cameras it left out. Over HTTP pass "cameras" on POST /v1/sessions; a wrong set is 400 bad_request naming the required and optional cameras before anything is leased.

Over HTTP the session is created with {"model":"hf:<org>/<repo>"} (and "hf_token" for a private repo). It answers 201 with state: "warming" and ready_in_s (ACT about 10 s, SmolVLA about 45 s, measured on the runner); poll GET /v1/sessions/:id until leased, then connect within 60 s of ready_at as usual. model comes back pinned to the commit that was validated (hf:<org>/<repo>@<sha>), semantics is so101-normalized-v1, and features, cameras and action_horizon are the checkpoint's own. Mind the two spellings: features keys are fully qualified (observation.images.front), while an observation's images[].cam takes the short name — and hello.cameras lists exactly those (a checkpoint trained on one front camera answers ["front"], not top/wrist). Getting it wrong is silent: a cam the policy does not know is not refused, that camera arrives black, and the chunk you get back is billed like any other (wire frames). The SDK does the polling for you (agi.Policy returns once the checkpoint is leased; wait_ready=False to return at once).

Billing is unchanged: one token per returned action. The custom row of GET /v1/models shows the free custom slots (2 per runner), the accepted policy types and the size cap. Limits per account: 2 live custom sessions, 10 cold starts per hour. A repo that declares no licence is served, and hub.license is null — the licence of what you serve is yours to check. tokens_used on the lease can still read 0 for a second after your last chunk — metering is batched once a second, so a DELETE sent immediately answers before the last call is counted; GET /v1/usage and the dashboard settle within a second or two. A checkpoint the runner cannot load ends the session with end_reason: "load_failed"; refusals before anything is leased are unsupported_policy, too_large, missing_config, hf_forbidden and unknown_model (the Hub answers the same for a repo that does not exist and a private one without a token).

HTTP lifecycle#

GET /v1/models is public. POST /v1/sessions with {"model":"agi-act/so101","record":false} reserves a slot using bearer auth or the same-origin cookie; record:false disables the session recording, not usage accounting. Read /v1/sessions/:id, connect over Zenoh or the browser wire, then DELETE /v1/sessions/:id. Sending camera frames to the session-creation endpoint does not run inference.

Billing#

One action = one token = $0.0001. A call returns many actions. New accounts receive 200,000 tokens ($20). There is no self-serve top-up yet: when the grant is spent, running sessions end (credits) and new leases answer 402 insufficient_credits.

A 100-row chunk is billed for all 100 actions, even if you execute only part of it. At 30 Hz it represents about 3.3 seconds of planned motion; 50 rows represent about 1.7 seconds; the flagship's 32 rows at 15 Hz represent about 2.1 seconds. These are action durations, not response-latency promises.

New accounts receive 200,000 tokens ($20 in usage credit), granted once. Dollar figures describe inference credit, not a cash balance. Leasing, connecting and idling do not consume tokens. Synthetic probes, browser policy runs and replays do. Check the session's actual horizon rather than assuming every call costs 100 tokens.

Use the dashboard, agi usage or GET /v1/usage to inspect spending. Metering is asynchronous; the UI may briefly lag a call. A session ends on close, expiry, exhausted credits, runner loss or failure to connect in time. Never create a second lease merely to retry a request against a still-active session.

Traces#

A trace records what was sent and returned: camera frames, joint state, task text, action chunks and timing. Open it from sessions or the CLI's trace link. Black probe frames and rendered simulation frames are not physical-camera evidence. Returned actions are planned targets, not measured proof that the robot reached them.

• Viewing or scrubbing a saved trace does not call the policy or move hardware.
• Run inference again (POST /v1/sessions/:id/replay) feeds saved observations through a model again, creates a new session and is billed normally. It does not reenact the physical scene. Whether you get the same actions back depends on the row: our ACT is deterministic, the flagship's served deployment now fixes its sampler seed (the same observation returned byte-identical chunks on the served path, 2026-10-25), and our SmolVLA sets no seed, so treat that one as non-reproducible until measured. The run-to-run spread in eval still belongs to the published figures, which were measured unseeded — so a comparison of two scores must clear it even though a replay now repeats.
• Traces are available through the API for 24 hours from session creation. The runner sweeps inactive trace files after its retention window; this is not a guaranteed deletion at an exact wall-clock instant. Disk pressure can omit camera frames.
• Session recordings are on by default. Use record:false at session creation to opt out. Python Policy currently has no record keyword; use REST or JavaScript agi.session(model, { record: false }) when you need this control.
• Owner cookies and the leasing key can read a trace; authorized operator accounts can also inspect sessions and traces, with administrative reads logged. Frames are not public, but this is not zero-retention inference. Usage/session accounting is separate from the short-lived trace.

Reading latency#

Three numbers appear in the tools; they measure different spans, so never compare one with another:

Compare your round trip with the refill threshold, not with the chunk length. The client asks for the next chunk while a little motion is still queued, so that threshold — and not the horizon — is how long a call may take before the arm starts holding its last target; a chunk twice as long only means you pay for a call half as often. The number, and what happens past it, are under Your SO-101, generated from LeRobot's own constant. Measured turnarounds per model, with the margin each leaves, are in docs/ROLLOUT.md. A large gap between round trip and time on the runner is network, not the model. More in how it works.

Troubleshooting#

An HTTP error carries error and message, with retry_after_s when available, and a doc_url that links the row below. Every error answer also has an x-request-id header — quote it when something looks wrong and we can find that exact request (curl -sD - … prints it; cross-origin browser code can read it too — we expose it). Treat unavailable status data as unknown, not idle capacity.

The five hf_…/hub_…/missing_config/unsupported_policy codes below concern bring-your-own Hub checkpoints, which are enabled per account; you will not see them while you use the catalog models.

predict() in Python can reconnect after a relay-port change; relay_wait_s=20 allows waiting for recovery. A hardware rollout cannot transparently recover every interruption: stop, inspect the physical state and establish a safe restart. See status before retrying.

Limits#

Every number in this table is read out of the API source on each release, so the table and the running service cannot disagree.

What is served today: SO-101 only, the checkpoints in the table above — ours, the third-party originals, a world-foundation-model preview and any hf: repo you bring — across a small shared pool whose size and runners you can read live: live capacity, what changed. Latency, scene mismatch and policy mistakes stay inside your control loop. There is no general robot support, no self-calibration, no fine-tuning endpoint, and no redundancy or uptime SLA.

References#

• Dashboard · Simulation · Sessions · Status · Changelog
• OpenAPI 3.1 description — every public route, field and error code, checked against the API source on each release; load it into Postman, a code generator or an agent.
• Python package · JavaScript package
• LeRobot remote inference