How it works.

The model behind every agi.diy call, in the words the product uses. Each diagram is generated at build time from the deployment files, the Worker constants and one recorded session (web/scripts/concepts.mjs); the numbers are labelled with where they came from.

Three ways in, depending on why you are here:

• Deciding whether this is for you — Policies (what is served today), The loop (one call), Tokens and billing (what a minute costs).
• About to connect an arm — Normalized units (the one setting that can break a real arm), Observations and cameras, Compatibility, When the link drops.
• Judging what came back — Did it work? (what the service does and does not tell you), Latency, Traces, Simulation and real.

Policies#

A policy is a trained model that turns what a robot sees and feels into what it should do next. agi.diy serves policies over an API: you send an observation, it returns actions. Your robot, your control loop and your safety checks stay on your side.

Two policies are served today, both for the SO-101 arm:

Both checkpoints are task-specific. ACT takes no text at all. SmolVLA reads the task sentence, but it was fine-tuned for one task in one kind of scene, so a different sentence is not general instruction following.

Each model has one page with everything you need to judge it — inputs, price, measured times, licences, a one-call test and live availability: /models/act-so101 and /models/smolvla-so101.

Licences, as each Hub card declares them at the revision the runner serves (checked 2026-10-25):

These are the upstream authors' declarations, not a legal review; check them against your own use. The per-model facts are also available as data: /concepts/models.json.

Your own checkpoint. You are not limited to the two: model: "hf:<org>/<repo>[@<revision>]" leases a dynamic deployment of any LeRobot checkpoint on the Hugging Face Hub. The hf: prefix is what tells us the id is yours and not one of ours — leave it off and vineetr196/act_so101_pickplace_v1 is read as a catalog id and refused 404 unknown_model, which is the first mistake most people make. The checkpoint has to be something the runner can load: an act, smolvla or diffusion policy, weights as safetensors — a pickled file is refused rather than executed — at most 3 GB, declaring observation.state and at least one camera. We validate the repo before charging anything, pin the commit you get (model comes back as hf:<org>/<repo>@<sha>), and read the features and the action horizon out of the checkpoint itself, so your own training decides the shape of the loop. A private or gated repo needs an hf_token; it is handed to the runner once and erased when the deployment is up.

Two things come from the checkpoint and not from us, and both will catch you out if you assume otherwise. The camera names are whatever its config.json declares — lerobot/smolvla_base asks for camera1, camera2, camera3, not our top and wrist — so read features on the lease (or hello.cameras on the wire) and send exactly those keys. And the action horizon is the checkpoint's, so a chunk may be a different length than ours.

Loading is the one place a custom checkpoint differs from ours: the session starts warming instead of leased — ready_in_s estimates roughly 10 s for an ACT and 45 s for a SmolVLA — and you poll until leased, then connect within 60 s. Measured on production, the same SmolVLA repo took 33.7 s, then 25.1 s, then 20.8 s across three leases: the weight cache is what shrinks it, so the slow one is the first lease of a repo nobody has run yet. A checkpoint the runner cannot load, or 300 s of warming, ends load_failed. Two live custom sessions and ten cold starts an hour per account; the price per action is the same as ours.

Direction, not inventory. Larger VLAs and world foundation models (WFMs) — models that predict how a scene will evolve — are where the platform is going. No WFM endpoint and no other robot is served today; Limits says what is. One checkpoint of ours is loaded on the runner for evaluation and is deliberately not listed in the catalog: it does not appear in GET /v1/models, and a session request for it from an ordinary account is refused like any unknown model (operators can lease it to run the evaluations, which is logged like their other reads) — availability is what the catalog means, not existence.

The loop#

One call is one observation in and one chunk out. The robot starts playing the chunk while it already sends the next observation, so the network round trip hides inside the motion the previous chunk bought. Two transports carry the same call:

• Python / LeRobot — the lease names a Zenoh endpoint and a private deployment name; lerobot connects with its RemoteClient, images JPEG-encoded. agi run, agi rollout and agi.Policy do this for you.
• Browser — GET /v1/sessions/:id/wire returns a short-lived signed WebSocket URL speaking agi-wire/1; /play and the dashboard test call use it. No Python, no Zenoh in the page. Its own limits: open the socket within 120 s of minting, one message at most 8 MB (two JPEGs are about 60 KB), and send an observation at least every 120 s or the gateway closes the socket with idle. When it refuses something it says so in one line: on production today that is {type: "error", code, message} — e.g. bad_frame, "Text frames must be JSON." (probed 2026-10-25). The merged gateway adds request_id and a request_id_origin that says whether the id is the one you sent or one the gateway minted; it reaches you when the runner's gateway next restarts, because that process is deployed separately from the API.

Either way the bill and the recording come from the same place on the runner, so there is one meter, not two.

Sessions#

POST /v1/sessions leases one warm slot — a policy process already loaded on the GPU — and returns a session: its id, the deployment name, the endpoint, the features the model expects, the horizon and the fps. Nobody else can use that slot until the session ends.

• warming — only for your own checkpoint (hf:<org>/<repo>): the runner is loading it, so there is no slot to attach to yet. Poll the session until it reads leased, then connect. A checkpoint the runner cannot load ends the session load_failed.
• leased — reserved, waiting for your client. If nothing attaches within 60 s the session ends never_connected and the slot goes back to the pool.
• active — a client attached (reported within about a second).
• ended — client_done (you closed it, or the SDK did on exit), expired (ttl_s, default 600, allowed 60–3600), credits (usage credit reached zero), runner_lost (the runner stopped reporting the slot), replay_failed, or load_failed (a checkpoint of yours the runner could not load).

A session costs nothing to hold; only returned actions are billed. Closing is still good manners: the slot is free for the next person only after the runner respawns it (10–30 s measured). Revoking an API key refuses future calls made with it; sessions already running keep going until they end.

Availability#

GET /v1/models (public, no key) reports each model's state and its free slots. Today one GPU runner keeps 2 ACT slots and 3 SmolVLA slots warm. The words the product uses:

A ready model is a promise about capacity, not about inference: the first call is what proves the policy answers.

More than one arm#

A slot serves one client at a time, so two arms means two sessions and the pool is the ceiling: 2 ACT slots and 3 SmolVLA slots on one runner today (Availability). The lease after the last free slot is 409 no_slots with retry_after_s: 15 — a refusal you retry, not a queue you wait in.

Running them together costs time as well as slots. Measured on the runner (runtime/CAPACITY.md, 2026-10-24, local client): SmolVLA 365 ms alone, about 495 ms each with two streams — +35 %, not double, because one 500M-parameter forward pass does not saturate the GPU. Extrapolated from that same increment: 3 streams about 620 ms and 5 about 880 ms per call, still inside the 1.67 s a 50-action chunk buys, so each arm stays ahead of its queue. Five concurrent streams have not been run for real yet. Your round trip is on top of these numbers (Latency).

Credits are per account, not per session: two SmolVLA arms at one call per second spend about twice 2,900 tokens a minute. Each session bills itself, and the dashboard total is the sum.

Observations and cameras#

An observation is one snapshot from the robot:

• observation.state — six float32 joint values, in this order: shoulder_pan, shoulder_lift, elbow_flex, wrist_flex, wrist_roll, gripper.
• two RGB camera images, 640×480 each, under the model's camera names: ACT observation.images.top + observation.images.wrist; SmolVLA observation.images.front + observation.images.wrist.
• for SmolVLA, the task text — over the browser wire only the first 512 characters reach the policy and the rest is cut silently (the echo and the trace show the short version); a rollout over the public relay passes your task through unchanged.

The names matter because each policy learned from particular viewpoints: top/front is a fixed camera looking at the table, wrist rides on the gripper.

front · what smolvla/so101 received (simulation)

wrist · what smolvla/so101 received (simulation)

That is where /play places them: the table camera's pose and field of view were taken from SmolVLA's training rig. ACT's training camera pose is not published; in /play its top image comes from this same camera, which is a stand-in rather than ACT's rig. For ACT, compare your top image with the checkpoint's training data on the Hub rather than trusting a number.

agi rollout maps your two cameras onto the model's names by name, refuses a wrong count, and refuses names it cannot place — all before anything is leased; the exact rule, with examples run against the SDK, is under Compatibility. A missing image is not an error on the browser wire — the gateway fills it with black so a connection test can run — but a policy fed black frames is not seeing anything.

Chunks, horizon and fps#

A chunk is the sequence of actions one call returns: n rows × 6 joints, float32. Each row is one action — six target joint values for one tick of the control loop. The horizon is n: 100 for ACT, 50 for SmolVLA, fixed when the session is leased.

fps is the rate the chunk is meant to be played at: 30 actions per second for both models. So one ACT chunk is about 3.3 s of planned motion and one SmolVLA chunk about 1.7 s. That is an execution rate, not a promise about how many requests per second the service answers.

The chart above is the first chunk of a recorded session: the camera images it answered were rendered by the browser simulation; the chunk itself is the live policy's output.

Tokens and billing#

One returned action is one token, and one token costs $0.0001. A call is billed for every row in its chunk, whether or not your robot plays all of them:

A minute of motion costs more than 30 tokens a second. Billing counts every returned row, but the robot does not play a whole chunk before the next one arrives: LeRobot asks for the next chunk while 0.5 s of motion (or one round trip, if that is longer) is still queued, and the new chunk replaces the unplayed rest. Recorded with agi rollout --fake (agi-diy 0.4.9, over the public relay, 2026-10-25): smolvla/so101 settled at one call every 1.03 s — about 48 actions a second — about 2,900 tokens, $0.29, per minute of motion; act/so101 played 3.12 s of its first 3.33 s chunk before the next call — about 32 actions a second — about 1,900 tokens, $0.19, per minute (one interval). A slower network makes the requests come earlier and the minute cost more; the session total on the dashboard is the number to plan with.

The picture is that recorded SmolVLA run, event by event: the first calls were slow enough that the queue ran dry and the arm held its last target, then the loop settled into a call every 1.03 s — so roughly a third of every chunk was replaced before it could be played, and billed all the same.

A token is a robot action, not an LLM text token. New accounts receive 200,000 tokens once — shown as $20 in usage credit. Usage credit is a balance for calls, not cash. A session can start only while the balance is above zero; when it reaches zero, live sessions end with credits (the balance can dip below zero by at most one metering batch). The first CLI command in the docs, agi run act/so101 --probe -n 1, is exactly one call: 100 tokens, $0.01.

Normalized units#

Both live checkpoints were trained on LeRobot's normalized motor positions (use_degrees=false, the default when their datasets were recorded): the five arm joints run from −100 to 100 across each joint's calibrated range, the gripper from 0 to 100. The values are not degrees, even though the session's semantics label still reads so101-degrees-gripper-percent-v1. That label is a known mismatch kept for compatibility until the contract changes; trust the data, not the name.

Set use_degrees=false on a real arm. Current LeRobot defaults the SO-101 follower to use_degrees=True: the arm would then report its state in degrees and execute these normalized actions as degrees. Pass --robot.use_degrees=false to lerobot-rollout. agi rollout from agi-diy 0.4.6 adds it for you and refuses true; on 0.4.5 or earlier, pass it yourself after -- (pip install -U agi-diy to upgrade). The session label cannot catch this — it says degrees either way.

What ±100 means physically depends on your arm's LeRobot calibration file. Confirm joint order, sign and limits against the checkpoint before you let a chunk move hardware. The browser simulation converts its own radians to these units; that conversion does not calibrate a real arm.

Latency#

Three numbers, never one:

• policy time — the model's forward pass (server_ms.policy in a trace).
• time on the runner — from the observation reaching the runner to the chunk leaving it: policy time plus decoding and post-processing. A trace's rtt_ms is this number; it does not include your network.
• round trip — what your client measures from sending the observation to holding the chunk: time on the runner plus the network both ways. Only your client can measure it.

A recorded example: three act/so101 test calls from a Mac over the public relay (session ses_fxA3qHVEJekI5JH3TleZUf, 2026-10-25, synthetic inputs). Policy time 18.7–67.2 ms, time on the runner 35.3–98.4 ms, round trip at the client 59–157 ms. Here the first call was the slowest, and for ACT the network was most of the wait.

The same test for smolvla/so101 (session ses_L2hzN236hB45ArRQkeSf1d, 2026-10-25): policy time 366.6–416.6 ms, time on the runner 381.9–446.1 ms, round trip at the client 552–1141 ms. The runner's share stayed flat; the public relay's did not — the third call spent about 760 ms on the network. Plan your loop around the round trip you measure, not the runner's.

Measured on the runner with a local client (runtime/CAPACITY.md, 2026-10-24): ACT 18–20 ms per call; SmolVLA 365 ms alone, about 495 ms each with two sessions at once. The public relay has added roughly 60–150 ms to a round trip. What matters for a smooth loop is the ratio: a round trip must finish before the chunk you are playing runs out.

In the recorded SmolVLA session above, time on the runner per call (389–444 ms, 373–413 ms of it policy time) was about a quarter of the 1.67 s its chunk bought; that trace does not hold the browser's own round trip. Your numbers will differ with network, load and model; these are dated measurements, not a service level.

When the link drops#

The robot plays chunks from a local queue, so a network drop does not stop the arm at once — and it does not let it run on forever either. The figure shows the worst case; the four steps under it are the same paragraph /docs carries, generated from one source (web/lib/linkdrop.mjs).

If the link drops mid-run — what LeRobot's remote rollout does with the values agi rollout sets (LeRobot @d3793b7e, the commit the runners serve):

1. It finishes the queued chunk, open loop. The arm keeps playing the actions it already has, with no new camera image or joint reading reaching the policy: at most one chunk: 100 rows, about 3.3 s for act/so101; 50 rows, 1.67 s for smolvla/so101. Usually less — LeRobot asks for the next chunk when 0.5 s of motion is left.
2. The moment the queue is empty, it holds. LeRobot re-sends the last position targets, so the motors stay energized at the last commanded pose. This is a position hold, not a brake: the arm can still settle toward that target. A chunk that arrives now resumes the run from a fresh camera frame.
3. After the grace, the run ends. If no chunk arrives within the 3 s grace agi rollout sets (starvation grace, --inference.action_starvation_grace_s; LeRobot's own default is 1 s), the run faults with "action starvation grace expired".
4. On the way out it moves once more, then lets go. When the run ends LeRobot first drives the arm back to its initial position, the pose it had when the run started (a smooth 3 s move; --return_to_initial_position=false skips it), then disconnects with the follower's default disable_torque_on_disconnect=true: torque is released and the arm can fall under gravity. Support the arm, or pass --robot.disable_torque_on_disconnect=false to keep the motors powered after exit.

Plan the workspace for the open-loop part and for the move home: whatever the last chunk was doing, the arm completes it, holds, travels back to where it started (unless you turned that off), and then lets go. A hosted policy cannot protect a physical arm through a network outage; your local stop can.

Traces, playback and running inference again#

Every session is recorded by default as a trace: each observation the policy saw (joint state, task, camera frames) and each chunk it returned, with timing.

Camera images and actions are recorded for session inspection — frames may show your room, so treat them like any camera recording.

Who can read a recording. The account that owns the session, through its browser cookie, or the API key that leased it. One more case, said plainly because the alternative is a surprise: an operator account — the logins in the Worker's ADMIN_LOGINS, today one — can read any session and read its trace and frames. It stops there: an operator cannot open a session's browser-wire URL, because a wire socket sends observations and returns chunks and every action is billed to the session's owner — that capability was removed, and an attempt is refused and logged {admin_denied, login, session}. Every operator read is written to the Worker's log as {admin_read, login, session, owner}. Nothing else reaches a recording: traces are not public, a frame is not served without the owner's cookie or the leasing key, and record: false at session create means there is nothing to read at all — the bill and the session row stay either way.

• Playback — viewing a trace on the dashboard. Free; nothing runs.
• Run inference again (POST /v1/sessions/:id/replay) — feeds the recorded observations to a policy in a new session, at the recorded cadence. It is billed like any session, moves no robot, and may return different actions. The target has to be a model served right now and it has to expect the same cameras (act/so101 top/wrist cannot run on a smolvla/so101 front/wrist recording). With no model in the body the target defaults to the recording's own — which for a custom checkpoint is the pinned hf:<org>/<repo>@<sha> that nobody serves once the session ended, so a custom session must be replayed with an explicit model.
• Retention — traces are readable through the API for 24 hours from session creation; the runner deletes older recordings periodically, and skips frames (keeps events) if its trace disk passes 20 GB. record: false on POST /v1/sessions turns recording off; billing and session history are kept either way.

Did it work?#

Nothing in the API answers that. A session tells you the policy answered — a chunk came back, in this many milliseconds, for this many tokens — and how the run ended (end_reason, see Sessions). There is no success, reward or score field in a session, a trace event or any other response: whether the orange ended up on the plate is your measurement, in your workspace, against your task.

So judging a policy is three things you do yourself:

1. Watch the run. On a real arm that is your own cameras and your own eyes; in the browser it is /play, where the same policy drives a simulated SO-101 you can see from any angle. Count attempts and completions by hand — a dozen honest trials say more than any number we could return.
2. Read the trace. The recording holds exactly what the policy was asked and what it answered, frame by frame. A failed grasp usually shows up there first: a camera pointing at the wrong thing, a state the checkpoint never saw in training, a chunk that barely moves.
3. Ask the same questions again — run inference again on a recorded session (POST /v1/sessions/:id/replay) and the recorded observations go to a policy a second time, at the recorded cadence, as a new billed session. That is the honest way to compare two policies: identical inputs, different model, no robot in the room. The refusals come in the order the slot is resolved, not the order you would guess: a model nobody serves is 404 unknown_model, a model whose slots are all busy is 409 no_slots with retry_after_s (retry it), and only a target that is actually available gets as far as the camera and state comparison — 409 incompatible_features, which names both camera lists.

What re-running does not give you is a different world. The recording answers the questions the first run asked; the moment a policy acts differently the real scene would have diverged, so a replay compares judgement on fixed inputs, not task success. For that, go back to step 1.

Simulation and real#

/play is an SO-101 simulated in your browser — MuJoCo physics and rendered cameras — driven by real calls to the live policies. Simulated inputs, live policy output. Two views exist there: the camera images sent to the policy are rendered from the simulation's geometry, and the photographic room behind the arm is a human view only — a policy never sees it. Running a policy in /play spends usage credit like any other call.

The recorded session on the landing page and in the charts above is from that simulation, labelled as such. No result here shows a physical arm completing a task: a returned chunk is not proof the task succeeded.

What it models, and what it cannot. Each action row is written as a position target for a modelled STS3215 servo (kp 998.22, kv 2.731, torque limited to ±2.94 N·m), stepped at a 0.005 s timestep — 200 steps a second — with joint damping 1, friction loss 0.1 and armature 0.005. So a chunk in /play is followed the way a servo follows it, with its lag and its torque ceiling, not teleported joint by joint. Objects get the mass and friction the world manifest declares (0.15 kg when it declares none), and the contact model is MuJoCo's.

What no number in that list can stand in for: your arm's own calibration, backlash and wear; the grip your gripper actually has on the thing in front of it; and your cameras — exposure, white balance, motion blur, lighting. The policy's images in /play are rendered from the scene's geometry, which is a cleaner picture than a camera takes. Treat a run there as a test of the loop, the units and the bill; the only test of a task is your own arm in your own room.

Compatibility#

Your setup matches a served model when all of these hold:

1. Robot — an SO-101 follower arm driven through LeRobot (--robot.type=so101_follower).
2. State — six joint values in the order above, in normalized units from your own calibration (--robot.use_degrees=false).
3. Cameras — exactly two, 640×480 RGB, placed like the model's: a fixed table view (top for ACT, front for SmolVLA) and a wrist camera.
4. Rate — your loop plays actions at the model's 30 fps.
5. Scene — close to what the checkpoint was trained on. Task-specific checkpoints degrade quickly with a different table, lighting or objects.
6. Python — 3.12 for agi rollout and wire inference. LeRobot at the pinned commit requires Python ≥ 3.12, and on Python 3.14 lerobot-rollout crashes while parsing its own flags (TypeError: str | None is not callable, raised in draccus; the session closes with 0 tokens). The REST calls alone (agi models, --dry-run) work on any supported Python.

agi rollout checks the first four before it leases anything (the Python version it does not check yet), so a mismatch costs nothing. What it does with each (sentences quoted from sdk/python/agi/rollout.py):

Name each camera after the view it is. agi-diy 0.4.9 and later match cameras by name, never by order: a camera whose name contains exactly one view word as a whole word takes that slot (wrist_cam → wrist, Front → front; laptop never counts as top). One leftover camera takes the remaining slot only if its name does not say it is another view: {"wrist": …, "table": …} → table → top for act/so101, but {"top": …, "front": …} is refused (front is not wrist), and so is a name with two views (top_wrist). Two cameras it cannot place are refused too. Every refusal prints the exact --rename_map to pass — check that each one really is that view, and read the cameras: line the plan prints. agi-diy 0.4.8 keeps only exact names and maps the rest by position, and 0.4.7 and earlier map purely by position: both can swap the two views silently. agi --version says which one you have.

agi rollout --dry-run composes the full lerobot-rollout command from the public model list without a key and without moving anything; --fake runs the loop with a hardware-free robot and black frames (calls are still billed). See Your SO-101.

Glossary#

The product's vocabulary, generated from web/lib/vocab.ts. The same list as data: /docs/concepts/glossary.json.

Say this, not that#

The words to avoid, from the same avoid[] lists in web/lib/vocab.ts that the drift guard reads.

Today the build refuses 12 phrasings outright; the other 33 are review rules — a row with a context names the one place the word is wrong, because elsewhere it is fine ("no robot moves" is not a billing unit).

12 further phrasings are refused outright by the build and are not quoted here — quoting one would fail the guard that bans it. They live in DEPRECATED in web/lib/vocab.ts (2 of the rows above would have repeated them).

Where you meet these words#

Every term above is printed somewhere in the product. This is where to look, so a definition can be checked against the thing itself rather than taken on faith.

Where to go next#

• /docs — the reference: every route, field, error code and limit, with runnable examples.
• /play — a simulated SO-101 driven by the live policies; costs usage credit like any call.
• /models — one page per model: inputs, price, measured times, licences, availability.
• /dashboard — your keys, sessions, spend and traces.
• /docs/concepts/glossary.json — this vocabulary as data, for agents and writers.