Introducing the Sidekick APIBeta

Robotics is in the middle of its own foundation-model moment: a fast-growing menu of genuinely capable foundation and world models, each one good at something a little different. Every serious model in that menu comes with its own inference server, its own checkpoint format, its own idea of what an "action" looks like, and its own opinion about how many camera views it needs and what to call them. Getting one of these models onto an arm is a satisfying few days of engineering. Getting several of them onto the same arm, so you can reach for whichever one is actually best at the task in front of you, is a bigger integration project, and it's the reason most teams pick one checkpoint and standardize on it rather than treating model choice as a decision they can revisit later.

Sidekick's answer is to put a single, unified API in front of all of it. No one has done this before in robotics. You send an observation and an instruction; the service decides which model is best positioned to answer, calls it, and falls back to another if the first choice is unavailable. The Python client, sidekick-sdk, is a thin, single-dependency wrapper around that API, built to run comfortably on a robot control computer or inside a ROS container rather than in a data center. Here's what it actually does.

pip install sidekick-sdk

from sidekick_sdk import Sidekick

sk = Sidekick(api_key="sk-sidekick-...")

act = sk.act(
    model="sidekick/auto:manipulation",   # a job, not a checkpoint
    instruction="fold the towel",
    action_space="joint_pos_14",
    observations=[sk.observation(image_path="frame.jpg", proprio=robot.joints())],
)

"sidekick/auto:manipulation" isn't a model; it's a domain alias. It tells the router "give me whatever live manipulation model is the best fit right now," and lets Sidekick handle the part you don't actually want to think about: which checkpoint is currently healthy, which is fastest, which is cheapest, and which one just started throwing errors. You can still address a specific checkpoint by name ("physical-intelligence/pi-0") when you need reproducibility, but the alias is the default posture the API is designed around.

Every response (from act(), predict(), or ground()) carries a route object that makes the routing decision an artifact you can inspect rather than a black box:

act.route["model"]          # which checkpoint actually answered
act.route["fallback_used"]  # True if the first choice didn't respond
act.route["simulated"]      # True means placeholder output: no GPU ran

That last field matters enough that it gets its own section below.

predict() rolls a world model forward from the current observation and returns the frames it imagines: useful for planning, for sanity-checking a policy before you trust it, or for visual model-predictive control. It also has a streaming counterpart, stream_predict(), which yields (event, payload) tuples over server-sent events as frames are generated, rather than waiting on the full rollout.

for event, payload in sk.stream_predict(
    model="sidekick/auto:world-model",
    observations=[obs],
    instruction="what happens if I push the mug",
    horizon=16,
):
    if event == "frame":
        ...  # a new predicted frame just arrived

ground() is the API's spatial-reasoning primitive. It takes an instruction like "point at every graspable object" and returns labeled points and boxes in normalized image coordinates, not actions, on purpose:

g = sk.ground(observations=[obs], instruction="point at every graspable object")
print(g.points)       # [{"label": "mug", "x": 0.42, "y": 0.61}, ...]
print(g.labels())     # sorted, deduplicated label set across points and boxes

Grounding output lives in image space, not robot space. Feeding it straight to an actuator would mean commanding joint targets with pixel coordinates, so the return type (Grounding) is structurally distinct from an Action; there's no accidental path from one to the other.

evaluate() runs a policy against a named benchmark scenario and returns a job you poll:

job = sk.evaluate(model="physical-intelligence/pi-0",
                   policy_endpoint="https://my-policy.example/act",
                   scenario="pick-and-place-cluttered", episodes=25)

result = sk.wait_for_evaluation(job.id, poll_s=2.0, timeout_s=3600)
print(result.status, result.results)

obs = sk.observation(image_path="frame.jpg", proprio=robot.joints())
obs = sk.observation(image_url="https://cdn.example/frame.jpg", proprio=...)

sk.observation() inspects whatever you pass (a URL, a local file path, or a raw base64 string) and figures out which one it's looking at, so a caller assembling a multi-camera rig doesn't have to annotate the encoding of each entry by hand. For rigs with more than one view, the cameras keyword takes a name-to-source mapping directly:

obs = sk.observation(
    cameras={
        "cam_high":        grab("cam_high"),
        "cam_low":         grab("cam_low"),
        "cam_left_wrist":  grab("cam_left_wrist"),
        "cam_right_wrist": grab("cam_right_wrist"),
    },
    proprio=robot.joints(),
    tactile=robot.tactile(),
)

This exists because the checkpoints most people actually reach for need it: pi0.5 on ALOHA expects four named views. The API also accepts depth images and camera intrinsics per frame, for models that use them.

Two properties of this design are worth understanding well, since the API is intentionally permissive here and leaves them to the caller rather than validating every possibility server-side. First, camera names are contractual: they have to match what the checkpoint declares. Sending a wrist view under the name the model expects for a head view doesn't error; it produces a well-formed, confident action chunk computed from the wrong scene. Second, proprio has to be exactly as wide as the action space you're requesting, in the order the policy was trained on. Neither of these is enforced by a type system, because they can't be, so getting the contract right is a matter of discipline in the calling code rather than something a runtime check can catch for you.

sk.act(..., preference="reliable")

provider is a hard constraint, not a preference; the router will refuse rather than violate it:

sk.act(..., provider={"max_latency_ms": 900, "allow_simulated": False})

allow_simulated: False is worth dwelling on. Sidekick can return simulated placeholder output (structurally identical to a real inference result, useful for pipeline development before a GPU is warm or a key has credits), and by default that's allowed. Setting the flag refuses it outright, which is what you want the moment you're pointing at actual hardware.

Before spending anything, preview_route() runs the router logic without touching a model at all:

plan = sk.preview_route(model="sidekick/auto:manipulation", contract="act")
print(plan[0]["estimated_list_usd"])   # the published rate
print(plan[0]["estimated_cost_usd"])   # what it would cost right now

The two numbers diverge specifically when the top candidate is cold: waking a serverless GPU container costs real money, and that cost is charged to whichever call triggers it, which is also why calling preview_route() twice in a row typically shows the second estimate drop.

stream = sk.stream_actions(
    model="sidekick/auto:manipulation",
    instruction="fold the towel",
    action_space="joint_pos_14", dof=14,
    preference="fastest",
    provider={"max_latency_ms": 900, "allow_simulated": False},
    observe=lambda: sk.observation(cameras=rig.capture(), proprio=robot.joints()),
)

stream.warm_up()          # blocking, run once, while the arm is braked
with stream:              # starts the background policy thread
    while running:
        step = stream.next_action()
        if step is None:
            robot.hold()  # decelerate, never repeat the last command
        else:
            robot.set_joint_positions(step)
        time.sleep(stream.dt_s)

Your control thread only ever calls next_action(), which reads from a local deque and is guaranteed not to block and not to raise. A separate background thread owns all network I/O: it fetches new chunks, appends them to the buffer (never replacing it, since steps already queued were planned from a real observation and discarding them would reopen the exact gap the class exists to close), and tracks a running average of fetch latency.

The refill trigger is where the implementation gets genuinely careful. It would be easy to say "fetch a new chunk when 5 steps remain," but that's wrong in a way that looks reasonable: at a 66 ms step interval, five steps is 330 ms of runway against a measured ~684 ms fetch time for a warm pi0 call: the buffer would run dry for roughly a third of every cycle. ActionStream instead expresses the threshold in time: it triggers a refetch once remaining runway drops below 1.5x the currently measured fetch latency, so the lead time scales with how slow the network and model actually are right now, not with an arbitrary step count decided in advance.

next_action() returning None is a first-class outcome, not a degraded one. It means the buffer is empty or the current chunk has aged past a staleness threshold (2 seconds), and the correct response, which the API pushes you toward at the type level, is to decelerate or hold, never to replay the last command. A robot repeating a stale action is a robot acting on a world that has moved on.

Before any chunk reaches the buffer, ActionStream runs the same three checks you'd otherwise have to remember to write yourself, and raises UnsafeResponse if any of them fail: the route isn't simulated, the returned action space matches what you asked for, and every step in the chunk has exactly dof values. warm_up() runs this same fetch-and-check path once, synchronously, before the robot is enabled, deliberately, since a cold serverless container loading several gigabytes of weights is a multi-second stall you want to happen with the arm braked, not on the first tick of a live loop. The stream also exposes running counters (stream.calls, stream.failures, stream.spend_usd) so a control process can report exactly what its policy dependency has cost and how often it's degraded, in real time.

from sidekick_sdk import SidekickError

try:
    act = sk.act(...)
except SidekickError as e:
    print(e.status, e.code)   # e.g. 409, "no_route_available"

UnsafeResponse covers the subtler case: the call succeeded, the response is well-formed, and it's still wrong for this robot, whether that's a simulated route, a mismatched action space, or a wrong-width action vector. Nothing about the payload looks broken, which is exactly why it gets its own exception type rather than blending into ordinary error handling: catching this class of issue takes a deliberate check, and the SDK makes that check easy to run. ActionStream.check() runs it automatically; calling act() directly, you run it yourself with a couple of lines.

sk.models(domain="manipulation")        # the live catalog, filterable
sk.model("physical-intelligence/pi-0")  # one model's card
sk.providers()                          # inference providers behind the router
sk.routes()                             # aliases that are live right now
sk.taxonomy()                           # every valid domain, family, action space
sk.action_space_dims("joint_pos_14")    # -> 14
sk.scenarios(runnable=True)             # the evaluate() benchmark registry
sk.literature(kind="survey")            # related papers, catalogued, not servable
sk.usage()                              # spend so far
sk.key()                                # what this key is scoped to do

taxonomy() in particular is meant to replace hard-coded strings anywhere in your code: it returns every domain, model family, and action space the catalog currently recognizes, with dimensions attached, so action_space_dims() can answer "how many numbers does one step of joint_pos_14 carry?" from the live registry instead of a comment.

with Sidekick(api_key="sk-sidekick-...") as sk:
    act = sk.act(model="sidekick/auto:manipulation", ...)

from sidekick_sdk import Sidekick

sk = Sidekick(api_key="sk-sidekick-...")

image = ("iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAIAAACQd1PeAAAADElEQVR42mNgYGAA"
         "AAAEAAHI6uv5AAAAAElFTkSuQmCC")   # a 1x1 placeholder, no camera required

act = sk.act(
    model="sidekick/auto:manipulation",
    instruction="fold the towel",
    action_space="joint_pos_14",
    horizon=4,
    observations=[{
        "frames": [{"camera": "primary", "b64": image}],
        "proprio": [0] * 14,
    }],
)

print(act.route["model"], act.usage["cost_usd"], act.steps)

That call runs end-to-end with no robot and no camera attached: the placeholder image is enough to exercise the full request path, including routing and billing, before any hardware is in the loop.