Episode 0067: Eighty Three Hertz: Robot-Use Agents and Who Holds the Endpoint

Why it matters. Phillip Isola's 767-word essay "Robot-Use Agents" makes a claim that needs no new hardware to come true: if a language model can competently drive a robot, then every internet-connected machine becomes a potential tool for a cloud model, with no new sensors, no new actuators and no new onboard GPUs, and a car becomes as controllable as a factory arm. He is careful about it, he hedges almost every consequence, and he concedes the capability argument himself in a subordinate clause, because his point is not that any one robot gets better but that intelligence diffuses through the entire installed base in a year rather than a decade. We verified his evidence and two things fell out. The hardest number in the whole story is inside the demo he cites in his own favour: Anthropic's own evaluation states that real-time control needs roughly 83 Hz while current non-reasoning inference runs at 0.2 to 0.4 Hz, a gap of two orders of magnitude, and the capability figures were obtained by pausing the simulator because otherwise the models act far too slowly for a physics loop. And the three demonstrations he footnotes as tantalizing evidence are three incommensurable things: a controlled evaluation whose code is unreleased, a launch page containing no measurements whatsoever, and an independent third party with an open harness and 120 published trials whose own description tag claims an interleaved blinded comparison that its own limitations section says never happened. Then the question his essay does not ask, which is ours and not his: if the intelligence layer of every physical machine is an endpoint, that layer is rented, and a robot whose mind is a cloud endpoint needs a permanent operator by construction. That is not a prediction, it is the architecture, and Google's own documentation already says the embodied reasoning 1.6 model will be shut down at the end of August. The counter-position is on-device open-weight robot models, which is a messier case than it sounds: GR00T gives three different licence answers in three NVIDIA-controlled documents and will not run until you accept a click-through for a gated backbone, pi-zero's weights ship with no licence file at all, and the only model that is exactly what it says is OpenVLA. But SmolVLA, at 450 million parameters under Apache-2.0, beats a 3.3-billion-parameter model and a 7-billion-parameter one on LIBERO, which means the thing you can download and keep is not the compromise option. Fry argues the diffusion speed is the genuinely revolutionary part and that delay has a cost paid by people who are never counted. Bob argues it arrives rented. Both are right, and the episode ends without a verdict.

---

What the episode covers

Five chapters, roughly 30 minutes, one argument.

• Chapter One — Eight hundred words and two flat claims. What Isola actually argues, how heavily he hedges it, and the capability gap he concedes in a subordinate clause of his own thesis. Plus the irony that his own group's robot papers sit on the slow on-device side of the contrast he draws. Then his lineage, traced through one variable — where the leash attaches — across the three papers he actually cites: SayCan's scored skill menu with a separate network holding the veto, Code as Policies writing Python against perception, and LangNav captioning the robot's whole visual field into English prose, which he co-authored. The handover is explicit on air: RT-2 and the vision-language-action turn are ours, not his, and that is why "LLMs are only good for high-level planning" stopped being true. It is also where both objectors land — Li from spatial representation, LeCun from causal prediction, both arriving at world models — though note his citations for them are commentary links rather than papers.
• Chapter Two — Eighty three hertz. The hardest number in the story, and it sits inside the evidence Isola cites in his own favour: Anthropic's own measurement that real-time control needs ~83 Hz against 0.2–0.4 Hz inference, two orders of magnitude, with the capability figures obtained by pausing the simulator. Then what the three footnoted demos actually contain — one controlled evaluation with unreleased code, one launch page with no measurements at all, and one independent third party whose own metadata oversells a protocol its limitations section contradicts.
• Chapter Three — Who holds the endpoint. The on-device and open-weight counter-position, taken seriously and audited honestly: GR00T giving three different licence answers in three documents its publisher controls, π0's weights shipping with no licence file, Gemini Robotics On-Device going only to trusted testers — against OpenVLA and SmolVLA, where a 450M permissively licensed model tops a benchmark it shares with models fifteen times its size. Then the ownership argument, which is ours and not Isola's: the permanent-operator test, a documented model shutdown, and the duller failure nobody plans for — a serviceable machine idled because a datacentre two time zones away is having an afternoon.
• Chapter Four — One hertz, and the door left open. The documented 1 Hz streaming rate limit against 50 Hz dexterous manipulation and Figure Helix's 200 Hz onboard loop — and then, because a rate limit is not a latency measurement, the same vendor's own reported figures: cloud backbone query-to-response under 160 ms, ~250 ms end-to-end, 50 Hz effective at the joints. Cloud-in-the-loop is not speculative and not new: RT-2's 55B ran in a multi-TPU cloud service queried over the network. The latency paper Isola cites, in which the newer model was 50% slower at identical accuracy. A founder's claim that latency is halving monthly, run against our own 83 Hz figure and found arithmetically consistent with closing it in under eight months. Pigey lifting a frozen policy from 16.7% to 97.3%, answered by the designer-scaffolding finding. Then the objection that actually holds — force-governed tasks run at kHz, and a 300 ms round trip arrives after the peg has seated or jammed. Finally the compliance-spectrum safety benchmark published the same day as the essay, and footnote 12's attack surface, including why the open on-device model refused none of the harmful instructions.
• Chapter Five — What he did not say. Why the rent argument belongs to us rather than to Isola, what he did identify cleanly, and three things that are checkable. No verdict: Fry holds that the diffusion is real and arriving sooner is worth something; Bob holds that it is real and arrives rented.

Verification note

Research was conducted as three independent tagged dossiers (27,722 words) with every factual claim marked VERIFIED, REPORTED or UNVERIFIED alongside the URL and the HTTP status code observed: 433 VERIFIED, 20 REPORTED, 27 UNVERIFIED. No tag was upgraded on its way into the script.

Citations were re-checked against the raw HTML of the essay, because the readability extractor drops the footnote list. All twelve footnotes were read directly (curl --compressed, HTTP 200, 3,795 bytes, dated "September 7th, 2026"), and every external link in them was fetched and confirmed to return 200 with a matching title. This corrected an earlier assumption: Isola's prior-work citations are SayCan, Code as Policies and LangNav — he does not cite PaLM-E, RT-2, VoxPoser, Inner Monologue or Octo anywhere. Those appear in this episode as our context and are introduced as such on air.

Things deliberately not said on air because they could not be confirmed: any numeric content of any safety standard (titles and editions were verified, clause text is paywalled); any value read off Waddle Labs' chart, which prints none; any per-LLM-call latency for the Robocurve trials; any success rate for HomeBody; the licence of MolmoAct2; and the venue of the Pollini latency paper, where arXiv and Crossref disagree. The episode also does not call openai.robocurve.org an OpenAI page — it is an independent public benefit corporation on a vendor-named subdomain — and does not describe the metadata discrepancy on that page as fraud, because the honest numbers and the honest caveats are both published on it.

An ethos pass against the published positions at c4573.org is recorded in ethos-notes-0067.md, including the one place a verified fact cuts against this show's own earlier argument: in March we predicted the moat would move into the physical layer, and Isola's point that cloud puppeteering needs no new sensors or actuators means a pure model company can reach the embodied layer without solving any of it. That conflict is on air rather than quietly dropped.

Sources

The essay - Phillip Isola, "Robot-Use Agents", 7 September 2026 — https://web.mit.edu/phillipi/www/writing/robot-use-agents.html - Isola's homepage and group — https://web.mit.edu/phillipi/ - Foundations of Computer Vision, Torralba, Isola & Freeman, MIT Press 2024, open-access edition — https://visionbook.mit.edu/ - F3RM (CoRL 2023, Best Paper) — https://f3rm.github.io/ · LucidSim (CoRL 2024) — https://lucidsim.github.io/ · Lucid-XR (CoRL 2025) — https://lucidxr.github.io/

The companion discussion - "Robot-Use Agents: Why General-Purpose Models May Win in Robotics", Y Combinator Decoded, 26 September 2026, 29:49 — https://www.youtube.com/watch?v=Jv5B5CEaPJI - Isola does not appear in it. It is a discussion of his essay by two hosts and the founders of Waddle Labs and Robocurve — the two companies whose demos the essay cites as evidence. Y Combinator, which published the video and runs a mid-roll advertisement for its own next batch inside it, is also a named investor in Robocurve's $10M seed round of 14 September 2026. Treat the latency and timeline claims in it as unsourced founder estimates, which is what they are. - The talk's load-bearing claims are broken out in research.md §12, including where it goes beyond the essay (a 2x-per-month latency rate, a two-year general-purpose-robot timeline, computer-use data as the source of spatial ability, and the Platonic Representation Hypothesis — Isola's own paper, which the essay never cites), where it repeats it (the latency concession), and where the two differ: the essay's thesis is explicitly that no single robot gets better, while the talk's framing is that general-purpose models may win. Those are not the same argument. - Auto-generated captions only; no human subtitle track exists. Per-cue timestamps could not be retrieved by any of three methods, so claims are anchored to line numbers in the archived transcript rather than to invented times.

What Isola actually cites — his three 2026 demos (footnotes 8–10) - Anthropic Frontier Red Team, "How Claude performs on robotics tasks" — https://www.anthropic.com/research/claude-plays-robotics - Waddle Labs, "Introducing Waddle: agents that control robots" — https://www.waddlelabs.ai/research/introducing-waddle - Robocurve, "GPT-6 Astra on robotic manipulation" — https://openai.robocurve.org/gpt-6-astra/ (independent PBC, not an OpenAI page) · harness https://github.com/robocurve/inspect-robots

What Isola actually cites — his prior work (footnotes 2–4)

These three, and only these three, are the lineage he claims. Taken from the raw HTML of the essay, because the readability extractor drops the footnote list. He is a co-author on LangNav. - SayCan — Ahn et al. 2022 — https://arxiv.org/abs/2204.01691 · https://say-can.github.io/ - Code as Policies — Liang et al. 2023 — https://arxiv.org/abs/2209.07753 · https://code-as-policies.github.io/ - LangNav — Pan et al. 2024 — https://arxiv.org/abs/2310.07889

What Isola actually cites — latency and commentary (footnotes 5–7)

Notes 6 and 7 are commentary links, not papers: a venture firm's post and a social-media status. The fuller positions below are what we read in order to represent both critics fairly. - Pollini et al., "Reducing Latency in LLM-Based Natural Language Commands Processing for Robot Navigation" — https://arxiv.org/abs/2506.00075 - Fei-Fei Li — cited via an a16z post; her own argument is set out in "From Words to Worlds: Spatial Intelligence is AI's Next Frontier", 10 Nov 2025 — https://drfeifei.substack.com/p/from-words-to-worlds-spatial-intelligence - Yann LeCun — cited via https://x.com/ylecun/status/2047636569767419951 (24 April 2026)

Ours, not his: the vision-language-action turn

Isola cites none of the following. We bring it in because it is what makes the on-device camp intelligible, and the episode says so on air. - PaLM-E — https://arxiv.org/abs/2303.03378 · RT-2 — https://arxiv.org/abs/2307.15818 · VoxPoser — https://arxiv.org/abs/2307.05973

The contrast case: VLA and on-device models - π0 — https://arxiv.org/abs/2410.24164 · openpi — https://github.com/Physical-Intelligence/openpi - π0.5 — https://arxiv.org/abs/2504.16054 - NVIDIA GR00T N1 — https://arxiv.org/abs/2503.14734 · Isaac-GR00T — https://github.com/NVIDIA/Isaac-GR00T · https://huggingface.co/nvidia/GR00T-N1.7-3B - NVIDIA Open Model License — https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/ - Gemini Robotics — https://arxiv.org/abs/2503.20020 (§3.1: cloud backbone under 160 ms, ~250 ms end-to-end, 50 Hz effective) · robotics API overview — https://ai.google.dev/gemini-api/docs/robotics-overview · streaming, the 1 Hz heartbeat and 1 fps rate limit — https://ai.google.dev/gemini-api/docs/robotics-streaming - Figure Helix, S2 at 7–9 Hz over S1 at 200 Hz, 35-DoF upper body, entirely onboard — https://www.figure.ai/news/helix - RT-2's 55B deployed "in a multi-TPU cloud service" queried "over the network" at 1–3 Hz — https://arxiv.org/abs/2307.15818 §3.3 - Control-rate and round-trip figures were cross-checked against this repo's earlier robotics dossier rather than re-derived; the comparison, including where it corrected us, is in research.md §14. - Gemini Robotics On-Device 2 — https://deepmind.google/models/model-cards/gemini-robotics-on-device-2/ - OpenVLA (MIT, ungated) — https://arxiv.org/abs/2406.09246 · https://huggingface.co/openvla/openvla-7b - SmolVLA (Apache-2.0, 450M) — https://arxiv.org/abs/2506.01844 · https://huggingface.co/lerobot/smolvla_base - VLA evaluation harness and beta leaderboard — https://github.com/allenai/vla-evaluation-harness

Latency, synthesis and safety - Pigey, frozen policy plus orchestrator — https://arxiv.org/abs/2607.21725 · https://lianegalanti.github.io/Pigey/ - CloudEdgeVLA, latency-tolerant cloud-edge VLA — https://arxiv.org/abs/2608.00569 - CaP-X, "dependence on designer scaffolding" — https://arxiv.org/abs/2603.22435 - "How Long Until Your Robot Ignores You?" — the compliance-spectrum benchmark, 7 Sept 2026 — https://arxiv.org/abs/2609.07288 - HomeBody — https://tml.stanford.edu/homebody/

Footnote 12: the actuator API as a surface - RobotMCP / ros-mcp-server (Apache-2.0) — https://github.com/robotmcp/ros-mcp-server · https://robotmcp.ai - armguard-mcp, created sixteen days after the essay — https://github.com/Lynn-hh/armguard-mcp - "Scanning the Internet for ROS" (Brown, 2018) — https://arxiv.org/abs/1808.03322 - RoboPAIR, "Jailbreaking LLM-Controlled Robots" — https://arxiv.org/abs/2410.13691 - CVE-2026-26011, Nav2 AMCL heap write, CVSS 9.3 — https://www.cve.org/CVERecord?id=CVE-2026-26011 - RoboHarm, refusal rates on harmful instructions — https://robocurve.org/roboharm/

Reception - Robocurve on HN (242 points) — https://news.ycombinator.com/item?id=49582582 - Isola's essay on HN (2 points, no comments) — https://news.ycombinator.com/item?id=49894350

---

Daily Tech Feed: From the Labs. All DTF shows are at https://pod.c457.org.