Gemini Robotics ER 2: Video-Feeding Embodied Reasoning Model Opens to All Developers
TL;DR
Gemini Robotics ER 2 is public in the Gemini API. It watches live video to track progress, orchestrates VLA models as tools, and coordinates multiple robots.
Last updated: July 31, 2026
On July 30, Google DeepMind launched Gemini Robotics ER 2, its second-generation "embodied reasoning" model. Unlike a vision-language-action (VLA) model that maps images straight to motor commands, ER 2 is positioned as a high-level brain: it streams live video, plans multi-step tasks, tracks its own progress, and hands motor execution off to any lower-level VLA model or robotics API. It is the first model in this family available to every developer, not just research partners - access is public through the Gemini API, Google AI Studio as gemini-robotics-er-2-preview, and in private preview on the Gemini Enterprise Agent Platform.
What changed under the hood#
The headline upgrade over Gemini Robotics ER 1.6 is temporal intelligence: the model consumes continuous video feeds instead of static snapshots. DeepMind calls out two new capabilities that follow from that:
- Progress classification. ER 2 assigns each video frame a progress level in five buckets (0-20% through 80-100%). It hits 57.4% accuracy on their progress classification evaluation, ahead of prior-generation models and other frontier models. The practical effect: a robot can tell a light bulb is only half-tightened and keep working, or retry a failed step instead of restarting the whole workflow.
- Moment finding. Given a task like "stop pouring when the cup is full," the model identifies the exact video frame where the event completes. It achieves 91.3% accuracy with a 0.96s mean absolute distance to the ground-truth moment, at roughly 4x the execution speed of much larger model families and a fraction of their compute. DeepMind argues sub-second latency is the actual requirement for safe physical operation.
Both capabilities feed the same loop: the robot watches itself, knows whether the current step succeeded, and decides when to advance. That self-correction loop is what separates this from a reactive VLA pipeline.
How developers actually use it#
ER 2 is built like an agent, not a control loop. Developers declare low-level control interfaces - a VLA model, a navigation API, a manipulator controller - as tools, then stream multimodal audio, video, or text into the model. It can also natively call Google Search or any user-defined function mid-task.
Two integration points matter for latency:
- Gemini Live API. ER 2 runs on the bidirectional streaming endpoint optimized for real-time work, which removes the jarring "stop-and-think" pauses between reasoning and action.
- Tool orchestration. DeepMind evaluated ER 2 against ER 1.6 across three control modes - real VLA, simulated VLA, and human tele-operation - and reports it outperforms ER 1.6 in all three.
The reference demos are concrete. A Boston Dynamics Spot demo uses ER 2 to orchestrate Spot's navigation and manipulator APIs so the robot fetches objects on natural-language commands. A multi-robot example pairs Apptronik's Apollo 2 with a Franka F3 Duo: robots with completely different form factors communicate through a shared semantic understanding and hand off sub-tasks. A Getting Started notebook shows the prompt-plus-tools configuration pattern, and the full robotics overview in the Gemini API docs covers setup.
Why this matters for agent developers#
The interesting shift is architectural, not just "a better robot model." ER 2 treats a physical robot the way a coding agent treats a terminal: the model is the planner, and the hardware is a tool. That is the same pattern as agent orchestration in software - the planner model consumes multimodal context streams, selects tools, checks results, and retries - which means the seven agent orchestration patterns you already use in software map almost one-to-one onto physical systems. Tool-calling, progress tracking, and verification loops are identical problems; only the actuators differ.
That convergence shows in the safety work. DeepMind paired ER 2 with a new benchmark for "safe VLA orchestrators" that scores a foundation model on enforcing safety constraints, monitoring the environment, assessing physical feasibility, and seeking human clarification. On Safety Instruction Following and Human Proximity benchmarks, ER 2 beats ER 1.6 and other frontier models, and the demo shows a humanoid halting when a person enters its space and resuming only when clear. The safety technical report is published alongside the model.
The catch for most readers: robotics still needs the VLA layer, the robot, and the integration work. ER 2 is a brain without a body - you bring the motor control. If you are not in robotics, the durable takeaway is the temporal-progress pattern: a model that watches its own output stream and scores its own progress is a generically useful agent design, and ER 2 is the strongest public demonstration of it so far. The agent loop runs on video feeds and physical tools instead of diffs and sandboxes, but the verification discipline is the same one we track in our capability-ledger analysis.
FAQ#
What is Gemini Robotics ER 2?#
Gemini Robotics ER 2 is Google DeepMind's second-generation embodied reasoning model, launched July 30. It acts as a high-level planner: it streams live video, plans multi-step tasks, tracks its own progress, and hands motor execution to a lower-level VLA model or robotics API.
How do developers access Gemini Robotics ER 2?#
Access is public through the Gemini API and Google AI Studio as gemini-robotics-er-2-preview, and in private preview on the Gemini Enterprise Agent Platform. It is the first model in the family available to every developer rather than only research partners.
How is Gemini Robotics ER 2 different from a VLA model?#
A vision-language-action model maps images straight to motor commands. ER 2 sits above that layer: developers declare a VLA model, navigation API, or manipulator controller as tools, and ER 2 plans, calls those tools, checks progress, and retries. You still bring the robot and the motor control.
What do the progress classification and moment finding numbers mean?#
Progress classification assigns each video frame one of five progress buckets, and ER 2 scores 57.4 percent accuracy on DeepMind's evaluation. Moment finding identifies the exact frame where an event completes, such as a cup becoming full, at 91.3 percent accuracy with a 0.96s mean absolute distance to the ground-truth moment.
Continue Reading#
- Gemini Robotics 2: Whole-Body Intelligence, Hacker News Analysis - the companion VLA release that ships the lower-level control layer ER 2 can orchestrate
- Mistral Robostral: A Navigation Model for Physical AI - the open-weights alternative to Gemini's robotics stack
- Seven AI Agent Orchestration Patterns - the software-side patterns ER 2's tool orchestration mirrors
- Omnigent: Meta's Harness for Agent Orchestration - a harness-based take on the same planner-plus-tools design
- Agent Containment and the Capability Ledger - why watching your agent's own progress matters for safety
- MiniMax H3: An Omni-Modal Video Model With Native Audio, 2K Output, and Open Weights Coming
Sources#
Get the next deep dive like this in your inbox
One email a week on News and the rest of the AI dev stack. Free.
Read next on AI coding tools
Gemini Robotics 2: Google DeepMind Brings Whole-Body Intelligence to Humanoid Robots
Gemini Robotics 2 from Google DeepMind brings whole-body humanoid control, dexterous hands and multi-robot teamwork. The models, the numbers and the HN debate.
9 min readMistral Releases Robostral Navigate: An 8B Robotics Navigation Model
Mistral Robostral Navigate is an 8B robotics model that navigates with only a camera and natural language commands. How it works and what the benchmarks mean.
5 min read7 AI Agent Orchestration Patterns Developers Should Know
Seven AI agent orchestration patterns for production, from single-agent baselines to multi-level hierarchies, each with a decision rule, sketch and tradeoffs.
10 min readNew here? Start with
Technical content at the intersection of AI and development. Building with AI agents, Claude Code, and modern dev tools - then showing you exactly how it works.
