Gemini Robotics ER 2: Video-Feeding Embodied Reasoning Model Opens to All Developers

On July 30, Google DeepMind launched Gemini Robotics ER 2, its second-generation "embodied reasoning" model. Unlike a vision-language-action (VLA) model that maps images straight to motor commands, ER 2 is positioned as a high-level brain: it streams live video, plans multi-step tasks, tracks its own progress, and hands motor execution off to any lower-level VLA model or robotics API. It is the first model in this family available to every developer, not just research partners - access is public through the Gemini API, Google AI Studio as gemini-robotics-er-2-preview, and in private preview on the Gemini Enterprise Agent Platform.

What changed under the hood#

The headline upgrade over Gemini Robotics ER 1.6 is temporal intelligence: the model consumes continuous video feeds instead of static snapshots. DeepMind calls out two new capabilities that follow from that:

• Progress classification. ER 2 assigns each video frame a progress level in five buckets (0-20% through 80-100%). It hits 57.4% accuracy on their progress classification evaluation, ahead of prior-generation models and other frontier models. The practical effect: a robot can tell a light bulb is only half-tightened and keep working, or retry a failed step instead of restarting the whole workflow.
• Moment finding. Given a task like "stop pouring when the cup is full," the model identifies the exact video frame where the event completes. It achieves 91.3% accuracy with a 0.96s mean absolute distance to the ground-truth moment, at roughly 4x the execution speed of much larger model families and a fraction of their compute. DeepMind argues sub-second latency is the actual requirement for safe physical operation.

Both capabilities feed the same loop: the robot watches itself, knows whether the current step succeeded, and decides when to advance. That self-correction loop is what separates this from a reactive VLA pipeline.

How developers actually use it#

ER 2 is built like an agent, not a control loop. Developers declare low-level control interfaces - a VLA model, a navigation API, a manipulator controller - as tools, then stream multimodal audio, video, or text into the model. It can also natively call Google Search or any user-defined function mid-task.

Two integration points matter for latency:

• Gemini Live API. ER 2 runs on the bidirectional streaming endpoint optimized for real-time work, which removes the jarring "stop-and-think" pauses between reasoning and action.
• Tool orchestration. DeepMind evaluated ER 2 against ER 1.6 across three control modes - real VLA, simulated VLA, and human tele-operation - and reports it outperforms ER 1.6 in all three.

The reference demos are concrete. A Boston Dynamics Spot demo uses ER 2 to orchestrate Spot's navigation and manipulator APIs so the robot fetches objects on natural-language commands. A multi-robot example pairs Apptronik's Apollo 2 with a Franka F3 Duo: robots with completely different form factors communicate through a shared semantic understanding and hand off sub-tasks. A Getting Started notebook shows the prompt-plus-tools configuration pattern, and the full robotics overview in the Gemini API docs covers setup.

Why this matters for agent developers#

The interesting shift is architectural, not just "a better robot model." ER 2 treats a physical robot the way a coding agent treats a terminal: the model is the planner, and the hardware is a tool. That is the same pattern as agent orchestration in software - the planner model consumes multimodal context streams, selects tools, checks results, and retries - which means the seven agent orchestration patterns you already use in software map almost one-to-one onto physical systems. Tool-calling, progress tracking, and verification loops are identical problems; only the actuators differ.

That convergence shows in the safety work. DeepMind paired ER 2 with a new benchmark for "safe VLA orchestrators" that scores a foundation model on enforcing safety constraints, monitoring the environment, assessing physical feasibility, and seeking human clarification. On Safety Instruction Following and Human Proximity benchmarks, ER 2 beats ER 1.6 and other frontier models, and the demo shows a humanoid halting when a person enters its space and resuming only when clear. The safety technical report is published alongside the model.

The catch for most readers: robotics still needs the VLA layer, the robot, and the integration work. ER 2 is a brain without a body - you bring the motor control. If you are not in robotics, the durable takeaway is the temporal-progress pattern: a model that watches its own output stream and scores its own progress is a generically useful agent design, and ER 2 is the strongest public demonstration of it so far. The agent loop runs on video feeds and physical tools instead of diffs and sandboxes, but the verification discipline is the same one we track in our capability-ledger analysis.

Continue Reading#

• Gemini Robotics 2: Whole-Body Intelligence, Hacker News Analysis - the companion VLA release that ships the lower-level control layer ER 2 can orchestrate
• Mistral Robostral: A Navigation Model for Physical AI - the open-weights alternative to Gemini's robotics stack
• Seven AI Agent Orchestration Patterns - the software-side patterns ER 2's tool orchestration mirrors
• Omnigent: Meta's Harness for Agent Orchestration - a harness-based take on the same planner-plus-tools design
• Agent Containment and the Capability Ledger - why watching your agent's own progress matters for safety

Sources#

• Introducing Gemini Robotics ER 2 - Google DeepMind blog
• Gemini Robotics ER 2 model card
• Gemini API robotics overview
• Gemini Live API documentation
• Gemini Robotics samples repository
• Gemini Robotics 2 safety technical report