ORCA-bench: Frontier Agents Score 10% on Hard Oncall RCA

Coding agents can fix bugs on frozen repositories. A new benchmark from Cornell Tech and Traversal asks a harder question: can they run oncall?

ORCA-bench (arXiv 2607.28545, submitted July 30) drops five frontier agents into a live, OpenTelemetry-instrumented microservice system with six days of real telemetry, and hands them the kind of report a user actually files: "checkout is broken", or worse, "users are having issues on the site". The results are a reality check for anyone assuming agentic coding extends to production reliability.

What the benchmark is#

ORCA-bench is built on the OpenTelemetry Astronomy Shop demo: 19 microservices in 13 languages, running under continuous simulated load for six days. Agents investigate through the same interfaces a human SRE would use - Prometheus for metrics, OpenSearch for logs, Jaeger for traces, all queried through the Grafana API - plus full source code access in a terminal. The environment generates 1,079 root cause analysis (RCA) tasks: 884 incident tasks and 195 control tasks where the correct answer is "nothing is wrong".

The design choices matter as much as the environment:

• Issue specificity ladder. Prompts vary from easy (error message included) to hard ("users are reporting site issues"). Hard tasks average 4.41 plausible root causes versus 2.00 for easy.
• Time-to-detection. Tasks start investigation 15 minutes to 24 hours after the incident begins, with exact, exact-range, and broad-range report times.
• Co-occurring faults. Scenarios span five types - isolated, independent, conflicting, cascading, and sequential. Day 6 ("FAFO Friday" in the paper) runs six feature flags concurrently.
• Ground truth and judge both human-checked. Expert SREs signed off the symptom rubrics, and a 40-task Verified subset has every label hand-confirmed. The GPT-5.4 LLM judge agrees with human re-scoring at Cohen's kappa 0.90.

What it found#

Across Claude Opus 4.7, Claude Sonnet 4.6, GPT-5.5, GLM-5, and DeepSeek-V4-Pro running in the Terminus-2 harness, with Claude Fable 5 added on the Verified subset:

• Best RCA accuracy on Medium difficulty: 25.3%. On Hard, where the report gives the least context, the best agent scores 10.0%.
• Claude Fable 5 on the 32 Verified incident tasks: 40.6% RCA accuracy and 58.2% RCA depth, versus 21.9% / 49.2% for GPT-5.5 and 25.0% / 47.6% for Claude Opus 4.7. The frontier model is meaningfully ahead, and still wrong in most cases.
• Hallucination is the scariest number. Agents named an implausible root cause in 7% (DeepSeek-V4-Pro) to 40% (GLM-5) of incident reports.
• Removing source code access drops RCA accuracy by 9 to 16 points for every model and spikes hallucination - yet agents spend only 16-20% of their commands reading code versus 70-72% on telemetry.
• Telemetry usage is broken in practice: 26-40% of telemetry calls error out or return empty, and agents get distracted by louder symptoms when multiple faults are active.

The Day 6 case study is the most concrete failure. Six events co-occurring: three models each identified exactly one root cause - all different ones - and missed the other five. GPT-5.5 found four of six. DeepSeek-V4-Pro found none.

Why it matters#

The SWE-bench shape does not transfer to RCA. Bug-fix benchmarks give agents a failing test and a frozen repo; success is a green test suite. ORCA-bench gives them an ambiguous report, a live distributed system, and a success criterion of a defensible diagnosis against human-curated ground truth. The 10-25% scores show this is not a harder version of the same task, it is a different skill. We made the same point about benchmark scope in Agent Evals Need Baseline Receipts and the SWE-NFI benchmark: what you measure is what your agents optimize, and nobody was measuring diagnosis.

The gap is a lower bound. The authors argue this directly: the agents investigate a 50 GB testbed whose code and instrumentation are public - almost certainly in pretraining - with each task isolated and no memory between incidents. Real production is larger by orders of magnitude, private, and drifting. Read the numbers as "this good at best", which makes the oncall delegation case weaker, not stronger.

Concurrency is the failure mode to watch. The Day 6 result - three agents each naming a single, different root cause, GPT-5.5 finding four of six, DeepSeek-V4-Pro finding none - matches the pattern we flagged in Agent Containment and the Capability Ledger: agents lock onto the loudest signal and stop. For teams running parallel agents, this is direct evidence about when multi-agent fan-out helps (independent tasks) and when it does not (one system, many interacting faults).

The eval infrastructure is worth borrowing even if the scores are not. A benchmark with human-verified ground truth, a judge calibrated against humans at kappa 0.90, and control tasks to catch "always says there is an incident" behavior is the shape every agent harness should copy. Agentic AI reliability case studies keep showing the same thing: teams ship agent behavior without baseline receipts, and only discover the hallucination rate after an incident.

My take#

Three conclusions.

First, oncall is the right next frontier for agent evals, and this is the first one built correctly. SRE benchmarks before it stripped out at least one of telemetry, source code, or realistic report ambiguity. ORCA-bench is the first to include all three, and the result is that everything collapses - that is a signal about the task, not the benchmark.

Second, the hallucination rate should be the headline for operators. A coding agent that hallucinates a root cause is worse than one that says "I do not know" - it sends a human chasing a plausible-sounding wrong lead during an outage. Until hallucination rates are near zero, the safe play is agents as investigation assistants that produce evidence trails, not agents that write the incident report. This is the same division of labor the 12-factor agents in production argument makes for all agent output: require receipts.

Third, expect the frontier to move fast here. Claude Fable 5 already roughly doubles Opus 4.7's Verified accuracy, and the paper's own limitation section notes structured workflows and hybrid causal-inference RCA are unexplored territory. The benchmark is public on the Harbor hub, so this is now a trackable metric like any other - and the first vendor to publish a real SRE-style score will have something to say.

The honest summary: agents write code better than they diagnose systems, and the gap is now quantified. If your team was wondering when AI can take pager duty, the answer from this benchmark is not this quarter - but the eval to track the progress now exists.

Continue Reading#

• Agent Evals Need Baseline Receipts
• SWE-NFI: The Benchmark That Catches What Coding Agents Miss
• Frontier Code Benchmarks: What They Mean for AI Coding
• Agentic AI Reliability: Case Studies
• Agent Containment and the Capability Ledger

Sources#

• ORCA-bench: How Ready Are Language Model Agents for Oncall? - arXiv
• ORCA-bench dataset - Harbor hub