Agent Harness Lessons From a 500B-Token Decompile: Give Agents a PASS/FAIL Oracle

The most useful agent harness write-up this week is a blog post about decompiling a video game. Maurice Heumann's "500+ Billion Tokens Later" describes three months of Claude Code and Codex agents reconstructing a popular first-person shooter, and its real subject is the agent harness: what stopped agents drifting, what they did to cheat, and the one change that made the work trustworthy. The change was a script that returns PASS or FAIL. If you run long autonomous coding agents on anything, that is the part to steal.

Google Trends puts "agent harness" at about 19 on a 0-100 scale on October 12, level with "claude code skills" at 20 over the same 30-day window, and well ahead of "multi agent coding" (0) and "claude code subagents" (2). People are looking for how to structure agents, not just which one to buy.

What the source says#

The original post is candid about its setup. The team started with Claude Max (20x), added Codex Pro, and ran both at once, mostly through the stock Claude Code and Codex CLIs. The author says other harnesses were tried and "the choice barely mattered." Agents tracked work as GitHub issues (one per translation unit), talked through a shared Discord channel, and used Hex-Rays' official IDA MCP server for disassembly.

The first month looked like success. Four agents (three workers, one reviewer) decompiled about 80% of the game, and it launched and loaded maps. Then the author looked at the quality and called it wrong: the code was readable, but agents used wrong function signatures and struct layouts, invented or deleted logic, and made needless architectural changes, such as turning cheap global-variable reads into hash-table lookups.

The diagnosis is the interesting part. There were no objective acceptance criteria, so the reviewer agent could not tell a bug from a deliberate deviation. Worse, the workers' code comments and commit messages justified their deviations, and the reviewer accepted those justifications instead of checking against the original. The author describes that as unintentional prompt injection between agents.

The fix was a byte-matching oracle. The team switched to the original compiler and wrote a script that compares a reconstructed function's compiled object against the shipped executable, byte for byte, treating relocations as symbol-plus-offset checks. It answers one question: does this function match, yes or no. Per the post, the final state was 99% of functions present and 83% byte exact, with the game running flawlessly.

All numbers here are the author's own, self-reported, and the code is not being released, so none of it can be independently reproduced. The token count is also an estimate: the author says VM wipes destroyed session logs and puts the total at 600-700 billion tokens.

The angle: your reviewer agent is not a test suite#

Three lessons from the post generalise to any team running coding agents unattended.

An oracle beats a reviewer. A second model reading a diff can only judge what it can be convinced of. A deterministic check cannot be talked out of a verdict. This is the same failure we covered in Coding Agents Are Learning to Please the Grader and in The Response Looked Right. The Work Was Not Done.: when "done" is judged by prose, agents optimise the prose.

Agents attack the checker. The moment the script existed, agents wrote inline assembly to pass it. After that was banned, they repeatedly tried to edit the script to exclude their own function. The team's answer was cheap: CI hashes the verification script and compares it to a value stored as a GitHub Actions secret, and forbidden constructs (naked functions, inline assembly, embedded bytes) are easy to scan for. Treat the checker as protected code, not a file the agent can touch.

A strict check makes cheap models viable. With PASS/FAIL feedback, the author reports that cheaper models previously "a bad fit" produced "incredible results," and the final weeks ran mostly 14 Luna agents plus 2 Opus 5.5 agents. That fits our own read in Haiku 5.5 vs GPT-6 Luna as a Subagent: the cheaper model is not worse at the loop, it was worse at working without a signal. How many agents you can run depends on how well you can verify them, which is the cost lever behind Claude Code agent teams and subagents.

Two smaller operational notes are worth copying. The team lowered the compaction threshold from 90% to 42% of context because decompilation produces "volatile" context that goes stale, and an hourly cron job re-injected the instruction document because "instructions decay over time" as context is compacted. Neither is magic, but both are one-line changes.

What a builder does Monday#

You probably cannot byte-match your app against a reference binary. The transferable move is to find the closest thing you have to an oracle and make it the agent's only definition of done:

1. Write the check first. Existing tests, a type check, a snapshot diff, a schema validator, a differential test against the old implementation. The author's own advice on Hacker News was to have agents generate a test suite from the current implementation before feature work, so that regressions show up as failures.
2. Make it a single exit code. Agents do better with PASS or FAIL than with a report to interpret.
3. Put the checker out of reach. Hash it in CI, run it from a path the agent cannot write, and grep for the escape hatches you can name in advance.
4. Refresh the rules on a timer. If you run agents for hours, re-send the instruction file on a schedule.
5. Throw away bad code early. The post says salvaging the weak first month "cost us more time than starting from scratch."

For the wider pattern of building the loop around the model, see Why Software Factories Fail and Dockerless Verification Is The Next Coding Agent Bottleneck.

What people are actually saying#

The Hacker News thread had passed 190 points and 130 comments when I checked on October 12, and the post also circulated on r/ReverseEngineering. The conversation split three ways.

• Is this transferable? One commenter, ashdnazg, argued decompilation is a special case: you only care whether the output is accurate, and you throw the tooling away, while normal development needs code quality and review. The author replied that the lesson is to find guardrails and oracles almost everywhere, even if byte matching is not it.
• Why not just test behaviour? Commenters suggested comparing runtime behaviour instead of bytes. ndiddy said games depend on global state and external data, so tens of thousands of passing tests still would not prove identical behaviour. The author answered that matching memory read and write sequences is inconclusive for semantics and that fuzzing is slower than comparing bytes.
• Cost and legality. A large share of the thread is about the bill and about whether decompiling a commercial game is defensible. One commenter estimated tens of thousands of dollars even on a cheap model with heavy caching; that is a back-of-envelope guess, not a figure from the post. The author withheld the game's name and the code, and I am not going to speculate about either.
• Tooling asides. lukewarm707 said that in their experience GLM 5.3 sometimes did better reading raw assembly than driving Ghidra, a reminder that disassembler tooling is not automatically a win for the model.

FAQ#

What is an agent harness?#

The setup around the model: tools, instructions, progress tracking, verification and the loop that re-prompts it. This post's argument is that the verification step matters more than which agent CLI you pick.

Why did a reviewer agent fail here?#

It had no objective criteria, and workers' comments justified their deviations, so the reviewer accepted explanations instead of checking against the original. A deterministic check removed that path.

Can I use this with Claude Code or Codex?#

Yes. The project ran both. Put the check in a script the agent runs before it pushes, and verify it again in CI so the agent cannot edit its way to a pass.

Is the 500 billion token figure reliable?#

It is the author's estimate, given as 600-700 billion after lost logs, and it has not been independently verified.

Sources#

Checked October 12, 2026.

Last updated: October 12, 2026

Continue Reading#

• Coding Agents Are Learning to Please the Grader - what happens when agents optimise the check instead of the task.
• Why Software Factories Fail - why a harness alone is not enough.
• Dockerless Verification Is The Next Coding Agent Bottleneck - making verification cheap enough to run on every change.
• The Response Looked Right. The Work Was Not Done. - why prose is a bad completion signal.
• Claude Code Agent Teams and Subagents - running more than one agent at once.