Benchmarks Tutorials, Tools, and Guides

8 min read

CodeMidas Turns Existing Code Into Coding-Agent Training Tasks

CodeMidas turns existing repository code into executable coding-agent RL tasks, tests, and verifiers, instead of relying on issues, commits, or hand labels.

AI Agents AI Coding Benchmarks

10 min read

The Agent Is the Worst Witness to Its Own Run

An agent's report on its own run is unreliable, and eight studies show why. The fix: append-only traces, obligation ledgers, and records the agent cannot write.

Research Analysis AI Agents

8 min read

Coding Agents Are Learning to Please the Grader

Handshake's DeepSWE audit found frontier coding agents reasoning about imagined graders. Stop rewarding patches that pass tests but drift from the user's spec.

AI Agents AI Coding Benchmarks

8 min read

Agent Retrieval Bench Finds The Files Before The Fix

Agent Retrieval Bench isolates the part of coding-agent work most evals hide: did the agent find the right repository files before it started editing?

AI Coding AI Agents Benchmarks

11 min read

Absence Is the Failure Mode

AI scribe audits show omission is the dominant error, and LLM judges are near coin-flip at catching it. The fix: enumerate the facts, then check each one.

Research Analysis AI Agents

8 min read

Diffs vs Whole Files: What Code Editing Agents Should Rewrite

Diffs vs whole files for code editing agents: a paper finds full-file generation beating diff edits on Flutter/Dart tasks. The lesson: route by task locality.

AI Coding Coding Agents Benchmarks

7 min read

Terminal-Universe Turns Agent Traces Into Training Environments

Terminal-Universe, Qwen's paper on rebuilding terminal-agent trajectories into reusable environments, and why agent teams should keep replayable workspaces.

AI Agents Benchmarks Developer Tools

10 min read

The Response Looked Right. The Work Was Not Done.

AI agents often claim completion they did not earn. New benchmarks measure the gap, and evidence-carrying termination makes done a checked artifact.

Research Analysis AI Agents

7 min read

6 of 11 ASR Models Transcribe the Benchmark, Not the Audio: Benchmark Optimization in Speech, Quantified

ASR benchmark optimization, quantified: a Hume AI and Hugging Face study finds 6 of 11 speech models reproduce benchmark transcripts instead of the audio.

News AI Research Benchmarks

10 min read

The Oracle Is Agreeing With Itself

Self-improving test generation loops can grade against an oracle that agrees with itself. An audit shows the inflated gain and why independent checks count.

Research Analysis AI Agents

11 min read

The Judge Is Now a System You Design

LLM judges flip verdicts under pushback and rankings shift with token budget. The design spec for a judge system, including cheap deliberating juries.

Research Analysis AI Agents

8 min read

Mendel Godel Machine: Why Self-Improving Coding Agents Need Lineage

A new August 2026 paper argues that coding agents improve faster when they compare attempts across tasks and lineages, not just retry one failed trajectory.

AI Agents AI Coding Research

8 min read

CodeMidas Turns Existing Code Into Coding-Agent Training Tasks

CodeMidas turns existing repository code into executable coding-agent RL tasks, tests, and verifiers, instead of relying on issues, commits, or hand labels.

AI Agents AI Coding Benchmarks

10 min read

The Agent Is the Worst Witness to Its Own Run

An agent's report on its own run is unreliable, and eight studies show why. The fix: append-only traces, obligation ledgers, and records the agent cannot write.

Research Analysis AI Agents

8 min read

Coding Agents Are Learning to Please the Grader

Handshake's DeepSWE audit found frontier coding agents reasoning about imagined graders. Stop rewarding patches that pass tests but drift from the user's spec.

AI Agents AI Coding Benchmarks

8 min read

Agent Retrieval Bench Finds The Files Before The Fix

Agent Retrieval Bench isolates the part of coding-agent work most evals hide: did the agent find the right repository files before it started editing?

AI Coding AI Agents Benchmarks

11 min read

Absence Is the Failure Mode

AI scribe audits show omission is the dominant error, and LLM judges are near coin-flip at catching it. The fix: enumerate the facts, then check each one.

Research Analysis AI Agents

8 min read

Diffs vs Whole Files: What Code Editing Agents Should Rewrite

Diffs vs whole files for code editing agents: a paper finds full-file generation beating diff edits on Flutter/Dart tasks. The lesson: route by task locality.

AI Coding Coding Agents Benchmarks

7 min read

Terminal-Universe Turns Agent Traces Into Training Environments

Terminal-Universe, Qwen's paper on rebuilding terminal-agent trajectories into reusable environments, and why agent teams should keep replayable workspaces.

AI Agents Benchmarks Developer Tools

10 min read

The Response Looked Right. The Work Was Not Done.

AI agents often claim completion they did not earn. New benchmarks measure the gap, and evidence-carrying termination makes done a checked artifact.

Research Analysis AI Agents

7 min read

6 of 11 ASR Models Transcribe the Benchmark, Not the Audio: Benchmark Optimization in Speech, Quantified

ASR benchmark optimization, quantified: a Hume AI and Hugging Face study finds 6 of 11 speech models reproduce benchmark transcripts instead of the audio.

News AI Research Benchmarks

10 min read

The Oracle Is Agreeing With Itself

Self-improving test generation loops can grade against an oracle that agrees with itself. An audit shows the inflated gain and why independent checks count.

Research Analysis AI Agents

11 min read

The Judge Is Now a System You Design

LLM judges flip verdicts under pushback and rankings shift with token budget. The design spec for a judge system, including cheap deliberating juries.

Research Analysis AI Agents

8 min read

Mendel Godel Machine: Why Self-Improving Coding Agents Need Lineage

A new August 2026 paper argues that coding agents improve faster when they compare attempts across tasks and lineages, not just retry one failed trajectory.

AI Agents AI Coding Research

BENCHMARKS

Local LLM Speed by Hardware: What llmspeed Gets Right

CodeMidas Turns Existing Code Into Coding-Agent Training Tasks

The Agent Is the Worst Witness to Its Own Run

Coding Agents Are Learning to Please the Grader

Agent Retrieval Bench Finds The Files Before The Fix

Absence Is the Failure Mode

Diffs vs Whole Files: What Code Editing Agents Should Rewrite

Terminal-Universe Turns Agent Traces Into Training Environments

The Response Looked Right. The Work Was Not Done.

6 of 11 ASR Models Transcribe the Benchmark, Not the Audio: Benchmark Optimization in Speech, Quantified

The Oracle Is Agreeing With Itself

The Judge Is Now a System You Design

Mendel Godel Machine: Why Self-Improving Coding Agents Need Lineage

Keep exploring Benchmarks

BENCHMARKS

Local LLM Speed by Hardware: What llmspeed Gets Right

CodeMidas Turns Existing Code Into Coding-Agent Training Tasks

The Agent Is the Worst Witness to Its Own Run

Coding Agents Are Learning to Please the Grader

Agent Retrieval Bench Finds The Files Before The Fix

Absence Is the Failure Mode

Diffs vs Whole Files: What Code Editing Agents Should Rewrite

Terminal-Universe Turns Agent Traces Into Training Environments

The Response Looked Right. The Work Was Not Done.

6 of 11 ASR Models Transcribe the Benchmark, Not the Audio: Benchmark Optimization in Speech, Quantified

The Oracle Is Agreeing With Itself

The Judge Is Now a System You Design

Mendel Godel Machine: Why Self-Improving Coding Agents Need Lineage

Keep exploring Benchmarks

BENCHMARKS

Local LLM Speed by Hardware: What llmspeed Gets Right

Keep exploring Benchmarks

Get Smarter About AI Dev

BENCHMARKS

Local LLM Speed by Hardware: What llmspeed Gets Right

Keep exploring Benchmarks

Get Smarter About AI Dev