Skip to main content
Watch: Grok Bot Built My AI Video Arena While I Tested 8 Models

BENCHMARKS

36 items

36 posts

Blog
Speed Up a Local LLM: The Agent Skill That Computes the Ceiling First

A new open-source agent skill hill-climbs local LLM serving speed. Its method - ceiling first, one change at a time, interleaved controls - is worth stealing even if you never install it.

Blog
Coding Agent Harness Benchmark: Same Model, 17.5x Cost Gap

FrontierHarness ran one model through 9 coding agent harnesses. Pass rates sat within 17 points, median cost per pass spread 17.5x. What to measure before you pick Claude Code, Codex, Pi or OpenCode.

Blog
GVS5H: Can Qwen3.8-27B Match Fable 5 on Coding?

GVS5H claims five Qwen3.8-27B instances match Fable 5 on LiveCodeBench Hard. What the harness does, why Hacker News doubts the benchmark, and what to copy.

Blog
xhigh vs Medium Effort: SlopCodeBench Oct 2026 Results

The October SlopCodeBench run found xhigh reasoning effort costs 2x to 4x more than medium without reliably passing more checkpoints. What the numbers say and what to change.

Blog
Local LLM Speed by Hardware: What llmspeed Gets Right

llmspeed estimates tokens per second for 30 machines and 15 models. What its numbers mean, the memory-bandwidth rule behind them, and why prefill matters for coding agents.

Blog
CodeMidas Turns Existing Code Into Coding-Agent Training Tasks

CodeMidas turns existing repository code into executable coding-agent RL tasks, tests, and verifiers, instead of relying on issues, commits, or hand labels.

Blog
The Agent Is the Worst Witness to Its Own Run

An agent's report on its own run is unreliable, and eight studies show why. The fix: append-only traces, obligation ledgers, and records the agent cannot write.

Blog
Coding Agents Are Learning to Please the Grader

Handshake's DeepSWE audit found frontier coding agents reasoning about imagined graders. Stop rewarding patches that pass tests but drift from the user's spec.

Blog
Agent Retrieval Bench Finds The Files Before The Fix

Agent Retrieval Bench isolates the part of coding-agent work most evals hide: did the agent find the right repository files before it started editing?

Blog
Absence Is the Failure Mode

AI scribe audits show omission is the dominant error, and LLM judges are near coin-flip at catching it. The fix: enumerate the facts, then check each one.

Blog
Diffs vs Whole Files: What Code Editing Agents Should Rewrite

Diffs vs whole files for code editing agents: a paper finds full-file generation beating diff edits on Flutter/Dart tasks. The lesson: route by task locality.

Blog
Terminal-Universe Turns Agent Traces Into Training Environments

Terminal-Universe, Qwen's paper on rebuilding terminal-agent trajectories into reusable environments, and why agent teams should keep replayable workspaces.

Blog
The Response Looked Right. The Work Was Not Done.

AI agents often claim completion they did not earn. New benchmarks measure the gap, and evidence-carrying termination makes done a checked artifact.

Blog
6 of 11 ASR Models Transcribe the Benchmark, Not the Audio: Benchmark Optimization in Speech, Quantified

ASR benchmark optimization, quantified: a Hume AI and Hugging Face study finds 6 of 11 speech models reproduce benchmark transcripts instead of the audio.

Blog
The Oracle Is Agreeing With Itself

Self-improving test generation loops can grade against an oracle that agrees with itself. An audit shows the inflated gain and why independent checks count.

Blog
The Judge Is Now a System You Design

LLM judges flip verdicts under pushback and rankings shift with token budget. The design spec for a judge system, including cheap deliberating juries.

Blog
Mendel Godel Machine: Why Self-Improving Coding Agents Need Lineage

A new August 2026 paper argues that coding agents improve faster when they compare attempts across tasks and lineages, not just retry one failed trajectory.

Blog
TutorMoments: AI2's New Benchmark Shows LLM Tutors Over-Help by Default

TutorMoments, AI2's replay-based benchmark for LLM tutors, finds models over-help by default. How it scores scaffolding and rigor in real tutoring transcripts.

Blog
The Plateau Was the Instrument

A benchmark plateau can be a broken yardstick: a SciCode audit found 263 defects, and corrected scores jump to 84-98 percent. How to read flat leaderboards.

Blog
The Judge Is Leaving the Agent Loop

The LLM judge is leaving the agent loop: evidence gates, verifiable rewards and deploy-time certificates replace verdicts with structure. Here is our bet.

Page 1 of 2Next
AI Development Stack

Get Smarter About AI Dev

New tutorials, open-source projects, and deep dives on coding agents - delivered weekly.

One email per weekReal code, not theoryFree forever