Topic
RESEARCH
Research briefings and fact-rooted posts - what the papers, HN threads, and benchmarks actually say.
56 resources - 51 posts, 1 tool, 4 guides
Blog Posts
View in blog →
Your Agent Has a Five-Constraint Budget
Give an agent one instruction and it obeys. Give it eight and it obeys all of them about five percent of the time, no matter which frontier model you bought. The phase transition is measured, the constraints also die in compaction and handoff notes, and in security-critical code the failure ships as infrastructure. The fix is not a better prompt. It is a smaller simultaneous budget and a side channel for the rules that must survive.
10 min read
The Response Looked Right. The Work Was Not Done.
Across three independent benchmarks this week, agents claimed completion they had not earned: 75.5 percent of non-passing Claude Code trajectories end in language that says done, high partial scores hide 0 to 4 percent real delivery, and answer-only evals count invalid traces as wins. The same week produced the fix: completion is becoming a certifiable artifact - a typed certificate bound to a replayable trace - and it works. Our bet: by end of 2027, 'done' stops being the model's claim and becomes a checked artifact in any consequence-bearing workflow.
10 min read
The Quiet Tax on Your Cheap Agent Tier
Compression is the default answer to the agent bill, and a new three-model, eleven-method audit says the bill is the wrong place to look: quantized and pruned agents lose their head knowledge first, stay confidently wrong about what they lost, and hide subgroup preference flips behind flat bias scores. The same week, the serving side produced cost cuts that touch none of that. Our bet: cheapness comes from the cache before it comes from the weights.
9 min read
The Oracle Is Agreeing With Itself
A feedback-driven test-generation loop reported steady improvement. An audit found a single-reference oracle had inflated the measured gain by 9.46 to 14.85 points, independent resampling beat the evolution at equal budget, and a placebo arm erased the feedback benefit. The judge was never the only layer that lied - the reference underneath shares the disease. Independent verification is the only real verification.
10 min read
The Judge Is Now a System You Design
LLM judges flip 25 to 71 percent of their verdicts under pushback and 62 to 91 percent under a trainable persuader, model rankings reverse across token budgets, and a deliberating jury of cheap open-weight models beats frontier single judges at 8 to 15 percent of the cost. The single-judge era is over. Here is the design spec that replaces it.
11 min read
We Read DeepSeek Harness: What 453K Lines of Agent Runtime Actually Say
DeepSeek open-sourced its agent harness today. We cloned it and read the code: a 453K-line plugin runtime on a vendored Cordis fork, three patterns worth stealing, V4 line signals hiding in the model adapter, and a 3-line BENCHMARK.md from a lab that published zero eval claims.
9 min read
Mendel Godel Machine: Why Self-Improving Coding Agents Need Lineage
A new August 2026 paper argues that coding agents improve faster when they compare attempts across tasks and lineages, not just retry one failed trajectory.
8 min read
Skill Files Are the New Supply Chain Attack Surface
Adversarial skill files - folders of instructions agents load dynamically - exploit a mainstream enterprise coding agent in 95.5 to 96.1 percent of runs, while the agent recognizes danger 1.99 percent of the time. The skill folder is now a measured attack surface, and the defense is admission engineering, not better prompts.
11 min read
ACE vs ALTK-Evolve: How You Deliver Agent Memory Determines the Token Bill
ACE and IBM's ALTK-Evolve both turn agent trajectories into reusable lessons. The difference is delivery: one injects the whole playbook every step, the other calibrates. On AppWorld, calibration wins with the same accuracy at a fraction of the tokens.
7 min read
Encrypted Chain-of-Thought Is Not Private: New Paper Decodes Reasoning Traces From Anthropic, OpenAI, and Google APIs
A new arXiv paper shows the encrypted reasoning blocks that Anthropic, OpenAI, and Google return to API clients can be replayed into weaker models from the same provider and transcribed verbatim. The authors decoded 315,320 blocks from public repositories and recovered 367 PII artifacts and 182 credentials.
7 min read
Stop Means Stop: New Paper Finds Agent Approval Gates and Cancellation Leak in Six Frameworks
A new arXiv paper probes six widely used open-source agent frameworks and finds the barrier semantics of approval gates, cancellation, and timeouts hold on none of them. A sibling branch can execute while the user is rejecting another one, and replay can double-execute. The fix is a verified external gate called SoundGate.
6 min read
AgentChaos: Fault Injection Shows Agent Robustness Is a Systems Problem, Not a Model Problem
A new ASE 2026 framework injects server errors, truncated responses, and corrupted tool calls into live agent systems at the HTTP layer. Every system degrades, pass@1 drops up to 50 points, and the ranking stays the same no matter which LLM is behind it.
6 min read
LivePlan: Monitoring and Corrective Steering for Coding Agents, Without the LLM Tax
A new arXiv paper builds a deterministic monitor on top of SWE-agent that watches long agent trajectories and only calls an advisor LLM when the run actually drifts. Resolution rates go up by up to 15.2 points at an extra $0.08 per instance, and the paper argues the expensive approach is re-planning from inside the loop.
7 min read
DeepMind Open-Sources WeatherNext Cyclones After a Nature-Verified Breakthrough
WeatherNext Cyclones adds a full day of lead time to tropical cyclone forecasts - roughly a decade of meteorological progress - and now the weights, code, and data feeds are public. What the paper actually shows and how to run it.
8 min read
DCAS: Why Fine-Tuned Coding Agents Fall Apart When You Switch Scaffolds
A Huawei-Queen's study finds open coding models fine-tuned under OpenHands degrade sharply under other scaffolds - SWE-Lego-Qwen3-32B drops from 52.6% to 8.4% Pass@1 on OpenCode. The fix: train planning as a model capability, not a scaffold artifact.
8 min read
TutorMoments: AI2's New Benchmark Shows LLM Tutors Over-Help by Default
AI2 released TutorMoments, a replay-based benchmark that drops seven LLMs into real math tutoring transcripts and scores whether they scaffold when help is needed or push for rigor when the student can do more. The default finding: models over-help, and spelling out the trade-off in the prompt lifts every score but does not close the gap to a consistent human call.
7 min read
SkillSV: A Shapley Framework That Values the Lines Inside an Agent Skill
Automated skill optimizers write long SKILL.md files whose credit is a black box. SkillSV attributes value to rules, examples, and scripts inside a skill: pruning to 69% of tokens without significant loss on four benchmarks.
8 min read
The Plateau Was the Instrument
Twelve frontier models sat at 60 percent on scientific coding, successors tying predecessors - a textbook saturation curve. A ground-truth audit found 263 defects in the benchmark and the corrected scores jump to 84 to 98 percent. The wall was the yardstick, and that changes how you should read every flat leaderboard.
10 min read
Kill Your Agent Runs Early
The first production-scale trace of agentic coding says the context you keep paying for is already dead at every turn boundary. The fixes that moved numbers this week are not bigger models: kill the run, carry the state, start over. Here is the bet you can grade us on.
11 min read
13.5 Million Copilot Sessions: What Production Coding Agent Traffic Actually Looks Like
A Microsoft Research analysis of 3.2M users and 761M LLM calls shows coding agent traffic is 87% agent-initiated, burns KV cache at turn boundaries, and punishes every tool failure with up to 4x compute.
7 min read
The Judge Is Leaving the Agent Loop
Evidence gates, verifiable reward games, deploy-time certificates: the fixes that moved agent quality this week did not make judges better, they removed the judge. We think the LLM verdict inside the agent loop is a transitional technology, and here is the bet you can grade us on.
10 min read
The Fix for Broken Benchmarks Is Architecture, Not Smarter Models
Since we published your-benchmark-is-lying-to-you, roughly 25 new results have landed on the eval-integrity question. The surprise: every fix that works is structural - ledgers, counterfactuals, decompositions, personas, readout discipline - and none of them asks the model to be smarter.
11 min read
Your Benchmark Is Lying to You
A wave of audits in the last two days measured the noise floor of agent benchmarks: misaligned ground truth, lenient model judges, and aggregate scalars that hide real failures. Here is what the numbers actually mean, what to trust, and how to buy agents without being played.
10 min read
Agent Memory Is Moving Into the Model
A late-July research wave - native in-backbone memory, pretrained parametric memory at scale, memory reconstruction, and transactional memory writes - challenges the external-store paradigm every agent memory product is built on. Here is what changes by late 2027 and what developers should do now.
8 min read
Blind Resampling Beats Self-Repair in Small Code Models: Retry Without the Failed Code
A placebo-controlled study on MBPP+ finds that when small code models fail, resampling from scratch beats repair loops that feed the failed code back - at 2.5-5.5x fewer tokens. The failed attempt is the anchor.
7 min read
CAPA Benchmark: Why Coding Agents Should Learn Your Habits Across Sessions
A new 600-session benchmark shows coding assistants that read a user's resolved session history resolve ambiguous requests with far fewer clarifying questions - Claude Opus 4.8's first-turn success jumps from 24.3% to 60.3% when history is available.
7 min read
Change2Task: The Assembly Line for Coding Agent Training Data
Microsoft's Change2Task turns merged pull requests into verified, executable coding agent tasks: 79.6% construction success across 1,130 repo changes, 29.2% more verified tasks than PR baselines, and tasks that stay current with the codebase.
6 min read
DeepSeek V4 Flash 0731: The Budget Tier Just Overtook Pro Preview on Agent Benchmarks
DeepSeek re-post-trained V4 Flash into an agent workhorse: Terminal Bench 82.7, DeepSWE 54.4, native Responses API, and first-party Codex support - all at $0.14/$0.28 per million tokens. What changed, what the numbers actually mean, and how to wire it up today.
9 min read
Microsoft's CLI Coding Agent Study: Adoption Is a Workflow Problem
A July 2026 Microsoft study of Claude Code and GitHub Copilot CLI found roughly 24% more merged pull requests among adopters, but the interesting lesson is rollout design, not magic productivity.
8 min read
ORCA-bench: Frontier Agents Score 10% on Hard Oncall RCA
A new benchmark drops five frontier coding agents into a live OpenTelemetry microservice system with real Prometheus, Jaeger, and OpenSearch telemetry. Best RCA accuracy: 25.3% on Medium, 10.0% on Hard. Even Claude Fable 5 is far from oncall-ready.
6 min read
OwlPath: Ontology-Based Code Retrieval Cuts Agent Tokens 29%
A new paper wraps code into an OWL2 ontology with SPARQL property paths to answer multi-hop structural queries for coding agents - 2.06x retrieval recall and 28.8% fewer tokens on SWE-bench Pro, versus treating code as plain text.
6 min read
PAIChecker: 13.6% of SWE-bench Verified Instances Have Misaligned PR-Issue Pairs
A systematic audit of SWE-bench Verified finds 68 of 500 instances (13.6%) pair a pull request with an issue it does not actually resolve, penalizing agents that correctly solve the stated problem. PAIChecker, a three-phase multi-agent checker, flags them with up to 92.12% binary accuracy.
7 min read
SIGIL Compiles Agent Skills into Harnesses: Prose Runs Skip 44% of Mandated Steps
A Michigan team measures prose SKILL.md files against compiled harnesses: agents execute only 56% of the steps their own skill mandates. SIGIL compiles skills into typed graph harnesses, hitting 86% compliance with 0.58x the tokens.
8 min read
SWE-NFI: The Benchmark That Catches What Coding Agents Miss
A new 188-task benchmark for non-functional improvements finds coding agents hit 70% on functional correctness but lag humans on refactors and structural changes - the quality gap that becomes tech debt.
6 min read
What If AI Was Free Tomorrow, at Exactly Today's Capabilities?
A thought experiment with the sci-fi removed: freeze the models at today's capability, drop the price to zero overnight, and work out what actually changes for a working developer. Less than you fear, more than you think, and not where you expect.
11 min read
$500 RL Fine-Tune of a 9B Open Model Beat GPT-5.6 Sol and Claude Opus 4.8 on Catalog Review
FermiSense fine-tuned Qwen 3.5 9B with 2,500 GRPO steps on a single GPU for $500 and beat GPT-5.6 Sol (93%) and Opus 4.8 (91%) on automotive catalog review, reaching 97% accuracy at 68x lower cost per listing.
7 min read
FLUX 3: Black Forest Labs Ships a Unified Multimodal Foundation Model for Image, Video, Audio, and Robotics
Black Forest Labs released FLUX 3, a single multimodal model trained jointly on images, video, and audio that also drives robots on Audi production lines. Here is what it does, how it works, and how to try it.
10 min read
Terence Tao Digests the Jacobian Conjecture Counterexample: How Claude Fable 5 Broke an 87-Year-Old Math Problem
Terence Tao published a deep mathematical digestion of the Jacobian conjecture counterexample discovered by Claude Fable 5. Here is what happened, what HN is saying, and what it means for AI-assisted research.
9 min read
Dockerless Verification Is The Next Coding Agent Bottleneck
ByteDance's Dockerless paper asks whether coding-agent patches can be verified without spinning up per-repo environments. The practical answer is not replace CI. It is use cheaper evidence before CI.
8 min read
Ghost Font: Text That Humans Can Read But AI Cannot
A new experimental technology encodes messages in video using motion-based steganography, exploiting how AI models process video as individual frames rather than continuous motion.
6 min read
Harness Engineering and the Path to Self-Improving AI
Lilian Weng argues self-improving AI won't start with models rewriting their weights - it starts with the harness. Here's what that means for developers building agents.
7 min read
Ilya Sutskever's 30 Papers: The Reading List That Covers 90% of What Matters
A CS student built 30papers.com to make Ilya's legendary ML reading list more accessible. HN has thoughts on the source, the format, and why compression equals intelligence.
7 min read
AI Tutor Shows 0.71-1.30 SD Effect Size in Dartmouth Statistics Course
A new study from Dartmouth measures the impact of an AI tutoring platform on introductory statistics performance. Full engagement with the system correlated with significant exam score improvements, though selection bias remains a key limitation.
8 min read
Clean Code Makes AI Agents 34% More Efficient - New Research
A controlled study of 660 Claude Code trials shows clean codebases reduce token usage by 7-8% and file revisitations by 34%, while pass rates stay the same. Traditional maintainability principles still matter in the age of AI coding.
7 min read
Does Code Cleanliness Affect AI Coding Agents?
A new SonarSource study finds clean code doesn't boost agent pass rates - but it cuts token usage by 8% and file revisitations by 34%. Here's what that means for your codebase.
5 min read
Program-as-Weights Turns Prompts Into Local Fuzzy Functions
The Program-as-Weights paper is a useful signal for developers: some LLM calls may move from per-request API prompts into compact local artifacts that behave like reusable fuzzy functions.
7 min read
Claude Science Developer Guide 2026: AI Workbench for Research
Anthropic's Claude Science combines scientific tools, local code execution, and HPC integration into one AI workbench. Here is how to access it, what it costs, and where it fits alongside Claude Code.
8 min read
Prompt Injection is Role Confusion - New ICML Research Explains Why LLMs Can't Tell Friend from Foe
New research from MIT reveals that LLMs identify speakers by writing style, not by tags - meaning attackers who sound like the system effectively become the system. The findings explain why prompt injection remains unsolved.
7 min read
Multi-Stream LLMs Hint at the Next Agent Architecture
The Multi-Stream LLMs paper argues that agents are bottlenecked by single chat streams. The practical takeaway is not to rebuild everything today, but to design agent runtimes around separated channels.
8 min read
Refusal Directions Are a Systems Problem
A trending refusal-direction paper is a reminder that model safety cannot be treated as a thin refusal layer. Builders need layered controls around the model.
8 min read
Over-Editing: Why Your AI Coding Agent Rewrites What Isn't Broken
A new study from nrehiew quantifies a problem every Claude Code, Cursor, and Codex user has felt: models making huge diffs for tiny fixes. Here is why it happens, why tests do not catch it, and what to do about it.
8 min read
Guides
All guides →Chronicle Research Preview Setup Guide
Set up Codex Chronicle on macOS, manage permissions, and understand privacy, security, and troubleshooting.
GuideFast Mode - Claude Code
2.5x faster Opus at a higher token cost (research preview).
GuideBuilt-in Subagents - Claude Code
Researcher, auditor, reviewer, and other ready-made subagent types.
GuideSubagent Context Isolation - Claude Code
Prevent bloating the main conversation with research or exploration.
GuideKeep exploring
More on Research
- - NotebookLM - recommended Research tool from the Developers Digest directory
- - Tools Directory - dive deeper across the Developers Digest knowledge base
- - All Research articles in the blog archive
- - Developers Digest on YouTube - video tutorials covering Research and more
Get Smarter About AI Dev
New tutorials, open-source projects, and deep dives on coding agents - delivered weekly.
One email per weekReal code, not theoryFree forever
Explore 921 topics
Browse All Topics