Why Software Factories Fail: Harness Engineering Is Not Enough

The dream of "lights-off" software factories - where AI agents autonomously ship code without human review - has been a recurring promise since coding agents emerged. But a new analysis from Dex Horthy at HumanLayer argues that the fundamental limitation is not harness engineering. It is the models themselves.

Last updated: October 14, 2026 - re-checked against the source: fixed the timeline of the failed factory, replaced two pull-quotes that did not match the text, and refreshed the thread numbers.

The article, Why Software Factories Fail, landed on Hacker News in July and has since reached 394 points and 272 comments. The discussion revealed a community deeply invested in making AI coding work - and increasingly realistic about what it cannot do yet.

The Core Argument#

Horthy's thesis is straightforward: current AI models excel at solving isolated problems but systematically degrade codebase quality over time. The issue is not that we lack good testing infrastructure or sophisticated agent harnesses. The issue is that models cannot reliably distinguish good architecture from bad architecture.

The article traces the evolution of "software factories" through three phases:

1. 2022 Model: Humans decide, build, review, and ship
2. Agentic Model: Agents replace humans in building, but review still bottlenecks
3. Lights-Off Model: Remove human review entirely; invest in testing and monitoring

Horthy says his team went "full lights-off" in July 2025. After repeated incidents where the agent could not fix a gnarly bug in code nobody had read for months, they decided around the third one, in November, to rewrite from scratch, and his cofounder spent two weeks plumbing the patterns out by hand. He is open that HumanLayer sells human-agent collaboration tooling, so the argument may be "a tad biased" in his own words.

Why Tests Are Not Enough#

The central insight is about feedback loops. Tests provide feedback in seconds. Architectural problems surface weeks or months later. Reinforcement learning cannot capture this delayed cost.

From the article:

The article's own heading says it plainly: "THERE ARE NO GOOD BENCHMARKS for a model's ability to maintain codebase quality." It admits it cannot prove the broader claim and offers it as an argument from how coding models are trained and evaluated.

Models trained via RL optimize for whatever metrics the training loop rewards - typically binary pass/fail signals. If a model could reliably distinguish good code from bad, it might have written the good version initially. But maintainability has no fast oracle.

What HN Is Saying#

The Hacker News discussion surfaced several perspectives that extend Horthy's analysis.

On the discipline gap: One commenter from the StrongDM AI Lab pushed back on the framing, noting that teams seeing the best results with AI were already high-discipline and high-hygiene before AI. "In order for coding with LLMs to go well, there has to be more rigor, more discipline, more good engineering hard-assedness." (link to thread)

On formal verification: Multiple commenters noted that prompts alone cannot steer agents to the precision needed. One developer at Autodesk shared a taxonomy of constraint mechanisms: generative constraints (to shape output), interpretive constraints (to shape how the model understands code), and elicitative constraints (to help the model ask the right questions).

On the consulting pivot: StrongDM's dark factory experiment is now a consulting company (Diffusion.io). Several commenters pointed out this is a familiar pattern - when products cannot stand alone, companies pivot to services. Palantir and Salesforce follow similar trajectories.

On review itself: One of the most upvoted threads questioned whether AI should be used to replace code review or to improve it. The consensus: review serves multiple purposes beyond correctness - knowledge sharing, architecture consistency, maintainability signaling. Automating only the "find bugs" slice misses most of the value.

The Context Engineering Response#

If harness engineering is insufficient, what actually helps? Horthy points to "context engineering" - front-loading alignment before code generation begins.

The practical recommendations:

1. Invest in planning discussions before coding. Thirty minutes of architecture conversation with the model can save hours of review and rework downstream.

2. Maintain human involvement at the design level. Strategic oversight on system architecture prevents problems that no amount of testing will catch.

3. Apply the Theory of Constraints. Recognize what models do well (isolated problem-solving, boilerplate generation, refactoring with clear patterns) versus what they do poorly (system design, long-term maintainability, architectural judgment). Optimize workflows around those constraints.

The takeaway is not that AI coding is broken. It is that the "lights-off" framing was always wrong. Human-agent collaboration is the productive model. Full automation is not.

Why This Matters for Developers#

The timing is notable. We are in a period where AI coding agents are shipping features faster than ever - Codex, Claude Code, Cursor, Kiro - yet production deployments keep surfacing the same patterns. Agents work well inside tight loops with clear verification. They struggle when the feedback signal is diffuse or delayed.

For teams evaluating AI coding tools, the implication is clear: invest in the context layer, not just the model. The right CLAUDE.md, the right test scaffolding, the right architecture documentation - these matter more than which model you choose.

The article also validates a pattern we have covered before in Agent Evals Need Baseline Receipts and AI Code Review Bottleneck: benchmarks measure the wrong things. Pass rates on SWE-bench do not predict production success. What predicts production success is whether the agent can operate inside a context that encodes your team's standards.

The Honest Position#

Horthy's section titled "This is not a skill issue" says that no amount of harness engineering can fix what he calls "fundamentally a model-training issue". His practical answer is to keep humans in the loop at the steps where it matters, such as his "30 minutes of planning saves hours of review".

The Hacker News discussion largely agreed. The arguments were about degree, not direction. How much human involvement is needed? At what points in the loop? How do you scale review when agents generate code 10x faster?

Those are the right questions. They assume AI coding is here to stay - and that the work is figuring out the collaboration model, not waiting for a magical fully-autonomous future.

Continue Reading#

• 12 Factor Agents: Production Principles
• Agent Evals Need Baseline Receipts
• AI Code Review Bottleneck
• Context Engineering for Coding Agents
• Why Code Cleanliness Affects AI Coding Agents
• Agent Harness Lessons From a 500B-Token Decompile - a byte-matching PASS/FAIL oracle that replaced the reviewer agent
• Intro to Agents 101 course - the harness, the loop, and verify loops for coding agents, built hands-on
• Best AI Agent Frameworks (2026) - the frameworks teams use to build their own harness, compared with code

Sources#

• Why Software Factories Fail - HumanLayer, July 2026
• Hacker News Discussion - 394 points, 272 comments (as of Oct 14, 2026)
• StrongDM AI Lab Weather Report
• Diffusion.io - StrongDM's consulting pivot