Change2Task: The Assembly Line for Coding Agent Training Data

Coding agents are eating benchmarks, but the benchmarks themselves are running out of food. Every agent release is now sold with numbers on SWE-bench-style tasks, and every one of those tasks has to couple a realistic software state with a specification, development tools, and reliable verification. Building that data is slow, manual, and it goes stale the moment the repository moves on. A new paper from Microsoft, Change2Task (arXiv 2607.28591, submitted July 30), treats that problem as a manufacturing problem: a pipeline that converts merged pull requests into verified, executable agent tasks that stay current with the codebase.

What it is#

Change2Task is a system grounded in repository history. It takes a merged pull request and turns it into a task an agent can actually run: a starting state, a specification, tooling, and automated verification. The hard part is that repositories change after a PR lands, so the paper's core contribution is aligning historical evidence with the evolved code. Three reconstruction strategies cover different cases:

• Patch Reversal: when the code around the change is stable, reverse the patch to rebuild the pre-change state.
• Code Mapping: map the historical change onto the modern revision where files moved or were renamed.
• Agent Reconstruction: when the change is too entangled with history, have an agent rebuild the task state and validate it.

The pipeline validates a full lifecycle for every task: a healthy base state, a task state, and a restored state after the fix. No verified lifecycle, no task.

The scope covers five task families that map to real agent work: bug fix, feature addition, test generation, API migration, and security repair. Results from the paper, starting from 1,130 eligible source changes:

• 79.6 percent verified task construction success across all five families
• 29.2 percent more verified tasks than a construction baseline based on pull requests, on a matched candidate set
• Up to 98.0 percent matched outcome agreement between historical and reconstructed tasks under agent evaluation
• 10.8 percent lower measured expenditure across the full pipeline by reusing modern base states

The last number matters more than it looks. Task construction has real cost, mostly environment setup and storage, and Change2Task's design makes the base environment a shared, reused asset instead of a per-task snapshot.

Why it matters#

The trend line across recent agent research is that the data is the product. SWE-NFI built 188 tasks from real merged PRs to test whether agents can refactor without breaking behavior. DeepSeek's V4 Flash 0731 update is a model re-post-trained on agent work, which only happens when the training data pipeline exists. Change2Task is the plumbing underneath both: it makes the supply of executable tasks expandable rather than hand-curated.

Three concrete implications for developers:

Benchmark freshness becomes a feature. Most benchmark tasks are frozen at the commit where they were written, so an agent can "solve" a repository that no longer looks like the real one. Change2Task deliberately builds tasks on healthy modern revisions, so evaluation tracks the codebase agents will actually touch. That is the right bias: agents should be graded on today's tree, not last year's snapshot.

PR history is the untapped data source. Every merged pull request is a labeled example of a human doing the thing we want agents to do. The paper's 98 percent outcome agreement between reconstructed and historical tasks is the evidence that this source is trustworthy, not just plentiful. Teams with active repos could, in principle, generate their own in-house agent eval sets from their own PR history, which is closer to their real workload than any public benchmark.

Cost is being engineered out of evaluation. A 10.8 percent expenditure reduction on construction, plus shared base environments, changes the economics of continuous evaluation. Instead of a quarterly benchmark run, teams can afford evaluation on a rolling basis, which matters because review queues are already the bottleneck in agent-heavy workflows.

What it does not solve#

The paper is honest about limits. Agent Reconstruction tasks carry the risk of the agent contaminating the task with its own guesses, which is why the lifecycle validation and matched-outcome checks exist. And construction success is not task quality: 79.6 percent of changes become verified tasks, but verified means runnable, not necessarily useful for training. The harder filtering problem, which tasks teach agents anything, is still open.

There is also a public-good gap. The paper does not ship a released dataset or leaderboard with the preprint, so the 1,130-change corpus is not yet something you can pull and run against your own stack. If the authors publish the artifact, this becomes immediately more valuable; until then it is a method paper with strong internal numbers.

The pattern behind it#

Change2Task is one more entry in the running argument that the harness and the data beat the model. The context-files ablation showed agent performance is gated by what context the harness feeds the model, and baseline-receipt evals showed most agent evaluations are not measuring what they claim. Change2Task attacks the same theme from the data side: if the task supply is the bottleneck, industrialize the supply. That is the right instinct, and it is a safe bet that every frontier lab is building something like it right now.

Continue Reading#

• SWE-NFI: The Benchmark That Catches What Coding Agents Miss
• GitHub Stacked PRs Hit Public Preview: Small Reviews for the Agent Era
• DeepSeek V4 Flash 0731: The Budget Tier Just Overtook Pro Preview
• What Is an AI Coding Agent?
• Agent Evals Need Baseline Receipts

Sources#

• Change2Task: From Repository Changes to Executable Coding Agent Tasks and Environments, arXiv 2607.28591 (fetched July 31, 2026; abstract and submission metadata only)