AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement Learning

Authors: Zi-Han Wang, Zhengxi Lu, Zhiyuan Yao, Jinyang Wu, Jie Wu, Zhengzhou Cai, Yueqing Sun, Ziang Ye, Linji Hao, Qi Gu, Xunliang Cai, Yongliang Shen, Yujiu Yang (Tsinghua University)

arXiv ID: 2608.05987

Problem: RL with verifiable rewards builds trajectory-level advantage estimates, but in long-horizon, multi-turn agentic tasks the outcome usually hinges on a few pivotal decisions, and trajectory-level credit fails to single them out. Privileged self-distillation provides denser supervision, but it was unclear how local signals should represent sequential credit.

Key Methodology:

Aggregates token-level teacher-student log-probability gaps into turn-level evidence
Recursively updates a Bayesian belief state in log-odds space, reweighting sparse outcome supervision into per-turn credit signals
Identifies pivotal turns through the marginal belief revision between consecutive states
Critic-free, no additional rollouts, compatible with standard policy optimization

Key Results:

Outperforms GRPO and strong self-distillation baselines on ALFWorld, WebShop, and Search-QA with Qwen2.5-3B and Qwen2.5-7B
89.1% success on ALFWorld with Qwen2.5-7B
Ablations attribute gains to turn-level aggregation and history-dependent recursive belief updates

Applied Context: For teams training long-horizon agents with verifiable rewards, this is a cheap upgrade path: turn-level credit without a learned critic. The belief-state mechanism also gives a principled way to detect which turns mattered after training - useful for trajectory auditing and reward design.