Multimodal / Distillation / Training
Denser $ eq$ Better: Limits of On-Policy Self-Distillation for Continual Post-Training
** Meng Wang, Haohan Zhao, Wenzhuo Liu, Lu Yang, Geng Liu, Haiyang Guo, Guo-Sen Xie, Gaofeng Meng, Hongbin Liu, Fei Zhu
Denser $\neq$ Better: Limits of On-Policy Self-Distillation for Continual Post-Training
Authors: Meng Wang, Haohan Zhao, Wenzhuo Liu, Lu Yang, Geng Liu, Haiyang Guo, Guo-Sen Xie, Gaofeng Meng, Hongbin Liu, Fei Zhu
arXiv ID: 2607.01763
Problem: Does injecting denser (token-level) on-policy self-distillation signals improve continual post-training for foundation models, or can it actually hurt?
Key Methodology:
- Introduces Self-Distillation Policy Optimization (SDPO), which adds token-level KL supervision from a teacher (the model itself at an earlier step) on top of on-policy preference optimization.
- Compares SDPO against on-policy RL methods (GRPO) across in-domain specialization, out-of-distribution generalization, and multi-task continual post-training settings.
- Analyzes parameter-space drift, response-space drift, and frequency-domain artifacts to diagnose failure modes.
Key Results:
- SDPO accelerates in-domain specialization (+3.2% on GSM8K) but collapses on OOD benchmarks (e.g. -12.8% on MATH, -7.1% on HumanEval) compared to GRPO.
- In continual post-training, SDPO exhibits stronger forgetting (average accuracy drop of 9.4% across tasks) versus GRPO (2.1% drop).
- Denser self-distillation induces 2.3× larger parameter drift and amplifies high-frequency formatting artifacts by up to 4.7 dB in spectral analysis.
Applied Context: Don't treat dense self-distillation as a default stabilizer for continual post-training - it accelerates convergence only when teacher targets are perfectly stable and token-level supervision is reliable; otherwise it amplifies distribution shift and collapse. GRPO-style on-policy RL is the safer default for builders doing continual fine-tuning.