Recovering Reliable Supervision via Critique-Guided On-Policy Distillation
Advances model training techniques for mathematical reasoning with measurable benchmark improvements
AI Summary
CriticOPD improves on-policy distillation by using teacher critiques to recover reliable supervision when students generate incorrect rollouts.
Excerpt
On-policy distillation (OPD) trains a student model on its own rollouts with token-level supervision from a teacher, assuming that the teacher can reach a correct answer from any prefix the student writes. We call this property the recoverability of the teacher and show it fails on incorrect rollouts: the teacher follows student errors, although it recognizes them. We propose CriticOPD, using error recognition to restore recoverability. The teacher critiques an incorrect rollout, allowing the st
