← Back to feed

Recovering Reliable Supervision via Critique-Guided On-Policy Distillation

L5 · ResearcherResearcharXiv· 10/14/2026

Advances model training techniques for mathematical reasoning with measurable benchmark improvements

AI Summary

CriticOPD improves on-policy distillation by using teacher critiques to recover reliable supervision when students generate incorrect rollouts.

Excerpt

On-policy distillation (OPD) trains a student model on its own rollouts with token-level supervision from a teacher, assuming that the teacher can reach a correct answer from any prefix the student writes. We call this property the recoverability of the teacher and show it fails on incorrect rollouts: the teacher follows student errors, although it recognizes them. We propose CriticOPD, using error recognition to restore recoverability. The teacher critiques an incorrect rollout, allowing the st

Read Original
0 upvotes · 0 downvotes · 1 min read

Related Articles

L5 · ResearcherResearcharXiv
Latent Compaction with Test-Time Training

Researchers introduce Latent Compaction, a test-time training method that compresses KV caches 32× while maintaining 96.7% accuracy on retrieval tasks.

L5 · ResearcherResearcharXiv
Price of Anarchy of First-Price Auctions in Agentic Markets

Researchers prove first-price auctions in AI agent markets can achieve near-optimal welfare under competitive floors and conditional independence assumptions.

L5 · ResearcherResearcharXiv
Your Transformer Is Secretly Elastic

Researchers introduce a retraining-free method to make transformers elastic, enabling on-the-fly adaptation across computational budgets by jointly ranking MLP neurons and attention heads.

L5 · ResearcherResearcharXiv
Co-Evolving Acting and Forecasting for Language Agents

Researchers propose Co-Evolving Acting and Forecasting (name), a method that improves language agent training efficiency by adding multi-step action targets to standard RL approaches.

L5 · ResearcherResearcharXiv
Hierarchical Residual Policy Learning for Real-World Mobile Manipulation with Sparse Human Guidance

Researchers propose HiRe-MoMa, a hierarchical residual learning framework that enables real-world robot fine-tuning with only 30 minutes of human-guided interaction.