Hugging Face Daily Papers · September 1, 2026 · 4 min read

Does On-Policy Distillation Really Distill? From Noisy Teacher to Self-Improvement

Mirrored from Hugging Face Daily Papers for archival readability. Support the source by reading on the original site.

Like Read original ↗

On-policy distillation can improve reasoning without meaningful teacher guidance: its gains largely come from suppressing unlikely tokens sampled by the student, motivating OPSA—a teacher-free method that uses the model’s own uncertainty for self-improvement.</p>\n","updatedAt":"2026-09-01T02:58:43.305Z","author":{"_id":"662678dfdd43e904ef1dcd03","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/662678dfdd43e904ef1dcd03/9bbnrnsRV3ylSuZXwcSNv.jpeg","fullname":"Yi Ding","name":"Tuwhy","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":3,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9402663111686707},"editors":["Tuwhy"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/662678dfdd43e904ef1dcd03/9bbnrnsRV3ylSuZXwcSNv.jpeg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2608.31046","authors":[{"_id":"6a963e8acd6ebc484732ec74","name":"Yi Ding","hidden":false},{"_id":"6a963e8acd6ebc484732ec75","name":"Ruqi Zhang","hidden":false}],"publishedAt":"2026-08-31T00:00:00.000Z","submittedOnDailyAt":"2026-09-01T00:00:00.000Z","title":"Does On-Policy Distillation Really Distill? From Noisy Teacher to Self-Improvement","submittedOnDailyBy":{"_id":"662678dfdd43e904ef1dcd03","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/662678dfdd43e904ef1dcd03/9bbnrnsRV3ylSuZXwcSNv.jpeg","isPro":false,"fullname":"Yi Ding","user":"Tuwhy","type":"user","name":"Tuwhy"},"summary":"On-policy distillation (OPD) offers dense token-level supervision as an alternative to the sparse outcome-level advantages of reinforcement learning with verifiable rewards (RLVR). However, the teacher scores student-generated trajectories that are inherently off-policy for it, so the reliability of its supervision, and hence the source of the student's improvement, remains unclear. We quantitatively analyze teacher supervision during OPD training and find substantial noise whose prevalence increases with teacher scale. Surprisingly, the student policy is insensitive to such noise, converging to comparable performance regardless of whether noisy supervision is retained or removed. Does OPD distill at all? By analyzing what drives its gains, we find that learning concentrates on low log-probability tokens, and using a single fixed negative advantage matches the performance of teacher-provided ones. This suggests that OPD works largely by suppressing low log-probability tokens, which requires no teacher. These findings motivate On-Policy Self-Adaptation (OPSA), a supervision-free method using entropy-adaptive negative advantages. It assigns stronger learning signals to high-entropy positions, suppressing tail tokens, and evenly redistributing probability mass among head tokens. Compared with the base Qwen3-1.7B, OPSA improves Avg@32 by 35.41 points on AIME24, corresponding to a 263\\% relative gain, and more than doubles Pass@32 across all three benchmarks. It also outperforms OPD by 16.77 points in Avg@32 on AIME24. Extensive experiments and analyses across model families and tasks further demonstrate its effectiveness and generalizability.","upvotes":36,"discussionId":"6a963e8acd6ebc484732ec76","projectPage":"https://dripnowhy.github.io/On-Policy-Self-Adaptation/","githubRepo":"https://github.com/DripNowhy/On-Policy-Self-Adaptation","githubRepoAddedBy":"user","ai_summary":"On-policy distillation relies mainly on suppressing low-probability tokens rather than teacher guidance, motivating a supervision-free entropy-adaptive method that substantially improves reasoning performance.","ai_keywords":["on-policy distillation","RLVR","token-level supervision","off-policy","teacher supervision","negative advantage","low log-probability tokens","On-Policy Self-Adaptation","entropy-adaptive negative advantages"],"ai_summary_model":"thinkingmachines/Inkling-Small","githubStars":1,"organization":{"_id":"6400300fe7767a89533fa2c0","name":"Purdue","fullname":"Purdue University","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/1677733893266-64002f32cafc9d54986439cb.jpeg"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"662678dfdd43e904ef1dcd03","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/662678dfdd43e904ef1dcd03/9bbnrnsRV3ylSuZXwcSNv.jpeg","isPro":false,"fullname":"Yi Ding","user":"Tuwhy","type":"user"},{"_id":"67990449a5da36c49b9cf4ae","avatarUrl":"/avatars/85fd1dbd65a07175528f808ee3c927b1.svg","isPro":false,"fullname":"YM","user":"Yuanmoo","type":"user"},{"_id":"6799071cd5baec25f5d09c2e","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6799071cd5baec25f5d09c2e/rKICY8M8CKZxKwbu6wLwr.jpeg","isPro":false,"fullname":"MZG","user":"MingZIgan","type":"user"},{"_id":"698af5771dd8f1eea4779b34","avatarUrl":"/avatars/ab581a11f185f2310a867bed1433f2c8.svg","isPro":false,"fullname":"Tang zy","user":"ziyipposur","type":"user"},{"_id":"67c724b4d0396c281c8a52ab","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/67c724b4d0396c281c8a52ab/F7U0F9qfeLpb2_6Jao77A.jpeg","isPro":false,"fullname":"InoueNatsu","user":"SANsan-IN","type":"user"},{"_id":"6696843b65ae510864899000","avatarUrl":"/avatars/3e2ede88741965028246f22f6f5be00c.svg","isPro":false,"fullname":"PIKA","user":"PIKALIANG","type":"user"},{"_id":"69840936129ec40e4159c99f","avatarUrl":"/avatars/ca72a988fbc91dc9b3af9daac266e6b1.svg","isPro":false,"fullname":"Jifeng Luo","user":"Jierfurry","type":"user"},{"_id":"67b1a18a1af43b636063ef89","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/67b1a18a1af43b636063ef89/fZt7UmMXKC92c4Szn-xEJ.jpeg","isPro":false,"fullname":"youngin","user":"Genevieveve","type":"user"},{"_id":"699fb5ef198bf87527ece7af","avatarUrl":"/avatars/2badd63ef1135c97c3748bef7d303a65.svg","isPro":false,"fullname":"XiaoHY","user":"Xiaohuiyao","type":"user"},{"_id":"678898762bfbe4a597e78cf5","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/678898762bfbe4a597e78cf5/h9MVCAEbsppMCGvZCgzuj.jpeg","isPro":false,"fullname":"HU-ZSS","user":"GINng","type":"user"},{"_id":"698588cb6d36197a005dd8d5","avatarUrl":"/avatars/0e2c775d34e6eb0cfd1c88a2c69be76f.svg","isPro":false,"fullname":"Xiao Zhao","user":"Zxiaojin","type":"user"},{"_id":"693077d48797646011270196","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/693077d48797646011270196/a16O1aXUbUVyXWgjeLMFx.png","isPro":false,"fullname":"Zheke Jin","user":"JinzheNea","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":3,"organization":{"_id":"6400300fe7767a89533fa2c0","name":"Purdue","fullname":"Purdue University","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/1677733893266-64002f32cafc9d54986439cb.jpeg"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2608/2608.31046.md","query":{}}">

Papers

arxiv:2608.31046

Does On-Policy Distillation Really Distill? From Noisy Teacher to Self-Improvement

Published on Aug 31

· Submitted by

Yi Ding on Sep 1

#3 Paper of the day

Purdue University

Upvote

Authors:

Abstract

On-policy distillation relies mainly on suppressing low-probability tokens rather than teacher guidance, motivating a supervision-free entropy-adaptive method that substantially improves reasoning performance.

Generated by thinkingmachines/Inkling-Small

On-policy distillation (OPD) offers dense token-level supervision as an alternative to the sparse outcome-level advantages of reinforcement learning with verifiable rewards (RLVR). However, the teacher scores student-generated trajectories that are inherently off-policy for it, so the reliability of its supervision, and hence the source of the student's improvement, remains unclear. We quantitatively analyze teacher supervision during OPD training and find substantial noise whose prevalence increases with teacher scale. Surprisingly, the student policy is insensitive to such noise, converging to comparable performance regardless of whether noisy supervision is retained or removed. Does OPD distill at all? By analyzing what drives its gains, we find that learning concentrates on low log-probability tokens, and using a single fixed negative advantage matches the performance of teacher-provided ones. This suggests that OPD works largely by suppressing low log-probability tokens, which requires no teacher. These findings motivate On-Policy Self-Adaptation (OPSA), a supervision-free method using entropy-adaptive negative advantages. It assigns stronger learning signals to high-entropy positions, suppressing tail tokens, and evenly redistributing probability mass among head tokens. Compared with the base Qwen3-1.7B, OPSA improves Avg@32 by 35.41 points on AIME24, corresponding to a 263\% relative gain, and more than doubles Pass@32 across all three benchmarks. It also outperforms OPD by 16.77 points in Avg@32 on AIME24. Extensive experiments and analyses across model families and tasks further demonstrate its effectiveness and generalizability.