Code: <a href=\"https://github.com/furiosa-ai/async-opd\" rel=\"nofollow\">https://github.com/furiosa-ai/async-opd</a></p>\n","updatedAt":"2026-06-30T01:33:58.791Z","author":{"_id":"63e1d5247fbb6ae4d4f4cc8e","avatarUrl":"/avatars/8a8f700adf9e8000641c2c2f6bd56080.svg","fullname":"Wonjun Kang","name":"wjkang","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":1,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.6342362761497498},"editors":["wjkang"],"editorAvatarUrls":["/avatars/8a8f700adf9e8000641c2c2f6bd56080.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2606.24143","authors":[{"_id":"6a3f4bb80dbbc53604b66602","name":"Wonjun Kang","hidden":false},{"_id":"6a3f4bb80dbbc53604b66603","name":"Kevin Galim","hidden":false},{"_id":"6a3f4bb80dbbc53604b66604","name":"Seunghyuk Oh","hidden":false},{"_id":"6a3f4bb80dbbc53604b66605","name":"Minjun Kang","hidden":false},{"_id":"6a3f4bb80dbbc53604b66606","name":"Sanghyun Park","hidden":false},{"_id":"6a3f4bb80dbbc53604b66607","name":"Donghoon Kim","hidden":false},{"_id":"6a3f4bb80dbbc53604b66608","name":"Minjae Lee","hidden":false},{"_id":"6a3f4bb80dbbc53604b66609","name":"Minseo Kim","hidden":false},{"_id":"6a3f4bb80dbbc53604b6660a","name":"Rishabh Tiwari","hidden":false},{"_id":"6a3f4bb80dbbc53604b6660b","name":"Yuchen Zeng","hidden":false},{"_id":"6a3f4bb80dbbc53604b6660c","name":"Hyung Il Koo","hidden":false},{"_id":"6a3f4bb80dbbc53604b6660d","name":"Kangwook Lee","hidden":false}],"publishedAt":"2026-06-23T00:00:00.000Z","submittedOnDailyAt":"2026-06-30T00:00:00.000Z","title":"AsyncOPD: How Stale Can On-Policy Distillation Be?","submittedOnDailyBy":{"_id":"63e1d5247fbb6ae4d4f4cc8e","avatarUrl":"/avatars/8a8f700adf9e8000641c2c2f6bd56080.svg","isPro":false,"fullname":"Wonjun Kang","user":"wjkang","type":"user","name":"wjkang"},"summary":"On-policy distillation (OPD) trains a student on its own rollouts guided by teacher feedback and is becoming increasingly important for large language model (LLM) post-training. Like reinforcement learning (RL), however, OPD faces an on-policy systems bottleneck, as rollouts can dominate training time for reasoning workloads. Asynchronous training pipelines can alleviate this bottleneck by decoupling rollout generation from learner updates, but doing so introduces stale-policy data. While prior work has studied stale data in asynchronous RL, its effects in OPD remain underexplored. We present the first systematic study of staleness in asynchronous OPD, focusing on a practical setting where teacher feedback is implemented through local KL losses and full-vocabulary teacher logits are too expensive to store or transfer, necessitating finite teacher-score caches. We first show that KL direction changes the stale-data problem: teacher-weighted forward KL is more robust to stale rollouts, whereas student-weighted reverse KL is vulnerable. Second, for this vulnerable reverse-KL case, we study whether methods designed to stabilize asynchronous RL can mitigate OPD staleness. In our experiments, they do not improve over a simpler OPD-specific surrogate: recomputing the reverse-KL signal under the current student at learner time. Third, we analyze how finite teacher-score caches create a bias-variance tradeoff for sparse and sampled reverse-KL OPD estimators. This motivates multi-sample Monte Carlo (MC), which preserves MC correctability while reducing one-sample variance. Finally, we present and open-source AsyncOPD, a fully asynchronous OPD training pipeline built from these estimator choices. Experiments show that AsyncOPD improves training throughput by 1.6times to 3.8times over strict synchronous training while reaching comparable accuracy.","upvotes":23,"discussionId":"6a3f4bb80dbbc53604b6660e","githubRepo":"https://github.com/furiosa-ai/async-opd","githubRepoAddedBy":"user","ai_summary":"Asynchronous on-policy distillation addresses training bottlenecks in large language model post-training by decoupling rollout generation from learner updates, though it introduces challenges with stale policy data that require specialized solutions.","ai_keywords":["on-policy distillation","asynchronous training","stale-policy data","KL divergence","reverse KL","forward KL","teacher-student framework","Monte Carlo estimation","policy gradient","reinforcement learning"],"ai_summary_model":"Qwen/Qwen2.5-Coder-32B-Instruct","githubStars":15,"organization":{"_id":"6213a1dcb670cb63a38074a1","name":"furiosa-ai","fullname":"FuriosaAI","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/620b6dc29412b0861cb2474a/Bl7ua2mXSFxk9rVo8vMA8.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"63e1d5247fbb6ae4d4f4cc8e","avatarUrl":"/avatars/8a8f700adf9e8000641c2c2f6bd56080.svg","isPro":false,"fullname":"Wonjun Kang","user":"wjkang","type":"user"},{"_id":"68f131f0f7a26fddda93aa90","avatarUrl":"/avatars/d9a8b8b306f958214f5999b97ca2ffc3.svg","isPro":false,"fullname":"Wonjun Kang","user":"wjkang-furiosa","type":"user"},{"_id":"6a308db1114dc7ae7d02c605","avatarUrl":"/avatars/4e2603b671bfe79255e8c45da1ad4d87.svg","isPro":false,"fullname":"Younghyun Kim","user":"yhyunkim","type":"user"},{"_id":"69dc4ed08b76de135f9f5c48","avatarUrl":"/avatars/f29cea8e50aa045358d2de4d31666ad7.svg","isPro":false,"fullname":"Donghoon Kim","user":"DHKim-Furiosa","type":"user"},{"_id":"64ae35dc00781825350e880b","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/64ae35dc00781825350e880b/VuON41yUDzNDACbvWAVhz.jpeg","isPro":false,"fullname":"Seunghyuk Oh","user":"JakeOh","type":"user"},{"_id":"630c90123dc31beba6e8f406","avatarUrl":"/avatars/2188b41fff122d4f5683b46c529ed79d.svg","isPro":false,"fullname":"Kevin Galim","user":"kev95","type":"user"},{"_id":"665994a4fe8d926f8ea1eaaf","avatarUrl":"/avatars/ef8f742ccd83a88a9f0694a38c0568f7.svg","isPro":false,"fullname":"SanghyunPark","user":"shxyzn","type":"user"},{"_id":"650167a5f355a888f9610bc6","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/ziuPES8WVhWv8XAd-xu47.png","isPro":false,"fullname":"MinJun_Kang","user":"baramshu","type":"user"},{"_id":"65ded6ec6efc2a4ebb18d306","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/SPsUBPd_CKMBzACSCgwP5.png","isPro":false,"fullname":"Yuchen Zeng","user":"yzeng58","type":"user"},{"_id":"66ab9a9c661901e7adacbba0","avatarUrl":"/avatars/ee146edc35a6ac5e669188af35bf34d0.svg","isPro":false,"fullname":"MinseoKim","user":"minseo25","type":"user"},{"_id":"6463554dd2044cd1d7c6e0bf","avatarUrl":"/avatars/d7653623117268c545a7063fec69664b.svg","isPro":false,"fullname":"Bingzheng Wei","user":"Bingzheng","type":"user"},{"_id":"6a2da6c8ca070ee12c6e396c","avatarUrl":"/avatars/0355287dcabaa67dbc7f0b10b87451f9.svg","isPro":false,"fullname":"Joe Mama","user":"JoeMama123123123","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"6213a1dcb670cb63a38074a1","name":"furiosa-ai","fullname":"FuriosaAI","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/620b6dc29412b0861cb2474a/Bl7ua2mXSFxk9rVo8vMA8.png"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2606/2606.24143.md","query":{}}">
AsyncOPD: How Stale Can On-Policy Distillation Be?
Authors: ,
,
,
,
,
,
,
,
,
,
,
Abstract
Asynchronous on-policy distillation addresses training bottlenecks in large language model post-training by decoupling rollout generation from learner updates, though it introduces challenges with stale policy data that require specialized solutions.
On-policy distillation (OPD) trains a student on its own rollouts guided by teacher feedback and is becoming increasingly important for large language model (LLM) post-training. Like reinforcement learning (RL), however, OPD faces an on-policy systems bottleneck, as rollouts can dominate training time for reasoning workloads. Asynchronous training pipelines can alleviate this bottleneck by decoupling rollout generation from learner updates, but doing so introduces stale-policy data. While prior work has studied stale data in asynchronous RL, its effects in OPD remain underexplored. We present the first systematic study of staleness in asynchronous OPD, focusing on a practical setting where teacher feedback is implemented through local KL losses and full-vocabulary teacher logits are too expensive to store or transfer, necessitating finite teacher-score caches. We first show that KL direction changes the stale-data problem: teacher-weighted forward KL is more robust to stale rollouts, whereas student-weighted reverse KL is vulnerable. Second, for this vulnerable reverse-KL case, we study whether methods designed to stabilize asynchronous RL can mitigate OPD staleness. In our experiments, they do not improve over a simpler OPD-specific surrogate: recomputing the reverse-KL signal under the current student at learner time. Third, we analyze how finite teacher-score caches create a bias-variance tradeoff for sparse and sampled reverse-KL OPD estimators. This motivates multi-sample Monte Carlo (MC), which preserves MC correctability while reducing one-sample variance. Finally, we present and open-source AsyncOPD, a fully asynchronous OPD training pipeline built from these estimator choices. Experiments show that AsyncOPD improves training throughput by 1.6times to 3.8times over strict synchronous training while reaching comparable accuracy.
Community
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2606.24143 in a model README.md to link it from this page.
Cite arxiv.org/abs/2606.24143 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2606.24143 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.