We find that knowledge distillation exhibits a surprising capability tradeoff behavior in mid-training (compared to pre-training) and propose a simple drop-in fix that mitigates the tradeoff.</p>\n","updatedAt":"2026-09-02T22:51:32.258Z","author":{"_id":"627c52e51b02d7b5bf671dca","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1666240522285-627c52e51b02d7b5bf671dca.png","fullname":"Jacqueline He","name":"jacquelinehe","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":3,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9161221981048584},"editors":["jacquelinehe"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1666240522285-627c52e51b02d7b5bf671dca.png"],"reactions":[],"isReport":false}},{"id":"6a98cd873e9c8eac21730082","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":379,"isUserFollowing":false},"createdAt":"2026-09-03T01:29:43.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"This is an automated message from the [Librarian Bot](https://huggingface.co/librarian-bots). I found the following papers similar to this paper. \n\nThe following papers were recommended by the Semantic Scholar API \n\n* [ACTD: Anchor-Based Cross-Tokenizer Distillation with Residual Regularization](https://huggingface.co/papers/2608.29662) (2026)\n* [Rethinking Reverse KL as Adaptive Entropy Distillation](https://huggingface.co/papers/2608.14685) (2026)\n* [On-Policy Self-Distillation without Any Supervision](https://huggingface.co/papers/2608.06296) (2026)\n* [Distilled Reinforcement Learning for LLM Post-training](https://huggingface.co/papers/2607.17247) (2026)\n* [Adaptive Supervised Anchoring for On-Policy Self-Distillation](https://huggingface.co/papers/2608.07935) (2026)\n* [Trace-Based On-Policy Distillation for Masked Diffusion Language Models](https://huggingface.co/papers/2607.16872) (2026)\n* [Post-Training Shifts Confidence: A Three-Stage Analysis of How SFT, RL, and OPD Shape CoT Calibration](https://huggingface.co/papers/2607.13753) (2026)\n\n\n Please give a thumbs up to this comment if you found it helpful!\n\n If you want recommendations for any Paper on Hugging Face checkout [this](https://huggingface.co/spaces/librarian-bots/recommend_similar_papers) Space\n\n You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: `@librarian-bot recommend`","html":"<p>This is an automated message from the <a href=\"https://huggingface.co/librarian-bots\">Librarian Bot</a>. I found the following papers similar to this paper. </p>\n<p>The following papers were recommended by the Semantic Scholar API </p>\n<ul>\n<li><a href=\"https://huggingface.co/papers/2608.29662\">ACTD: Anchor-Based Cross-Tokenizer Distillation with Residual Regularization</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2608.14685\">Rethinking Reverse KL as Adaptive Entropy Distillation</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2608.06296\">On-Policy Self-Distillation without Any Supervision</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2607.17247\">Distilled Reinforcement Learning for LLM Post-training</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2608.07935\">Adaptive Supervised Anchoring for On-Policy Self-Distillation</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2607.16872\">Trace-Based On-Policy Distillation for Masked Diffusion Language Models</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2607.13753\">Post-Training Shifts Confidence: A Three-Stage Analysis of How SFT, RL, and OPD Shape CoT Calibration</a> (2026)</li>\n</ul>\n<p> Please give a thumbs up to this comment if you found it helpful!</p>\n<p> If you want recommendations for any Paper on Hugging Face checkout <a href=\"https://huggingface.co/spaces/librarian-bots/recommend_similar_papers\">this</a> Space</p>\n<p> You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: <code>@librarian-bot recommend</code></p>\n","updatedAt":"2026-09-03T01:29:43.798Z","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":379,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.7625230550765991},"editors":["librarian-bot"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2609.01532","authors":[{"_id":"6a9889e7fea818274321fda9","user":{"_id":"627c52e51b02d7b5bf671dca","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1666240522285-627c52e51b02d7b5bf671dca.png","isPro":false,"fullname":"Jacqueline He","user":"jacquelinehe","type":"user","name":"jacquelinehe"},"name":"Jacqueline He","status":"claimed_verified","statusLastChangedAt":"2026-09-03T00:45:04.491Z","hidden":false},{"_id":"6a9889e7fea818274321fdaa","name":"Howard Yen","hidden":false},{"_id":"6a9889e7fea818274321fdab","name":"Shuyue Stella Li","hidden":false},{"_id":"6a9889e7fea818274321fdac","name":"Margaret Li","hidden":false},{"_id":"6a9889e7fea818274321fdad","name":"Hanqing Zeng","hidden":false},{"_id":"6a9889e7fea818274321fdae","name":"Yinglong Xia","hidden":false},{"_id":"6a9889e7fea818274321fdaf","name":"Benyu Zhang","hidden":false},{"_id":"6a9889e7fea818274321fdb0","name":"Zhuokai Zhao","hidden":false},{"_id":"6a9889e7fea818274321fdb1","name":"Qiang Zhang","hidden":false},{"_id":"6a9889e7fea818274321fdb2","name":"Pang Wei Koh","hidden":false},{"_id":"6a9889e7fea818274321fdb3","name":"Luke Zettlemoyer","hidden":false},{"_id":"6a9889e7fea818274321fdb4","name":"Wen-tau Yih","hidden":false}],"publishedAt":"2026-09-01T00:00:00.000Z","submittedOnDailyAt":"2026-09-02T00:00:00.000Z","title":"Knowledge Distillation During Mid-Training Favors Reasoning over Factual Recall","submittedOnDailyBy":{"_id":"627c52e51b02d7b5bf671dca","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1666240522285-627c52e51b02d7b5bf671dca.png","isPro":false,"fullname":"Jacqueline He","user":"jacquelinehe","type":"user","name":"jacquelinehe"},"summary":"Logit-based knowledge distillation (KD) is used to train smaller language models (LMs) via supervision from stronger teachers, but whether its benefits are consistent across training stages remains unclear. Through controlled experiments, we find that forward Kullback-Leibler (KL) distillation--the standard KD formulation--with post-trained teachers behaves fundamentally differently during mid-training, an intermediate phase of self-supervised learning on curated corpora. Surprisingly, while forward KD simultaneously improves reasoning and factual recall during pre-training relative to standard next-token prediction (NTP), it instead slows factual recall acquisition during mid-training despite continued reasoning gains. We trace this stage dependence to an asymmetry in teacher confidence across data domains and the student's evolving knowledge state: teachers are more confident on procedural than knowledge-intensive data, while students acquire low-entropy factual knowledge earlier in training. To mitigate this imbalance, we propose Switch Distillation, a simple mid-training objective that distills on tokens where the teacher is confident, using teacher predictive entropy as a lightweight routing signal, and otherwise falls back to cross-entropy. Switch Distillation consistently outperforms existing distillation objectives across teacher sizes. Relative to standard NTP, it achieves 1.61-1.71x the reasoning performance and 1.13-1.19x the knowledge and commonsense performance while preserving 96.7-96.8% of factual recall. Crucially, these benefits persist after post-training: Switch Distillation closes the factual recall gap while maintaining 1.25-1.32x and 1.13-1.20x gains in reasoning and knowledge and commonsense, respectively.","upvotes":1,"discussionId":"6a9889e7fea818274321fdb5","githubRepo":"https://github.com/facebookresearch/midtraining-distillation","githubRepoAddedBy":"user","ai_summary":"Switch Distillation selectively applies logit-based knowledge distillation during mid-training based on teacher confidence to improve reasoning and preserve factual recall in smaller language models.","ai_keywords":["logit-based knowledge distillation","forward KL divergence","post-trained teachers","mid-training","self-supervised learning","next-token prediction","Switch Distillation","predictive entropy","cross-entropy","reasoning","factual recall"],"ai_summary_model":"thinkingmachines/Inkling-Small","organization":{"_id":"6a78cbce04f58391d2d7c93b","name":"meta","fullname":"Meta","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/5dd96eb166059660ed1ee413/Yq_gbM7i78tR3nfEvcwSA.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"63ac5701c21e60a3e9b58aa7","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/63ac5701c21e60a3e9b58aa7/g6EX7diOpuA94R2ab-rZC.png","isPro":true,"fullname":"Dipankar Sarkar","user":"dipankarsarkar","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"6a78cbce04f58391d2d7c93b","name":"meta","fullname":"Meta","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/5dd96eb166059660ed1ee413/Yq_gbM7i78tR3nfEvcwSA.png"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2609/2609.01532.md","query":{}}">

Get this paper in your agent:

Don't have the latest CLI?

Cite arxiv.org/abs/2609.01532 in a model README.md to link it from this page.

Cite arxiv.org/abs/2609.01532 in a dataset README.md to link it from this page.

Cite arxiv.org/abs/2609.01532 in a Space README.md to link it from this page.

Add this paper to a collection to link it from this page.