Prismix
Hugging Face Daily Papers · · 6 min read

Don't Drop Dropout: Optimizing Layer Sparsity for Efficient LLM Training and Inference

Mirrored from Hugging Face Daily Papers for archival readability. Support the source by reading on the original site.

Like Read original ↗
\n\t<a id=\"paper-summary\" class=\"block pr-1.5 text-lg md:absolute md:p-1.5 md:opacity-0 md:group-hover:opacity-100 md:right-full\" href=\"#paper-summary\" rel=\"nofollow\">\n\t\t<span class=\"header-link\"><svg class=\"text-gray-500 hover:text-black dark:hover:text-gray-200 w-4\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\" aria-hidden=\"true\" role=\"img\" width=\"1em\" height=\"1em\" preserveAspectRatio=\"xMidYMid meet\" viewBox=\"0 0 256 256\"><path d=\"M167.594 88.393a8.001 8.001 0 0 1 0 11.314l-67.882 67.882a8 8 0 1 1-11.314-11.315l67.882-67.881a8.003 8.003 0 0 1 11.314 0zm-28.287 84.86l-28.284 28.284a40 40 0 0 1-56.567-56.567l28.284-28.284a8 8 0 0 0-11.315-11.315l-28.284 28.284a56 56 0 0 0 79.196 79.197l28.285-28.285a8 8 0 1 0-11.315-11.314zM212.852 43.14a56.002 56.002 0 0 0-79.196 0l-28.284 28.284a8 8 0 1 0 11.314 11.314l28.284-28.284a40 40 0 0 1 56.568 56.567l-28.285 28.285a8 8 0 0 0 11.315 11.314l28.284-28.284a56.065 56.065 0 0 0 0-79.196z\" fill=\"currentColor\"></path></svg></span>\n\t</a>\n\t<span>\n\t\tPaper Summary\n\t</span>\n</h2>\n<p><a href=\"https://cdn-uploads.huggingface.co/production/uploads/63c9725ebedad7e2bf160bdc/505nl6-ydOmjns58X58vj.png\" rel=\"nofollow\"><img src=\"https://cdn-uploads.huggingface.co/production/uploads/63c9725ebedad7e2bf160bdc/505nl6-ydOmjns58X58vj.png\" alt=\"image\"></a></p>\n<ul>\n<li>Layer dropout (stochastic depth) has largely vanished from LLM pretraining recipes due to reported accuracy degradation — this paper shows those degradations came from suboptimal configurations, not a fundamental limitation.</li>\n<li>With the right recipe, layer dropout gives <strong>lower validation loss at equal training FLOPs</strong>, or up to 25% FLOP savings for the same validation loss.</li>\n<li>Four key ingredients, validated across 2,400+ runs from 271M–8.2B params and up to 160B tokens:<ul>\n<li><strong>Scaling factor</strong> 1/(1−p) for hyperparameter transfer across dropout rates (no retuning needed)</li>\n<li><strong>Granularity</strong>: drop whole transformer blocks, per-sequence (not per-batch, not attention/FFN separately)</li>\n<li><strong>Increasing Layer Distribution (ILD)</strong>: protect early layers, drop later layers more aggressively — advantage grows with scale</li>\n<li><strong>Decreasing Time Schedule (DTS)</strong>: start at max dropout, decay to zero over training</li>\n</ul>\n</li>\n</ul>\n<p><a href=\"https://cdn-uploads.huggingface.co/production/uploads/63c9725ebedad7e2bf160bdc/RK79u42SXPceeksvgXq8g.png\" rel=\"nofollow\"><img src=\"https://cdn-uploads.huggingface.co/production/uploads/63c9725ebedad7e2bf160bdc/RK79u42SXPceeksvgXq8g.png\" alt=\"findings_waterfall\"></a></p>\n<ul>\n<li>Beyond training speed, layer dropout enables \"free\" post-training depth elasticity: early exit, intermediate-layer skipping, and self-speculative decoding — <strong>up to 1.5x inference speedup</strong> with negligible accuracy loss.</li>\n<li>Bigger models tolerate higher max dropout rates (up to 0.99 on the 8.2B model), so the benefits compound with scale.</li>\n<li>All experiments run on Cerebras CS-3 systems.</li>\n</ul>\n","updatedAt":"2026-09-07T06:21:01.783Z","author":{"_id":"63c9725ebedad7e2bf160bdc","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/63c9725ebedad7e2bf160bdc/wzPuyhOXCYBNGwZDshbnL.jpeg","fullname":"Mostafa Elhoushi","name":"melhoushi","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":40,"isUserFollowing":false}},"numEdits":7,"identifiedLanguage":{"language":"en","probability":0.8628265857696533},"editors":["melhoushi"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/63c9725ebedad7e2bf160bdc/wzPuyhOXCYBNGwZDshbnL.jpeg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2609.05275","authors":[{"_id":"6a9e1ad3de5ea82090db6550","name":"Mostafa Elhoushi","hidden":false},{"_id":"6a9e1ad3de5ea82090db6551","name":"Alex Pretko","hidden":false},{"_id":"6a9e1ad3de5ea82090db6552","name":"Nolan Dey","hidden":false},{"_id":"6a9e1ad3de5ea82090db6553","name":"Bin Claire Zhang","hidden":false},{"_id":"6a9e1ad3de5ea82090db6554","name":"Gavia Gray","hidden":false},{"_id":"6a9e1ad3de5ea82090db6555","name":"Gurpreet Gosal","hidden":false},{"_id":"6a9e1ad3de5ea82090db6556","name":"Abdulrahman Mahmoud","hidden":false},{"_id":"6a9e1ad3de5ea82090db6557","name":"Shane Bergsma","hidden":false},{"_id":"6a9e1ad3de5ea82090db6558","name":"Joel Hestness","hidden":false}],"publishedAt":"2026-09-04T00:00:00.000Z","submittedOnDailyAt":"2026-09-07T00:00:00.000Z","title":"Don't Drop Dropout: Optimizing Layer Sparsity for Efficient LLM Training and Inference","submittedOnDailyBy":{"_id":"6039478ab3ecf716b1a5fd4d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6039478ab3ecf716b1a5fd4d/_Thy4E7taiSYBLKxEKJbT.jpeg","isPro":true,"fullname":"taesiri","user":"taesiri","type":"user","name":"taesiri"},"summary":"Layer dropout (a.k.a. stochastic depth) has been shown to enable faster training, higher accuracy, and robustness to zero-shot layer pruning in both language and vision transformers. However, as models and datasets have scaled, dropout - particularly layer dropout - has largely disappeared from large language models (LLMs) pre-training recipes. While some prior work has reported that dropout can degrade accuracy, no comprehensive study has quantified, let alone mitigated, this effect. In this study, we show that layer dropout should be used in state-of-the-art LLM training, establishing best practices and scaling analysis for both training and post-training benefits. Concretely, with optimal layer distribution, time schedule, and optimizer hyperparameters, we observe that at the same training FLOPs layer dropout leads to lower loss. For a given number of training steps, LLMs can achieve lower or similar validation loss while saving upto 25% of training FLOPs. Moreover, layer dropout enables significant post-training optimizations, such as early exit, intermediate-layer skipping, and self-speculative decoding, yielding up to 1.5x inference speedup with negligible accuracy loss. Across more than 2400 training experiments, spanning models from 271M to 8.2B parameters and datasets up to 160B tokens, we demonstrate that these findings extend reliably to large-scale training regimes. All pre-training experiments were run on Cerebras CS-3 systems.","upvotes":6,"discussionId":"6a9e1ad4de5ea82090db6559","ai_summary":"Layer dropout improves large language model training efficiency and enables faster inference via early exit and speculative decoding without sacrificing accuracy.","ai_keywords":["layer dropout","stochastic depth","transformers","LLM pre-training","early exit","intermediate-layer skipping","self-speculative decoding"],"ai_summary_model":"thinkingmachines/Inkling-Small","organization":{"_id":"640fbc14208821a59b767b3e","name":"cerebras","fullname":"Cerebras","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/1678752782877-640fbb91bd54b1efde3678b2.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"63ac5701c21e60a3e9b58aa7","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/63ac5701c21e60a3e9b58aa7/g6EX7diOpuA94R2ab-rZC.png","isPro":true,"fullname":"Dipankar Sarkar","user":"dipankarsarkar","type":"user"},{"_id":"63c9725ebedad7e2bf160bdc","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/63c9725ebedad7e2bf160bdc/wzPuyhOXCYBNGwZDshbnL.jpeg","isPro":false,"fullname":"Mostafa Elhoushi","user":"melhoushi","type":"user"},{"_id":"6638d8ca315dd25cf7804943","avatarUrl":"/avatars/f2cf78a1bdeac8c43ad8af65e2c338d5.svg","isPro":false,"fullname":"Amr Al-Hamad","user":"amralhamad","type":"user"},{"_id":"6a9e4474aee10999a11259d2","avatarUrl":"/avatars/854d99c9c1ebc02242175aa453090124.svg","isPro":false,"fullname":"Sameer Zaheer","user":"smrzhr","type":"user"},{"_id":"695dafe425a700d784b10198","avatarUrl":"/avatars/97d3097a689e4e447e49f228d4722cd6.svg","isPro":false,"fullname":"Ahmad Hafez","user":"aihafez","type":"user"},{"_id":"68b2a4157f881fc640ba7d80","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/lMTgr3pe7pOHtMe7bVF7F.png","isPro":false,"fullname":"khtsly","user":"khtsly","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"640fbc14208821a59b767b3e","name":"cerebras","fullname":"Cerebras","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/1678752782877-640fbb91bd54b1efde3678b2.png"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2609/2609.05275.md","query":{}}">
Papers
arxiv:2609.05275

Don't Drop Dropout: Optimizing Layer Sparsity for Efficient LLM Training and Inference

Published on Sep 4
· Submitted by
taesiri on Sep 7
Authors:
,

Abstract

Layer dropout improves large language model training efficiency and enables faster inference via early exit and speculative decoding without sacrificing accuracy.

Layer dropout (a.k.a. stochastic depth) has been shown to enable faster training, higher accuracy, and robustness to zero-shot layer pruning in both language and vision transformers. However, as models and datasets have scaled, dropout - particularly layer dropout - has largely disappeared from large language models (LLMs) pre-training recipes. While some prior work has reported that dropout can degrade accuracy, no comprehensive study has quantified, let alone mitigated, this effect. In this study, we show that layer dropout should be used in state-of-the-art LLM training, establishing best practices and scaling analysis for both training and post-training benefits. Concretely, with optimal layer distribution, time schedule, and optimizer hyperparameters, we observe that at the same training FLOPs layer dropout leads to lower loss. For a given number of training steps, LLMs can achieve lower or similar validation loss while saving upto 25% of training FLOPs. Moreover, layer dropout enables significant post-training optimizations, such as early exit, intermediate-layer skipping, and self-speculative decoding, yielding up to 1.5x inference speedup with negligible accuracy loss. Across more than 2400 training experiments, spanning models from 271M to 8.2B parameters and datasets up to 160B tokens, we demonstrate that these findings extend reliably to large-scale training regimes. All pre-training experiments were run on Cerebras CS-3 systems.

Community

Paper Summary

image

  • Layer dropout (stochastic depth) has largely vanished from LLM pretraining recipes due to reported accuracy degradation — this paper shows those degradations came from suboptimal configurations, not a fundamental limitation.
  • With the right recipe, layer dropout gives lower validation loss at equal training FLOPs, or up to 25% FLOP savings for the same validation loss.
  • Four key ingredients, validated across 2,400+ runs from 271M–8.2B params and up to 160B tokens:
    • Scaling factor 1/(1−p) for hyperparameter transfer across dropout rates (no retuning needed)
    • Granularity: drop whole transformer blocks, per-sequence (not per-batch, not attention/FFN separately)
    • Increasing Layer Distribution (ILD): protect early layers, drop later layers more aggressively — advantage grows with scale
    • Decreasing Time Schedule (DTS): start at max dropout, decay to zero over training

findings_waterfall

  • Beyond training speed, layer dropout enables "free" post-training depth elasticity: early exit, intermediate-layer skipping, and self-speculative decoding — up to 1.5x inference speedup with negligible accuracy loss.
  • Bigger models tolerate higher max dropout rates (up to 0.99 on the 8.2B model), so the benefits compound with scale.
  • All experiments run on Cerebras CS-3 systems.
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images

· Sign up or log in to comment

Get this paper in your agent:

hf papers read 2609.05275
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper

No model linking this paper

Cite arxiv.org/abs/2609.05275 in a model README.md to link it from this page.

Datasets citing this paper

No dataset linking this paper

Cite arxiv.org/abs/2609.05275 in a dataset README.md to link it from this page.

Spaces citing this paper

No Space linking this paper

Cite arxiv.org/abs/2609.05275 in a Space README.md to link it from this page.

Collections including this paper

No Collection including this paper

Add this paper to a collection to link it from this page.

Discussion (0)

Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.

Sign in →

No comments yet. Sign in and be the first to say something.