Looped Transformers increase effective depth by iterating a shared block of layers, but most evaluations compare at fixed model size, conflating architectural advantage with extra FLOPs. We study looping on Mixture-of-Experts Transformers while closely matching per-token FLOPs, total non-embedding parameters, and KV cache. Through a series of ablations, we arrive at a recipe we call SMELT (Sparse MoE Transformer, middle layers Loop Twice), which loops the middle half of layers twice while matching the unlooped Baseline on all three budgets. We scale SMELT across four sizes up to 54B non-embedding parameters and fit a separate Chinchilla-style scaling law for each architecture. SMELT's loss drops faster with compute, saving 6.8--18.0% of training FLOPs on the compute-optimal frontier. The advantage transfers to downstream benchmarks beyond what validation loss predicts, is largest on Code, and grows with sample length and the number of in-context examples. Mechanistic analysis shows that the second visit reduces the attention sink and redirects mass toward content-relevant tokens, an inductive bias that may underlie the observed performance gains. These results show that looping can improve Transformers even under budget matching, offering a practical recipe that turns depth reuse into measurable gains.</p>\n","updatedAt":"2026-09-02T03:03:54.441Z","author":{"_id":"65377c30e48353201e6fdda0","avatarUrl":"/avatars/a8f803b6f2e598eaee9c52c0d2ddfc16.svg","fullname":"Jiaheng Liu","name":"CheeryLJH","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":29,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8530933856964111},"editors":["CheeryLJH"],"editorAvatarUrls":["/avatars/a8f803b6f2e598eaee9c52c0d2ddfc16.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2609.01343","authors":[{"_id":"6a977b2bfe3c2f89286c38a6","name":"Shaowen Wang","hidden":false},{"_id":"6a977b2bfe3c2f89286c38a7","name":"Ge Zhang","hidden":false},{"_id":"6a977b2bfe3c2f89286c38a8","name":"Kairong Luo","hidden":false},{"_id":"6a977b2bfe3c2f89286c38a9","name":"Yuhao Wu","hidden":false},{"_id":"6a977b2bfe3c2f89286c38aa","name":"Shaofan Liu","hidden":false},{"_id":"6a977b2bfe3c2f89286c38ab","name":"Jiaheng Liu","hidden":false},{"_id":"6a977b2bfe3c2f89286c38ac","name":"Wenhao Huang","hidden":false},{"_id":"6a977b2bfe3c2f89286c38ad","name":"Shen Yan","hidden":false},{"_id":"6a977b2bfe3c2f89286c38ae","name":"Jian Li","hidden":false}],"publishedAt":"2026-09-01T00:00:00.000Z","submittedOnDailyAt":"2026-09-02T00:00:00.000Z","title":"SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers","submittedOnDailyBy":{"_id":"65377c30e48353201e6fdda0","avatarUrl":"/avatars/a8f803b6f2e598eaee9c52c0d2ddfc16.svg","isPro":false,"fullname":"Jiaheng Liu","user":"CheeryLJH","type":"user","name":"CheeryLJH"},"summary":"Looped Transformers increase effective depth by iterating a shared block of layers, but most evaluations compare at fixed model size, conflating architectural advantage with extra FLOPs. We study looping on Mixture-of-Experts Transformers while closely matching per-token FLOPs, total non-embedding parameters, and KV cache. Through a series of ablations, we arrive at a recipe we call SMELT (Sparse MoE Transformer, middle layers Loop Twice), which loops the middle half of layers twice while matching the unlooped Baseline on all three budgets. We scale SMELT across four sizes up to 54B non-embedding parameters and fit a separate Chinchilla-style scaling law for each architecture. SMELT's loss drops faster with compute, saving 6.8--18.0\\% of training FLOPs on the compute-optimal frontier. The advantage transfers to downstream benchmarks beyond what validation loss predicts, is largest on Code, and grows with sample length and the number of in-context examples. Mechanistic analysis shows that the second visit reduces the attention sink and redirects mass toward content-relevant tokens, an inductive bias that may underlie the observed performance gains. These results show that looping can improve Transformers even under budget matching, offering a practical recipe that turns depth reuse into measurable gains.","upvotes":53,"discussionId":"6a977b2bfe3c2f89286c38af","ai_summary":"Looping middle layers in sparse Mixture-of-Experts Transformers improves training efficiency and downstream performance while matching per-token FLOPs, parameters, and cache budgets.","ai_keywords":["Mixture-of-Experts","sparse MoE","looped Transformers","SMELT","attention sink","scaling laws","Chinchilla","KV cache","cross-attention"],"ai_summary_model":"thinkingmachines/Inkling-Small","organization":{"_id":"67d1140985ea0644e2f14b99","name":"ByteDance-Seed","fullname":"ByteDance Seed","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/6535c9e88bde2fae19b6fb25/flkDUqd_YEuFsjeNET3r-.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"60ffcb37eb91118ab563670a","avatarUrl":"/avatars/547875810aa7766b9ca39ac0f08ebdc6.svg","isPro":false,"fullname":"Shaowen Wang","user":"Outsider565","type":"user"},{"_id":"604714a0c82d59b7347b55ae","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/604714a0c82d59b7347b55ae/WZFKDDUPi8JS0BK8t7mIv.jpeg","isPro":true,"fullname":"YangWang92","user":"yangwang92","type":"user"},{"_id":"644a28463b274e131a0de314","avatarUrl":"/avatars/980eafafbd1373fac143d76aa931f1e2.svg","isPro":false,"fullname":"zhanshijin","user":"gqszhanshijin","type":"user"},{"_id":"65556c0ee0169cf32ce13ce6","avatarUrl":"/avatars/c3648910cf53875ce953f23be59323b2.svg","isPro":true,"fullname":"Maggie Huan","user":"Ibisbill","type":"user"},{"_id":"66a06670f89441587fd0792b","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/2OF0z7O_vSGnFt6XDuFNu.png","isPro":false,"fullname":"Jane Jiang","user":"Jane532","type":"user"},{"_id":"65d2251f98b4a470bf6a26e3","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/65d2251f98b4a470bf6a26e3/C4T0LHYGejrI9mu_k3M8p.jpeg","isPro":false,"fullname":"xts","user":"xtsssss","type":"user"},{"_id":"6377366acc034ef804cf0aef","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6377366acc034ef804cf0aef/JHsET8LJE2w1E7vJD-R2o.png","isPro":false,"fullname":"ShawnYing","user":"ShawnYing","type":"user"},{"_id":"65377c30e48353201e6fdda0","avatarUrl":"/avatars/a8f803b6f2e598eaee9c52c0d2ddfc16.svg","isPro":false,"fullname":"Jiaheng Liu","user":"CheeryLJH","type":"user"},{"_id":"64b22e6b0a54158d66f18688","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/64b22e6b0a54158d66f18688/Cbl3oMMMANbnCMoSUYenI.png","isPro":true,"fullname":"Benhao Huang","user":"HuskyDoge","type":"user"},{"_id":"68f5d9dc2dabe9aec07a1664","avatarUrl":"/avatars/eb9051526a79603d27f5530b6facc292.svg","isPro":false,"fullname":"Yihao Xiao","user":"ririririri2","type":"user"},{"_id":"63ac5701c21e60a3e9b58aa7","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/63ac5701c21e60a3e9b58aa7/g6EX7diOpuA94R2ab-rZC.png","isPro":true,"fullname":"Dipankar Sarkar","user":"dipankarsarkar","type":"user"},{"_id":"66ea643899af9ac3463639b1","avatarUrl":"/avatars/252d470e761a57834dee3dbc60dfefed.svg","isPro":false,"fullname":"Disen Lan","user":"landisen","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"67d1140985ea0644e2f14b99","name":"ByteDance-Seed","fullname":"ByteDance Seed","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/6535c9e88bde2fae19b6fb25/flkDUqd_YEuFsjeNET3r-.png"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2609/2609.01343.md","query":{}}">
SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers
Abstract
Looping middle layers in sparse Mixture-of-Experts Transformers improves training efficiency and downstream performance while matching per-token FLOPs, parameters, and cache budgets.
Looped Transformers increase effective depth by iterating a shared block of layers, but most evaluations compare at fixed model size, conflating architectural advantage with extra FLOPs. We study looping on Mixture-of-Experts Transformers while closely matching per-token FLOPs, total non-embedding parameters, and KV cache. Through a series of ablations, we arrive at a recipe we call SMELT (Sparse MoE Transformer, middle layers Loop Twice), which loops the middle half of layers twice while matching the unlooped Baseline on all three budgets. We scale SMELT across four sizes up to 54B non-embedding parameters and fit a separate Chinchilla-style scaling law for each architecture. SMELT's loss drops faster with compute, saving 6.8--18.0\% of training FLOPs on the compute-optimal frontier. The advantage transfers to downstream benchmarks beyond what validation loss predicts, is largest on Code, and grows with sample length and the number of in-context examples. Mechanistic analysis shows that the second visit reduces the attention sink and redirects mass toward content-relevant tokens, an inductive bias that may underlie the observed performance gains. These results show that looping can improve Transformers even under budget matching, offering a practical recipe that turns depth reuse into measurable gains.
Community
Looped Transformers increase effective depth by iterating a shared block of layers, but most evaluations compare at fixed model size, conflating architectural advantage with extra FLOPs. We study looping on Mixture-of-Experts Transformers while closely matching per-token FLOPs, total non-embedding parameters, and KV cache. Through a series of ablations, we arrive at a recipe we call SMELT (Sparse MoE Transformer, middle layers Loop Twice), which loops the middle half of layers twice while matching the unlooped Baseline on all three budgets. We scale SMELT across four sizes up to 54B non-embedding parameters and fit a separate Chinchilla-style scaling law for each architecture. SMELT's loss drops faster with compute, saving 6.8--18.0% of training FLOPs on the compute-optimal frontier. The advantage transfers to downstream benchmarks beyond what validation loss predicts, is largest on Code, and grows with sample length and the number of in-context examples. Mechanistic analysis shows that the second visit reduces the attention sink and redirects mass toward content-relevant tokens, an inductive bias that may underlie the observed performance gains. These results show that looping can improve Transformers even under budget matching, offering a practical recipe that turns depth reuse into measurable gains.
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2609.01343 in a model README.md to link it from this page.
Cite arxiv.org/abs/2609.01343 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2609.01343 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.