Dion3 is a full stack optimization of muon-like updates yielding up to an observed factor of 6 reduction in optimizer time and an extra centinat of performance with code available at <a href=\"https://github.com/microsoft/dion\" rel=\"nofollow\">https://github.com/microsoft/dion</a> .</p>\n<p>The time improvements include Gram Newton Schulz, Row Selection, Megabatching, Cudagraph Capture, and Symmetric Kernels.</p>\n<p>The statistical improvements include joint learning rate+fraction optimization, row normalization that works with row selection, and fused rounding.</p>\n<p>Enjoy :-)</p>\n","updatedAt":"2026-08-17T01:48:53.596Z","author":{"_id":"67165cb237e3cfacd9a5571d","avatarUrl":"/avatars/a7387a90d56f47af17f3bf6b6ac87ad7.svg","fullname":"John Langford","name":"JohnLangford","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8849174380302429},"editors":["JohnLangford"],"editorAvatarUrls":["/avatars/a7387a90d56f47af17f3bf6b6ac87ad7.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2608.11612","authors":[{"_id":"6a7fe472b601d59c65281047","name":"Noah Amsel","hidden":false},{"_id":"6a7fe472b601d59c65281048","name":"Jack Zhang","hidden":false},{"_id":"6a7fe472b601d59c65281049","name":"Kwangjun Ahn","hidden":false},{"_id":"6a7fe472b601d59c6528104a","name":"Ali Naeimi","hidden":false},{"_id":"6a7fe472b601d59c6528104b","name":"Austin Feng","hidden":false},{"_id":"6a7fe472b601d59c6528104c","name":"Berlin Chen","hidden":false},{"_id":"6a7fe472b601d59c6528104d","name":"Tri Dao","hidden":false},{"_id":"6a7fe472b601d59c6528104e","user":{"_id":"67165cb237e3cfacd9a5571d","avatarUrl":"/avatars/a7387a90d56f47af17f3bf6b6ac87ad7.svg","isPro":false,"fullname":"John Langford","user":"JohnLangford","type":"user","name":"JohnLangford"},"name":"John Langford","status":"claimed_verified","statusLastChangedAt":"2026-08-17T00:45:04.636Z","hidden":false}],"mediaUrls":["https://cdn-uploads.huggingface.co/production/uploads/67165cb237e3cfacd9a5571d/CVY6X-DgkNXrF9vVlOefU.png"],"publishedAt":"2026-08-12T00:00:00.000Z","submittedOnDailyAt":"2026-08-17T00:00:00.000Z","title":"Dion3: Full-Stack Orthogonal Updates","submittedOnDailyBy":{"_id":"67165cb237e3cfacd9a5571d","avatarUrl":"/avatars/a7387a90d56f47af17f3bf6b6ac87ad7.svg","isPro":false,"fullname":"John Langford","user":"JohnLangford","type":"user","name":"JohnLangford"},"summary":"The Muon optimizer incurs a significant overhead cost due to its cubic-time Newton-Schulz orthogonalization step. When weights are sharded, communication overhead compounds this computational cost, eroding the benefits of Muon in many settings. We present Dion3, a revision of Muon that targets this overhead at every level of the stack. Our Gram Newton-Schulz algorithm reduces the FLOP cost of orthogonalization, our CuteDSL kernels accelerate it by exploiting symmetry, and our megabatching strategy reduces communication overhead. Moreover, we propose a simple change to the update rule that cuts costs even further: selecting only a fraction of the momentum matrix's rows to orthogonalize at each step. This update rule improves on Dion (another \"compressed\" version of Muon), in both speed and performance. Overall, Dion3 matches or improves on the loss achieved by Muon but reduces optimizer step time by up to 6x. Dion3 is available via the dion package (https://github.com/microsoft/dion) as a drop-in replacement for Muon.","upvotes":2,"discussionId":"6a7fe473b601d59c6528104f","projectPage":"https://github.com/microsoft/dion","ai_summary":"Dion3 accelerates the Muon optimizer by reducing orthogonalization and communication overhead through algorithmic, kernel-level, and update-rule improvements.","ai_keywords":["Muon optimizer","Newton-Schulz orthogonalization","Gram Newton-Schulz","CuteDSL kernels","megabatching","momentum matrix","Dion3"],"ai_summary_model":"thinkingmachines/Inkling-Small","organization":{"_id":"68151d0f51add3813f3f7d1b","name":"MicrosoftResearch","fullname":"Microsoft Research","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/6529a4f2f1205983224fa513/PeuVr7jSuJflmDBBGxoDX.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"63ac5701c21e60a3e9b58aa7","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/63ac5701c21e60a3e9b58aa7/g6EX7diOpuA94R2ab-rZC.png","isPro":true,"fullname":"Dipankar Sarkar","user":"dipankarsarkar","type":"user"},{"_id":"671257538b7afb772404caa6","avatarUrl":"/avatars/c09bbf34f21e7080e0bd4c3f70f772c6.svg","isPro":false,"fullname":"Ali NT","user":"AliNT99","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"68151d0f51add3813f3f7d1b","name":"MicrosoftResearch","fullname":"Microsoft Research","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/6529a4f2f1205983224fa513/PeuVr7jSuJflmDBBGxoDX.png"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2608/2608.11612.md","query":{}}">

Get this paper in your agent:

Don't have the latest CLI?

Cite arxiv.org/abs/2608.11612 in a model README.md to link it from this page.

Cite arxiv.org/abs/2608.11612 in a dataset README.md to link it from this page.

Cite arxiv.org/abs/2608.11612 in a Space README.md to link it from this page.

Add this paper to a collection to link it from this page.