SGD collapses deep neural networks toward sparse, low-rank representations generated by architectural symmetry. We show how this collapse progresses over time, and that the mechanism extends to Adam and AdamW.</p>\n","updatedAt":"2026-09-04T06:22:02.305Z","author":{"_id":"6335696e6ee22307763b091f","avatarUrl":"/avatars/935c4a7c58d09f43a86728ca7cc1f621.svg","fullname":"Sai Niranjan","name":"rsn243","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8872398138046265},"editors":["rsn243"],"editorAvatarUrls":["/avatars/935c4a7c58d09f43a86728ca7cc1f621.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2609.02373","authors":[{"_id":"6a98f883fea818274321ff7c","name":"Sai Niranjan Ramachandran","hidden":false},{"_id":"6a98f883fea818274321ff7d","name":"Suvrit Sra","hidden":false}],"publishedAt":"2026-09-02T00:00:00.000Z","submittedOnDailyAt":"2026-09-04T00:00:00.000Z","title":"Percolation Dynamics in Optimization : Variance Cascades and Discrete Scale Invariance","submittedOnDailyBy":{"_id":"6335696e6ee22307763b091f","avatarUrl":"/avatars/935c4a7c58d09f43a86728ca7cc1f621.svg","isPro":false,"fullname":"Sai Niranjan","user":"rsn243","type":"user","name":"rsn243"},"summary":"We study the dynamics of Stochastic Gradient Descent (SGD), which is known to steer deep neural networks toward invariant sets that correspond to simpler subnetworks. How this steering unfolds over time remains poorly understood. We answer this by modeling the stochastic gradient flow (SGF) as a percolation process, in which architectural symmetries force subnetworks to merge in discrete simultaneous blocks rather than one at a time. These structural transitions register as variance spikes in a macroscopic order parameter, echoing physical phase transitions. We further show this trapping mechanism and its associated scaling cascade extend to Adam and AdamW under an explicit heavy-tailed noise model.","upvotes":0,"discussionId":"6a98f883fea818274321ff7e","ai_summary":"Stochastic gradient descent dynamics are modeled as a percolation process where architectural symmetries cause subnetworks to merge in discrete blocks, producing variance spikes resembling phase transitions, with similar trapping behavior extending to Adam and AdamW under heavy-tailed noise.","ai_keywords":["Stochastic Gradient Descent","stochastic gradient flow","percolation process","architectural symmetries","subnetworks","variance spikes","order parameter","phase transitions","trapping mechanism","scaling cascade","Adam","AdamW","heavy-tailed noise"],"ai_summary_model":"thinkingmachines/Inkling-Small"},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[],"acceptLanguages":["en"],"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2609/2609.02373.md","query":{}}">

Get this paper in your agent:

Don't have the latest CLI?

Cite arxiv.org/abs/2609.02373 in a model README.md to link it from this page.

Cite arxiv.org/abs/2609.02373 in a dataset README.md to link it from this page.

Cite arxiv.org/abs/2609.02373 in a Space README.md to link it from this page.

Add this paper to a collection to link it from this page.