We present the first systematic study of Massive activations (MAs) in layer-interleaved HLA LLMs and uncover two architecture-aligned morphologies: MAs consistently spike immediately before full attention layers, forming pre-attention spikes (PAS), and can persist through intervening linear attention layers, giving rise to inter-spike plateaus (ISP). As full attention becomes denser, successive PAS become increasingly connected through ISP, ultimately recovering the stable MA morphology of full attention LLMs. We establish the recurrence of this organization across five linear attention architectures, six hybridization configurations, five data domains, and representative open-source hybrid models spanning 1.2B to 397B total parameters. Controlled pretraining of GDN-based hybrids at scales up to 1.3B shows that both morphologies emerge early and respond asymmetrically to output gating: full attention output gating strongly attenuates their absolute magnitudes without eliminating their layerwise organization, whereas removing GDN gates yields comparatively modest amplification. Mechanistically, our systematic-outlier analysis supports a shared lifecycle account governed by the timing of MA cancellation. PAS follows a localized write-sink-cancel process, while the extended persistence of ISP is consistent with delayed cancellation. At the full attention limit, this account recovers the stable MA morphology characteristic of full attention LLMs.</p>\n","updatedAt":"2026-08-14T02:48:22.312Z","author":{"_id":"655b4f5cc11dee7f7e882a0c","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/655b4f5cc11dee7f7e882a0c/4GaAC5Qt55eYlbNRJpVD5.png","fullname":"zunhaisu","name":"zunhai","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":1,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8942615389823914},"editors":["zunhai"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/655b4f5cc11dee7f7e882a0c/4GaAC5Qt55eYlbNRJpVD5.png"],"reactions":[],"isReport":false}},{"id":"6a7eef5b6734ead23d8ceacd","author":{"_id":"6a7eef1dff35ce2ac73d72e4","avatarUrl":"/avatars/0b3c26da519bfcda921c014d19c3fbcf.svg","fullname":"tran huyen","name":"tran073huyen","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false},"createdAt":"2026-08-14T10:35:07.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"ultimately recovering the stable MA morphology of full attention LLMs. We establish the recurrence of this organization across five linear attention architectures, six hybridization configurations, five data domains, and representative open-source hybrid models spanning 1.2B to 397B total parameters. Controlled pretraining of GDN-based hybrids at scales up to 1.3B shows that both morphologies emerge early and respond asymmetrically to output gating: full attention output gating strongly attenuates their absolute magnitudes without eliminating their layerwise organization, whereas removing GDN gates yields comparatively modest amplification.","html":"<p>ultimately recovering the stable MA morphology of full attention LLMs. We establish the recurrence of this organization across five linear attention architectures, six hybridization configurations, five data domains, and representative open-source hybrid models spanning 1.2B to 397B total parameters. Controlled pretraining of GDN-based hybrids at scales up to 1.3B shows that both morphologies emerge early and respond asymmetrically to output gating: full attention output gating strongly attenuates their absolute magnitudes without eliminating their layerwise organization, whereas removing GDN gates yields comparatively modest amplification.</p>\n","updatedAt":"2026-08-14T10:35:07.455Z","author":{"_id":"6a7eef1dff35ce2ac73d72e4","avatarUrl":"/avatars/0b3c26da519bfcda921c014d19c3fbcf.svg","fullname":"tran huyen","name":"tran073huyen","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8710896372795105},"editors":["tran073huyen"],"editorAvatarUrls":["/avatars/0b3c26da519bfcda921c014d19c3fbcf.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2608.12149","authors":[{"_id":"6a7e818a42823931a1f17665","user":{"_id":"655b4f5cc11dee7f7e882a0c","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/655b4f5cc11dee7f7e882a0c/4GaAC5Qt55eYlbNRJpVD5.png","isPro":false,"fullname":"zunhaisu","user":"zunhai","type":"user","name":"zunhai"},"name":"Zunhai Su","status":"claimed_verified","statusLastChangedAt":"2026-08-14T08:45:04.684Z","hidden":false},{"_id":"6a7e818a42823931a1f17666","name":"Bohan Sun","hidden":false},{"_id":"6a7e818a42823931a1f17667","name":"Xialie Zhuang","hidden":false},{"_id":"6a7e818a42823931a1f17668","name":"Shuibai Zhang","hidden":false},{"_id":"6a7e818a42823931a1f17669","name":"He Xiao","hidden":false},{"_id":"6a7e818a42823931a1f1766a","name":"Jing Xiong","hidden":false},{"_id":"6a7e818a42823931a1f1766b","name":"Hengyuan Zhang","hidden":false},{"_id":"6a7e818a42823931a1f1766c","name":"Zhongzhu Zhou","hidden":false},{"_id":"6a7e818a42823931a1f1766d","name":"Tiantian Zhang","hidden":false},{"_id":"6a7e818a42823931a1f1766e","name":"Ngai Wong","hidden":false},{"_id":"6a7e818a42823931a1f1766f","name":"Chuan-Wei Kuo","hidden":false}],"publishedAt":"2026-08-12T00:00:00.000Z","submittedOnDailyAt":"2026-08-14T00:00:00.000Z","title":"Massive Activations in Hybrid Linear Attention Large Language Models: Pre-Attention Spikes and Inter-Spike Plateaus","submittedOnDailyBy":{"_id":"655b4f5cc11dee7f7e882a0c","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/655b4f5cc11dee7f7e882a0c/4GaAC5Qt55eYlbNRJpVD5.png","isPro":false,"fullname":"zunhaisu","user":"zunhai","type":"user","name":"zunhai"},"summary":"We present the first systematic study of Massive activations (MAs) in layer-interleaved HLA LLMs and uncover two architecture-aligned morphologies: MAs consistently spike immediately before full attention layers, forming pre-attention spikes (PAS), and can persist through intervening linear attention layers, giving rise to inter-spike plateaus (ISP). As full attention becomes denser, successive PAS become increasingly connected through ISP, ultimately recovering the stable MA morphology of full attention LLMs. We establish the recurrence of this organization across five linear attention architectures, six hybridization configurations, five data domains, and representative open-source hybrid models spanning 1.2B to 397B total parameters. Controlled pretraining of GDN-based hybrids at scales up to 1.3B shows that both morphologies emerge early and respond asymmetrically to output gating: full attention output gating strongly attenuates their absolute magnitudes without eliminating their layerwise organization, whereas removing GDN gates yields comparatively modest amplification. Mechanistically, our systematic-outlier analysis supports a shared lifecycle account governed by the timing of MA cancellation. PAS follows a localized write-sink-cancel process, while the extended persistence of ISP is consistent with delayed cancellation. At the full attention limit, this account recovers the stable MA morphology characteristic of full attention LLMs. Our code is available at https://github.com/StartluxLabs/Massive-Activations-HLA.","upvotes":13,"discussionId":"6a7e818a42823931a1f17670","githubRepo":"https://github.com/StartluxLabs/Massive-Activations-HLA","githubRepoAddedBy":"user","ai_summary":"Massive activations in hybrid linear-attention LLMs exhibit pre-attention spikes and inter-spike plateaus governed by cancellation timing, with morphology recovering at full-attention limits.","ai_keywords":["massive activations","layer-interleaved HLA LLMs","pre-attention spikes","inter-spike plateaus","linear attention","full attention","GDN","output gating","systematic-outlier analysis"],"ai_summary_model":"thinkingmachines/Inkling-Small","githubStars":4,"organization":{"_id":"6a7c4cc758cb28f623454ea4","name":"startlux-models","fullname":"Startlux","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/64d33626465b6039259b9c72/ze5erAvHj6QzIS5hiLEZs.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"655b4f5cc11dee7f7e882a0c","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/655b4f5cc11dee7f7e882a0c/4GaAC5Qt55eYlbNRJpVD5.png","isPro":false,"fullname":"zunhaisu","user":"zunhai","type":"user"},{"_id":"64c9c79ccb2f1bf0e7ee194a","avatarUrl":"/avatars/67dc1c37f79c65716c571b2c25b3f08b.svg","isPro":false,"fullname":"Ri Liu","user":"danielliu99","type":"user"},{"_id":"6714a24f9c61dafbdaf619f4","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/WfDozR1nUVEvvDkPlluvu.png","isPro":false,"fullname":"Zane Zhao","user":"fishzzz","type":"user"},{"_id":"6a29618705fb99aae069caa1","avatarUrl":"/avatars/fd962e7898e6ddc0a12e9efeb8a10c98.svg","isPro":false,"fullname":"anping","user":"Ethan20260508","type":"user"},{"_id":"671c6ade11009e5ac0d5e217","avatarUrl":"/avatars/c7c837807afd3bf7b2350c2f9f205201.svg","isPro":false,"fullname":"He Xiao","user":"River555","type":"user"},{"_id":"611dbe252f524cfc42ba1830","avatarUrl":"/avatars/fd3de6e6746abac88e44b214f2769366.svg","isPro":false,"fullname":"piao ya","user":"piao","type":"user"},{"_id":"682d0afd6bb5a942313ae9c5","avatarUrl":"/avatars/35a0b3bbd25ff3044a3e9a4164ba5b1e.svg","isPro":false,"fullname":"Tiantian Zhang","user":"crystal-cheung","type":"user"},{"_id":"666ff0e9251af54f960e9870","avatarUrl":"/avatars/c6b727821e29f9d40afdf0ef6935cd0b.svg","isPro":false,"fullname":"Shuibai Zhang","user":"Shuibai12138","type":"user"},{"_id":"63870c8388b39a64e1e8cdfa","avatarUrl":"/avatars/1813b49eca6eb7396fa18cccc6e24342.svg","isPro":false,"fullname":"zhanghengyuan","user":"hengyuanya","type":"user"},{"_id":"67cee21329143cf9ea714e55","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/soEd3qhTb5vzrw-WKJC6D.jpeg","isPro":false,"fullname":"Bohan Sun","user":"Bohansss","type":"user"},{"_id":"6474e1afb68461d5cf7c41cc","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6474e1afb68461d5cf7c41cc/bcoiD_qPrjHUBlB259djg.png","isPro":false,"fullname":"Dawei Li","user":"wjldw","type":"user"},{"_id":"6575a625b951d40e7a4d8685","avatarUrl":"/avatars/2c78027f501975568cf06094fc20ee5d.svg","isPro":false,"fullname":"Xialie Zhuang","user":"Lie24","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"6a7c4cc758cb28f623454ea4","name":"startlux-models","fullname":"Startlux","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/64d33626465b6039259b9c72/ze5erAvHj6QzIS5hiLEZs.png"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2608/2608.12149.md","query":{}}">
Massive Activations in Hybrid Linear Attention Large Language Models: Pre-Attention Spikes and Inter-Spike Plateaus
Abstract
Massive activations in hybrid linear-attention LLMs exhibit pre-attention spikes and inter-spike plateaus governed by cancellation timing, with morphology recovering at full-attention limits.
We present the first systematic study of Massive activations (MAs) in layer-interleaved HLA LLMs and uncover two architecture-aligned morphologies: MAs consistently spike immediately before full attention layers, forming pre-attention spikes (PAS), and can persist through intervening linear attention layers, giving rise to inter-spike plateaus (ISP). As full attention becomes denser, successive PAS become increasingly connected through ISP, ultimately recovering the stable MA morphology of full attention LLMs. We establish the recurrence of this organization across five linear attention architectures, six hybridization configurations, five data domains, and representative open-source hybrid models spanning 1.2B to 397B total parameters. Controlled pretraining of GDN-based hybrids at scales up to 1.3B shows that both morphologies emerge early and respond asymmetrically to output gating: full attention output gating strongly attenuates their absolute magnitudes without eliminating their layerwise organization, whereas removing GDN gates yields comparatively modest amplification. Mechanistically, our systematic-outlier analysis supports a shared lifecycle account governed by the timing of MA cancellation. PAS follows a localized write-sink-cancel process, while the extended persistence of ISP is consistent with delayed cancellation. At the full attention limit, this account recovers the stable MA morphology characteristic of full attention LLMs. Our code is available at https://github.com/StartluxLabs/Massive-Activations-HLA.
Community
We present the first systematic study of Massive activations (MAs) in layer-interleaved HLA LLMs and uncover two architecture-aligned morphologies: MAs consistently spike immediately before full attention layers, forming pre-attention spikes (PAS), and can persist through intervening linear attention layers, giving rise to inter-spike plateaus (ISP). As full attention becomes denser, successive PAS become increasingly connected through ISP, ultimately recovering the stable MA morphology of full attention LLMs. We establish the recurrence of this organization across five linear attention architectures, six hybridization configurations, five data domains, and representative open-source hybrid models spanning 1.2B to 397B total parameters. Controlled pretraining of GDN-based hybrids at scales up to 1.3B shows that both morphologies emerge early and respond asymmetrically to output gating: full attention output gating strongly attenuates their absolute magnitudes without eliminating their layerwise organization, whereas removing GDN gates yields comparatively modest amplification. Mechanistically, our systematic-outlier analysis supports a shared lifecycle account governed by the timing of MA cancellation. PAS follows a localized write-sink-cancel process, while the extended persistence of ISP is consistent with delayed cancellation. At the full attention limit, this account recovers the stable MA morphology characteristic of full attention LLMs.
ultimately recovering the stable MA morphology of full attention LLMs. We establish the recurrence of this organization across five linear attention architectures, six hybridization configurations, five data domains, and representative open-source hybrid models spanning 1.2B to 397B total parameters. Controlled pretraining of GDN-based hybrids at scales up to 1.3B shows that both morphologies emerge early and respond asymmetrically to output gating: full attention output gating strongly attenuates their absolute magnitudes without eliminating their layerwise organization, whereas removing GDN gates yields comparatively modest amplification.
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2608.12149 in a model README.md to link it from this page.
Cite arxiv.org/abs/2608.12149 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2608.12149 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.