Fast Weight Attention for Continual Learning</p>\n","updatedAt":"2026-08-31T06:40:35.445Z","author":{"_id":"647bf082aba7062fe5c51ca9","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/647bf082aba7062fe5c51ca9/VvKAhQC_LxBcBuy3XROSX.jpeg","fullname":"Yifan Zhang","name":"yifAI","type":"user","isPro":true,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":28,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.7100426554679871},"editors":["yifAI"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/647bf082aba7062fe5c51ca9/VvKAhQC_LxBcBuy3XROSX.jpeg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2608.27763","authors":[{"_id":"6a95216d073195fee51572d7","name":"Yifan Zhang","hidden":false},{"_id":"6a95216d073195fee51572d8","name":"Steve Ta","hidden":false},{"_id":"6a95216d073195fee51572d9","name":"Jasper Zhang","hidden":false},{"_id":"6a95216d073195fee51572da","name":"Jichen Feng","hidden":false},{"_id":"6a95216d073195fee51572db","name":"Shuzhen Li","hidden":false},{"_id":"6a95216d073195fee51572dc","name":"Yongxin Zhang","hidden":false},{"_id":"6a95216d073195fee51572dd","name":"Yifeng Liu","hidden":false},{"_id":"6a95216d073195fee51572de","name":"Huizhuo Yuan","hidden":false},{"_id":"6a95216d073195fee51572df","name":"Mengdi Wang","hidden":false},{"_id":"6a95216d073195fee51572e0","name":"Quanquan Gu","hidden":false},{"_id":"6a95216d073195fee51572e1","name":"Andrew Chi-Chih Yao","hidden":false}],"publishedAt":"2026-08-27T00:00:00.000Z","submittedOnDailyAt":"2026-08-31T00:00:00.000Z","title":"Fast Weight Attention for Continual Learning","submittedOnDailyBy":{"_id":"647bf082aba7062fe5c51ca9","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/647bf082aba7062fe5c51ca9/VvKAhQC_LxBcBuy3XROSX.jpeg","isPro":true,"fullname":"Yifan Zhang","user":"yifAI","type":"user","name":"yifAI"},"summary":"Recurrent fast-weight memories and selective state-space models compress an expanding context into a fixed-size recurrent state, making the state transition an online learning rule. We study this rule under read-after-write autoregressive semantics. For the prefix-prediction objective considered here, the local fast-memory example revealed at step t is the prefix-aligned pair (x_t,y_t)=(ϕ(k_{t-1}),v_t). The common same-step association (ϕ(k_t),v_t) remains causal, but optimizes a different internal objective. We derive normalized first-order updates for squared-error regression and negative inner-product objectives. The regression family comprises Falcon-1 (a scalar NLMS update), Falcon-2 (its per-column extension), and Falcon-3 (a sliding-window mini-batch update); Falcon-1A/Falcon-2A/Falcon-3A are the corresponding inner-product variants. We provide recurrent, masked-parallel, and chunk-parallel forms, together with numerically stable positive-decay renormalization. Representative variants remain competitive in language modeling and improve length extrapolation on variable-digit addition. This framework separates temporal alignment, plasticity, forgetting, and bounded rehearsal in recurrent sequence models.","upvotes":3,"discussionId":"6a95216d073195fee51572e2","projectPage":"https://github.com/yifanzhang-pro/fast-weight-attention","githubRepo":"https://github.com/yifanzhang-pro/fast-weight-attention","githubRepoAddedBy":"user","ai_summary":"Recurrent fast-weight memories and selective state-space models are analyzed as online learning rules under autoregressive semantics, yielding normalized update families with stable renormalization that improve length extrapolation.","ai_keywords":["fast-weight memories","selective state-space models","recurrent state","online learning rule","read-after-write autoregressive semantics","prefix-prediction","normalized first-order updates","squared-error regression","negative inner-product","Falcon-1","Falcon-2","Falcon-3","masked-parallel","chunk-parallel","positive-decay renormalization","length extrapolation"],"ai_summary_model":"thinkingmachines/Inkling-Small","githubStars":6},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"647bf082aba7062fe5c51ca9","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/647bf082aba7062fe5c51ca9/VvKAhQC_LxBcBuy3XROSX.jpeg","isPro":true,"fullname":"Yifan Zhang","user":"yifAI","type":"user"},{"_id":"64b2f97434a92b848c7e941e","avatarUrl":"/avatars/c699c50f3b43cd1641469521127753bb.svg","isPro":false,"fullname":"Nagori","user":"MohammedNaeem","type":"user"},{"_id":"620783f24e28382272337ba4","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/620783f24e28382272337ba4/zkUveQPNiDfYjgGhuFErj.jpeg","isPro":false,"fullname":"GuoLiangTang","user":"Tommy930","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"query":{}}">

Cite arxiv.org/abs/2608.27763 in a model README.md to link it from this page.

Cite arxiv.org/abs/2608.27763 in a dataset README.md to link it from this page.

Cite arxiv.org/abs/2608.27763 in a Space README.md to link it from this page.

Add this paper to a collection to link it from this page.