Hugging Face Daily Papers · September 3, 2026 · 4 min read

Language Models Can Control Their Own Attention

Mirrored from Hugging Face Daily Papers for archival readability. Support the source by reading on the original site.

Like Read original ↗

Language models can control their own attention. Zero-shot evaluation of Gemma 4 31B shows a 52% reduction in global attention cost during decoding across 15 long-context benchmarks with 1.52pp accuracy drop.</p>\n","updatedAt":"2026-09-03T05:26:13.655Z","author":{"_id":"62d0f7faad741b94f5d13744","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1677826870505-62d0f7faad741b94f5d13744.jpeg","fullname":"Namgyu Ho","name":"itsnamgyu","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":9,"isUserFollowing":false}},"numEdits":1,"identifiedLanguage":{"language":"en","probability":0.9139766693115234},"editors":["itsnamgyu"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1677826870505-62d0f7faad741b94f5d13744.jpeg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2609.02737","authors":[{"_id":"6a98e2cafea818274321fed3","name":"Namgyu Ho","hidden":false},{"_id":"6a98e2cafea818274321fed4","name":"Huzama Ahmad","hidden":false},{"_id":"6a98e2cafea818274321fed5","name":"Woosung Koh","hidden":false},{"_id":"6a98e2cafea818274321fed6","name":"Se-Young Yun","hidden":false},{"_id":"6a98e2cafea818274321fed7","name":"Tal Schuster","hidden":false},{"_id":"6a98e2cafea818274321fed8","name":"Cicero Nogueira dos Santos","hidden":false}],"mediaUrls":["https://cdn-uploads.huggingface.co/production/uploads/62d0f7faad741b94f5d13744/2kQ6pgkbsBa9L9ftrmueY.png"],"publishedAt":"2026-09-02T00:00:00.000Z","submittedOnDailyAt":"2026-09-03T00:00:00.000Z","title":"Language Models Can Control Their Own Attention","submittedOnDailyBy":{"_id":"62d0f7faad741b94f5d13744","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1677826870505-62d0f7faad741b94f5d13744.jpeg","isPro":false,"fullname":"Namgyu Ho","user":"itsnamgyu","type":"user","name":"itsnamgyu"},"summary":"Language models spend most of their attention on a small fraction of context, yet they read the entire KV cache to find the few tokens that matter. If the user asks about a previous detail in a 1M-token conversation, global attention layers must scan the full context to generate each token of the reply. A prominent approach mitigates this cost by pre-selecting relevant tokens via lightweight proxy scores, but this extrinsic scoring still incurs O(N) per step. We take an intrinsic approach motivated by the simple question: wouldn't the model already know which parts of the context are relevant? To this end, we introduce Declarative Attention (DA), a protocol that elicits the model to declare where it needs to attend within its chain-of-thought, partitioning generation into three modes: <global> (full context), <focus> (a specific region), and <local> (recent output only). The inference engine parses these declarations like tool calls and skips most of the KV cache read. Under zero-shot evaluation across 15 long-context tasks, DA on off-the-shelf models (Gemma-4-31B, Qwen-3.6-27B) significantly reduces total attended tokens during decoding (52.0%, 31.1%) with modest accuracy drops (1.27pp, 2.75pp) that shrink with model scale. DA unlocks a new axis of sparse attention, with further potential under training-based methods that future work can explore.","upvotes":38,"discussionId":"6a98e2cafea818274321fed9","ai_summary":"Declarative Attention lets language models declare relevant context regions during reasoning to skip most KV cache reads, reducing attended tokens with small accuracy trade-offs.","ai_keywords":["Declarative Attention","KV cache","chain-of-thought","global attention","sparse attention","long-context"],"ai_summary_model":"thinkingmachines/Inkling-Small","organization":{"_id":"6475760c33192631bad2bb38","name":"kaist-ai","fullname":"KAIST AI","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/6469949654873f0043b09c22/aaZFiyXe1qR-Dmy_xq67m.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"62d0f7faad741b94f5d13744","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1677826870505-62d0f7faad741b94f5d13744.jpeg","isPro":false,"fullname":"Namgyu Ho","user":"itsnamgyu","type":"user"},{"_id":"67d823d86d6baf84fba04392","avatarUrl":"/avatars/8dfe7800aa3f9d9166963a1f5da5657a.svg","isPro":false,"fullname":"segyu lee","user":"segyulee","type":"user"},{"_id":"64c9cf490d3d1b209d487308","avatarUrl":"/avatars/5db24ec87db0080198bf233f045fb0d4.svg","isPro":false,"fullname":"SehyeoKKang","user":"SH0329","type":"user"},{"_id":"69a042f86f16394da5517225","avatarUrl":"/avatars/67c6ce7cdd6d20e94a7df598f5236b99.svg","isPro":false,"fullname":"Juhyeong Kim","user":"myjju","type":"user"},{"_id":"64b214c4f4361a032002cdcf","avatarUrl":"/avatars/3188a4edec4d6c945a0ed9f36050a03c.svg","isPro":false,"fullname":"Andrew Wan Ju Kang","user":"soarhigh","type":"user"},{"_id":"684bd350b203be6c15d0dd30","avatarUrl":"/avatars/2b533d99c9107aad70d970a96ea316c3.svg","isPro":false,"fullname":"Minsu2","user":"Minsu2","type":"user"},{"_id":"698fbbb49d17ff43291a932c","avatarUrl":"/avatars/46498083e5406d4485281224858534d3.svg","isPro":false,"fullname":"Sujin Kim","user":"niirhv","type":"user"},{"_id":"699bc8d3d9f567ac21153c1f","avatarUrl":"/avatars/767e2b4a19b6cbcc862902628e2609f8.svg","isPro":false,"fullname":"Operezvrh","user":"operezvrh","type":"user"},{"_id":"63f5f7754b831cc179ba5410","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1677064273670-63f5f7754b831cc179ba5410.jpeg","isPro":false,"fullname":"Reiss Koh","user":"Reiss","type":"user"},{"_id":"65558caf38985840b368134f","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/XQSBrMezsvAcS78HzYiYs.png","isPro":false,"fullname":"Yujin Kim","user":"kimyuji","type":"user"},{"_id":"6452213165339508ce35e8a9","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6452213165339508ce35e8a9/4ngD1cQ9WcTLXtxaX9b7L.png","isPro":false,"fullname":"Huzama Ahmad","user":"huzama","type":"user"},{"_id":"62e167813eb0730f6217c95e","avatarUrl":"/avatars/95a18009f725832646643d6169c2bc08.svg","isPro":false,"fullname":"Seo Hyun Kim","user":"shkim0116","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"6475760c33192631bad2bb38","name":"kaist-ai","fullname":"KAIST AI","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/6469949654873f0043b09c22/aaZFiyXe1qR-Dmy_xq67m.png"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2609/2609.02737.md","query":{}}">

Papers

arxiv:2609.02737

Language Models Can Control Their Own Attention

Published on Sep 2

· Submitted by

Namgyu Ho on Sep 3

KAIST AI

Upvote

Authors:

Abstract

Declarative Attention lets language models declare relevant context regions during reasoning to skip most KV cache reads, reducing attended tokens with small accuracy trade-offs.

Generated by thinkingmachines/Inkling-Small

Language models spend most of their attention on a small fraction of context, yet they read the entire KV cache to find the few tokens that matter. If the user asks about a previous detail in a 1M-token conversation, global attention layers must scan the full context to generate each token of the reply. A prominent approach mitigates this cost by pre-selecting relevant tokens via lightweight proxy scores, but this extrinsic scoring still incurs O(N) per step. We take an intrinsic approach motivated by the simple question: wouldn't the model already know which parts of the context are relevant? To this end, we introduce Declarative Attention (DA), a protocol that elicits the model to declare where it needs to attend within its chain-of-thought, partitioning generation into three modes: <global> (full context), <focus> (a specific region), and <local> (recent output only). The inference engine parses these declarations like tool calls and skips most of the KV cache read. Under zero-shot evaluation across 15 long-context tasks, DA on off-the-shelf models (Gemma-4-31B, Qwen-3.6-27B) significantly reduces total attended tokens during decoding (52.0%, 31.1%) with modest accuracy drops (1.27pp, 2.75pp) that shrink with model scale. DA unlocks a new axis of sparse attention, with further potential under training-based methods that future work can explore.

View arXiv page View PDF Add to collection