A fresh perspective on retrieval heads!</p>\n","updatedAt":"2026-07-03T13:14:40.203Z","author":{"_id":"61001311e043e15c13412d30","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/61001311e043e15c13412d30/6yAbTweYR16XtxMBEyOWl.png","fullname":"Pasquale Minervini","name":"pminervini","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":62,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.7535076141357422},"editors":["pminervini"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/61001311e043e15c13412d30/6yAbTweYR16XtxMBEyOWl.png"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2607.01002","authors":[{"_id":"6a46246cf4333ef0ca1d026d","name":"Aryo Pradipta Gema","hidden":false},{"_id":"6a46246cf4333ef0ca1d026e","name":"Beatrice Alex","hidden":false},{"_id":"6a46246cf4333ef0ca1d026f","name":"Pasquale Minervini","hidden":false}],"publishedAt":"2026-07-01T00:00:00.000Z","submittedOnDailyAt":"2026-07-03T00:00:00.000Z","title":"Logit-Contribution Scoring Identifies Non-Literal Retrieval Heads","submittedOnDailyBy":{"_id":"61001311e043e15c13412d30","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/61001311e043e15c13412d30/6yAbTweYR16XtxMBEyOWl.png","isPro":false,"fullname":"Pasquale Minervini","user":"pminervini","type":"user","name":"pminervini"},"summary":"In long-context use, large language models frequently synthesize answers from the meaning of a relevant context span rather than literally copy-pasting them. Identifying which attention heads perform this synthesis matters for interpreting long-context model behavior. Yet existing detectors miss these heads by construction: they reward heads whose attended token matches the generated token, a literal-copy criterion that captures where a head reads but not what it writes through its output-value (OV) circuit, the very mechanism that carries non-literal retrieval. We introduce Logit-Contribution Scoring (LOCOS), a write-aware detector that scores each head by the projection of its OV-circuit output onto the answer-token unembedding direction, contrasting needle and off-needle source positions in a single forward pass. Across three model families (Qwen3, Gemma-3, OLMo-3.1), mean-ablating the top LOCOS heads on the NoLiMa non-literal retrieval benchmark collapses ROUGE-L at lower head counts than prior attention-based detections; on Qwen3-8B, ablating 50 heads drives ROUGE-L from 0.401 to 0.000 while the strongest baseline still retains 0.292. The selected heads are retrieval-specific: parametric recall and arithmetic reasoning stay at baseline under the same ablation. On Qwen3-8B, the same ablation also drops MuSiQue from 0.55 to 0.08 and BABI-Long from 0.62 to 0.20, while a random-heads control stays within 0.05 of baseline.","upvotes":11,"discussionId":"6a46246cf4333ef0ca1d0270","projectPage":"https://aryopg.com/locos/","githubRepo":"https://github.com/aryopg/locos","githubRepoAddedBy":"user","ai_summary":"Logit-Contribution Scoring (LOCOS) identifies attention heads responsible for non-literal context synthesis in large language models by measuring their output-value circuit's contribution to answer tokens, outperforming existing methods on retrieval benchmarks.","ai_keywords":["attention heads","logit-contribution scoring","OV-circuit","answer-token unembedding","non-literal retrieval","ROUGE-L","parametric recall","arithmetic reasoning","MuSiQue","BABI-Long"],"ai_summary_model":"Qwen/Qwen2.5-Coder-32B-Instruct","githubStars":0},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"61001311e043e15c13412d30","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/61001311e043e15c13412d30/6yAbTweYR16XtxMBEyOWl.png","isPro":false,"fullname":"Pasquale Minervini","user":"pminervini","type":"user"},{"_id":"63dcfaaaf37111482522fbd6","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/63dcfaaaf37111482522fbd6/-NkJznxAJum4fMPy8zoUT.jpeg","isPro":false,"fullname":"Waylon Li","user":"waylonli","type":"user"},{"_id":"644f895e23d7eb05ca695054","avatarUrl":"/avatars/3fb04dd8544b403262bf98507de05453.svg","isPro":true,"fullname":"Aryo Pradipta Gema","user":"aryopg","type":"user"},{"_id":"68d4346704deb6c5c23970e7","avatarUrl":"/avatars/dd365161955399d87ebbc0d77d45219c.svg","isPro":false,"fullname":"Monica Sekoyan","user":"monica-sekoyan","type":"user"},{"_id":"685a42c1f1005a8fa0f70769","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/6QWSMPcIzYjd856w_VzxC.png","isPro":false,"fullname":"CHEN，TAN","user":"RichardLChen","type":"user"},{"_id":"632f5ffdcc0b3661318ced3b","avatarUrl":"/avatars/52f9a56028098128dd558132d31dee18.svg","isPro":false,"fullname":"Federico Tiblias","user":"akatief","type":"user"},{"_id":"6707a05c371a647d6c76b0eb","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6707a05c371a647d6c76b0eb/mc4FPxyICt5Lx0vfHG_2r.jpeg","isPro":false,"fullname":"Alessandro Cagiano","user":"AlessandroFrancescoBruno","type":"user"},{"_id":"664b4b4b10bf933450bfd728","avatarUrl":"/avatars/0f40775ebb2e26fb2f68b3159ffdc8bd.svg","isPro":false,"fullname":"raj b","user":"raj999","type":"user"},{"_id":"649a98de7970baf75fc7d07a","avatarUrl":"/avatars/b9bc8d082f63e06dd852467dfd1750af.svg","isPro":false,"fullname":"Cyrus Kwan","user":"wckwan","type":"user"},{"_id":"5e7749883d77a72421292d07","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/5e7749883d77a72421292d07/M4AmBReZk_otxCIG3o0bL.jpeg","isPro":false,"fullname":"Gabriele Sarti","user":"gsarti","type":"user"},{"_id":"6473543e8b7a55cfa91d75cd","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6473543e8b7a55cfa91d75cd/6glN97Z3tcYV30a7a08Ed.jpeg","isPro":false,"fullname":"Lorenzo Molfetta","user":"LorMolf","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2607/2607.01002.md","query":{}}">
Logit-Contribution Scoring Identifies Non-Literal Retrieval Heads
Abstract
Logit-Contribution Scoring (LOCOS) identifies attention heads responsible for non-literal context synthesis in large language models by measuring their output-value circuit's contribution to answer tokens, outperforming existing methods on retrieval benchmarks.
In long-context use, large language models frequently synthesize answers from the meaning of a relevant context span rather than literally copy-pasting them. Identifying which attention heads perform this synthesis matters for interpreting long-context model behavior. Yet existing detectors miss these heads by construction: they reward heads whose attended token matches the generated token, a literal-copy criterion that captures where a head reads but not what it writes through its output-value (OV) circuit, the very mechanism that carries non-literal retrieval. We introduce Logit-Contribution Scoring (LOCOS), a write-aware detector that scores each head by the projection of its OV-circuit output onto the answer-token unembedding direction, contrasting needle and off-needle source positions in a single forward pass. Across three model families (Qwen3, Gemma-3, OLMo-3.1), mean-ablating the top LOCOS heads on the NoLiMa non-literal retrieval benchmark collapses ROUGE-L at lower head counts than prior attention-based detections; on Qwen3-8B, ablating 50 heads drives ROUGE-L from 0.401 to 0.000 while the strongest baseline still retains 0.292. The selected heads are retrieval-specific: parametric recall and arithmetic reasoning stay at baseline under the same ablation. On Qwen3-8B, the same ablation also drops MuSiQue from 0.55 to 0.08 and BABI-Long from 0.62 to 0.20, while a random-heads control stays within 0.05 of baseline.
Community
A fresh perspective on retrieval heads!
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2607.01002 in a model README.md to link it from this page.
Cite arxiv.org/abs/2607.01002 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.