Long-video MLLMs can't process every frame; an hour at 1fps is 3,600 images, and most systems keep a small fixed slice. This paper holds selection, spatial compression, and reinvestment separate and varies one at a time, across 6 selection rules, 3 benchmarks, 2 answering models.</p>\n<ul>\n<li>Which frames you pick matters a lot — Orthogonal Matching Pursuit (OMP - a classic algorithm), used with no changes, picks frames about as well as newer purpose-built methods.</li>\n<li>Shrinking each frame's resolution costs almost nothing in accuracy.</li>\n<li>The real win: take the tokens you saved from shrinking, and use them to look at more frames instead of banking the savings. That's worth a few extra points of accuracy - shrinking alone isn't the point, spending the savings is.</li>\n</ul>\n<p>Side-by-side comparison of using 8 frames at 100% visual token budget and 16 frames at 50% visual token budget(Effectively the same in terms of tokens).<br><a href=\"https://cdn-uploads.huggingface.co/production/uploads/68c914f2ba41e98e803f809b/NloYYKnDezDEZAzE7XLGo.png\" rel=\"nofollow\"><img src=\"https://cdn-uploads.huggingface.co/production/uploads/68c914f2ba41e98e803f809b/NloYYKnDezDEZAzE7XLGo.png\" alt=\"Screenshot 2026-09-04 100943\"></a></p>\n","updatedAt":"2026-09-04T04:51:42.358Z","author":{"_id":"68c914f2ba41e98e803f809b","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/NP72KVXx81X5MCIG8HafS.png","fullname":"Prakhar Khatri","name":"Prakh25","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false}},"numEdits":1,"identifiedLanguage":{"language":"en","probability":0.9173583388328552},"editors":["Prakh25"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/NP72KVXx81X5MCIG8HafS.png"],"reactions":[],"isReport":false}},{"id":"6a9aa9aefc64b85e100f48c4","author":{"_id":"68c914f2ba41e98e803f809b","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/NP72KVXx81X5MCIG8HafS.png","fullname":"Prakhar Khatri","name":"Prakh25","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false},"createdAt":"2026-09-04T11:21:18.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"Long-video MLLMs can't process every frame; an hour at 1fps is 3,600 images, and most systems keep a small fixed slice. This paper holds selection, spatial compression, and reinvestment separate and varies one at a time, across 6 selection rules, 3 benchmarks, 2 answering models.\n\n- Which frames you pick matters a lot — Orthogonal Matching Pursuit (OMP - a classic algorithm), used with no changes, picks frames about as well as newer purpose-built methods.\n- Shrinking each frame's resolution costs almost nothing in accuracy.\n- The real win: take the tokens you saved from shrinking, and use them to look at more frames instead of banking the savings. That's worth a few extra points of accuracy - shrinking alone isn't the point, spending the savings is.\n\nSide-by-side comparison of using 8 frames at 100% visual token budget and 16 frames at 50% visual token budget(Effectively the same in terms of tokens).\n\n ","html":"<p>Long-video MLLMs can't process every frame; an hour at 1fps is 3,600 images, and most systems keep a small fixed slice. This paper holds selection, spatial compression, and reinvestment separate and varies one at a time, across 6 selection rules, 3 benchmarks, 2 answering models.</p>\n<ul>\n<li>Which frames you pick matters a lot — Orthogonal Matching Pursuit (OMP - a classic algorithm), used with no changes, picks frames about as well as newer purpose-built methods.</li>\n<li>Shrinking each frame's resolution costs almost nothing in accuracy.</li>\n<li>The real win: take the tokens you saved from shrinking, and use them to look at more frames instead of banking the savings. That's worth a few extra points of accuracy - shrinking alone isn't the point, spending the savings is.</li>\n</ul>\n<p>Side-by-side comparison of using 8 frames at 100% visual token budget and 16 frames at 50% visual token budget(Effectively the same in terms of tokens).</p>\n<p> <a href=\"https://cdn-uploads.huggingface.co/production/uploads/68c914f2ba41e98e803f809b/CtNHUjqkXeoG93ZeiBQnt.png\" rel=\"nofollow\"><img src=\"https://cdn-uploads.huggingface.co/production/uploads/68c914f2ba41e98e803f809b/CtNHUjqkXeoG93ZeiBQnt.png\" alt=\"image\"></a></p>\n","updatedAt":"2026-09-04T11:21:18.187Z","author":{"_id":"68c914f2ba41e98e803f809b","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/NP72KVXx81X5MCIG8HafS.png","fullname":"Prakhar Khatri","name":"Prakh25","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8925553560256958},"editors":["Prakh25"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/NP72KVXx81X5MCIG8HafS.png"],"reactions":[],"isReport":false}},{"id":"6a9b6ec4673790efdfac7c87","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":379,"isUserFollowing":false},"createdAt":"2026-09-05T01:22:12.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"This is an automated message from the [Librarian Bot](https://huggingface.co/librarian-bots). I found the following papers similar to this paper. \n\nThe following papers were recommended by the Semantic Scholar API \n\n* [Efficient Frame Selection for Long Videos at Test Time with Attention-Based MLLM Selectors](https://huggingface.co/papers/2607.15689) (2026)\n* [Allocation Before Ranking: Decoupled Token Compression for OmniLLMs](https://huggingface.co/papers/2608.01665) (2026)\n* [RIDGE: Region-Informed Derivative-Guided Evidence Selection for Long Video Understanding](https://huggingface.co/papers/2608.29958) (2026)\n* [FORGE: Frame Orthogonality in Relevance Geometry for Long-Form Video Understanding](https://huggingface.co/papers/2607.25266) (2026)\n* [Pre-Decoding Acoustic Triage for Budgeted Vision-Language Captioning of Untrimmed Egocentric Video](https://huggingface.co/papers/2608.22359) (2026)\n* [One Ranking, Any Budget: Matryoshka Evidence-to-Context Frame Selection for Long-Video Understanding](https://huggingface.co/papers/2608.05707) (2026)\n* [Coverage-Driven Adaptive Keyframe Selection for Video Understanding](https://huggingface.co/papers/2608.00714) (2026)\n\n\n Please give a thumbs up to this comment if you found it helpful!\n\n If you want recommendations for any Paper on Hugging Face checkout [this](https://huggingface.co/spaces/librarian-bots/recommend_similar_papers) Space\n\n You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: `@librarian-bot recommend`","html":"<p>This is an automated message from the <a href=\"https://huggingface.co/librarian-bots\">Librarian Bot</a>. I found the following papers similar to this paper. </p>\n<p>The following papers were recommended by the Semantic Scholar API </p>\n<ul>\n<li><a href=\"https://huggingface.co/papers/2607.15689\">Efficient Frame Selection for Long Videos at Test Time with Attention-Based MLLM Selectors</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2608.01665\">Allocation Before Ranking: Decoupled Token Compression for OmniLLMs</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2608.29958\">RIDGE: Region-Informed Derivative-Guided Evidence Selection for Long Video Understanding</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2607.25266\">FORGE: Frame Orthogonality in Relevance Geometry for Long-Form Video Understanding</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2608.22359\">Pre-Decoding Acoustic Triage for Budgeted Vision-Language Captioning of Untrimmed Egocentric Video</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2608.05707\">One Ranking, Any Budget: Matryoshka Evidence-to-Context Frame Selection for Long-Video Understanding</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2608.00714\">Coverage-Driven Adaptive Keyframe Selection for Video Understanding</a> (2026)</li>\n</ul>\n<p> Please give a thumbs up to this comment if you found it helpful!</p>\n<p> If you want recommendations for any Paper on Hugging Face checkout <a href=\"https://huggingface.co/spaces/librarian-bots/recommend_similar_papers\">this</a> Space</p>\n<p> You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: <code>@librarian-bot recommend</code></p>\n","updatedAt":"2026-09-05T01:22:12.144Z","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":379,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.7129331827163696},"editors":["librarian-bot"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2609.03820","authors":[{"_id":"6a9a48e58f7c3b7557239558","user":{"_id":"68c914f2ba41e98e803f809b","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/NP72KVXx81X5MCIG8HafS.png","isPro":false,"fullname":"Prakhar Khatri","user":"Prakh25","type":"user","name":"Prakh25"},"name":"Prakhar Khatri","status":"claimed_verified","statusLastChangedAt":"2026-09-04T08:45:04.300Z","hidden":false}],"mediaUrls":["https://cdn-uploads.huggingface.co/production/uploads/68c914f2ba41e98e803f809b/1dW1kjdTfQE51l8LIcnbt.png"],"publishedAt":"2026-09-03T00:00:00.000Z","submittedOnDailyAt":"2026-09-04T00:00:00.000Z","title":"Select, Compress, Reinvest: A Controlled Study of Visual-Token Allocation in Long-Video MLLMs","submittedOnDailyBy":{"_id":"68c914f2ba41e98e803f809b","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/NP72KVXx81X5MCIG8HafS.png","isPro":false,"fullname":"Prakhar Khatri","user":"Prakh25","type":"user","name":"Prakh25"},"summary":"Long-video language models cannot look at every frame: an hour sampled once per second is 3,600 images, and a system keeps only a small fixed slice of that pool. Which frames survive that slice is usually treated as a preprocessing detail; we test whether it should be. Published selectors make the comparison hard because they change the frame scorer, the prompt boundary, the resolution policy, and the answering model all at once. We hold each fixed and vary one decision at a time: selection, spatial compression, and reinvestment of the savings, across six training-free selection rules, three long-video benchmarks, and two answering models. Selection is the largest single lever: on LongVideoBench's hour-long bin, eight query-selected frames beat sixteen uniformly spaced ones by 6.9 points, and Orthogonal Matching Pursuit, an unmodified decades-old sparse-approximation algorithm, matches or comes within a point of every purpose-built selector we compare it against, across all three benchmarks. Compression is close to free: halving each frame's spatial budget at fixed timestamps costs at most 0.44 points. Reinvestment is where that budget turns back into accuracy: spending the freed tokens on twice as many compressed frames, at a measured cost no higher than the original eight, returns a further two to three points; compression only pays off once its savings are spent this way. Along the way, an implementation bug in our own AKS baseline and a 0.07 to 3.74 point gap between two harnesses running the same published rules at the same budget show why these comparisons need to happen inside one controlled harness rather than across papers.","upvotes":15,"discussionId":"6a9a48e58f7c3b7557239559","githubRepo":"https://github.com/codeprakhar25/omp-keyframe-sampling","githubRepoAddedBy":"user","ai_summary":"Frame selection dominates accuracy in long-video language models, while spatial compression is nearly free if savings are reinvested into more frames, and a unified harness is needed to compare selectors fairly.","ai_keywords":["long-video language models","frame selection","Orthogonal Matching Pursuit","sparse approximation","spatial compression","token reinvestment"],"ai_summary_model":"thinkingmachines/Inkling-Small","githubStars":1},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"68c914f2ba41e98e803f809b","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/NP72KVXx81X5MCIG8HafS.png","isPro":false,"fullname":"Prakhar Khatri","user":"Prakh25","type":"user"},{"_id":"6a2da6c8ca070ee12c6e396c","avatarUrl":"/avatars/0355287dcabaa67dbc7f0b10b87451f9.svg","isPro":false,"fullname":"Joe Mama","user":"JoeMama123123123","type":"user"},{"_id":"6a6a9c3dbbca071c7189c742","avatarUrl":"/avatars/1420f2ad85cb529e3ad9ce5becef48d2.svg","isPro":false,"fullname":"Richard Anderson","user":"Quiet-Eli","type":"user"},{"_id":"6a6c80fcee0efdfcf121ccaa","avatarUrl":"/avatars/9baa2276d08cf82f287b681fdcb96f1d.svg","isPro":false,"fullname":"Matthew Lee","user":"Wild-Ian","type":"user"},{"_id":"6a6aa720a698558ca7632e02","avatarUrl":"/avatars/d82c5171d9e05a56cdfd1fa9c097da12.svg","isPro":false,"fullname":"Patricia Miller","user":"novabeacon","type":"user"},{"_id":"6a6dca7f0127a9d4aa987cff","avatarUrl":"/avatars/b48ce8b65d85837d2f04239a67d09828.svg","isPro":false,"fullname":"Patricia Martinez","user":"Cedar-Patricia","type":"user"},{"_id":"6a70200b1512236b81991681","avatarUrl":"/avatars/16928ae879188300266c13939bee0afc.svg","isPro":false,"fullname":"Susan Martinez","user":"rapidScope","type":"user"},{"_id":"6a9aa67c6a801fc3df1308d0","avatarUrl":"/avatars/bfb32956c8c76b0833b7aab953f13923.svg","isPro":false,"fullname":"Jingyi Li","user":"Quiet-Jingyi581","type":"user"},{"_id":"63ac5701c21e60a3e9b58aa7","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/63ac5701c21e60a3e9b58aa7/g6EX7diOpuA94R2ab-rZC.png","isPro":true,"fullname":"Dipankar Sarkar","user":"dipankarsarkar","type":"user"},{"_id":"6a6aa6de726441725a30a027","avatarUrl":"/avatars/2e777075d9d0d6dcdaaf0e97860f551e.svg","isPro":false,"fullname":"Sarah Clark","user":"granitecore","type":"user"},{"_id":"6a6c7b20efff7e7025883f83","avatarUrl":"/avatars/7871f4700b31baadb6a5ab889f88d04c.svg","isPro":false,"fullname":"Andrew Martinez","user":"SableKiteA","type":"user"},{"_id":"6a6c9c217d5ca057d90c7cf1","avatarUrl":"/avatars/2a27fb930a6493f2b9aa26516ae4a99e.svg","isPro":false,"fullname":"John Taylor","user":"AzureWisp","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2609/2609.03820.md","query":{}}">

Get this paper in your agent:

Don't have the latest CLI?

Cite arxiv.org/abs/2609.03820 in a model README.md to link it from this page.

Cite arxiv.org/abs/2609.03820 in a dataset README.md to link it from this page.

Cite arxiv.org/abs/2609.03820 in a Space README.md to link it from this page.