Code is publicly released at:<br><a href=\"https://github.com/Patrick-Ni/ReFreeKV\" rel=\"nofollow\">https://github.com/Patrick-Ni/ReFreeKV</a></p>\n","updatedAt":"2026-06-30T03:13:10.616Z","author":{"_id":"650f0fac11f3210cf7a8a849","avatarUrl":"/avatars/687d56c3a6d4f5cdb34e424cdcff954d.svg","fullname":"Liyan Xu","name":"lxucs","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":1,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8299158811569214},"editors":["lxucs"],"editorAvatarUrls":["/avatars/687d56c3a6d4f5cdb34e424cdcff954d.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2502.16886","authors":[{"_id":"6a433393763f63ca3757e90b","name":"Xuanfan Ni","hidden":false},{"_id":"6a433393763f63ca3757e90c","name":"Liyan Xu","hidden":false},{"_id":"6a433393763f63ca3757e90d","name":"Chenyang Lyu","hidden":false},{"_id":"6a433393763f63ca3757e90e","name":"Longyue Wang","hidden":false},{"_id":"6a433393763f63ca3757e90f","name":"Mo Yu","hidden":false},{"_id":"6a433393763f63ca3757e910","name":"Lemao Liu","hidden":false},{"_id":"6a433393763f63ca3757e911","name":"Fandong Meng","hidden":false},{"_id":"6a433393763f63ca3757e912","name":"Jie Zhou","hidden":false},{"_id":"6a433393763f63ca3757e913","name":"Piji Li","hidden":false}],"publishedAt":"2026-06-26T00:00:00.000Z","submittedOnDailyAt":"2026-06-30T00:00:00.000Z","title":"ReFreeKV: Towards Threshold-Free KV Cache Compression","submittedOnDailyBy":{"_id":"650f0fac11f3210cf7a8a849","avatarUrl":"/avatars/687d56c3a6d4f5cdb34e424cdcff954d.svg","isPro":false,"fullname":"Liyan Xu","user":"lxucs","type":"user","name":"lxucs"},"summary":"To reduce memory consumption during LLM inference, a handful of methods have been proposed for KV cache pruning. While these techniques can accomplish lossless memory reduction on many datasets, they often hinge on an under-emphasized condition: an input/domain-specific threshold for KV cache budget needs to be pre-determined to achieve the optimal performance. However, such input-sensitive design may be considerably limited in real-world scenarios, as open-domain inputs span diverse domains, lengths and difficulty levels, without clear boundaries for threshold selection. As a result, the dependence of such input-sensitive threshold can be a fundamental limitation that causes large degradation on arbitrary inputs. In this work, we propose a new objective that lifts the threshold constraints for robust KV compression, advocating for \"threshold-free\" methods that adaptively adjust budget allocation while preserving full-cache performance. We then propose a novel method, ReFreeKV, serving as the first instantiation of this objective. Extensive experiments across 13 datasets with diverse context lengths, task types, and model sizes demonstrate its efficacy and efficiency. Our code is publicly released at https://github.com/Patrick-Ni/ReFreeKV.","upvotes":43,"discussionId":"6a433394763f63ca3757e914","githubRepo":"https://github.com/Patrick-Ni/ReFreeKV","githubRepoAddedBy":"user","ai_summary":"ReFreeKV addresses the limitations of threshold-dependent KV cache pruning by introducing a threshold-free approach that adaptively allocates compression budgets while maintaining full-cache performance across diverse datasets and model sizes.","ai_keywords":["KV cache pruning","threshold-free methods","adaptive budget allocation","KV cache compression","LLM inference"],"ai_summary_model":"Qwen/Qwen2.5-Coder-32B-Instruct","githubStars":2,"organization":{"_id":"66543b6e420092799d2f625c","name":"tencent","fullname":"Tencent","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/5dd96eb166059660ed1ee413/Lp3m-XLpjQGwBItlvn69q.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"650f0fac11f3210cf7a8a849","avatarUrl":"/avatars/687d56c3a6d4f5cdb34e424cdcff954d.svg","isPro":false,"fullname":"Liyan Xu","user":"lxucs","type":"user"},{"_id":"6982e25f631413333a261d21","avatarUrl":"/avatars/2c7526ce4cf5c28c53836581d67abe5b.svg","isPro":false,"fullname":"Vasalin","user":"vasalin56","type":"user"},{"_id":"68a3ee837aca8caa6e9ba15b","avatarUrl":"/avatars/9ff95f08ff64fd696e82436b4c14d32b.svg","isPro":false,"fullname":"Mango","user":"mango-eater","type":"user"},{"_id":"68a3edf729422fdc9fd7dac4","avatarUrl":"/avatars/ec3039389b9ffd4d4c08680668d5eeee.svg","isPro":false,"fullname":"Jovi","user":"jovi281","type":"user"},{"_id":"68a3ed6d114e4b968eabd81f","avatarUrl":"/avatars/59db1dbc09cfffd75e69c971c46d4198.svg","isPro":false,"fullname":"Alex B","user":"hf-2816","type":"user"},{"_id":"68a3f638814d0858f586f4be","avatarUrl":"/avatars/0f7f6c7a49bb796817cc1b0fa775538a.svg","isPro":false,"fullname":"gshock","user":"gshock888","type":"user"},{"_id":"68a53641664013867789bcce","avatarUrl":"/avatars/ad9e0b1d903687ab5bd6539775530a11.svg","isPro":false,"fullname":"wewin","user":"wewin678","type":"user"},{"_id":"68a3f70b1603e124879b7e4e","avatarUrl":"/avatars/700115c4da775143b3b0ed623d12be7b.svg","isPro":false,"fullname":"vitamin","user":"vitamin788","type":"user"},{"_id":"6982c600b07a2a393eb35e44","avatarUrl":"/avatars/039f0b2feb7f5a67bf5845528257a22a.svg","isPro":false,"fullname":"IvyC","user":"Ivy5813","type":"user"},{"_id":"6982d81986db6868fdeda1d5","avatarUrl":"/avatars/d43f3dd2b06e61854d26810158f388b6.svg","isPro":false,"fullname":"Tummy","user":"tummy-moon","type":"user"},{"_id":"6982d8c50bf4ca5eeff87e91","avatarUrl":"/avatars/dcf30f7a21f4b783ed32fd8a5cb5edcd.svg","isPro":false,"fullname":"bonheur-de","user":"bonheur999","type":"user"},{"_id":"68a3f32b3f8836479984e316","avatarUrl":"/avatars/650dad1f05e584110389b780bcf0c7d3.svg","isPro":false,"fullname":"Coco","user":"coco-187","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"66543b6e420092799d2f625c","name":"tencent","fullname":"Tencent","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/5dd96eb166059660ed1ee413/Lp3m-XLpjQGwBItlvn69q.png"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2502/2502.16886.md","query":{}}">
ReFreeKV: Towards Threshold-Free KV Cache Compression
Abstract
ReFreeKV addresses the limitations of threshold-dependent KV cache pruning by introducing a threshold-free approach that adaptively allocates compression budgets while maintaining full-cache performance across diverse datasets and model sizes.
To reduce memory consumption during LLM inference, a handful of methods have been proposed for KV cache pruning. While these techniques can accomplish lossless memory reduction on many datasets, they often hinge on an under-emphasized condition: an input/domain-specific threshold for KV cache budget needs to be pre-determined to achieve the optimal performance. However, such input-sensitive design may be considerably limited in real-world scenarios, as open-domain inputs span diverse domains, lengths and difficulty levels, without clear boundaries for threshold selection. As a result, the dependence of such input-sensitive threshold can be a fundamental limitation that causes large degradation on arbitrary inputs. In this work, we propose a new objective that lifts the threshold constraints for robust KV compression, advocating for "threshold-free" methods that adaptively adjust budget allocation while preserving full-cache performance. We then propose a novel method, ReFreeKV, serving as the first instantiation of this objective. Extensive experiments across 13 datasets with diverse context lengths, task types, and model sizes demonstrate its efficacy and efficiency. Our code is publicly released at https://github.com/Patrick-Ni/ReFreeKV.
Community
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2502.16886 in a model README.md to link it from this page.
Cite arxiv.org/abs/2502.16886 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2502.16886 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.