A major challenge for scaling recursive self-improvement, especially in harness optimization, is the cost of evolution.</p>\n<p>We introduce a fundamentally different approach that cuts the evolution cost of existing harness optimization by up to 80%.</p>\n<p>🚀 Introducing Task-CoEvolve<br>Existing approaches focus on using fewer candidates or fewer iterations. Task-CoEvolve takes an orthogonal direction: instead of reducing the search itself, we reduce the number of evaluation tasks at each iteration.</p>\n<p>💡 The key idea is simple: not every validation task needs to be evaluated at every iteration.</p>\n<p>Task-CoEvolve co-evolves the validation tasks together with the harness:</p>\n<ul>\n<li>Select informative tasks based on the evolving harnesses</li>\n<li>Adaptively update which tasks are evaluated</li>\n<li>Estimate full-set performance from partial evaluations</li>\n</ul>\n<p>📊 Results<br>On online text classification and Terminal-Bench 2.1, Task-CoEvolve matches the performance of full-set search using only 20% of the evaluations (reducing the overall search cost by 67–80% in Terminal-Bench 2.1)🔥</p>\n<p>📄 <a href=\"https://arxiv.org/abs/2608.20169\" rel=\"nofollow\">https://arxiv.org/abs/2608.20169</a><br>Github:<br><a href=\"https://github.com/Agent4Science-UTokyo/Task-CoEvolve\" rel=\"nofollow\">https://github.com/Agent4Science-UTokyo/Task-CoEvolve</a></p>\n","updatedAt":"2026-08-25T04:18:35.587Z","author":{"_id":"6527b37c0ae663e384eb1b85","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6527b37c0ae663e384eb1b85/zKWa8h6YU4BWfcitpM5Pl.png","fullname":"Atsuyuki Miyai","name":"AtsuMiyai","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":7,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8557443618774414},"editors":["AtsuMiyai"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/6527b37c0ae663e384eb1b85/zKWa8h6YU4BWfcitpM5Pl.png"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2608.20169","authors":[{"_id":"6a8d13785add2537c32e9777","name":"Atsuyuki Miyai","hidden":false},{"_id":"6a8d13785add2537c32e9778","name":"Kiyoharu Aizawa","hidden":false},{"_id":"6a8d13785add2537c32e9779","name":"Toshihiko Yamasaki","hidden":false}],"publishedAt":"2026-08-24T00:00:00.000Z","submittedOnDailyAt":"2026-08-25T00:00:00.000Z","title":"Task-CoEvolve: Efficient Harness Optimization via Adaptive Validation Task Selection","submittedOnDailyBy":{"_id":"6527b37c0ae663e384eb1b85","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6527b37c0ae663e384eb1b85/zKWa8h6YU4BWfcitpM5Pl.png","isPro":false,"fullname":"Atsuyuki Miyai","user":"AtsuMiyai","type":"user","name":"AtsuMiyai"},"summary":"We present a novel approach to efficient LLM harness optimization through adaptive validation task selection. Harness optimization iteratively rewrites the harness code based on validation performance, enabling substantial performance gains without updating the underlying model weights. Existing approaches, however, evaluate a fixed validation set in full at every iteration, incurring substantial evaluation costs even on tasks that become less discriminative as the harness evolves. We propose Task-CoEvolve, which co-evolves the validation tasks with the harness by addressing two challenges: selecting informative tasks and estimating full-set performance from partial evaluations. Task-CoEvolve builds on the observation that tasks on which candidate harnesses disagree are more informative for distinguishing among them than tasks that are consistently solved or failed. It uses variance-weighted sampling based on past outcomes to focus evaluation on tasks near the capability frontier, with the sampling distribution adapting as the harness evolves. It then estimates full-set scores from the sampled tasks by accounting for their sampling probabilities, enabling consistent comparisons across iterations despite evaluating different subsets. Experiments on online text classification and Terminal-Bench 2.1 show that Task-CoEvolve consistently outperforms subset-based baselines and matches the final performance of full-set search while reducing the number of evaluations during optimization by 80%. Code will be released at https://github.com/Agent4Science-UTokyo/Task-CoEvolve.","upvotes":3,"discussionId":"6a8d13795add2537c32e977a","githubRepo":"https://github.com/Agent4Science-UTokyo/Task-CoEvolve","githubRepoAddedBy":"user","ai_summary":"Task-CoEvolve improves LLM harness optimization by adaptively selecting validation tasks and estimating full-set performance from partial evaluations, cutting evaluation costs by 80%.","ai_keywords":["LLM harness optimization","adaptive validation task selection","variance-weighted sampling","capability frontier","Task-CoEvolve"],"ai_summary_model":"thinkingmachines/Inkling-Small","githubStars":7,"organization":{"_id":"6730c40cfab94f649285ded2","name":"hal-utokyo","fullname":"Hal Lab UTokyo","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/65d86a59bdb95b4bbc744e10/FOjSKBb6nGV-BBRp9fQU5.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"6527b37c0ae663e384eb1b85","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6527b37c0ae663e384eb1b85/zKWa8h6YU4BWfcitpM5Pl.png","isPro":false,"fullname":"Atsuyuki Miyai","user":"AtsuMiyai","type":"user"},{"_id":"6342796a0875f2c99cfd313b","avatarUrl":"/avatars/98575092404c4197b20c929a6499a015.svg","isPro":false,"fullname":"Yuseung \"Phillip\" Lee","user":"phillipinseoul","type":"user"},{"_id":"63ac5701c21e60a3e9b58aa7","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/63ac5701c21e60a3e9b58aa7/g6EX7diOpuA94R2ab-rZC.png","isPro":true,"fullname":"Dipankar Sarkar","user":"dipankarsarkar","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"6730c40cfab94f649285ded2","name":"hal-utokyo","fullname":"Hal Lab UTokyo","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/65d86a59bdb95b4bbc744e10/FOjSKBb6nGV-BBRp9fQU5.png"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2608/2608.20169.md","query":{}}">
Task-CoEvolve: Efficient Harness Optimization via Adaptive Validation Task Selection
Abstract
Task-CoEvolve improves LLM harness optimization by adaptively selecting validation tasks and estimating full-set performance from partial evaluations, cutting evaluation costs by 80%.
We present a novel approach to efficient LLM harness optimization through adaptive validation task selection. Harness optimization iteratively rewrites the harness code based on validation performance, enabling substantial performance gains without updating the underlying model weights. Existing approaches, however, evaluate a fixed validation set in full at every iteration, incurring substantial evaluation costs even on tasks that become less discriminative as the harness evolves. We propose Task-CoEvolve, which co-evolves the validation tasks with the harness by addressing two challenges: selecting informative tasks and estimating full-set performance from partial evaluations. Task-CoEvolve builds on the observation that tasks on which candidate harnesses disagree are more informative for distinguishing among them than tasks that are consistently solved or failed. It uses variance-weighted sampling based on past outcomes to focus evaluation on tasks near the capability frontier, with the sampling distribution adapting as the harness evolves. It then estimates full-set scores from the sampled tasks by accounting for their sampling probabilities, enabling consistent comparisons across iterations despite evaluating different subsets. Experiments on online text classification and Terminal-Bench 2.1 show that Task-CoEvolve consistently outperforms subset-based baselines and matches the final performance of full-set search while reducing the number of evaluations during optimization by 80%. Code will be released at https://github.com/Agent4Science-UTokyo/Task-CoEvolve.
Community
A major challenge for scaling recursive self-improvement, especially in harness optimization, is the cost of evolution.
We introduce a fundamentally different approach that cuts the evolution cost of existing harness optimization by up to 80%.
🚀 Introducing Task-CoEvolve
Existing approaches focus on using fewer candidates or fewer iterations. Task-CoEvolve takes an orthogonal direction: instead of reducing the search itself, we reduce the number of evaluation tasks at each iteration.
💡 The key idea is simple: not every validation task needs to be evaluated at every iteration.
Task-CoEvolve co-evolves the validation tasks together with the harness:
- Select informative tasks based on the evolving harnesses
- Adaptively update which tasks are evaluated
- Estimate full-set performance from partial evaluations
📊 Results
On online text classification and Terminal-Bench 2.1, Task-CoEvolve matches the performance of full-set search using only 20% of the evaluations (reducing the overall search cost by 67–80% in Terminal-Bench 2.1)🔥
📄 https://arxiv.org/abs/2608.20169
Github:
https://github.com/Agent4Science-UTokyo/Task-CoEvolve
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2608.20169 in a model README.md to link it from this page.
Cite arxiv.org/abs/2608.20169 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2608.20169 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.