Hugging Face Daily Papers · September 3, 2026 · 5 min read

HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?

Mirrored from Hugging Face Daily Papers for archival readability. Support the source by reading on the original site.

Like Read original ↗

We introduce HarnessDev, a benchmark that evaluates whether LLMs can create and iteratively improve the agent harnesses that shape their downstream performance. Across six creator LLMs, four domains, and five benchmarks, generated harnesses show promise but still lag mature human-engineered systems in code, search, and research, while evolution gains are often unstable and dependent on the runtime model. Project page: <a href=\"https://self-developing-agents.github.io/\" rel=\"nofollow\">https://self-developing-agents.github.io/</a></p>\n","updatedAt":"2026-09-03T03:44:32.431Z","author":{"_id":"63369da91ba5d5ece24118a4","avatarUrl":"/avatars/67889e1ecadb04100a77bc8b5284c6fd.svg","fullname":"wuyuhao","name":"mozhu","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":6,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8996512293815613},"editors":["mozhu"],"editorAvatarUrls":["/avatars/67889e1ecadb04100a77bc8b5284c6fd.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2609.01437","authors":[{"_id":"6a98ec9cfea818274321ff28","name":"Yuhao Wu","hidden":false},{"_id":"6a98ec9cfea818274321ff29","name":"Jingyuan Zhang","hidden":false},{"_id":"6a98ec9cfea818274321ff2a","name":"Jiajun Shi","hidden":false},{"_id":"6a98ec9cfea818274321ff2b","name":"Xinping Lei","hidden":false},{"_id":"6a98ec9cfea818274321ff2c","name":"Qingshui Gu","hidden":false},{"_id":"6a98ec9cfea818274321ff2d","name":"Yuxuan Zhang","hidden":false},{"_id":"6a98ec9cfea818274321ff2e","name":"Zexuan Wang","hidden":false},{"_id":"6a98ec9cfea818274321ff2f","name":"Chen He","hidden":false},{"_id":"6a98ec9cfea818274321ff30","name":"Chen Huang","hidden":false},{"_id":"6a98ec9cfea818274321ff31","name":"Maojia Song","hidden":false},{"_id":"6a98ec9cfea818274321ff32","name":"Zhiyuan Zeng","hidden":false},{"_id":"6a98ec9cfea818274321ff33","name":"Shaowen Wang","hidden":false},{"_id":"6a98ec9cfea818274321ff34","name":"Jinkai Liu","hidden":false},{"_id":"6a98ec9cfea818274321ff35","name":"Yunfeng Shi","hidden":false},{"_id":"6a98ec9cfea818274321ff36","name":"Jiaheng Liu","hidden":false},{"_id":"6a98ec9cfea818274321ff37","name":"Shen Yan","hidden":false},{"_id":"6a98ec9cfea818274321ff38","name":"Wenhao Huang","hidden":false},{"_id":"6a98ec9cfea818274321ff39","name":"Ge Zhang","hidden":false},{"_id":"6a98ec9cfea818274321ff3a","name":"Wenxuan Zhang","hidden":false}],"publishedAt":"2026-09-01T00:00:00.000Z","submittedOnDailyAt":"2026-09-03T00:00:00.000Z","title":"HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?","submittedOnDailyBy":{"_id":"63369da91ba5d5ece24118a4","avatarUrl":"/avatars/67889e1ecadb04100a77bc8b5284c6fd.svg","isPro":false,"fullname":"wuyuhao","user":"mozhu","type":"user","name":"mozhu"},"summary":"As agents move from research prototypes to deployed tools, their capability increasingly depends on model-external execution infrastructure, commonly termed the agent harness. Changing this harness while holding model weights fixed can substantially alter task performance. Current agent evaluations typically report downstream performance under a chosen harness, leaving a model's ability to develop the harness itself comparatively underexplored. We introduce HarnessDev, a benchmark that shifts the unit of evaluation from task outputs to runnable infrastructure. HarnessDev covers two stages. In Creation, the agent starts from a minimal seed and a small number of cases, then builds a complete execution system. In Evolution, it starts from its own created harness and iteratively revises it using downstream execution feedback, with the goal of improving benchmark performance. We then evaluate each constructed harness on capability (task success on held-out benchmarks) and efficiency (execution-token cost). The reported Creation results cover six creator LLMs, four domains, and five downstream benchmarks totaling 2,207 unique downstream instances, with hidden evaluation tasks withheld from development. We find that generated harnesses remain substantially behind mature human-engineered references on code and on search and research, while matching or exceeding the selected references on writing and machine-learning experimentation, with large variation in execution cost. Evolution produces some performance gains, but they are unstable and transfer only partially to held-out tasks. Experiments with a fixed runtime model further show that the gains depend strongly on the model executing the harness, indicating limited transfer across models.","upvotes":17,"discussionId":"6a98ec9dfea818274321ff3b","projectPage":"https://self-developing-agents.github.io/","ai_summary":"HarnessDev evaluates agents by measuring their ability to build and iteratively improve execution infrastructure rather than final task outputs, revealing that self-built harnesses vary widely in capability and efficiency and transfer poorly across models.","ai_keywords":["agent harness","HarnessDev","execution infrastructure","LLMs","held-out benchmarks","execution-token cost","harness evolution"],"ai_summary_model":"thinkingmachines/Inkling-Small","organization":{"_id":"67d1140985ea0644e2f14b99","name":"ByteDance-Seed","fullname":"ByteDance Seed","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/6535c9e88bde2fae19b6fb25/flkDUqd_YEuFsjeNET3r-.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"63369da91ba5d5ece24118a4","avatarUrl":"/avatars/67889e1ecadb04100a77bc8b5284c6fd.svg","isPro":false,"fullname":"wuyuhao","user":"mozhu","type":"user"},{"_id":"6468ed2ae134d050a58b4c4e","avatarUrl":"/avatars/e54f43f80b5da6ab12ace5d34fef3d1f.svg","isPro":false,"fullname":"mathsfan-spark","user":"ml-llm-12345","type":"user"},{"_id":"639ae8dfb49b726255975f86","avatarUrl":"/avatars/3361477fb2de29eaea5484696b2721c6.svg","isPro":false,"fullname":"xushaoyang","user":"beiweixiaoxu","type":"user"},{"_id":"685a9f5b6f785247e5ed562f","avatarUrl":"/avatars/b391f5c045d40ee5afc69d072a4d7f3a.svg","isPro":false,"fullname":"ZexuanWang","user":"ZexuanWang","type":"user"},{"_id":"636f526a6cd69d9a36ff2b53","avatarUrl":"/avatars/8f2271a193fcac609d9be270552b5afa.svg","isPro":false,"fullname":"Qiguang Chen","user":"LightChen2333","type":"user"},{"_id":"64f0b56a87d05e740359279d","avatarUrl":"/avatars/8934a2e2bf4bcbbc3ae9d75e318c80b7.svg","isPro":false,"fullname":"Yiyang Zhang","user":"Anti-clockwise","type":"user"},{"_id":"638efcf4c67af472d316d424","avatarUrl":"/avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg","isPro":false,"fullname":"Ge Zhang","user":"zhangysk","type":"user"},{"_id":"6578265ddea7e2122d02f6ba","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6578265ddea7e2122d02f6ba/Bh6JjoVF5ceLSjV7Z7nTk.jpeg","isPro":false,"fullname":"king zhu","user":"kangz","type":"user"},{"_id":"6a2da6c8ca070ee12c6e396c","avatarUrl":"/avatars/0355287dcabaa67dbc7f0b10b87451f9.svg","isPro":false,"fullname":"Joe Mama","user":"JoeMama123123123","type":"user"},{"_id":"63ac5701c21e60a3e9b58aa7","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/63ac5701c21e60a3e9b58aa7/g6EX7diOpuA94R2ab-rZC.png","isPro":true,"fullname":"Dipankar Sarkar","user":"dipankarsarkar","type":"user"},{"_id":"6a6383d5db1f1187506ef6bd","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/Z9PQTUkr5xjFP-y20_guD.png","isPro":false,"fullname":"Ignacio Ramírez","user":"altairBASIC","type":"user"},{"_id":"620783f24e28382272337ba4","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/620783f24e28382272337ba4/zkUveQPNiDfYjgGhuFErj.jpeg","isPro":false,"fullname":"GuoLiangTang","user":"Tommy930","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"67d1140985ea0644e2f14b99","name":"ByteDance-Seed","fullname":"ByteDance Seed","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/6535c9e88bde2fae19b6fb25/flkDUqd_YEuFsjeNET3r-.png"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2609/2609.01437.md","query":{}}">

Papers

arxiv:2609.01437

HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?

Published on Sep 1

· Submitted by

wuyuhao on Sep 3

ByteDance Seed

Upvote

Authors:

Abstract

HarnessDev evaluates agents by measuring their ability to build and iteratively improve execution infrastructure rather than final task outputs, revealing that self-built harnesses vary widely in capability and efficiency and transfer poorly across models.

Generated by thinkingmachines/Inkling-Small

As agents move from research prototypes to deployed tools, their capability increasingly depends on model-external execution infrastructure, commonly termed the agent harness. Changing this harness while holding model weights fixed can substantially alter task performance. Current agent evaluations typically report downstream performance under a chosen harness, leaving a model's ability to develop the harness itself comparatively underexplored. We introduce HarnessDev, a benchmark that shifts the unit of evaluation from task outputs to runnable infrastructure. HarnessDev covers two stages. In Creation, the agent starts from a minimal seed and a small number of cases, then builds a complete execution system. In Evolution, it starts from its own created harness and iteratively revises it using downstream execution feedback, with the goal of improving benchmark performance. We then evaluate each constructed harness on capability (task success on held-out benchmarks) and efficiency (execution-token cost). The reported Creation results cover six creator LLMs, four domains, and five downstream benchmarks totaling 2,207 unique downstream instances, with hidden evaluation tasks withheld from development. We find that generated harnesses remain substantially behind mature human-engineered references on code and on search and research, while matching or exceeding the selected references on writing and machine-learning experimentation, with large variation in execution cost. Evolution produces some performance gains, but they are unstable and transfer only partially to held-out tasks. Experiments with a fixed runtime model further show that the gains depend strongly on the model executing the harness, indicating limited transfer across models.

View arXiv page View PDF Project page Add to collection