🚀 We are excited to share <strong>Beyond Final Scores: A Systematic Evaluation of Agents for Long-Horizon AI Research and Development</strong>.</p>\n<p>🔍 As AI agents increasingly tackle long-horizon research and engineering tasks, <strong>evaluating them by final scores alone is no longer enough. How do they actually approach, execute, and improve throughout the research process?</strong></p>\n<p>We systematically evaluate <strong>7 frontier models across 36 long-horizon tasks</strong>, going beyond final performance to examine how agents <strong>frame solutions, execute experiments, respond to feedback, and reuse experience</strong>.</p>\n<p>🧑‍🔬 Our results suggest that today's agents are better characterized as <strong>engineering optimizers than fully autonomous researchers</strong>: they can formulate and implement practical solutions, but performance remains highly variable across runs, strong solutions largely adapt or combine established techniques, and genuine methodological novelty is still rare.</p>\n<p>📊 We also find that:</p>\n<ul>\n<li>similar final scores can hide very different process bottlenecks;</li>\n<li>experience reuse can either help or mislead later decisions;</li>\n<li>harness design substantially affects performance stability.</li>\n</ul>\n<p>💡 We hope this study provides a more fine-grained view of where current research agents succeed, where they fail, and what needs to improve next.</p>\n<p>Would love to hear your thoughts and discussions!</p>\n","updatedAt":"2026-08-17T01:57:03.758Z","author":{"_id":"64e4090f222b232f03fe5f63","avatarUrl":"/avatars/1e97328de374d726f64bf16528d36ca4.svg","fullname":"Wanli Yang","name":"WenDingY","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.919795572757721},"editors":["WenDingY"],"editorAvatarUrls":["/avatars/1e97328de374d726f64bf16528d36ca4.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2608.13417","authors":[{"_id":"6a813687b601d59c65281206","name":"Yiwei Li","hidden":false},{"_id":"6a813687b601d59c65281207","name":"Wanli Yang","hidden":false},{"_id":"6a813687b601d59c65281208","name":"Hexiang Tan","hidden":false},{"_id":"6a813687b601d59c65281209","name":"Xiangzhou Huang","hidden":false},{"_id":"6a813687b601d59c6528120a","name":"Zhengyu Chen","hidden":false},{"_id":"6a813687b601d59c6528120b","name":"Ziran Li","hidden":false},{"_id":"6a813687b601d59c6528120c","name":"Borun Chen","hidden":false},{"_id":"6a813687b601d59c6528120d","name":"Shanglin Lei","hidden":false},{"_id":"6a813687b601d59c6528120e","name":"Huaisheng Zhu","hidden":false},{"_id":"6a813687b601d59c6528120f","name":"Hao Tian","hidden":false},{"_id":"6a813687b601d59c65281210","name":"Fei Sun","hidden":false},{"_id":"6a813687b601d59c65281211","name":"Xunliang Cai","hidden":false},{"_id":"6a813687b601d59c65281212","name":"Jingang Wang","hidden":false}],"mediaUrls":["https://cdn-uploads.huggingface.co/production/uploads/64e4090f222b232f03fe5f63/W0BAGDdouou6eC-bxo4Be.png"],"publishedAt":"2026-08-13T00:00:00.000Z","submittedOnDailyAt":"2026-08-17T00:00:00.000Z","title":"Beyond Final Scores: A Systematic Evaluation of Agents for Long-Horizon AI Research and Development","submittedOnDailyBy":{"_id":"64e4090f222b232f03fe5f63","avatarUrl":"/avatars/1e97328de374d726f64bf16528d36ca4.svg","isPro":false,"fullname":"Wanli Yang","user":"WenDingY","type":"user","name":"WenDingY"},"summary":"Autonomous agents are increasingly capable of improving models, systems, and other technical artifacts through long-horizon experimentation. To understand the current state of this capability, however, evaluation must go beyond final scores, which neither reveal where progress is gained or lost nor indicate whether accumulated experience improves later decisions. We therefore present a systematic evaluation of seven frontier models on 36 long-horizon tasks based on a new framework that uses rule-based metrics to characterize within-run behavior through Solution Framing, Execution, and Feedback Control and controlled comparisons to assess experience reuse within and across tasks. The results show that current agents operate more like engineering optimizers than fully autonomous researchers: they can formulate and implement practical solutions, but their performance varies substantially across runs, their strongest solutions mainly adapt or combine established techniques, and genuine methodological novelty remains rare. Detailed analysis reveals that observed performance is shaped by multiple factors, including distinct process bottlenecks behind similar final outcomes, experience reuse that can help or mislead subsequent decisions, and harness designs that affect performance stability. These findings suggest concrete directions for improving model training, inference-time strategies, experience management, and harness design.","upvotes":26,"discussionId":"6a813688b601d59c65281213","projectPage":"https://yiwei98.github.io/AutoResearchEval","ai_summary":"Frontier autonomous agents excel at engineering optimization but show unstable performance, limited novelty, and variable experience reuse across long-horizon tasks.","ai_keywords":["autonomous agents","long-horizon experimentation","rule-based metrics","Solution Framing","Execution","Feedback Control","experience reuse","engineering optimizers","methodological novelty","harness design"],"ai_summary_model":"thinkingmachines/Inkling-Small","organization":{"_id":"68b28d79a176a9beb30d2049","name":"meituan-longcat","fullname":"LongCat","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/68a2a29ab9d4c5698e02c747/CDCAx7X7rXDt7xjI-DoxG.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"64e4090f222b232f03fe5f63","avatarUrl":"/avatars/1e97328de374d726f64bf16528d36ca4.svg","isPro":false,"fullname":"Wanli Yang","user":"WenDingY","type":"user"},{"_id":"65f51459b6941db5c20512e0","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/65f51459b6941db5c20512e0/55RXnnpqFtBa4WtP3b8od.jpeg","isPro":false,"fullname":"antiquality","user":"antiquality","type":"user"},{"_id":"68da4c0442241d2b33e8c221","avatarUrl":"/avatars/4a4846028e5963cabf0f15111631a77a.svg","isPro":false,"fullname":"Fei Sun","user":"feisun-sf","type":"user"},{"_id":"69edaa8521554af9c4190c3e","avatarUrl":"/avatars/cc09691364d4a5b39c1c481ebf7a6173.svg","isPro":false,"fullname":"Shixuan Zhang","user":"Charlieaaa","type":"user"},{"_id":"64bf898d979949d2e2585c9a","avatarUrl":"/avatars/da77c856ec997e2b812c06272a01c8b2.svg","isPro":false,"fullname":"mengruwang","user":"mengru","type":"user"},{"_id":"647097cbcfd57849518e656b","avatarUrl":"/avatars/c66fe0add29c1bde9e3a98bf4a8793b9.svg","isPro":false,"fullname":"Jingang Wang","user":"bitwjg","type":"user"},{"_id":"6401690a52fb66b80d1f8975","avatarUrl":"/avatars/561dd0584150eb29b3a62ffdd650de7b.svg","isPro":false,"fullname":"Tan-Hexiang","user":"Tan-Hexiang","type":"user"},{"_id":"69819fe7bcb55c9348156f8e","avatarUrl":"/avatars/e2ce16ac9e205335fdabed2b36c1fa71.svg","isPro":false,"fullname":"Jing Tang","user":"asterqingyun","type":"user"},{"_id":"63ac5701c21e60a3e9b58aa7","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/63ac5701c21e60a3e9b58aa7/g6EX7diOpuA94R2ab-rZC.png","isPro":true,"fullname":"Dipankar Sarkar","user":"dipankarsarkar","type":"user"},{"_id":"6825a69b0797c68013e88a3e","avatarUrl":"/avatars/742d3e22b8b042f1a751653c995baafe.svg","isPro":false,"fullname":"Honglin Wang","user":"Leo-WHL","type":"user"},{"_id":"641a6f0619fc5647be192b39","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/641a6f0619fc5647be192b39/V2pWirFwkrxhweKuWY_4G.jpeg","isPro":false,"fullname":"Yihang Wang","user":"Bool1020","type":"user"},{"_id":"6227226bb6377a6ec0168400","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6227226bb6377a6ec0168400/VYRQdAKVtAF-O_voXYHMP.jpeg","isPro":false,"fullname":"Turbo Pascal","user":"TurboPascal","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":3,"organization":{"_id":"68b28d79a176a9beb30d2049","name":"meituan-longcat","fullname":"LongCat","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/68a2a29ab9d4c5698e02c747/CDCAx7X7rXDt7xjI-DoxG.png"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2608/2608.13417.md","query":{}}">

Get this paper in your agent:

Don't have the latest CLI?

Cite arxiv.org/abs/2608.13417 in a model README.md to link it from this page.

Cite arxiv.org/abs/2608.13417 in a dataset README.md to link it from this page.

Cite arxiv.org/abs/2608.13417 in a Space README.md to link it from this page.

Add this paper to a collection to link it from this page.