Agent harnesses have substantially improved performance on long-horizon tasks by coordinating agent interactions with the environment. However, reinforcement learning through complex harnesses remains largely unexplored, as scaling such training to long-horizon agent tasks introduces fundamental challenges. In this work, we present a unified black-box RL framework for stable and scalable optimization of general agents through complex harnesses. Concretely, we first build a sandbox-based execution infrastructure that isolates task environments and harnesses within temporary sandboxes for large-scale concurrent rollouts. We then decouple policy optimization from opaque harness execution and place a serving proxy at the model boundary to capture model calls. To reconstruct multi-turn trajectories and improve training efficiency, we organize the captured calls into prefix trees and further adapt both critic-based PPO and critic-free GRPO to optimize over the recovered tree structure. Meanwhile, we maintain training-inference consistency throughout the optimization process. Finally, we introduce mix-harness training, allowing a single model to be jointly optimized by heterogeneous harnesses. With Qwen3-30A3B, black-box RL improves Pass@1 on ClawGym-Bench by 9.98 and 14.81 points through OpenClaw and Claude Code, respectively, while remaining stable over 200-400 optimization steps. Moreover, the framework yields consistent gains on more challenging tasks such as JobBench and OfficeQA. Overall, our framework enables effective, stable, and scalable optimization of general agents through black-box harnesses, supporting unified training across heterogeneous execution systems.</p>\n","updatedAt":"2026-08-18T02:46:57.487Z","author":{"_id":"66163dc8c7f45b3f893ff40b","avatarUrl":"/avatars/801043dac0caae90bbca8c9d3e2e203b.svg","fullname":"Song Huatong","name":"XXsongLALA","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":4,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.917172908782959},"editors":["XXsongLALA"],"editorAvatarUrls":["/avatars/801043dac0caae90bbca8c9d3e2e203b.svg"],"reactions":[{"reaction":"🚀","users":["daixuancheng"],"count":1}],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2608.16798","authors":[{"_id":"6a83c715675db694db8cd4ad","name":"Huatong Song","hidden":false},{"_id":"6a83c715675db694db8cd4ae","name":"Fei Bai","hidden":false},{"_id":"6a83c715675db694db8cd4af","name":"Ming Yang","hidden":false},{"_id":"6a83c715675db694db8cd4b0","name":"Renyuan Li","hidden":false},{"_id":"6a83c715675db694db8cd4b1","name":"Jia Deng","hidden":false},{"_id":"6a83c715675db694db8cd4b2","name":"Jujie He","hidden":false},{"_id":"6a83c715675db694db8cd4b3","name":"Zhange Zhang","hidden":false},{"_id":"6a83c715675db694db8cd4b4","user":{"_id":"649e6761f9134a06ed1e0cea","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/649e6761f9134a06ed1e0cea/XNeKceE8xSwI0xWwWUwwJ.jpeg","isPro":false,"fullname":"Daixuan Cheng","user":"daixuancheng","type":"user","name":"daixuancheng"},"name":"Daixuan Cheng","status":"claimed_verified","statusLastChangedAt":"2026-08-18T08:45:05.135Z","hidden":false},{"_id":"6a83c715675db694db8cd4b5","name":"Yan Xing","hidden":false},{"_id":"6a83c715675db694db8cd4b6","name":"Qi Yun","hidden":false},{"_id":"6a83c715675db694db8cd4b7","name":"Xuxing Chen","hidden":false},{"_id":"6a83c715675db694db8cd4b8","name":"Danyang Li","hidden":false},{"_id":"6a83c715675db694db8cd4b9","name":"Feng Chang","hidden":false},{"_id":"6a83c715675db694db8cd4ba","name":"Chuan Hao","hidden":false},{"_id":"6a83c715675db694db8cd4bb","name":"Ran Tao","hidden":false},{"_id":"6a83c715675db694db8cd4bc","name":"Jian Yang","hidden":false},{"_id":"6a83c715675db694db8cd4bd","name":"Bryan Dai","hidden":false},{"_id":"6a83c715675db694db8cd4be","name":"Wayne Xin Zhao","hidden":false},{"_id":"6a83c715675db694db8cd4bf","name":"Mingjie Tang","hidden":false},{"_id":"6a83c715675db694db8cd4c0","name":"Ji-Rong Wen","hidden":false}],"publishedAt":"2026-08-17T00:00:00.000Z","submittedOnDailyAt":"2026-08-18T00:00:00.000Z","title":"ClawGym II: Exploring Black-Box RL on Agent Harness","submittedOnDailyBy":{"_id":"66163dc8c7f45b3f893ff40b","avatarUrl":"/avatars/801043dac0caae90bbca8c9d3e2e203b.svg","isPro":false,"fullname":"Song Huatong","user":"XXsongLALA","type":"user","name":"XXsongLALA"},"summary":"Agent harnesses have substantially improved performance on long-horizon tasks by coordinating agent interactions with the environment. However, reinforcement learning through complex harnesses remains largely unexplored, as scaling such training to long-horizon agent tasks introduces fundamental challenges. In this work, we present a unified black-box RL framework for stable and scalable optimization of general agents through complex harnesses. Concretely, we first build a sandbox-based execution infrastructure that isolates task environments and harnesses within temporary sandboxes for large-scale concurrent rollouts. We then decouple policy optimization from opaque harness execution and place a serving proxy at the model boundary to capture model calls. To reconstruct multi-turn trajectories and improve training efficiency, we organize the captured calls into prefix trees and further adapt both critic-based PPO and critic-free GRPO to optimize over the recovered tree structure. Meanwhile, we maintain training-inference consistency throughout the optimization process. Finally, we introduce mix-harness training, allowing a single model to be jointly optimized by heterogeneous harnesses. With Qwen3-30A3B, black-box RL improves Pass@1 on ClawGym-Bench by 9.98 and 14.81 points through OpenClaw and Claude Code, respectively, while remaining stable over 200-400 optimization steps. Moreover, the framework yields consistent gains on more challenging tasks such as JobBench and OfficeQA. Overall, our framework enables effective, stable, and scalable optimization of general agents through black-box harnesses, supporting unified training across heterogeneous execution systems.","upvotes":32,"discussionId":"6a83c716675db694db8cd4c1","projectPage":"https://github.com/ClawGym","githubRepo":"https://github.com/ClawGym/ClawGym-Agents","githubRepoAddedBy":"user","ai_summary":"A unified black-box reinforcement learning framework enables stable, scalable optimization of general agents through complex harnesses via sandbox execution, trajectory reconstruction, and mix-harness training.","ai_keywords":["black-box RL","reinforcement learning","PPO","GRPO","prefix trees","mix-harness training","sandbox-based execution","serving proxy","multi-turn trajectories"],"ai_summary_model":"thinkingmachines/Inkling-Small","githubStars":33,"organization":{"_id":"622177ac43826d6f261f8208","name":"RUC","fullname":"Renmin University of China","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/61ac8f8a00d01045fca0ad2f/670IAX9A2-BflqA5MiSBW.jpeg"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"66feabce98f30194f8b0c223","avatarUrl":"/avatars/d888eaaac303ddacd5826bb2434d9bb2.svg","isPro":false,"fullname":"Fei Bai","user":"Teaven","type":"user"},{"_id":"66163dc8c7f45b3f893ff40b","avatarUrl":"/avatars/801043dac0caae90bbca8c9d3e2e203b.svg","isPro":false,"fullname":"Song Huatong","user":"XXsongLALA","type":"user"},{"_id":"6599f69617edd1f0537ee8d8","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/GpIurlha91mUA-u1FDSnl.jpeg","isPro":false,"fullname":"Yanzipeng Gao","user":"O2iginal","type":"user"},{"_id":"674476e821e39628723f13ad","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/7iUE9VSdvuX979_vERXpI.png","isPro":false,"fullname":"mfzzzzzz","user":"mfzzzzzz","type":"user"},{"_id":"6334f71031a2be3938c8b3f3","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1664415607197-6334f71031a2be3938c8b3f3.jpeg","isPro":false,"fullname":"txy","user":"txy","type":"user"},{"_id":"61d78857a21a9b49c7e8e4a9","avatarUrl":"/avatars/c7e7f84cad775be2d13fab8530bf21f5.svg","isPro":false,"fullname":"Yifan Du","user":"Richard1999","type":"user"},{"_id":"63f06116f1a47aaea5bd497b","avatarUrl":"/avatars/7d99ffa59c4579599e852a0ffb261268.svg","isPro":false,"fullname":"Guoxin Chen","user":"GuoxinChen","type":"user"},{"_id":"65c747f1bbc318a59eceb452","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/65c747f1bbc318a59eceb452/W5ERLsLFmwhbt-blcNslJ.jpeg","isPro":false,"fullname":"Shuang Sun","user":"SNHE","type":"user"},{"_id":"655af1093ff5ba1b1b42a8ce","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/655af1093ff5ba1b1b42a8ce/HGM_BjTTfiZgXit_jaU4p.png","isPro":false,"fullname":"Miraclys","user":"Miraclys","type":"user"},{"_id":"63ac5701c21e60a3e9b58aa7","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/63ac5701c21e60a3e9b58aa7/g6EX7diOpuA94R2ab-rZC.png","isPro":true,"fullname":"Dipankar Sarkar","user":"dipankarsarkar","type":"user"},{"_id":"616648c84c0937d31946f21b","avatarUrl":"/avatars/7ca27de5c5116c91ff1db61ba6277ed5.svg","isPro":false,"fullname":"Ziyang","user":"hzy","type":"user"},{"_id":"649e6761f9134a06ed1e0cea","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/649e6761f9134a06ed1e0cea/XNeKceE8xSwI0xWwWUwwJ.jpeg","isPro":false,"fullname":"Daixuan Cheng","user":"daixuancheng","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":3,"organization":{"_id":"622177ac43826d6f261f8208","name":"RUC","fullname":"Renmin University of China","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/61ac8f8a00d01045fca0ad2f/670IAX9A2-BflqA5MiSBW.jpeg"},"query":{}}">
ClawGym II: Exploring Black-Box RL on Agent Harness
Abstract
A unified black-box reinforcement learning framework enables stable, scalable optimization of general agents through complex harnesses via sandbox execution, trajectory reconstruction, and mix-harness training.
Agent harnesses have substantially improved performance on long-horizon tasks by coordinating agent interactions with the environment. However, reinforcement learning through complex harnesses remains largely unexplored, as scaling such training to long-horizon agent tasks introduces fundamental challenges. In this work, we present a unified black-box RL framework for stable and scalable optimization of general agents through complex harnesses. Concretely, we first build a sandbox-based execution infrastructure that isolates task environments and harnesses within temporary sandboxes for large-scale concurrent rollouts. We then decouple policy optimization from opaque harness execution and place a serving proxy at the model boundary to capture model calls. To reconstruct multi-turn trajectories and improve training efficiency, we organize the captured calls into prefix trees and further adapt both critic-based PPO and critic-free GRPO to optimize over the recovered tree structure. Meanwhile, we maintain training-inference consistency throughout the optimization process. Finally, we introduce mix-harness training, allowing a single model to be jointly optimized by heterogeneous harnesses. With Qwen3-30A3B, black-box RL improves Pass@1 on ClawGym-Bench by 9.98 and 14.81 points through OpenClaw and Claude Code, respectively, while remaining stable over 200-400 optimization steps. Moreover, the framework yields consistent gains on more challenging tasks such as JobBench and OfficeQA. Overall, our framework enables effective, stable, and scalable optimization of general agents through black-box harnesses, supporting unified training across heterogeneous execution systems.
Community
Agent harnesses have substantially improved performance on long-horizon tasks by coordinating agent interactions with the environment. However, reinforcement learning through complex harnesses remains largely unexplored, as scaling such training to long-horizon agent tasks introduces fundamental challenges. In this work, we present a unified black-box RL framework for stable and scalable optimization of general agents through complex harnesses. Concretely, we first build a sandbox-based execution infrastructure that isolates task environments and harnesses within temporary sandboxes for large-scale concurrent rollouts. We then decouple policy optimization from opaque harness execution and place a serving proxy at the model boundary to capture model calls. To reconstruct multi-turn trajectories and improve training efficiency, we organize the captured calls into prefix trees and further adapt both critic-based PPO and critic-free GRPO to optimize over the recovered tree structure. Meanwhile, we maintain training-inference consistency throughout the optimization process. Finally, we introduce mix-harness training, allowing a single model to be jointly optimized by heterogeneous harnesses. With Qwen3-30A3B, black-box RL improves Pass@1 on ClawGym-Bench by 9.98 and 14.81 points through OpenClaw and Claude Code, respectively, while remaining stable over 200-400 optimization steps. Moreover, the framework yields consistent gains on more challenging tasks such as JobBench and OfficeQA. Overall, our framework enables effective, stable, and scalable optimization of general agents through black-box harnesses, supporting unified training across heterogeneous execution systems.
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2608.16798 in a model README.md to link it from this page.
Cite arxiv.org/abs/2608.16798 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2608.16798 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.