Reinforcement learning (RL) has emerged as a powerful approach for improving reasoning in language and vision-language models, yet its strongest successes still depend heavily on ground-truth supervision (e.g., verifiable reward). Such annotations are costly to obtain and become increasingly scarce as reasoning capabilities advance beyond what humans can reliably evaluate. Self-rewarding RL reduces this dependence by enabling models to derive reward signals from their own completions. However, training solely on self-generated feedback can reinforce existing biases and suboptimal behaviors, reduce response diversity, and ultimately lead to homogenized responses and training collapse. In this work, we show that unsupervised reasoning can emerge through cooperative multi-agent training. We introduce Co-RL, a framework in which multiple decoupled models, sharing no parameters, are simultaneously optimized through RL using rewards derived from their peers. We further show that increasing cohort diversity, through heterogeneous model families, sizes, and rephrased training samples, reduces the correlated errors that drive self-reinforcing feedback loops. This diversity consistently improves reasoning performance, maintains behavioral diversity, and mitigates training collapse. Across text-only and multimodal domains, Co-RL consistently outperforms the base models and prior label-free approaches, while matching or surpassing supervised methods, without access to any ground-truth labels. Concretely, Co-RL yields average gains of 3.0-8.6% across seven text-only benchmarks for LLMs and 2.3-7.2% across four multimodal benchmarks for VLMs.</p>\n","updatedAt":"2026-08-20T04:38:28.706Z","author":{"_id":"6419309f22270b3ccf177c77","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6419309f22270b3ccf177c77/KQa1586iBBKqucUlfpuPp.jpeg","fullname":"William Li","name":"williamium","type":"user","isPro":true,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":8,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.9299910664558411},"editors":["williamium"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/6419309f22270b3ccf177c77/KQa1586iBBKqucUlfpuPp.jpeg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2608.17253","authors":[{"_id":"6a8682addb13816030683f11","name":"Yunhao Yang","hidden":false},{"_id":"6a8682addb13816030683f12","name":"Yuexin Bian","hidden":false},{"_id":"6a8682addb13816030683f13","name":"Yunjie Tian","hidden":false},{"_id":"6a8682addb13816030683f14","name":"Di Fu","hidden":false},{"_id":"6a8682addb13816030683f15","name":"Tianjin Huang","hidden":false},{"_id":"6a8682addb13816030683f16","name":"Yuanyuan Shi","hidden":false},{"_id":"6a8682addb13816030683f17","name":"Ziang Xiao","hidden":false},{"_id":"6a8682addb13816030683f18","name":"Nuno Vasconcelos","hidden":false},{"_id":"6a8682addb13816030683f19","name":"Yijiang Li","hidden":false}],"publishedAt":"2026-08-19T00:00:00.000Z","submittedOnDailyAt":"2026-08-20T00:00:00.000Z","title":"Co-RL: Unsupervised Reasoning Emerges from Diverse Cohort in Multi-agent RL","submittedOnDailyBy":{"_id":"6419309f22270b3ccf177c77","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6419309f22270b3ccf177c77/KQa1586iBBKqucUlfpuPp.jpeg","isPro":true,"fullname":"William Li","user":"williamium","type":"user","name":"williamium"},"summary":"Reinforcement learning (RL) has emerged as a powerful approach for improving reasoning in language and vision-language models, yet its strongest successes still depend heavily on ground-truth supervision (e.g., verifiable reward). Such annotations are costly to obtain and become increasingly scarce as reasoning capabilities advance beyond what humans can reliably evaluate. Self-rewarding RL reduces this dependence by enabling models to derive reward signals from their own completions. However, training solely on self-generated feedback can reinforce existing biases and suboptimal behaviors, reduce response diversity, and ultimately lead to homogenized responses and training collapse. In this work, we show that unsupervised reasoning can emerge through cooperative multi-agent training. We introduce Co-RL, a framework in which multiple decoupled models, sharing no parameters, are simultaneously optimized through RL using rewards derived from their peers. We further show that increasing cohort diversity, through heterogeneous model families, sizes, and rephrased training samples, reduces the correlated errors that drive self-reinforcing feedback loops. This diversity consistently improves reasoning performance, maintains behavioral diversity, and mitigates training collapse. Across text-only and multimodal domains, Co-RL consistently outperforms the base models and prior label-free approaches, while matching or surpassing supervised methods, without access to any ground-truth labels. Concretely, Co-RL yields average gains of 3.0-8.6% across seven text-only benchmarks for LLMs and 2.3-7.2% across four multimodal benchmarks for VLMs. Code is available at https://github.com/DrStranded/Co-RL.","upvotes":73,"discussionId":"6a8682addb13816030683f1a","githubRepo":"https://github.com/DrStranded/Co-RL","githubRepoAddedBy":"user","ai_summary":"Co-RL enables unsupervised reasoning via cooperative multi-agent reinforcement learning with peer-derived rewards, improving performance across text and vision tasks without ground-truth labels.","ai_keywords":["reinforcement learning","self-rewarding RL","cooperative multi-agent training","Co-RL","peer-derived rewards","cohort diversity","correlated errors","training collapse","reasoning performance","multimodal benchmarks"],"ai_summary_model":"thinkingmachines/Inkling-Small","githubStars":1,"organization":{"_id":"697e87d12cc19315a8497001","name":"UCSanDiego","fullname":"University of California at San Diego","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/697e8687c00f332cf492d29e/KUQpvngxP4r9oBSDZwIwZ.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"6419309f22270b3ccf177c77","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6419309f22270b3ccf177c77/KQa1586iBBKqucUlfpuPp.jpeg","isPro":true,"fullname":"William Li","user":"williamium","type":"user"},{"_id":"6a7596d81edbab8018044390","avatarUrl":"/avatars/997866a2df54094f6b86512163dba8d5.svg","isPro":false,"fullname":"logan","user":"b1716523669","type":"user"},{"_id":"67bd2bdd8c436e4372ba27bb","avatarUrl":"/avatars/57a06a9463ceaeb85e0e9da3007e5b4a.svg","isPro":false,"fullname":"Yuexin Bian","user":"alwaysbyx","type":"user"},{"_id":"6a86858979a0d9529acd7ed9","avatarUrl":"/avatars/d369b401b845495b7e24164212696fc9.svg","isPro":false,"fullname":"Yuan Wang","user":"shakie77","type":"user"},{"_id":"694e6186c717ebb1689a274e","avatarUrl":"/avatars/53c8f9d4d4ac5068c466469c24095787.svg","isPro":true,"fullname":"Logan Yang","user":"q1716523669","type":"user"},{"_id":"662296e86bab5536e26a014e","avatarUrl":"/avatars/b0e774e7319a17462c2c0e5c7229c5ed.svg","isPro":false,"fullname":"YUAN ZHUANG","user":"paulzy","type":"user"},{"_id":"6a86866bcacf1083b5b04871","avatarUrl":"/avatars/3dd03f5daf06bbc961169254f32bc4e2.svg","isPro":false,"fullname":"Yujie ZHANG","user":"Croissant7","type":"user"},{"_id":"689f283f6563f6bd1b4bd3aa","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/cz5OrEj6d9-tnmAwEZ-aG.png","isPro":false,"fullname":"Haichao Zhang","user":"owlonoak","type":"user"},{"_id":"6a69eb40da85bffbe935dc52","avatarUrl":"/avatars/f7af2d579a31754981d459c35b9495c4.svg","isPro":false,"fullname":"Mary Hernandez","user":"hernandez-m","type":"user"},{"_id":"6a8689410d4882d585b0c2d4","avatarUrl":"/avatars/522fd6269251fce1dd56327d1eff693f.svg","isPro":false,"fullname":"Zidan Yu","user":"yyqxdetuimao","type":"user"},{"_id":"6a8689db9ce162693938fe1e","avatarUrl":"/avatars/8b6b4cdce4759d1060a1d55ee8411e04.svg","isPro":false,"fullname":"Ziqian Zhu","user":"zzqfankling","type":"user"},{"_id":"6a69eb62c5e36f5d1b3c77e4","avatarUrl":"/avatars/8481802389ca8628ac440d9b926ca9ac.svg","isPro":false,"fullname":"William Lopez","user":"wlopez-nlp","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":2,"organization":{"_id":"697e87d12cc19315a8497001","name":"UCSanDiego","fullname":"University of California at San Diego","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/697e8687c00f332cf492d29e/KUQpvngxP4r9oBSDZwIwZ.png"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2608/2608.17253.md","query":{}}">
Co-RL: Unsupervised Reasoning Emerges from Diverse Cohort in Multi-agent RL
Abstract
Co-RL enables unsupervised reasoning via cooperative multi-agent reinforcement learning with peer-derived rewards, improving performance across text and vision tasks without ground-truth labels.
Reinforcement learning (RL) has emerged as a powerful approach for improving reasoning in language and vision-language models, yet its strongest successes still depend heavily on ground-truth supervision (e.g., verifiable reward). Such annotations are costly to obtain and become increasingly scarce as reasoning capabilities advance beyond what humans can reliably evaluate. Self-rewarding RL reduces this dependence by enabling models to derive reward signals from their own completions. However, training solely on self-generated feedback can reinforce existing biases and suboptimal behaviors, reduce response diversity, and ultimately lead to homogenized responses and training collapse. In this work, we show that unsupervised reasoning can emerge through cooperative multi-agent training. We introduce Co-RL, a framework in which multiple decoupled models, sharing no parameters, are simultaneously optimized through RL using rewards derived from their peers. We further show that increasing cohort diversity, through heterogeneous model families, sizes, and rephrased training samples, reduces the correlated errors that drive self-reinforcing feedback loops. This diversity consistently improves reasoning performance, maintains behavioral diversity, and mitigates training collapse. Across text-only and multimodal domains, Co-RL consistently outperforms the base models and prior label-free approaches, while matching or surpassing supervised methods, without access to any ground-truth labels. Concretely, Co-RL yields average gains of 3.0-8.6% across seven text-only benchmarks for LLMs and 2.3-7.2% across four multimodal benchmarks for VLMs. Code is available at https://github.com/DrStranded/Co-RL.
Community
Reinforcement learning (RL) has emerged as a powerful approach for improving reasoning in language and vision-language models, yet its strongest successes still depend heavily on ground-truth supervision (e.g., verifiable reward). Such annotations are costly to obtain and become increasingly scarce as reasoning capabilities advance beyond what humans can reliably evaluate. Self-rewarding RL reduces this dependence by enabling models to derive reward signals from their own completions. However, training solely on self-generated feedback can reinforce existing biases and suboptimal behaviors, reduce response diversity, and ultimately lead to homogenized responses and training collapse. In this work, we show that unsupervised reasoning can emerge through cooperative multi-agent training. We introduce Co-RL, a framework in which multiple decoupled models, sharing no parameters, are simultaneously optimized through RL using rewards derived from their peers. We further show that increasing cohort diversity, through heterogeneous model families, sizes, and rephrased training samples, reduces the correlated errors that drive self-reinforcing feedback loops. This diversity consistently improves reasoning performance, maintains behavioral diversity, and mitigates training collapse. Across text-only and multimodal domains, Co-RL consistently outperforms the base models and prior label-free approaches, while matching or surpassing supervised methods, without access to any ground-truth labels. Concretely, Co-RL yields average gains of 3.0-8.6% across seven text-only benchmarks for LLMs and 2.3-7.2% across four multimodal benchmarks for VLMs.
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2608.17253 in a model README.md to link it from this page.
Cite arxiv.org/abs/2608.17253 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2608.17253 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.