How to train a critic reliably in LLM RL? </p>\n<p>No secret sauce. No novelty. Just implementation details that matter. </p>\n<p><a href=\"https://cdn-uploads.huggingface.co/production/uploads/63885f1d0bebb233d8ad6e5b/d_uD1oiKvnZ15F2jdjvBu.png\" rel=\"nofollow\"><img src=\"https://cdn-uploads.huggingface.co/production/uploads/63885f1d0bebb233d8ad6e5b/d_uD1oiKvnZ15F2jdjvBu.png\" alt=\"image\"></a></p>\n","updatedAt":"2026-08-26T02:38:21.334Z","author":{"_id":"63885f1d0bebb233d8ad6e5b","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1669881620925-noauth.jpeg","fullname":"Penghui Qi","name":"QPHutu","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":7,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.7479581832885742},"editors":["QPHutu"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1669881620925-noauth.jpeg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2608.23566","authors":[{"_id":"6a8e50d37bc881afa25f30bb","name":"Penghui Qi","hidden":false},{"_id":"6a8e50d37bc881afa25f30bc","name":"Xiangxin Zhou","hidden":false},{"_id":"6a8e50d37bc881afa25f30bd","name":"Wee Sun Lee","hidden":false}],"publishedAt":"2026-08-25T00:00:00.000Z","submittedOnDailyAt":"2026-08-26T00:00:00.000Z","title":"Best Practice Critic Optimization","submittedOnDailyBy":{"_id":"63885f1d0bebb233d8ad6e5b","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1669881620925-noauth.jpeg","isPro":false,"fullname":"Penghui Qi","user":"QPHutu","type":"user","name":"QPHutu"},"summary":"Group-based reinforcement learning methods such as GRPO for large language models avoid training a critic by sampling multiple responses for each prompt. A reliable critic could instead estimate token-level advantages from one response, but standard critic-based training recipes are often unstable. We study this instability and develop **Best Practice Critic Optimization (BPCO)**, a recipe that combines DPPO, value predictions bounded to the reward range, Monte Carlo value targets, unnormalized policy advantages, and length-adaptive generalized advantage estimation. Because the critic is used only during training, BPCO can also condition it on reward-defining information, such as a reference answer or grading rubric, that is hidden from the policy. Controlled experiments isolate the effect of each design choice. Across mathematical reasoning tasks with models ranging from 1.5B parameters to 30B-A3B mixtures of experts, BPCO improves a strong critic-based baseline consistently, and matches or exceeds a group-based baseline while sampling one response per prompt. The same recipe also improves learning with rubric-based rewards. These results show that a carefully designed critic provides a reliable alternative to group-relative advantage estimation. Code is available at https://github.com/QPHutu/golden_critic.","upvotes":7,"discussionId":"6a8e50d47bc881afa25f30be","githubRepo":"https://github.com/QPHutu/golden_critic","githubRepoAddedBy":"user","ai_summary":"BPCO stabilizes critic-based reinforcement learning for language models by combining bounded value predictions, Monte Carlo targets, and adaptive advantage estimation, matching group-based methods with single-response sampling.","ai_keywords":["GRPO","DPPO","critic optimization","value predictions","Monte Carlo value targets","generalized advantage estimation","token-level advantages","rubric-based rewards","mixture of experts"],"ai_summary_model":"thinkingmachines/Inkling-Small","githubStars":1,"organization":{"_id":"67c6b5eda8999849cc2739aa","name":"NUS-DSA3101","fullname":"National University of Singapore","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/67c6b59341330fa608335ff0/E8jB5V_iTM8isw_2tzpFv.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"63885f1d0bebb233d8ad6e5b","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1669881620925-noauth.jpeg","isPro":false,"fullname":"Penghui Qi","user":"QPHutu","type":"user"},{"_id":"67a1d21e33e92b4a1183f3bb","avatarUrl":"/avatars/43f9dd3fcb7d58ddc69562fd1fc12957.svg","isPro":false,"fullname":"Zhi Zheng","user":"zz1358m","type":"user"},{"_id":"66129c7b50350afe76757262","avatarUrl":"/avatars/a2f4fac076b9d658a0d904ed54960f6f.svg","isPro":false,"fullname":"Xiangxin Zhou","user":"zhouxiangxin","type":"user"},{"_id":"63ac5701c21e60a3e9b58aa7","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/63ac5701c21e60a3e9b58aa7/g6EX7diOpuA94R2ab-rZC.png","isPro":true,"fullname":"Dipankar Sarkar","user":"dipankarsarkar","type":"user"},{"_id":"65ae30620844d9e0d6215890","avatarUrl":"/avatars/6296f9afca06d31a960f5c220aa772f8.svg","isPro":false,"fullname":"XIA TIAN","user":"Summer66","type":"user"},{"_id":"682de3e6a8526009690754fa","avatarUrl":"/avatars/d24c55fdbaa709c84590bdaa559a9e6c.svg","isPro":false,"fullname":"Liu Yunhui","user":"liuyunhui","type":"user"},{"_id":"682de69104c8bebe87ce2051","avatarUrl":"/avatars/e1954981881f7b0b615aee910cd8e3cc.svg","isPro":false,"fullname":"Wang Yajun","user":"wangyejun","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"67c6b5eda8999849cc2739aa","name":"NUS-DSA3101","fullname":"National University of Singapore","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/67c6b59341330fa608335ff0/E8jB5V_iTM8isw_2tzpFv.png"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2608/2608.23566.md","query":{}}">
Best Practice Critic Optimization
Abstract
BPCO stabilizes critic-based reinforcement learning for language models by combining bounded value predictions, Monte Carlo targets, and adaptive advantage estimation, matching group-based methods with single-response sampling.
Group-based reinforcement learning methods such as GRPO for large language models avoid training a critic by sampling multiple responses for each prompt. A reliable critic could instead estimate token-level advantages from one response, but standard critic-based training recipes are often unstable. We study this instability and develop **Best Practice Critic Optimization (BPCO)**, a recipe that combines DPPO, value predictions bounded to the reward range, Monte Carlo value targets, unnormalized policy advantages, and length-adaptive generalized advantage estimation. Because the critic is used only during training, BPCO can also condition it on reward-defining information, such as a reference answer or grading rubric, that is hidden from the policy. Controlled experiments isolate the effect of each design choice. Across mathematical reasoning tasks with models ranging from 1.5B parameters to 30B-A3B mixtures of experts, BPCO improves a strong critic-based baseline consistently, and matches or exceeds a group-based baseline while sampling one response per prompt. The same recipe also improves learning with rubric-based rewards. These results show that a carefully designed critic provides a reliable alternative to group-relative advantage estimation. Code is available at https://github.com/QPHutu/golden_critic.
Community
How to train a critic reliably in LLM RL?
No secret sauce. No novelty. Just implementation details that matter.
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2608.23566 in a model README.md to link it from this page.
Cite arxiv.org/abs/2608.23566 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2608.23566 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.