We introduce TTPO, a label-free test-time training framework that remains robust even when majority-vote pseudo-labels are incorrect. TTPO uses an asymmetric objective: it distills agreeing rollouts while penalizing confident errors in disagreeing rollouts through grouped reinforcement learning.</p>\n","updatedAt":"2026-08-28T03:25:22.096Z","author":{"_id":"676127cf11b19ea602bb202a","avatarUrl":"/avatars/dfd802a24bd63e509728159ebb1769f6.svg","fullname":"Zhengxi Lu","name":"LZXzju","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":13,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8569393754005432},"editors":["LZXzju"],"editorAvatarUrls":["/avatars/dfd802a24bd63e509728159ebb1769f6.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2608.27448","authors":[{"_id":"6a90fd91a64059bab69c366f","name":"Aozhe Wang","hidden":false},{"_id":"6a90fd91a64059bab69c3670","user":{"_id":"676127cf11b19ea602bb202a","avatarUrl":"/avatars/dfd802a24bd63e509728159ebb1769f6.svg","isPro":false,"fullname":"Zhengxi Lu","user":"LZXzju","type":"user","name":"LZXzju"},"name":"Zhengxi Lu","status":"claimed_verified","statusLastChangedAt":"2026-08-28T08:45:04.711Z","hidden":false},{"_id":"6a90fd91a64059bab69c3671","name":"Jianze Wang","hidden":false},{"_id":"6a90fd91a64059bab69c3672","name":"Shangke Lv","hidden":false},{"_id":"6a90fd91a64059bab69c3673","name":"Ying Liu","hidden":false},{"_id":"6a90fd91a64059bab69c3674","name":"Weiming Lu","hidden":false},{"_id":"6a90fd91a64059bab69c3675","name":"Jun Xiao","hidden":false},{"_id":"6a90fd91a64059bab69c3676","name":"Yueting Zhuang","hidden":false},{"_id":"6a90fd91a64059bab69c3677","name":"Hua Yang","hidden":false},{"_id":"6a90fd91a64059bab69c3678","name":"Qianglong Chen","hidden":false},{"_id":"6a90fd91a64059bab69c3679","name":"Yongliang Shen","hidden":false}],"publishedAt":"2026-08-27T00:00:00.000Z","submittedOnDailyAt":"2026-08-28T00:00:00.000Z","title":"TTPO: Test-Time Policy Optimization","submittedOnDailyBy":{"_id":"676127cf11b19ea602bb202a","avatarUrl":"/avatars/dfd802a24bd63e509728159ebb1769f6.svg","isPro":false,"fullname":"Zhengxi Lu","user":"LZXzju","type":"user","name":"LZXzju"},"summary":"Recent prominent post-training methods, such as Reinforcement Learning (RL) and On-Policy Self-Distillation (OPSD), have driven rapid progress in mathematical reasoning for large language models, yet their reliance on ground-truth labels precludes test-time training (TTT). Replacing ground truth with majority-vote pseudo-labels is a natural alternative, yet it is fragile: an incorrect vote corrupts the teacher and misleads every token. We observe that this failure mode is asymmetric: rollouts that disagree with the pseudo-label are typically wrong regardless of whether the vote itself is correct. Building on this observation, we propose Test-Time Policy Optimization (TTPO), an asymmetric objective that distills agreeing rollouts via OPSD and penalizes disagreeing rollouts with Grouped RL. Token-level selection further refines both branches: distillation down-weights already-converged positions, while RL penalizes only confident errors. Both updates remain well-grounded even under frequent pseudo-label errors, and majority-vote routing yields tighter self-supervision as the model improves. Without any labels, TTPO matches label-supervised OPSD on five competition-level benchmarks, raises Qwen3-1.7B from 38.0% to 45.2% in TTT, yields +25.2% to +36.4% without thinking, and shows strong cross-task generalization.","upvotes":49,"discussionId":"6a90fd92a64059bab69c367a","projectPage":"https://zju-real.github.io/TTPO/","githubRepo":"https://github.com/ZJU-REAL/TTPO","githubRepoAddedBy":"user","ai_summary":"Test-Time Policy Optimization enables label-free test-time training for mathematical reasoning by asymmetrically distilling agreeing rollouts and penalizing disagreeing ones, matching supervised performance.","ai_keywords":["Reinforcement Learning","On-Policy Self-Distillation","test-time training","majority-vote pseudo-labels","Test-Time Policy Optimization","Grouped RL","token-level selection"],"ai_summary_model":"thinkingmachines/Inkling-Small","githubStars":8},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"681a1223600186111c8b017f","avatarUrl":"/avatars/d617188ad8c16df65fcf199047c4acde.svg","isPro":false,"fullname":"Aoshining","user":"Aoshining","type":"user"},{"_id":"67543820c3af453d7b3e1d5e","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/67543820c3af453d7b3e1d5e/RbAZ9AQlxpy5E5Is-QN8b.jpeg","isPro":false,"fullname":"Dingming Li","user":"lidingm","type":"user"},{"_id":"69be550b8bb23e7dc12f4306","avatarUrl":"/avatars/55ed79898dde00beba2d3ed14615e839.svg","isPro":false,"fullname":"Avicii Tom","user":"jianzewang","type":"user"},{"_id":"682c14409f1aeba16e13af66","avatarUrl":"/avatars/57660cd718e390b91134f8494ebefd3e.svg","isPro":false,"fullname":"Hai","user":"fiowhahf","type":"user"},{"_id":"6708edcae69f6e30a816af9f","avatarUrl":"/avatars/c4daa9b0cb2f4bb2a7db0e78b22034cb.svg","isPro":false,"fullname":"Yao","user":"distant-yuan","type":"user"},{"_id":"674572a99543fbaf3c63f35b","avatarUrl":"/avatars/6c891450c2ceeb7b034556548afc772d.svg","isPro":false,"fullname":"蔡正舟","user":"conctsai","type":"user"},{"_id":"66276727368ec2a0b933772c","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/66276727368ec2a0b933772c/ff5kAJuUr8tVlbNP1If_Y.jpeg","isPro":false,"fullname":"ccloud0525","user":"Ccloud0525","type":"user"},{"_id":"69045a2e9c3d523381fcf489","avatarUrl":"/avatars/8b1982e043a7ab28006b707dfff6c5dc.svg","isPro":false,"fullname":"JunlinLiu","user":"AaronLiu0702","type":"user"},{"_id":"6635c12178f0395f0b7f3602","avatarUrl":"/avatars/c6a86828266d7d3259b0c19b916227f7.svg","isPro":false,"fullname":"wu","user":"23davin","type":"user"},{"_id":"674214196d0146a7999cf6c3","avatarUrl":"/avatars/8b03275214aae60474e5a8d9099e3d68.svg","isPro":false,"fullname":"Hao Chen","user":"chhaounc","type":"user"},{"_id":"661d16214055a97ef95d031e","avatarUrl":"/avatars/2bbae234de009e592e8a5391c08802b6.svg","isPro":false,"fullname":"Xiao Jinwei","user":"ginreedcho","type":"user"},{"_id":"64913f1b24d9bc9bb8ff407e","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/64913f1b24d9bc9bb8ff407e/N1cdMd9_DJb5GymdKJ3Mb.jpeg","isPro":false,"fullname":"Haoxiang Zhang","user":"IPF","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":3,"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2608/2608.27448.md","query":{}}">
TTPO: Test-Time Policy Optimization
Abstract
Test-Time Policy Optimization enables label-free test-time training for mathematical reasoning by asymmetrically distilling agreeing rollouts and penalizing disagreeing ones, matching supervised performance.
Recent prominent post-training methods, such as Reinforcement Learning (RL) and On-Policy Self-Distillation (OPSD), have driven rapid progress in mathematical reasoning for large language models, yet their reliance on ground-truth labels precludes test-time training (TTT). Replacing ground truth with majority-vote pseudo-labels is a natural alternative, yet it is fragile: an incorrect vote corrupts the teacher and misleads every token. We observe that this failure mode is asymmetric: rollouts that disagree with the pseudo-label are typically wrong regardless of whether the vote itself is correct. Building on this observation, we propose Test-Time Policy Optimization (TTPO), an asymmetric objective that distills agreeing rollouts via OPSD and penalizes disagreeing rollouts with Grouped RL. Token-level selection further refines both branches: distillation down-weights already-converged positions, while RL penalizes only confident errors. Both updates remain well-grounded even under frequent pseudo-label errors, and majority-vote routing yields tighter self-supervision as the model improves. Without any labels, TTPO matches label-supervised OPSD on five competition-level benchmarks, raises Qwen3-1.7B from 38.0% to 45.2% in TTT, yields +25.2% to +36.4% without thinking, and shows strong cross-task generalization.
Community
We introduce TTPO, a label-free test-time training framework that remains robust even when majority-vote pseudo-labels are incorrect. TTPO uses an asymmetric objective: it distills agreeing rollouts while penalizing confident errors in disagreeing rollouts through grouped reinforcement learning.
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2608.27448 in a model README.md to link it from this page.
Cite arxiv.org/abs/2608.27448 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2608.27448 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.