How Can Rhetoric Reward Hack AI Reviewers? Dissecting Rhetorical Sensitivity in AI-Based Peer Review 🤔🤔</p>\n<p>We already know that rewriting a paper can affect AI review scores. But how exactly does rhetoric matter? Which rhetorical choices move AI reviewers, in which direction, and under what conditions?</p>\n<p>We systematically study 6 dimensions of scientific rhetoric using 4,200 rewritten full-paper manuscripts and 42K+ AI reviews, with nearly $30K in API costs.</p>\n<p>We find a clear hierarchy of rhetorical sensitivity: Evidence framing and novelty stance have the strongest effects, followed by scope framing, while technical register and linguistic complexity have much smaller or less stable effects.</p>\n","updatedAt":"2026-08-14T02:04:47.653Z","author":{"_id":"65031d01cccc7b28a388c719","avatarUrl":"/avatars/9d8c94b6ab8ad8b4faba3221b7e76053.svg","fullname":"Ming Li","name":"MingLiiii","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":6,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8991527557373047},"editors":["MingLiiii"],"editorAvatarUrls":["/avatars/9d8c94b6ab8ad8b4faba3221b7e76053.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2608.08975","authors":[{"_id":"6a7e774942823931a1f1761e","name":"Ming Li","hidden":false},{"_id":"6a7e774942823931a1f1761f","name":"Chenguang Wang","hidden":false},{"_id":"6a7e774942823931a1f17620","name":"Xirui Li","hidden":false},{"_id":"6a7e774942823931a1f17621","name":"Xinyue Zeng","hidden":false},{"_id":"6a7e774942823931a1f17622","name":"Dianqi Li","hidden":false},{"_id":"6a7e774942823931a1f17623","name":"Peng Shi","hidden":false},{"_id":"6a7e774942823931a1f17624","name":"Dawei Zhou","hidden":false},{"_id":"6a7e774942823931a1f17625","name":"Tianyi Zhou","hidden":false}],"publishedAt":"2026-08-10T00:00:00.000Z","submittedOnDailyAt":"2026-08-14T00:00:00.000Z","title":"How Can Rhetoric Reward-Hack AI Reviewers? Dissecting Rhetorical Sensitivity in AI-Based Peer Review","submittedOnDailyBy":{"_id":"65031d01cccc7b28a388c719","avatarUrl":"/avatars/9d8c94b6ab8ad8b4faba3221b7e76053.svg","isPro":false,"fullname":"Ming Li","user":"MingLiiii","type":"user","name":"MingLiiii"},"summary":"As large language models increasingly participate in scientific evaluation, we investigate a potential form of reward hacking: how rhetorical choices shape AI-review judgments when reported scientific content is preserved and how these effects vary across evaluation conditions. We construct a controlled corpus of 4,200 full-paper manuscripts derived from 120 anonymized ICLR 2026 submissions. Two LLM rewriters transform six rhetorical dimensions in opposing directions, and five LLM reviewers evaluate the resulting manuscripts under standard and strict protocols. We also test joint, recursive, and reviewer-guided rewriting. Our results show that rhetorical sensitivity is structured rather than uniform. Evidence framing and novelty stance produce the largest positive-negative contrasts in overall assessment, with scope framing forming a weaker second tier; the remaining dimensions have smaller or less stable effects. This hierarchy persists across human-assessed quality levels, but score movement depends strongly on the AI reviewer's original score: lower scores tend to rise, higher scores tend to fall, and directional contrasts are clearest in the middle ranges. More elaborate workflows do not reliably yield larger gains. Joint rewriting is strongly rewriter-dependent, reviewer guidance does not consistently outperform an unguided second pass, and repeated rewriting yields diminishing, configuration-dependent returns. Across conditions, the rewriter primarily determines the separation between opposing variants, whereas the reviewer determines the magnitude and sign of their score effects. Strict review lowers mean OA by 1.36 points without consistently changing rhetorical sensitivity. These findings identify when rhetorical presentation influences AI scientific review and motivate evaluation systems robust to content-preserving variation in scientific writing.","upvotes":9,"discussionId":"6a7e774a42823931a1f17626","githubRepo":"https://github.com/MingLiiii/Dissecting_AI_Reviews","githubRepoAddedBy":"user","ai_summary":"Rhetorical framing significantly biases AI scientific review scores in structured ways, with effects shaped by reviewer identity, score range, and evaluation strictness rather than rewriting complexity.","ai_keywords":["reward hacking","rhetorical dimensions","LLM reviewers","evidence framing","novelty stance","scope framing","joint rewriting","recursive rewriting","reviewer-guided rewriting","strict review protocol"],"ai_summary_model":"thinkingmachines/Inkling-Small","githubStars":0,"organization":{"_id":"696a0b42aea420708e18283a","name":"UMaryland","fullname":"University of Maryland","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/68e396f2b5bb631e9b2fac9a/ma3XHhwnYRf6aEy8rN0Yr.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"65031d01cccc7b28a388c719","avatarUrl":"/avatars/9d8c94b6ab8ad8b4faba3221b7e76053.svg","isPro":false,"fullname":"Ming Li","user":"MingLiiii","type":"user"},{"_id":"6534a434e778506c5b1e5be8","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6534a434e778506c5b1e5be8/349SdAnjEdIQJSzWvKfZ4.png","isPro":true,"fullname":"Xirui Li","user":"AIcell","type":"user"},{"_id":"64a8121e35fab7cd04c30ed0","avatarUrl":"/avatars/48849b84703158772f1022932331b143.svg","isPro":false,"fullname":"Chenrui Fan","user":"Fcr09","type":"user"},{"_id":"6425ce1eba51f8a21366b63b","avatarUrl":"/avatars/93dffc4a048b3f9c2ab703e943393345.svg","isPro":false,"fullname":"Chenguang Wang","user":"SteveWCG","type":"user"},{"_id":"6a7e7d563f18e32b07f4aaf9","avatarUrl":"/avatars/fb527bfc3c75e1f0efeb57590babe510.svg","isPro":false,"fullname":"Jianhao Shi","user":"SHIforreal","type":"user"},{"_id":"65c867cf1b1a5743b3d2fc58","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/65c867cf1b1a5743b3d2fc58/1c8vQtwY7ZNl4e4U4pAix.jpeg","isPro":false,"fullname":"Yaochen Wang","user":"MisakiWang","type":"user"},{"_id":"667b8db8b9e88cc054f147f2","avatarUrl":"/avatars/e0ac7d5631fa91121f116890506749f7.svg","isPro":false,"fullname":"aaaaaki","user":"aaaki","type":"user"},{"_id":"6449ccbc86e837e3d586d1f7","avatarUrl":"/avatars/a5477b1e881e9db12d3a852c85340c17.svg","isPro":false,"fullname":"Song","user":"kmno4","type":"user"},{"_id":"664d930f4b870dd167473c1c","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/664d930f4b870dd167473c1c/TXVEPGvkhftdI_xE1mluu.jpeg","isPro":false,"fullname":"Andy Guan","user":"andytonglove","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"696a0b42aea420708e18283a","name":"UMaryland","fullname":"University of Maryland","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/68e396f2b5bb631e9b2fac9a/ma3XHhwnYRf6aEy8rN0Yr.png"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2608/2608.08975.md","query":{}}">
How Can Rhetoric Reward-Hack AI Reviewers? Dissecting Rhetorical Sensitivity in AI-Based Peer Review
Abstract
Rhetorical framing significantly biases AI scientific review scores in structured ways, with effects shaped by reviewer identity, score range, and evaluation strictness rather than rewriting complexity.
As large language models increasingly participate in scientific evaluation, we investigate a potential form of reward hacking: how rhetorical choices shape AI-review judgments when reported scientific content is preserved and how these effects vary across evaluation conditions. We construct a controlled corpus of 4,200 full-paper manuscripts derived from 120 anonymized ICLR 2026 submissions. Two LLM rewriters transform six rhetorical dimensions in opposing directions, and five LLM reviewers evaluate the resulting manuscripts under standard and strict protocols. We also test joint, recursive, and reviewer-guided rewriting. Our results show that rhetorical sensitivity is structured rather than uniform. Evidence framing and novelty stance produce the largest positive-negative contrasts in overall assessment, with scope framing forming a weaker second tier; the remaining dimensions have smaller or less stable effects. This hierarchy persists across human-assessed quality levels, but score movement depends strongly on the AI reviewer's original score: lower scores tend to rise, higher scores tend to fall, and directional contrasts are clearest in the middle ranges. More elaborate workflows do not reliably yield larger gains. Joint rewriting is strongly rewriter-dependent, reviewer guidance does not consistently outperform an unguided second pass, and repeated rewriting yields diminishing, configuration-dependent returns. Across conditions, the rewriter primarily determines the separation between opposing variants, whereas the reviewer determines the magnitude and sign of their score effects. Strict review lowers mean OA by 1.36 points without consistently changing rhetorical sensitivity. These findings identify when rhetorical presentation influences AI scientific review and motivate evaluation systems robust to content-preserving variation in scientific writing.
Community
How Can Rhetoric Reward Hack AI Reviewers? Dissecting Rhetorical Sensitivity in AI-Based Peer Review 🤔🤔
We already know that rewriting a paper can affect AI review scores. But how exactly does rhetoric matter? Which rhetorical choices move AI reviewers, in which direction, and under what conditions?
We systematically study 6 dimensions of scientific rhetoric using 4,200 rewritten full-paper manuscripts and 42K+ AI reviews, with nearly $30K in API costs.
We find a clear hierarchy of rhetorical sensitivity: Evidence framing and novelty stance have the strongest effects, followed by scope framing, while technical register and linguistic complexity have much smaller or less stable effects.
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2608.08975 in a model README.md to link it from this page.
Cite arxiv.org/abs/2608.08975 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2608.08975 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.