<a href=\"https://cdn-uploads.huggingface.co/production/uploads/649e457d0068a3930ec81cfa/YNPuP_Yz1S233WAqICWV2.png\" rel=\"nofollow\"><img src=\"https://cdn-uploads.huggingface.co/production/uploads/649e457d0068a3930ec81cfa/YNPuP_Yz1S233WAqICWV2.png\" alt=\"image\"></a></p>\n","updatedAt":"2026-09-01T08:26:31.135Z","author":{"_id":"649e457d0068a3930ec81cfa","avatarUrl":"/avatars/458b2f2a5bd1571e8034201ffdf65deb.svg","fullname":"Alex Wang","name":"alexWang2002","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":1,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.5796641111373901},"editors":["alexWang2002"],"editorAvatarUrls":["/avatars/458b2f2a5bd1571e8034201ffdf65deb.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2608.28833","authors":[{"_id":"6a968b25cd6ebc484732ef81","name":"Yumeng Wang","hidden":false},{"_id":"6a968b25cd6ebc484732ef82","name":"Yuchen Wu","hidden":false},{"_id":"6a968b25cd6ebc484732ef83","name":"Cheng Qian","hidden":false},{"_id":"6a968b25cd6ebc484732ef84","name":"Zhiyuan Fan","hidden":false},{"_id":"6a968b25cd6ebc484732ef85","name":"Hyeonjeong Ha","hidden":false},{"_id":"6a968b25cd6ebc484732ef86","name":"Shujin Wu","hidden":false},{"_id":"6a968b25cd6ebc484732ef87","name":"Jiayu Liu","hidden":false},{"_id":"6a968b25cd6ebc484732ef88","name":"Heng Ji","hidden":false},{"_id":"6a968b25cd6ebc484732ef89","name":"Ge Wang","hidden":false}],"publishedAt":"2026-08-28T00:00:00.000Z","submittedOnDailyAt":"2026-09-01T00:00:00.000Z","title":"Evaluating the Hidden Costs of Personalization in Large Language Models","submittedOnDailyBy":{"_id":"649e457d0068a3930ec81cfa","avatarUrl":"/avatars/458b2f2a5bd1571e8034201ffdf65deb.svg","isPro":false,"fullname":"Alex Wang","user":"alexWang2002","type":"user","name":"alexWang2002"},"summary":"While Large language models (LLMs) incorporate user personalization signals to improve usability and helpfulness, they increasingly shift from providing balanced, informative responses toward optimizing for user satisfaction when conditioned on personal context such as conversation history, inferred preferences, and user profiles. Specifically, we identify three emerging risks: (1) irrelevant personalization, where models reference personal information in unnecessary contexts; (2) preference narrowing, where models reinforce informational echo chambers; and (3) sycophantic bias, where models agree excessively with user opinions. As a result, models may reference personal information in contexts where it is unnecessary, inadvertently collapse response diversity, or agree excessively with user opinions. Despite the growing use of personalization in AI assistants, there has been limited systematic evaluation of its potential side effects. To bridge this gap, we propose PRISK, a dynamic evaluation framework with automated data generation and tailored metrics that uncovers systematic limitations in current LLM personalization and how personalized information shapes its responses. Our empirical analysis across 13 LLMs demonstrates the presence of user profiles and retrieved memories consistently exacerbates biases, resulting in an average drop of 45.9% in irrelevant personalization, 41.7% in preference narrowing and 61.7% in sycophantic bias.","upvotes":8,"discussionId":"6a968b25cd6ebc484732ef8a","githubRepo":"https://github.com/yumeng-10/personalization_risk","githubRepoAddedBy":"user","ai_summary":"The study proposes PRISK, a framework that reveals how personalized context in LLMs increases irrelevant personalization, preference narrowing, and sycophantic bias.","ai_keywords":["large language models","personalization","sycophantic bias","preference narrowing","PRISK","user profiles","retrieved memories"],"ai_summary_model":"thinkingmachines/Inkling-Small","githubStars":0,"organization":{"_id":"60212a089f64108326fac7c2","name":"illinois","fullname":"University of Illinois at Urbana-Champaign","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/1612786274096-6021121cfb1b47827d667074.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"649e457d0068a3930ec81cfa","avatarUrl":"/avatars/458b2f2a5bd1571e8034201ffdf65deb.svg","isPro":false,"fullname":"Alex Wang","user":"alexWang2002","type":"user"},{"_id":"671a0e4aea0925fac8cd3ea5","avatarUrl":"/avatars/61a2c22ed05642049ee5bac53d563203.svg","isPro":false,"fullname":"R L","user":"rliuak","type":"user"},{"_id":"65f105cd6750342ce08a114a","avatarUrl":"/avatars/096a0795c05d8d9324e5f39cb80a66f8.svg","isPro":false,"fullname":"Weile Tan","user":"Waylon0625","type":"user"},{"_id":"68ff2027938a06754e4efe1a","avatarUrl":"/avatars/3b5fa122841e887c4aabac243f189268.svg","isPro":false,"fullname":"Kobe Bryant","user":"Kooobe","type":"user"},{"_id":"6a968e3031a7f8cc3d7f0fbf","avatarUrl":"/avatars/dacdbf442a7c37ab1b772f5ebfbd1b05.svg","isPro":false,"fullname":"SHI Chenwei","user":"helixes","type":"user"},{"_id":"6a3fccfd9b5a2d9c5a29c1ea","avatarUrl":"/avatars/81fd5b785be8bc51c8c21a6446362f7a.svg","isPro":false,"fullname":"Zhiyuan Fan","user":"ZhiyuanFanHY","type":"user"},{"_id":"68898dc18be8f79081bc185b","avatarUrl":"/avatars/4ce7eaaaa94c0f29e5fa2dbae402704f.svg","isPro":false,"fullname":"Xuefei Mao","user":"MXFFlorence","type":"user"},{"_id":"670418126fc16dac0f6ded29","avatarUrl":"/avatars/8a31df234d473854c52c46f61985f14a.svg","isPro":false,"fullname":"Wu Kam Man","user":"WJM-George","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"60212a089f64108326fac7c2","name":"illinois","fullname":"University of Illinois at Urbana-Champaign","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/1612786274096-6021121cfb1b47827d667074.png"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2608/2608.28833.md","query":{}}">
Evaluating the Hidden Costs of Personalization in Large Language Models
Abstract
The study proposes PRISK, a framework that reveals how personalized context in LLMs increases irrelevant personalization, preference narrowing, and sycophantic bias.
While Large language models (LLMs) incorporate user personalization signals to improve usability and helpfulness, they increasingly shift from providing balanced, informative responses toward optimizing for user satisfaction when conditioned on personal context such as conversation history, inferred preferences, and user profiles. Specifically, we identify three emerging risks: (1) irrelevant personalization, where models reference personal information in unnecessary contexts; (2) preference narrowing, where models reinforce informational echo chambers; and (3) sycophantic bias, where models agree excessively with user opinions. As a result, models may reference personal information in contexts where it is unnecessary, inadvertently collapse response diversity, or agree excessively with user opinions. Despite the growing use of personalization in AI assistants, there has been limited systematic evaluation of its potential side effects. To bridge this gap, we propose PRISK, a dynamic evaluation framework with automated data generation and tailored metrics that uncovers systematic limitations in current LLM personalization and how personalized information shapes its responses. Our empirical analysis across 13 LLMs demonstrates the presence of user profiles and retrieved memories consistently exacerbates biases, resulting in an average drop of 45.9% in irrelevant personalization, 41.7% in preference narrowing and 61.7% in sycophantic bias.
Community
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2608.28833 in a model README.md to link it from this page.
Cite arxiv.org/abs/2608.28833 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2608.28833 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.