Evolution Strategies (ES) have recently emerged as a memory-efficient post-training paradigm for LLM<br>reasoning. However, the optimization behavior of ES remains understudied, making it hard to define<br>its advantage scope compared to mainstream post-training paradigms (e.g., Group Relative Policy<br>Optimization (GRPO)). By systematically investigating ES dynamics and mechanisms, this paperfirst identifies a performance advantage of ES over GRPO, theoretically and empirically showing<br>that ES can lead to broader reasoning coverage, thereby better exploiting the reasoning capabilities of<br>pretrained LLMs. Theoretically, we show that verifier-projected Jensen–Shannon diversity across the<br>ES population is helpful to higher Pass@K performances. Empirically, unlike GRPO, which exhibits<br>entropy collapse, ES improves Pass@1 while attaining higher Pass@K than GRPO. We further develop<br>a sequential GRPO–ES training strategy that combines GRPO’s strength in Pass@1 with ES’s gains in<br>Pass@K. Second, we find that despite substantial whole-model parameter drift, the task-performance<br>gains of ES are only contributed to a sparse subset of larger-magnitude updates. This functional<br>sparsity suggests that large parameter movement need not imply widespread functional change, and<br>held-out evaluations further show that it does not necessarily lead to catastrophic forgetting. Finally,we study how hyperparameter design affects the effectiveness of ES, demonstrating that ES requires a<br>smaller population size in a larger LLM. These findings position ES as a distinct reasoning post-training<br>paradigm rather than a less effective, memory-efficient alternative to GRPO.</p>\n","updatedAt":"2026-08-28T02:11:39.385Z","author":{"_id":"67a1d21e33e92b4a1183f3bb","avatarUrl":"/avatars/43f9dd3fcb7d58ddc69562fd1fc12957.svg","fullname":"Zhi Zheng","name":"zz1358m","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":6,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8895478844642639},"editors":["zz1358m"],"editorAvatarUrls":["/avatars/43f9dd3fcb7d58ddc69562fd1fc12957.svg"],"reactions":[{"reaction":"👍","users":["kuangrepi"],"count":1}],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2608.27351","authors":[{"_id":"6a90ee2ba64059bab69c35d2","name":"Yunpeng Ba","hidden":false},{"_id":"6a90ee2ba64059bab69c35d3","user":{"_id":"67a1d21e33e92b4a1183f3bb","avatarUrl":"/avatars/43f9dd3fcb7d58ddc69562fd1fc12957.svg","isPro":false,"fullname":"Zhi Zheng","user":"zz1358m","type":"user","name":"zz1358m"},"name":"Zhi Zheng","status":"claimed_verified","statusLastChangedAt":"2026-08-28T08:45:04.599Z","hidden":false},{"_id":"6a90ee2ba64059bab69c35d4","name":"Yue Xie","hidden":false},{"_id":"6a90ee2ba64059bab69c35d5","name":"Jiaqing Li","hidden":false},{"_id":"6a90ee2ba64059bab69c35d6","name":"Xialiang Tong","hidden":false},{"_id":"6a90ee2ba64059bab69c35d7","name":"Tao Zhong","hidden":false},{"_id":"6a90ee2ba64059bab69c35d8","name":"Mingxuan Yuan","hidden":false},{"_id":"6a90ee2ba64059bab69c35d9","name":"Zhichao Lu","hidden":false},{"_id":"6a90ee2ba64059bab69c35da","name":"Xuyang Wu","hidden":false},{"_id":"6a90ee2ba64059bab69c35db","name":"Zhenkun Wang","hidden":false}],"publishedAt":"2026-08-27T00:00:00.000Z","submittedOnDailyAt":"2026-08-28T00:00:00.000Z","title":"Understanding Evolution Strategies for LLM Reasoning: Broader Reasoning Coverage than GRPO","submittedOnDailyBy":{"_id":"67a1d21e33e92b4a1183f3bb","avatarUrl":"/avatars/43f9dd3fcb7d58ddc69562fd1fc12957.svg","isPro":false,"fullname":"Zhi Zheng","user":"zz1358m","type":"user","name":"zz1358m"},"summary":"Evolution Strategies (ES) have recently emerged as a memory-efficient post-training paradigm for LLM reasoning. However, the optimization behavior of ES remains understudied, making it hard to define its advantage scope compared to mainstream post-training paradigms (e.g., Group Relative Policy Optimization (GRPO)). By systematically investigating ES dynamics and mechanisms, this paper first identifies a performance advantage of ES over GRPO, theoretically and empirically showing that ES can lead to broader reasoning coverage, thereby better exploiting the reasoning capabilities of pretrained LLMs. Theoretically, we show that verifier-projected Jensen-Shannon diversity across the ES population is helpful to higher Pass@K performances. Empirically, unlike GRPO, which exhibits entropy collapse, ES improves Pass@1 while attaining higher Pass@K than GRPO. We further develop a sequential GRPO-ES training strategy that combines GRPO's strength in Pass@1 with ES's gains in Pass@K. Second, we find that despite substantial whole-model parameter drift, the task-performance gains of ES are only contributed to a sparse subset of larger-magnitude updates. This functional sparsity suggests that large parameter movement need not imply widespread functional change, and held-out evaluations further show that it does not necessarily lead to catastrophic forgetting. Finally, we study how hyperparameter design affects the effectiveness of ES, demonstrating that ES requires a smaller population size in a larger LLM. These findings position ES as a distinct reasoning post-training paradigm rather than a less effective, memory-efficient alternative to GRPO.","upvotes":11,"discussionId":"6a90ee2ba64059bab69c35dc","ai_summary":"Evolution strategies improve reasoning diversity and Pass@K over GRPO through sparse functional updates and population diversity, supporting a hybrid training approach.","ai_keywords":["Evolution Strategies","GRPO","Group Relative Policy Optimization","Pass@K","verifier-projected Jensen-Shannon diversity","entropy collapse","functional sparsity","catastrophic forgetting","population size"],"ai_summary_model":"thinkingmachines/Inkling-Small"},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"67a1d21e33e92b4a1183f3bb","avatarUrl":"/avatars/43f9dd3fcb7d58ddc69562fd1fc12957.svg","isPro":false,"fullname":"Zhi Zheng","user":"zz1358m","type":"user"},{"_id":"67b2d9cb48d466697ba54563","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/N7zTMQvNNOJL4TMNmRNxt.png","isPro":false,"fullname":"Guyu","user":"kuangrepi","type":"user"},{"_id":"6a34f8ed2e96e4b61df5f68c","avatarUrl":"/avatars/17c2527e1bfcb403cb054a30a5afac4b.svg","isPro":false,"fullname":"Rongsheng Chen","user":"crssbl","type":"user"},{"_id":"6a1ef6772a65642f97ab44e1","avatarUrl":"/avatars/d17285f1efdb5e5973dd6f6bec34c1dd.svg","isPro":false,"fullname":"Jiaqing Li","user":"ljq34952","type":"user"},{"_id":"689b365a5cac3f570d13d4e6","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/4pZOuTbUBG8DhXQVBZyPM.png","isPro":false,"fullname":"Jimmy Luo","user":"Leo5223","type":"user"},{"_id":"63e087b6a98d931aa90c1b9c","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/63e087b6a98d931aa90c1b9c/4ZnfL0U8rrj3cNhj7WTgo.jpeg","isPro":false,"fullname":"Hyunwoo Ko","user":"Cartinoe5930","type":"user"},{"_id":"66d8512c54209e9101811e8e","avatarUrl":"/avatars/62dfd8e6261108f2508efe678d5a2a57.svg","isPro":false,"fullname":"M Saad Salman","user":"MSS444","type":"user"},{"_id":"6a911ba1949f9020f7fe7bf5","avatarUrl":"/avatars/cccfac316398145817bf31543a133bca.svg","isPro":false,"fullname":"null","user":"EricZzZzZz1Q24","type":"user"},{"_id":"67975fece83b28ed520ab5ac","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/e4ij-o_uyZusyjmuRF9KA.png","isPro":false,"fullname":"Zhuoliang Xie","user":"Acvine","type":"user"},{"_id":"6a1481b614524dbef227e9cf","avatarUrl":"/avatars/6950a2495e270f72af553803c7abbdb8.svg","isPro":false,"fullname":"Mario Ba","user":"marioba7","type":"user"},{"_id":"63ac5701c21e60a3e9b58aa7","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/63ac5701c21e60a3e9b58aa7/g6EX7diOpuA94R2ab-rZC.png","isPro":true,"fullname":"Dipankar Sarkar","user":"dipankarsarkar","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2608/2608.27351.md","query":{}}">
Understanding Evolution Strategies for LLM Reasoning: Broader Reasoning Coverage than GRPO
Abstract
Evolution strategies improve reasoning diversity and Pass@K over GRPO through sparse functional updates and population diversity, supporting a hybrid training approach.
Evolution Strategies (ES) have recently emerged as a memory-efficient post-training paradigm for LLM reasoning. However, the optimization behavior of ES remains understudied, making it hard to define its advantage scope compared to mainstream post-training paradigms (e.g., Group Relative Policy Optimization (GRPO)). By systematically investigating ES dynamics and mechanisms, this paper first identifies a performance advantage of ES over GRPO, theoretically and empirically showing that ES can lead to broader reasoning coverage, thereby better exploiting the reasoning capabilities of pretrained LLMs. Theoretically, we show that verifier-projected Jensen-Shannon diversity across the ES population is helpful to higher Pass@K performances. Empirically, unlike GRPO, which exhibits entropy collapse, ES improves Pass@1 while attaining higher Pass@K than GRPO. We further develop a sequential GRPO-ES training strategy that combines GRPO's strength in Pass@1 with ES's gains in Pass@K. Second, we find that despite substantial whole-model parameter drift, the task-performance gains of ES are only contributed to a sparse subset of larger-magnitude updates. This functional sparsity suggests that large parameter movement need not imply widespread functional change, and held-out evaluations further show that it does not necessarily lead to catastrophic forgetting. Finally, we study how hyperparameter design affects the effectiveness of ES, demonstrating that ES requires a smaller population size in a larger LLM. These findings position ES as a distinct reasoning post-training paradigm rather than a less effective, memory-efficient alternative to GRPO.
Community
Evolution Strategies (ES) have recently emerged as a memory-efficient post-training paradigm for LLM
reasoning. However, the optimization behavior of ES remains understudied, making it hard to define
its advantage scope compared to mainstream post-training paradigms (e.g., Group Relative Policy
Optimization (GRPO)). By systematically investigating ES dynamics and mechanisms, this paperfirst identifies a performance advantage of ES over GRPO, theoretically and empirically showing
that ES can lead to broader reasoning coverage, thereby better exploiting the reasoning capabilities of
pretrained LLMs. Theoretically, we show that verifier-projected Jensen–Shannon diversity across the
ES population is helpful to higher Pass@K performances. Empirically, unlike GRPO, which exhibits
entropy collapse, ES improves Pass@1 while attaining higher Pass@K than GRPO. We further develop
a sequential GRPO–ES training strategy that combines GRPO’s strength in Pass@1 with ES’s gains in
Pass@K. Second, we find that despite substantial whole-model parameter drift, the task-performance
gains of ES are only contributed to a sparse subset of larger-magnitude updates. This functional
sparsity suggests that large parameter movement need not imply widespread functional change, and
held-out evaluations further show that it does not necessarily lead to catastrophic forgetting. Finally,we study how hyperparameter design affects the effectiveness of ES, demonstrating that ES requires a
smaller population size in a larger LLM. These findings position ES as a distinct reasoning post-training
paradigm rather than a less effective, memory-efficient alternative to GRPO.
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2608.27351 in a model README.md to link it from this page.
Cite arxiv.org/abs/2608.27351 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2608.27351 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.