Skills have emerged as a practical and effective approach for enhancing LLM agents at inference time through structured packages of knowledge. However, existing evaluations largely measure whether skills improve aggregated task success, leaving a more fundamental question underexplored: When do skills help, why do they work, and where do they fail? Through controlled experiments across various benchmarks, agent harnesses and LLMs, we isolate the effects of representation, outcome annotation, retrieval difficulty, and cross-framework robustness of skills. To further answer this question, we design a contrastive study that combines controlled quantitative experiments with paired trajectory analysis. We normalize 8,135 trial records from controlled experiments and retain 238 valid unique labels from 240 open-coded records. We consolidate these observations into a taxonomy of three high-level categories and twelve skill-use modes: skills work when noisy trajectories become procedural anchors that stabilize execution. Skills improve over Workflow Memory by 6.06 points in matched comparisons. Procedural anchoring accounts for 65.7% of skill cases, versus 4.5% for explicit knowledge injection, showing that skills stabilize action rather than inject missing facts. Retrieval is a separate bottleneck: as pools grow from 5 to 100, actual-use precision falls from 29.6% to 3.3%. Confusable distractors impair offline identification, yet downstream success remains stable; exact ground-truth invocation is neither sufficient nor necessary. Skills fail under brittle assumptions, incompatible contexts, or insufficient adaptation. These findings move evaluation beyond aggregate success rates and guide reliable self-evolving agents.</p>\n","updatedAt":"2026-08-19T07:58:02.060Z","author":{"_id":"6419309f22270b3ccf177c77","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6419309f22270b3ccf177c77/KQa1586iBBKqucUlfpuPp.jpeg","fullname":"William Li","name":"williamium","type":"user","isPro":true,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":7,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8884046673774719},"editors":["williamium"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/6419309f22270b3ccf177c77/KQa1586iBBKqucUlfpuPp.jpeg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2608.14036","authors":[{"_id":"6a856190536bdd3bdd48f956","name":"Zhiyuan Jiang","hidden":false},{"_id":"6a856190536bdd3bdd48f957","name":"Fangrui Huang","hidden":false},{"_id":"6a856190536bdd3bdd48f958","name":"Hanwen Xing","hidden":false},{"_id":"6a856190536bdd3bdd48f959","name":"Xander Wu","hidden":false},{"_id":"6a856190536bdd3bdd48f95a","name":"Yipeng Gao","hidden":false},{"_id":"6a856190536bdd3bdd48f95b","name":"Rui Cao","hidden":false},{"_id":"6a856190536bdd3bdd48f95c","name":"Mengdi Wang","hidden":false},{"_id":"6a856190536bdd3bdd48f95d","name":"Shilong Liu","hidden":false},{"_id":"6a856190536bdd3bdd48f95e","name":"Yijiang Li","hidden":false}],"publishedAt":"2026-08-14T00:00:00.000Z","submittedOnDailyAt":"2026-08-19T00:00:00.000Z","title":"Demystifying Agent Skills: Why They Work-Until They Don't","submittedOnDailyBy":{"_id":"6419309f22270b3ccf177c77","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6419309f22270b3ccf177c77/KQa1586iBBKqucUlfpuPp.jpeg","isPro":true,"fullname":"William Li","user":"williamium","type":"user","name":"williamium"},"summary":"Skills have emerged as a practical and effective approach for enhancing LLM agents at inference time through structured packages of knowledge. However, existing evaluations largely measure whether skills improve aggregated task success, leaving a more fundamental question underexplored: \\textbf{When do skills help, why do they work, and where do they fail?} Through controlled experiments across various benchmarks, agent harnesses and LLMs, we isolate the effects of representation, outcome annotation, retrieval difficulty, and cross-framework robustness of skills. To further answer this question, we design a contrastive study that combines controlled quantitative experiments with paired trajectory analysis. We normalize 8,135 trial records from controlled experiments and retain 238 valid unique labels from 240 open-coded records. We consolidate these observations into a taxonomy of three high-level categories and twelve skill-use modes: skills work when noisy trajectories become procedural anchors that stabilize execution. Skills improve over Workflow Memory by 6.06 points in matched comparisons. Procedural anchoring accounts for 65.7\\% of skill cases, versus 4.5\\% for explicit knowledge injection, showing that skills stabilize action rather than inject missing facts. Retrieval is a separate bottleneck: as pools grow from 5 to 100, actual-use precision falls from 29.6\\% to 3.3\\%. Confusable distractors impair offline identification, yet downstream success remains stable; exact ground-truth invocation is neither sufficient nor necessary. Skills fail under brittle assumptions, incompatible contexts, or insufficient adaptation. These findings move evaluation beyond aggregate success rates and guide reliable self-evolving agents.","upvotes":31,"discussionId":"6a856190536bdd3bdd48f95f","projectPage":"https://zhiyuanjiang04.github.io/demystify-agent-skills","githubRepo":"https://github.com/zhiyuanjiang04/demystify-agent-skills","githubRepoAddedBy":"user","ai_summary":"Skills enhance LLM agents primarily by stabilizing execution through procedural anchoring rather than injecting missing knowledge, though retrieval bottlenecks and brittle assumptions limit their effectiveness.","ai_keywords":["LLM agents","skills","procedural anchoring","retrieval difficulty","cross-framework robustness","workflow memory","contrastive study","trajectory analysis"],"ai_summary_model":"thinkingmachines/Inkling-Small","githubStars":1},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"6419309f22270b3ccf177c77","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6419309f22270b3ccf177c77/KQa1586iBBKqucUlfpuPp.jpeg","isPro":true,"fullname":"William Li","user":"williamium","type":"user"},{"_id":"654427a0326cb9a32be7a0e9","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/654427a0326cb9a32be7a0e9/-yiqFr8v6luvuhfYaqFbA.jpeg","isPro":false,"fullname":"Icy Wang","user":"Icey444","type":"user"},{"_id":"63ef0af2bfe4ead22ca8f69a","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1676610243576-noauth.jpeg","isPro":false,"fullname":"Haozheng Luo","user":"robinzixuan","type":"user"},{"_id":"670f632944d497ffc693129f","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/670f632944d497ffc693129f/f6e9LDjRHCDlZs4AUMoqz.png","isPro":false,"fullname":"Zongyang (Zane) QIU","user":"Zane-QIU","type":"user"},{"_id":"6a14564a3789c92742679ffd","avatarUrl":"/avatars/43623d2e0331834f07b7193b28be494c.svg","isPro":false,"fullname":"Charles Lopez","user":"charleslopez81","type":"user"},{"_id":"669a6055a2dc7c3fb89044f9","avatarUrl":"/avatars/2dee758c64eb3c203bba492fbeeaef88.svg","isPro":false,"fullname":"yiyun deng","user":"yyoraa","type":"user"},{"_id":"68830b95910677c12e2aeed3","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/R3niXd_7P4-f58FY-KlCv.png","isPro":false,"fullname":"Rakuen","user":"puppetsasya","type":"user"},{"_id":"689f283f6563f6bd1b4bd3aa","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/cz5OrEj6d9-tnmAwEZ-aG.png","isPro":false,"fullname":"Haichao Zhang","user":"owlonoak","type":"user"},{"_id":"69d3560b037e731451c33bd2","avatarUrl":"/avatars/f361ecf1a5676b0de45ec95d778add10.svg","isPro":false,"fullname":"network","user":"n3two2k","type":"user"},{"_id":"657858d0dddc2360b01643a1","avatarUrl":"/avatars/00fb2410aaa74e220d69b15084dfebd6.svg","isPro":false,"fullname":"l","user":"gouerrrr","type":"user"},{"_id":"666f7c95fcbf9cefb481e88b","avatarUrl":"/avatars/225501fbb4e962766d0dd07794b0a0dd.svg","isPro":false,"fullname":"Fangrui Huang","user":"fangruih","type":"user"},{"_id":"652867b09903f7a1c9f7cbf1","avatarUrl":"/avatars/1d1d3665328943ccdbd6a25ce03f2f10.svg","isPro":false,"fullname":"Sijun Tan","user":"sijuntan","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2608/2608.14036.md","query":{}}">
Demystifying Agent Skills: Why They Work-Until They Don't
Abstract
Skills enhance LLM agents primarily by stabilizing execution through procedural anchoring rather than injecting missing knowledge, though retrieval bottlenecks and brittle assumptions limit their effectiveness.
Skills have emerged as a practical and effective approach for enhancing LLM agents at inference time through structured packages of knowledge. However, existing evaluations largely measure whether skills improve aggregated task success, leaving a more fundamental question underexplored: \textbf{When do skills help, why do they work, and where do they fail?} Through controlled experiments across various benchmarks, agent harnesses and LLMs, we isolate the effects of representation, outcome annotation, retrieval difficulty, and cross-framework robustness of skills. To further answer this question, we design a contrastive study that combines controlled quantitative experiments with paired trajectory analysis. We normalize 8,135 trial records from controlled experiments and retain 238 valid unique labels from 240 open-coded records. We consolidate these observations into a taxonomy of three high-level categories and twelve skill-use modes: skills work when noisy trajectories become procedural anchors that stabilize execution. Skills improve over Workflow Memory by 6.06 points in matched comparisons. Procedural anchoring accounts for 65.7\% of skill cases, versus 4.5\% for explicit knowledge injection, showing that skills stabilize action rather than inject missing facts. Retrieval is a separate bottleneck: as pools grow from 5 to 100, actual-use precision falls from 29.6\% to 3.3\%. Confusable distractors impair offline identification, yet downstream success remains stable; exact ground-truth invocation is neither sufficient nor necessary. Skills fail under brittle assumptions, incompatible contexts, or insufficient adaptation. These findings move evaluation beyond aggregate success rates and guide reliable self-evolving agents.
Community
Skills have emerged as a practical and effective approach for enhancing LLM agents at inference time through structured packages of knowledge. However, existing evaluations largely measure whether skills improve aggregated task success, leaving a more fundamental question underexplored: When do skills help, why do they work, and where do they fail? Through controlled experiments across various benchmarks, agent harnesses and LLMs, we isolate the effects of representation, outcome annotation, retrieval difficulty, and cross-framework robustness of skills. To further answer this question, we design a contrastive study that combines controlled quantitative experiments with paired trajectory analysis. We normalize 8,135 trial records from controlled experiments and retain 238 valid unique labels from 240 open-coded records. We consolidate these observations into a taxonomy of three high-level categories and twelve skill-use modes: skills work when noisy trajectories become procedural anchors that stabilize execution. Skills improve over Workflow Memory by 6.06 points in matched comparisons. Procedural anchoring accounts for 65.7% of skill cases, versus 4.5% for explicit knowledge injection, showing that skills stabilize action rather than inject missing facts. Retrieval is a separate bottleneck: as pools grow from 5 to 100, actual-use precision falls from 29.6% to 3.3%. Confusable distractors impair offline identification, yet downstream success remains stable; exact ground-truth invocation is neither sufficient nor necessary. Skills fail under brittle assumptions, incompatible contexts, or insufficient adaptation. These findings move evaluation beyond aggregate success rates and guide reliable self-evolving agents.
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2608.14036 in a model README.md to link it from this page.
Cite arxiv.org/abs/2608.14036 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2608.14036 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.