Excited to share <strong>ExecRetrieval</strong>! We ask a simple question: can code embeddings actually distinguish correct code from near-identical buggy code? Across 939 tasks and 24 retrieval systems, the best system reaches <strong>100% exec@10 but only 33.1% exec@1</strong>, with most rank-1 failures being the query’s own near-clone buggy variants. Curious to hear thoughts on how we should close this functional-correctness gap in retrieval for coding agents and code RAG. (<a href=\"https://arxiv.org/pdf/2609.01865\" title=\"ExecRetrieval: Measuring the Functional-Correctness Gap in Code-Embedding Retrieval\" rel=\"nofollow\">arxiv.org</a>)</p>\n<p><strong>Accepted to EMNLP 2026 - Main Conference (Budapest, October 2026)</strong></p>\n","updatedAt":"2026-09-03T02:13:31.419Z","author":{"_id":"64e1a459ff3fd4fd8eedb456","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/64e1a459ff3fd4fd8eedb456/pzfa_43ftEWGHDGRUZil3.png","fullname":"Aaryan Kapoor","name":"AaryanK","type":"user","isPro":true,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":38,"isUserFollowing":false}},"numEdits":2,"identifiedLanguage":{"language":"en","probability":0.7557879090309143},"editors":["AaryanK"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/64e1a459ff3fd4fd8eedb456/pzfa_43ftEWGHDGRUZil3.png"],"reactions":[{"reaction":"🚀","users":["AaryanK"],"count":1}],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2609.01865","authors":[{"_id":"6a98d5c7fea818274321fe66","user":{"_id":"64e1a459ff3fd4fd8eedb456","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/64e1a459ff3fd4fd8eedb456/pzfa_43ftEWGHDGRUZil3.png","isPro":true,"fullname":"Aaryan Kapoor","user":"AaryanK","type":"user","name":"AaryanK"},"name":"Aaryan Kapoor","status":"claimed_verified","statusLastChangedAt":"2026-09-03T08:17:03.307Z","hidden":false},{"_id":"6a98d5c7fea818274321fe67","name":"Md Abdullah Al Hafiz Khan","hidden":false}],"publishedAt":"2026-09-01T00:00:00.000Z","submittedOnDailyAt":"2026-09-03T00:00:00.000Z","title":"ExecRetrieval: Measuring the Functional-Correctness Gap in Code-Embedding Retrieval","submittedOnDailyBy":{"_id":"64e1a459ff3fd4fd8eedb456","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/64e1a459ff3fd4fd8eedb456/pzfa_43ftEWGHDGRUZil3.png","isPro":true,"fullname":"Aaryan Kapoor","user":"AaryanK","type":"user","name":"AaryanK"},"summary":"Embedding-based code retrieval is a core component of coding agents and retrieval-augmented code generation, where retrieving correct code matters more than retrieving lexically similar code. Existing code-retrieval benchmarks do not plant controlled, execution-verified single-edit variants of each query's canonical implementation in the search pool, leaving the question of whether embeddings can functionally discriminate correct from near-clone-but-incorrect code unanswered in a retrieval setting. Resolving this requires a benchmark whose search pool itself contains the relevant counterfactuals -- execution-verified buggy variants near-identical to each canonical -- so that a retriever's rank ordering can be directly tested for functional discrimination rather than topical or identity overlap. We introduce ExecRetrieval, 939 Python tasks each paired with one execution-verified canonical implementation and up to four execution-verified buggy distractors, each generated by a mechanical mutation making a single targeted edit, and evaluate 23 dense embedding configurations plus BM25 under provider-native invocation with paired McNemar tests and query-level bootstrap intervals. With near-clone counterfactuals in the pool, the top hosted system reaches exec@10 = 1.00 but only exec@1 = 0.331; rank-1 misses are paired buggy variants 91.5-99.4% of the time across the four leading systems, and the canonical scores below at least one of its four paired distractors in 67-78% of queries on the leading systems. The full dataset, execution oracle, embedding matrices, environment snapshot, and pairwise statistical tests are released at the URL in Appendix D.","upvotes":5,"discussionId":"6a98d5c8fea818274321fe68","projectPage":"https://huggingface.co/datasets/AaryanK/ExecRetrieval","ai_summary":"ExecRetrieval evaluates whether code embeddings can distinguish correct implementations from near-identical buggy variants, revealing that leading retrievers frequently rank incorrect near-clones above canonical solutions.","ai_keywords":["embedding-based code retrieval","execution-verified buggy variants","dense embedding configurations","BM25","exec@10","exec@1","McNemar tests","bootstrap intervals"],"ai_summary_model":"thinkingmachines/Inkling-Small","organization":{"_id":"66a3e96de141169f408b65dc","name":"kennesaw","fullname":"Kennesaw State University","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/66442fdde3bcf5c9f49707b3/TSQy_xAh_dsKKJrv7xQ5F.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"64e1a459ff3fd4fd8eedb456","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/64e1a459ff3fd4fd8eedb456/pzfa_43ftEWGHDGRUZil3.png","isPro":true,"fullname":"Aaryan Kapoor","user":"AaryanK","type":"user"},{"_id":"63ac5701c21e60a3e9b58aa7","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/63ac5701c21e60a3e9b58aa7/g6EX7diOpuA94R2ab-rZC.png","isPro":true,"fullname":"Dipankar Sarkar","user":"dipankarsarkar","type":"user"},{"_id":"6a2da6c8ca070ee12c6e396c","avatarUrl":"/avatars/0355287dcabaa67dbc7f0b10b87451f9.svg","isPro":false,"fullname":"Joe Mama","user":"JoeMama123123123","type":"user"},{"_id":"69bd021013b7f8d20b0c7c65","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/GNMz54OTWhxe8VH_jboRR.png","isPro":false,"fullname":"Максим Козлов","user":"ethlopez19","type":"user"},{"_id":"69bcc905518f6d1f3d2b86df","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/t-ZEqYEkqPI8-saats0nT.jpeg","isPro":false,"fullname":"Liu Jingyi","user":"gao-wenxuan2","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"66a3e96de141169f408b65dc","name":"kennesaw","fullname":"Kennesaw State University","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/66442fdde3bcf5c9f49707b3/TSQy_xAh_dsKKJrv7xQ5F.png"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2609/2609.01865.md","query":{}}">

Get this paper in your agent:

Don't have the latest CLI?

Cite arxiv.org/abs/2609.01865 in a model README.md to link it from this page.

Cite arxiv.org/abs/2609.01865 in a dataset README.md to link it from this page.

Cite arxiv.org/abs/2609.01865 in a Space README.md to link it from this page.

Add this paper to a collection to link it from this page.