Hugging Face Daily Papers · June 30, 2026 · 3 min read

SWE-Together: Evaluating Coding Agents in Interactive User Sessions

Mirrored from Hugging Face Daily Papers for archival readability. Support the source by reading on the original site.

Like Read original ↗

<a href=\"https://cdn-uploads.huggingface.co/production/uploads/64641a2938083255f6769953/2f6lwusD3q57YeOdrEkwx.png\" rel=\"nofollow\"><img src=\"https://cdn-uploads.huggingface.co/production/uploads/64641a2938083255f6769953/2f6lwusD3q57YeOdrEkwx.png\" alt=\"swe_together_x_composite\"></a></p>\n","updatedAt":"2026-06-30T16:52:49.111Z","author":{"_id":"64641a2938083255f6769953","avatarUrl":"/avatars/a4117357703607bd7b290dc2975acbef.svg","fullname":"Yifan Wu","name":"yfwu","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.35096997022628784},"editors":["yfwu"],"editorAvatarUrls":["/avatars/a4117357703607bd7b290dc2975acbef.svg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2606.29957","authors":[{"_id":"6a434f75763f63ca3757ea98","name":"Yifan Wu","hidden":false},{"_id":"6a434f75763f63ca3757ea99","name":"Zhuokai Zhao","hidden":false},{"_id":"6a434f75763f63ca3757ea9a","name":"Songlin Li","hidden":false},{"_id":"6a434f75763f63ca3757ea9b","name":"Ho Hin Lee","hidden":false},{"_id":"6a434f75763f63ca3757ea9c","name":"Jiacheng Zhu","hidden":false},{"_id":"6a434f75763f63ca3757ea9d","name":"Shirley Wu","hidden":false},{"_id":"6a434f75763f63ca3757ea9e","name":"Tianhe Yu","hidden":false},{"_id":"6a434f75763f63ca3757ea9f","name":"Serena Li","hidden":false},{"_id":"6a434f75763f63ca3757eaa0","name":"Lizhu Zhang","hidden":false},{"_id":"6a434f75763f63ca3757eaa1","name":"Xiangjun Fan","hidden":false},{"_id":"6a434f75763f63ca3757eaa2","name":"Shengzhi Li","hidden":false}],"publishedAt":"2026-06-29T00:00:00.000Z","submittedOnDailyAt":"2026-06-30T00:00:00.000Z","title":"SWE-Together: Evaluating Coding Agents in Interactive User Sessions","submittedOnDailyBy":{"_id":"64641a2938083255f6769953","avatarUrl":"/avatars/a4117357703607bd7b290dc2975acbef.svg","isPro":false,"fullname":"Yifan Wu","user":"yfwu","type":"user","name":"yfwu"},"summary":"Most coding-agent benchmarks are static: an agent receives a complete task description up front and is judged only by its final code. Real coding assistance is interactive, with users clarifying goals, adding constraints, and correcting mistakes over multiple turns. We introduce SWE-Together, a multi-turn benchmark reconstructed from real user-agent coding sessions. To make real interactions verifiable, we curate 109 repository-level tasks from 11,260 recorded sessions, selecting sessions with recoverable repository states, clear user goals, and observable outcomes. To replay these interactions across agents, we build a reactive LLM-based user simulator that preserves the original users' intents and provides feedback when the coding agent's progress requires it. To evaluate agents as collaborators, we measure both final repository correctness and the number of corrective feedback turns required during the interaction. Experiments with frontier coding agents show that stronger agents generally achieve higher final success rates while requiring fewer interventions, suggesting an improved user experience.","upvotes":11,"discussionId":"6a434f75763f63ca3757eaa3","projectPage":"https://togetherbench.com","githubRepo":"https://github.com/Togetherbench/SWE-Together","githubRepoAddedBy":"user","ai_summary":"SWE-Together is a multi-turn coding benchmark created from real user-agent interactions, featuring a reactive LLM simulator to evaluate agents based on both final correctness and interaction efficiency.","ai_keywords":["coding-agent benchmarks","multi-turn benchmark","user-agent coding sessions","repository-level tasks","user simulator","corrective feedback turns","final repository correctness","interaction efficiency"],"ai_summary_model":"Qwen/Qwen2.5-Coder-32B-Instruct","githubStars":2,"organization":{"_id":"66b54027408752ae16404b05","name":"metaresearch","fullname":"Meta Research","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/66b25f3f58babfaeb76112dc/2GmiaF075AZ7BcE538oPk.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"684aa92ec20b8b204249821e","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/684aa92ec20b8b204249821e/m63SVIldPZi5L3uskCrno.png","isPro":false,"fullname":"QuantaSpark","user":"QuantaSparkLabs","type":"user"},{"_id":"64641a2938083255f6769953","avatarUrl":"/avatars/a4117357703607bd7b290dc2975acbef.svg","isPro":false,"fullname":"Yifan Wu","user":"yfwu","type":"user"},{"_id":"63ac5701c21e60a3e9b58aa7","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/63ac5701c21e60a3e9b58aa7/g6EX7diOpuA94R2ab-rZC.png","isPro":true,"fullname":"Dipankar Sarkar","user":"dipankarsarkar","type":"user"},{"_id":"638913002a897944ea5bd2ab","avatarUrl":"/avatars/418e0d5fc61599ef8cb6a3ef7b9566bd.svg","isPro":false,"fullname":"alex shengzhi li","user":"alexshengzhili","type":"user"},{"_id":"665640a3a2d7a882a8c7f7d5","avatarUrl":"/avatars/56dbd937eec1681ab2837fc4e978b9d4.svg","isPro":false,"fullname":"Yuhang Zhou","user":"tonyzhou1998","type":"user"},{"_id":"678d60dae7b3986f59869666","avatarUrl":"/avatars/c78e30338af124001043191d1f4f864c.svg","isPro":false,"fullname":"Songlin Li","user":"vincent-sk-li","type":"user"},{"_id":"6543177102409f0994060dcf","avatarUrl":"/avatars/72f17451158dfce412ea9ad089c5441b.svg","isPro":false,"fullname":"Xiayan Ji","user":"xjiae","type":"user"},{"_id":"665773284018967357a62af8","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/665773284018967357a62af8/XZtkM-l4BapsD4KYRBNFZ.jpeg","isPro":false,"fullname":"Marvin Liu","user":"blueyefish","type":"user"},{"_id":"641fb016f8c8b04c0bad1df5","avatarUrl":"/avatars/e66bfff9befb964ba3a58d05ebb9180b.svg","isPro":false,"fullname":"Yu Feng","user":"AnnieFeng","type":"user"},{"_id":"65fc85dc91922fa7a768a93a","avatarUrl":"/avatars/beea6621cb229df67e812da731b17309.svg","isPro":false,"fullname":"Yucheng Zhang","user":"yuchengzhang","type":"user"},{"_id":"653801b167325b6218ddfdc8","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/653801b167325b6218ddfdc8/y3DjEDpcvgnRC30201dFc.jpeg","isPro":false,"fullname":"Jiacheng Zhu","user":"JiachengZhu","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"66b54027408752ae16404b05","name":"metaresearch","fullname":"Meta Research","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/66b25f3f58babfaeb76112dc/2GmiaF075AZ7BcE538oPk.png"},"markdownContentUrl":"https://huggingface.co/buckets/huggingchat/papers-content/resolve/2606/2606.29957.md","query":{}}">

Papers

arxiv:2606.29957

SWE-Together: Evaluating Coding Agents in Interactive User Sessions

Published on Jun 29

· Submitted by

Yifan Wu on Jun 30

Meta Research

Upvote

Authors:

Abstract

SWE-Together is a multi-turn coding benchmark created from real user-agent interactions, featuring a reactive LLM simulator to evaluate agents based on both final correctness and interaction efficiency.

Generated by Qwen/Qwen2.5-Coder-32B-Instruct

Most coding-agent benchmarks are static: an agent receives a complete task description up front and is judged only by its final code. Real coding assistance is interactive, with users clarifying goals, adding constraints, and correcting mistakes over multiple turns. We introduce SWE-Together, a multi-turn benchmark reconstructed from real user-agent coding sessions. To make real interactions verifiable, we curate 109 repository-level tasks from 11,260 recorded sessions, selecting sessions with recoverable repository states, clear user goals, and observable outcomes. To replay these interactions across agents, we build a reactive LLM-based user simulator that preserves the original users' intents and provides feedback when the coding agent's progress requires it. To evaluate agents as collaborators, we measure both final repository correctness and the number of corrective feedback turns required during the interaction. Experiments with frontier coding agents show that stronger agents generally achieve higher final success rates while requiring fewer interventions, suggesting an improved user experience.