We introduce SWE-Interact, a new testbed for evaluating coding agents on multi-turn, interactive, user-driven software engineering tasks. Existing frontier SWE benchmarks typically provide complete requirements upfront and evaluate agents on autonomous implementation. In contrast, SWE-Interact places agents in a realistic developer workflow: a carefully designed user simulator starts with vague or incomplete instructions, progressively reveals requirements, inspects the agent's workspace, and provides targeted feedback, revisions, and new constraints until the full task goal has been handed off.</p>\n","updatedAt":"2026-07-01T17:45:11.909Z","author":{"_id":"602b8ed6c27b258e705cffae","avatarUrl":"/avatars/7601c6db417ec6ea66acc455967664bb.svg","fullname":"Anisha Gunjal","name":"anisha2102","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.8869451880455017},"editors":["anisha2102"],"editorAvatarUrls":["/avatars/7601c6db417ec6ea66acc455967664bb.svg"],"reactions":[],"isReport":false}},{"id":"6a45c3a502dda729ed06eaad","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":372,"isUserFollowing":false},"createdAt":"2026-07-02T01:49:25.000Z","type":"comment","data":{"edited":false,"hidden":false,"latest":{"raw":"This is an automated message from the [Librarian Bot](https://huggingface.co/librarian-bots). I found the following papers similar to this paper. \n\nThe following papers were recommended by the Semantic Scholar API \n\n* [SWE-Together: Evaluating Coding Agents in Interactive User Sessions](https://huggingface.co/papers/2606.29957) (2026)\n* [SWE Atlas: Benchmarking Coding Agents Beyond Issue Resolution](https://huggingface.co/papers/2605.08366) (2026)\n* [EvoCode-Bench: Evaluating Coding Agents in Multi-Turn Iterative Interactions](https://huggingface.co/papers/2605.24110) (2026)\n* [SWE-Marathon: Can Agents Autonomously Complete Ultra-Long-Horizon Software Work?](https://huggingface.co/papers/2606.07682) (2026)\n* [$\\pi$-Bench: Evaluating Proactive Personal Assistant Agents in Long-Horizon Workflows](https://huggingface.co/papers/2605.14678) (2026)\n* [TerminalWorld: Benchmarking Agents on Real-World Terminal Tasks](https://huggingface.co/papers/2605.22535) (2026)\n* [WeaveBench: A Long-Horizon, Real-World Benchmark for Computer-Use Agents with Hybrid Interfaces](https://huggingface.co/papers/2606.09426) (2026)\n\n\n Please give a thumbs up to this comment if you found it helpful!\n\n If you want recommendations for any Paper on Hugging Face checkout [this](https://huggingface.co/spaces/librarian-bots/recommend_similar_papers) Space\n\n You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: `@librarian-bot recommend`","html":"<p>This is an automated message from the <a href=\"https://huggingface.co/librarian-bots\">Librarian Bot</a>. I found the following papers similar to this paper. </p>\n<p>The following papers were recommended by the Semantic Scholar API </p>\n<ul>\n<li><a href=\"https://huggingface.co/papers/2606.29957\">SWE-Together: Evaluating Coding Agents in Interactive User Sessions</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2605.08366\">SWE Atlas: Benchmarking Coding Agents Beyond Issue Resolution</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2605.24110\">EvoCode-Bench: Evaluating Coding Agents in Multi-Turn Iterative Interactions</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2606.07682\">SWE-Marathon: Can Agents Autonomously Complete Ultra-Long-Horizon Software Work?</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2605.14678\">$\\pi$-Bench: Evaluating Proactive Personal Assistant Agents in Long-Horizon Workflows</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2605.22535\">TerminalWorld: Benchmarking Agents on Real-World Terminal Tasks</a> (2026)</li>\n<li><a href=\"https://huggingface.co/papers/2606.09426\">WeaveBench: A Long-Horizon, Real-World Benchmark for Computer-Use Agents with Hybrid Interfaces</a> (2026)</li>\n</ul>\n<p> Please give a thumbs up to this comment if you found it helpful!</p>\n<p> If you want recommendations for any Paper on Hugging Face checkout <a href=\"https://huggingface.co/spaces/librarian-bots/recommend_similar_papers\">this</a> Space</p>\n<p> You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: <code>@librarian-bot recommend</code></p>\n","updatedAt":"2026-07-02T01:49:25.362Z","author":{"_id":"63d3e0e8ff1384ce6c5dd17d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg","fullname":"Librarian Bot (Bot)","name":"librarian-bot","type":"user","isPro":false,"isHf":false,"isHfAdmin":false,"isMod":false,"followerCount":372,"isUserFollowing":false}},"numEdits":0,"identifiedLanguage":{"language":"en","probability":0.7173247337341309},"editors":["librarian-bot"],"editorAvatarUrls":["https://cdn-avatars.huggingface.co/v1/production/uploads/1674830754237-63d3e0e8ff1384ce6c5dd17d.jpeg"],"reactions":[],"isReport":false}}],"primaryEmailConfirmed":false,"paper":{"id":"2606.30573","authors":[{"_id":"6a4551cf4f1dd35e48fb8d94","name":"Mohit Raghavendra","hidden":false},{"_id":"6a4551cf4f1dd35e48fb8d95","name":"Anisha Gunjal","hidden":false},{"_id":"6a4551cf4f1dd35e48fb8d96","name":"Aakash Sabharwal","hidden":false},{"_id":"6a4551cf4f1dd35e48fb8d97","name":"Yunzhong He","hidden":false}],"publishedAt":"2026-06-29T00:00:00.000Z","submittedOnDailyAt":"2026-07-01T00:00:00.000Z","title":"SWE-INTERACT: Reimagining SWE Benchmarks as User-Driven Long-Horizon Coding Sessions","submittedOnDailyBy":{"_id":"602b8ed6c27b258e705cffae","avatarUrl":"/avatars/7601c6db417ec6ea66acc455967664bb.svg","isPro":false,"fullname":"Anisha Gunjal","user":"anisha2102","type":"user","name":"anisha2102"},"summary":"We introduce SWE-Interact, a new testbed for evaluating coding agents on multi-turn, interactive, user-driven software engineering tasks. Existing frontier SWE benchmarks typically provide complete requirements upfront and evaluate agents on autonomous implementation. In contrast, SWE-Interact places agents in a realistic developer workflow: a carefully designed user simulator starts with vague or incomplete instructions, progressively reveals requirements, inspects the agent's workspace, and provides targeted feedback, revisions, and new constraints until the full task goal has been handed off. Grounded in large-scale studies of real coding-agent interactions, this setup tests whether agents can discover user intent, adapt to evolving requirements, and build on their own prior work. Across a suite of frontier and open-weight models, we find that strong performance on single-turn SWE tasks does not reliably transfer to multi-turn, user-driven workflows: the best-performing models solve roughly 50% of single-turn baseline tasks but only 25% of the corresponding SWE-Interact tasks. The strongest models in our evaluation, including Opus 4.8 and GPT 5.5, start strong even in the face of vague initial instructions, persevere until all the requirements are surfaced by the user, integrate them better and write clean code. However, they still suffer from over-agentic coding, forgetting requirements and technical mistakes. Weaker models start poorly under ambiguity, give up early, forget or ignore instructions and rework their code more. Overall, SWE-Interact measures an orthogonal, real-world capability axis for frontier model development: interactive goal discovery and iterative refinement with a user in the loop.","upvotes":2,"discussionId":"6a4551cf4f1dd35e48fb8d98","githubRepo":"https://github.com/scaleapi/SWE-Interact","githubRepoAddedBy":"user","ai_summary":"SWE-Interact presents a testbed that evaluates coding agents in realistic multi-turn, user-driven software engineering scenarios, revealing significant gaps between single-turn performance and interactive task completion.","ai_keywords":["coding agents","user simulator","interactive software engineering","multi-turn tasks","user-driven workflows","goal discovery","iterative refinement","large-scale studies","real-world capabilities"],"ai_summary_model":"Qwen/Qwen2.5-Coder-32B-Instruct","githubStars":12,"organization":{"_id":"6677220f8a4064c02bc81217","name":"ScaleAI","fullname":"Scale AI","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/65d6a5f94c28026a003581b4/uqHyTuNQ8fX7LheVhzPeO.png"}},"canReadDatabase":false,"canManagePapers":false,"canSubmit":false,"hasHfLevelAccess":false,"upvoted":false,"upvoters":[{"_id":"6039478ab3ecf716b1a5fd4d","avatarUrl":"https://cdn-avatars.huggingface.co/v1/production/uploads/6039478ab3ecf716b1a5fd4d/_Thy4E7taiSYBLKxEKJbT.jpeg","isPro":true,"fullname":"taesiri","user":"taesiri","type":"user"},{"_id":"69a5cba5ee290d6bb49457b8","avatarUrl":"/avatars/f80c17c13d6baf6bcd375d31efe21116.svg","isPro":true,"fullname":"Darrow O'Lykos","user":"darrowoflykos","type":"user"}],"acceptLanguages":["en"],"dailyPaperRank":0,"organization":{"_id":"6677220f8a4064c02bc81217","name":"ScaleAI","fullname":"Scale AI","avatar":"https://cdn-avatars.huggingface.co/v1/production/uploads/65d6a5f94c28026a003581b4/uqHyTuNQ8fX7LheVhzPeO.png"},"query":{}}">
SWE-INTERACT: Reimagining SWE Benchmarks as User-Driven Long-Horizon Coding Sessions
Abstract
SWE-Interact presents a testbed that evaluates coding agents in realistic multi-turn, user-driven software engineering scenarios, revealing significant gaps between single-turn performance and interactive task completion.
We introduce SWE-Interact, a new testbed for evaluating coding agents on multi-turn, interactive, user-driven software engineering tasks. Existing frontier SWE benchmarks typically provide complete requirements upfront and evaluate agents on autonomous implementation. In contrast, SWE-Interact places agents in a realistic developer workflow: a carefully designed user simulator starts with vague or incomplete instructions, progressively reveals requirements, inspects the agent's workspace, and provides targeted feedback, revisions, and new constraints until the full task goal has been handed off. Grounded in large-scale studies of real coding-agent interactions, this setup tests whether agents can discover user intent, adapt to evolving requirements, and build on their own prior work. Across a suite of frontier and open-weight models, we find that strong performance on single-turn SWE tasks does not reliably transfer to multi-turn, user-driven workflows: the best-performing models solve roughly 50% of single-turn baseline tasks but only 25% of the corresponding SWE-Interact tasks. The strongest models in our evaluation, including Opus 4.8 and GPT 5.5, start strong even in the face of vague initial instructions, persevere until all the requirements are surfaced by the user, integrate them better and write clean code. However, they still suffer from over-agentic coding, forgetting requirements and technical mistakes. Weaker models start poorly under ambiguity, give up early, forget or ignore instructions and rework their code more. Overall, SWE-Interact measures an orthogonal, real-world capability axis for frontier model development: interactive goal discovery and iterative refinement with a user in the loop.
Community
We introduce SWE-Interact, a new testbed for evaluating coding agents on multi-turn, interactive, user-driven software engineering tasks. Existing frontier SWE benchmarks typically provide complete requirements upfront and evaluate agents on autonomous implementation. In contrast, SWE-Interact places agents in a realistic developer workflow: a carefully designed user simulator starts with vague or incomplete instructions, progressively reveals requirements, inspects the agent's workspace, and provides targeted feedback, revisions, and new constraints until the full task goal has been handed off.
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.
Tap or paste here to upload images
Cite arxiv.org/abs/2606.30573 in a model README.md to link it from this page.
Cite arxiv.org/abs/2606.30573 in a dataset README.md to link it from this page.
Cite arxiv.org/abs/2606.30573 in a Space README.md to link it from this page.
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.