Qwen 3.8 27B SlopCodeBench results
Mirrored from r/LocalLLaMA for archival readability. Support the source by reading on the original site.
Howdy, I'm back again - running my favorite benchmark (it's still unsaturated for the time being so might as well!)
https://github.com/michaelasper/benchmarks/blob/main/qwen3.8-27b-pi-on-slop-code-bench.md
I ran this via OpenRouter because my mac would cry running 9 problems
It did pretty poorly on the strict checkpoints which means you probably don't want Qwen managing the codebase by itself as it'll grow unwieldy and disorganized, but did fairly well on the core checkpoints so it can solve issues with a copilot and clear direction
AI;DR here are the direct results
HumanLayer Opus 5 Benchmark Subset (3 Problems, 17 Checkpoints)
Qwen scored 3/17 (17.6%) strict.
| Reported System | Strict Score |
|---|---|
| DeepSeek V4 Flash 0731 · pi (run B) | 5/17 (29.4%) |
| Opus 5 · Claude Code | 4/17 (23.5%) |
| Qwen3.8-27B · pi | 3/17 (17.6%) |
| DeepSeek V4 Flash · OpenCode | 3/17 (17.6%) |
| Opus 4.8 · Claude Code | 1/17 (5.9%) |
| Sonnet 5 · Claude Code | 1/17 (5.9%) |
HumanLayer Fable, Sol, and Kimi Benchmark Subset (6 Problems, 30 Checkpoints)
Qwen scored 4/30 (13.3%) strict.
| Reported System | Strict Score |
|---|---|
| Fable 5 · Claude Code | 10/30 (33.3%) |
| GPT-5.6 Sol · Codex | 10/30 (33.3%) |
| Kimi K3 · Modal / OpenCode | 8/30 (26.7%) |
| Kimi K3 · Baseten / OpenCode | 7/30 (23.3%) |
| Qwen3.8-27B · pi | 4/30 (13.3%) |
[link] [comments]
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.