Qwen 3.8 27B Aider score

I ran the Aider benchmark on Qwen 3.8 27B FP8 with FP8 KV cache 256K context vLLM.

The score: 72.9

This matches Gemini 2.5 Pro from 2025-04-12 which also scored 72.9.

Beats Claude Opus 4 from 2025-05-25 which scored 72.0.

DeepSeek R1 2025-06-06 scored 71.4.

It may just be a random benchmark, an old one at that, but it is still wild that my MacBook is now matching those SOTA models just slightly over a year old.

In actual performance it is also wildly surpassing them because the harness got better. I have been playing the DeepSeek Harness and it would clearly solve most if not 100% of the Aider tests, although it would use more than 2 turns.