Qwen3.8-27B scored 29/30 on AIME 2026 with FP8 + xhigh reasoning — BF16 vs FP8 results
Mirrored from r/LocalLLaMA for archival readability. Support the source by reading on the original site.
I benchmarked Qwen3.8-27B on MathArena/aime_2026 dataset, comparing BF16 and FP8 weights at medium and xhigh reasoning effort.
Interesting findings are:
- quantized FP8 xhigh is better than BF 16 medium equally good as 16 BF xhigh with better speed.
- On problem 7, both BF16 xhigh and quantized FP8 xhigh exhausted the full context token generation budget without producing a final answer. So those were empty not wrong.
Final results
| Model · thinking | Score | Max tokens | Pre-fill tk/s | Decode tk/s |
|---|---|---|---|---|
| BF16 · medium | 28/30 (93.3%) | 58,747 | 1.9K | 28 |
| BF16 · xhigh | 29/30 (96.7%) | 258,048 | 1.9 | 28 |
| Quantized FP8 · medium | 26/30 (86.7%) | 75,024 | 3.4K | 76 |
| Quantized FP8 · xhigh | 29/30 (96.7%) | 258,048 | 3.4K | 76 |
Evaluation settings
- Dataset:
MathArena/aime_2026, - Dataset revision:
d2de22f3c656b4f56cf8981212186377d1e23bc3 - Exact-match scoring
- Sampling: disabled
- BF16 concurrency: 4
- FP8 concurrency: 7
- Same chat template and prompt format across all runs
- Temperature: zero
How it compares with frontier models
Note : looks like others ran it multiple times on their models and averaged, I ran it only once.
| Model and setting | Reported AIME 2026 score |
|---|---|
| GPT-5.6 Sol, xhigh | 99.9%* |
| GLM-5.2 | 99.2% |
| GPT-5.4, xhigh | 99.2% |
| Gemini 3.1 Pro, high | 98.3% |
| Inkling, effort 0.99 | 97.1% |
| Claude Opus 4.6, max | 96.7% |
| DeepSeek V4 Pro | 96.7% |
| Qwen3.8-27B FP8, xhigh — our pass@1 run | 96.7% (29/30) |
| Kimi K2.6, thinking | 96.4% |
| Qwen3.6-27B | 94.1% |
[link] [comments]
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.