Qwen3.8-Flash-Next-NVFP4 vs Qwen3.8-27B-FP Test Results

## Qwen3.8-Flash-Next-NVFP4 (inferact) vs Qwen3.8-27B-FP8 (qwen)

Slammed with work and no time to pretty this up. Qwen wrote most of this but I checked the data.

All tests done on the same rig, same prompts, and most tests are my real workloads.

Single-GPU local eval: one RTX PRO 6000 Blackwell Max-Q (96 GB, SM120) + 256 GB DDR5, vLLM nightly, both models served alternately under the same service alias and port that my agent stack actually consumes: text scoring pipelines, memory consolidation, local deep research, browser automation, etc. Minimal coding. Because downstream consumers key off the alias, swapping the model behind it is the honest way to find out what breaks.

Models tested:

- Qwen3.8-Flash-Next-NVFP4 (https://huggingface.co/Inferact/Qwen3.8-Flash-Next-NVFP4)

- Qwen3.8-27B-FP8 (https://huggingface.co/Qwen/Qwen3.8-27B-FP8).

Every prompt, fixture, and scorer below is byte-identical between the two passes — only the served model differs.

**TL;DR:*\* Flash-Next is *faster and mechanically flawless (strict JSON, injection resistance, SLAs: all zero failures) and wins the high-reasoning spatial/code-gen tier with better failure modes. The dense 27B still wins sustained multi-step symbolic work (bug-fixing, math proofs, abstract puzzles) and this is where Flash-Next exhibits a striking new failure shape: it promises the deliverable, declares "done", and outputs nothing. Same `reasoning_effort` knob, radically different semantics. Not a drop-in replacement; a conditional promotion.

## Serving recipes (what I actually ran)

**Flash-Next:**

``` docker run vllm/vllm-openai:qwen38-flash-next \
-e VLLM_PLE_CPU_OFFLOAD=1 \ # parks ~100GB n-gram embed table in host RAM
-e VLLM_API_KEY=*** \
--entrypoint vllm serve Inferact/Qwen3.8-Flash-Next-NVFP4 \
--max-model-len 200704 \ # ~200K (262K native)
--gpu-memory-utilization 0.91 \
--max-num-seqs 16 \ # latency-first single workstation
--no-enable-flashinfer-autotune \ # hybrid-attn path picks its own backend
--structured-outputs-config '{"backend":"xgrammar","disable_any_whitespace":true}' \
--enable-prefix-caching --enable-chunked-prefill \
--reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder \
--speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
--served-model-name llm-large ```

Notes from the trenches: PyPI wheels don't support this architecture — purpose-built image only. The `xgrammar` pin above was inherited from my 27B stack and turned out load-bearing (dropping it reintroduced silent stalls). Measured: ~177 tok/s generation, MTP draft acceptance length ~2.1.

**3.8-27B:**

```bash
python3 -m vllm.entrypoints.openai.api_server \
--model Qwen3.8-27B-FP8 --max-model-len 262144 --kv-cache-dtype fp8 \
--gpu-memory-utilization 0.52 --max-num-seqs 16 --attention-backend FLASHINFER \
--structured-outputs-config '{"backend":"xgrammar","disable_any_whitespace":true}' \
--reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder \
--speculative-config '{"method":"mtp","num_speculative_tokens":2}' \
--served-model-name llm-large ```

Deliberate confound control: sampler defaults (`enable_thinking`, MTP, xgrammar) stay fixed across both; where a task pins different sampling (below), it's pinned identically for both models.

## The three test suites

**1. Capability battery** — 8 tasks × 2–3 reps, temp 1.0 / effort `medium`: bugfix-from-traceback, long-context pipeline instructions, multi-edit email, document policy audit (everyday tier); Codeforces 1117-D, ARC-AGI 227, IMO Problem 5 sketch, contamination-controlled 2026 factual recall (hard tier). Deterministic scorers, 0–1 scores. *(Battery: Flash-Next N=2/task, 27B N=3/task — flagged where it matters.)*

**2. Production grading sweep** — my actual workload: clients' text documents scored against a rubric with a strict json_schema enforcement, temp 0.3 / `medium`, 300 s per-request SLA. 27B ran the full 320-request validation (edge + injection cases); Flash-Next ran the 12-case edge/injection subset × 3 reps (36 reqs), plus 27B's full-set numbers as reference.

**3. Chessboard spatial reconstruction** — the stress test: given a 7-move PGN, emit a valid SVG of the board with all 30 pieces on exact squares and the last move highlighted. Swept the `reasoning_effort` axis (xhigh/medium/low/off), Flash-Next N=5/arm, 27B N=3/arm, scorer checks geometry against ground truth; contested renders tied-broken by visual inspection.

## Results

### Scoring sweep (the workload this GPU actually pays rent for)

| | 27B-FP8 | Flash-Next-NVFP4 |

|---|---|---|

| Requests (validation sweep) | 320 | 36 (edge+injection subset) |

| Schema-invalid JSON | 0 | 0 |

| SLA breaches (>300 s) | 0 | 0 |

| Rubric-exact score | 319/320 (99.7%) | 33/36 (91.7%) |

| Prompt-injection resisted | 54/55 | 9/9 |

Every one of Flash-Next's three misses is the same cell: a legitimate item buried in keyboard-mash garbage. 27B awarded partial credit ten times straight; Flash-Next gave 0 with coherent rubric reasoning all three reps. This isn't flakiness — it's a stable, argued re-calibration of the noise-tolerance boundary. Mechanical guarantees (pure JSON, no timeouts, injection-proof) were perfect on both; semantic judgment at rubric edges did not. Adding one line to my system rubric ("noise-buried items still earn partial credit") would likely closes this gap but I have not tested it.

### Capability battery (effort `medium`, temp 1.0)

| Task | 27B (N=3) | F-N (N=2) | |

|---|---|---|---|

| Bugfix from traceback | **0.900** | 0.525 | regression |

| Long-context instructions | **0.917** | 0.675 | regression |

| Multi-edit email | **0.887** | 0.870 | ~tie |

| Document audit | **0.651** | 0.611 | ~tie |

| Codeforces 1117-D | **0.667** | 0.562 | coin-flip tier |

| ARC-AGI 227 | 0.615 (≤216 s) | **timed out >420 s, both reps** | regression |

| IMO P5 sketch | **0.533** | 0.300 | regression |

| 2026 factual recall | 0.250 | 0.250 | floor, both |

| **Everyday mean** | **0.839** | 0.670 | |

| **Hard mean** | **0.516** | 0.371 | |

Formatting-heavy everyday work barely budges. Sustained symbolic manipulation bleeds — and ARC is where Flash-Next got weirder: both reps burned 7+ minutes at 100% GPU with spec-decoding acceptance length collapsing toward ~1.0 (drafts rejected ~forever, greedy detritus grinding), never converging, while the dense model solved the same cells in ≤3.5 min. A livelock, effectively.

### Chessboard (exact board + correct highlight, per arm)

| Effort | 27B (N=3) | Flash-Next (N=5) |

|---|---|---|

| **xhigh** | 1/3 (best-of: 2/3 piece-exact; one **fully empty board**) | **3/5** — misses are 1–3-square near-misses; zero empty boards; ~20% faster wall (165 s vs 206 s) |

| medium | 1/3 | **0/5** — three reps emitted *no SVG at all* |

| low | 1/3 | 1/5* |

| off | 0/3 | 0/5 |

\* scored from SVG markup (renderer-crop artifact on the preview); excluding it makes low 0/5 — no conclusion changes either way.

Different failure taxonomies: 27B's xhigh occasionally **blows up catastrophically** (17.8k tokens consumed, empty board shipped, `finish_reason: stop`). Flash-Next's xhigh **rarely blows up** — its errors are localized 1–3 piece slippage. And at `medium`, Flash-Next's signature failure is the scariest thing in this whole benchmark: `finish_reason: stop`, response ends *"Here is the final SVG:"* — followed by nothing. It believes it delivered.

## Surprising findings

1. **`reasoning_effort` is not portable across architectures.** "Medium" on the dense 27B is a reliable workhorse setting. On Flash-Next it reliably produces *phantom deliverables* (generation declares done, artifact absent) and degraded boards. "xhigh" on Flash-Next is *better and faster* than "xhigh" on the 27B for this task. The reasoninf knob's semantics are model-specific.
2. **Failure morphology flips from gradient to cliff.** 27B: mediocre-but-present outputs, rare catastrophe. Flash-Next: bimodal — near-perfect or structurally absent, plus pathological token loops (18–22k detritus, finish=stop) and the ARC-style acceptance-collapse livelock. Design wrappers with *artifact validation*, not just timeout guards.
3. **Strict mechanics are perfect in both.** Zero invalid strict-JSON across 36/36, zero injection failures, zero SLA breaches. The xgrammar structured-output path is rock solid on this architecture.
4. **Throughput ≠ reasoning time.** Happy-path speed favored Flash-Next ~1.3–1.9× everywhere (MoE sparsity + MTP×3 at ~177 tok/s), yet it needed *hours of patience* on the puzzle where the smaller dense model finished in minutes.
5. **Token ceilings bit harder than expected.** Cutting Flash-Next's output budget at ~12k corrupted more of its generations than every other failure mode combined in the dense runs — its thinking chains are chattier. Budget ≥16k, or expect truncation-shaped corruption.
6. **Plumbing gotcha for anyone self-hosting this family:** PyPI vLLM can't load it (dedicated image only), `VLLM_PLE_CPU_OFFLOAD=1` is mandatory on a single 96 GB card, forced attention backends fight the hybrid path, and my naive sequential-requests harness hung in Python interpreter teardown after long generations — a poll-and-kill driver fixed it. All solvable, none documented anywhere I could find at the time.

## My Personal Conclusions (not LLM-written)

- Qwen3.8-27B-FP8 is a more reliable overall workhorse than Qwen3.8-Next-Flash-NVFP4. That may change with more mature vLLM support and better quants, but for right now Next-Flash is not reliable enough to run in production.

- Next-Flash has a clear speed advantage. It's noticeably faster, at least until it starts going on a wild thinking spree and burns 12k tokens before any outputs.

- 27B set reasoning to 'medium'. Flash-next set it to xhigh. 27B is much more reliable and stable as a production model at medium. 27B at xhigh has more catastrophic failures and thinking loops. BUT 27b at xhigh will also have some huge wins. It's bimodal in its quality. Flash-Next wants xhigh all the time. Medium of Flash-Next is a mess and unusable.

- Low is usable but poor quality and not really fewer tokens that medium on either model.

- **Ban `off` (no-thinking) on both models** — useless on either at any task we tried. Unlike Qwen3.6-27B, turning thinking/reasoning off cripples both 27B-FP8 and Flash-Next.

## Caveats

This was a small personal test based in part on hard edge cases but leaning heavily into my own daily workload.

Small-N territory on the battery (2 vs 3 reps) — treat sub-0.1 deltas as directional, the big ones (ARC, bugfix) as directional-but-real. Single machine, single operator, private fixtures (no public leaderboard overlap; the "hard" tier deliberately mixes contamination-controlled novel problems).

vLLM may be part of the problem. I can't 100% blame Flash-Next when vLLM support is much less mature than it is for the Qwen3.8-27B architecture.