I spent 3 weeks testing local Qwen3.8 on the new low-latency SGLang/vLLM recipes: DFlash2 2.8x. Builds: RadixArk + Inferact 27B NVFP4, 27B BF16, orcarouter 27B Uncensored, Flash-Next NVFP4
Mirrored from r/LocalLLaMA for archival readability. Support the source by reading on the original site.
| | Hey guys, Last time I tested Qwen3.8-Flash-Next on its own. This time I put three Qwen3.8 checkpoints through the same 10 tests on the same RTX PRO 6000:
Each model got the same prompts and its own model card's sampler, with one attempt per task. Video with the battles, the castles and the ball run: https://youtu.be/VOtfja_Toj4 Short version
Setup
One workstation, one model server at a time, and every number comes from a saved run. 1. Speed: drafters, SGLang vs vLLM, and long promptsFor the 27B I ran a speed matrix: every drafter, two engines, and four builds (three NVFP4 exports, one of them the Uncensored fine-tune, plus full-precision BF16). Each arm got its own server from a cold boot, the card's sampler and the 400W cap. "One user" is the Spec-Bench median over its 480 prompts. Engines: Which drafter (tok/s, one user):
DFlash2 wins on both engines. It keeps about 3.7 drafted tokens per step, against 2.9 for MTP. Which build, on which engine (tok/s):
The three models:
The 27B keeps writing at 223 tok/s with a full 245K-token window behind it. Speculative decoding depends a lot on the content: maths ran at 339 tok/s, roleplay at 149. The pattern was the same for both 27B builds. One important caveat. Flash-Next's speed test ran on 2026-09-12 at a 600W power cap. I later moved the card to 400W, and the 27B matrix ran at that cap. In my power sweep, prefill lost about 6% per 50W removed, so some of the gap is the cap. Moe also helps 2. Tool use: the dense 27B leadsThis is a 900-case BFCL v4 subset (11 categories), not the full leaderboard. Thinking was off, with temperature 0.7 and top_p 0.8 from the card.
These aren't comparable with my last post's Flash-Next BFCL numbers, which used temperature 0. 3. Long context: perfect for all threeI hid a fact in a log file that filled 33%, 66% or 99% of the 262K window, at three depths, with three needle types. The cache was flushed before every request.
The largest prompt was about 259.5K tokens. 4. Battle arena: the local 27B beat ClaudeEach model gets a rules sheet and a 1,000-point budget. In the open arena it designs one army blind and fights 13 armies: nine historical references plus the other entries. The score is 1,000 × its average win rate. Every matchup is 200 deterministic battles (100 seeds, sides swapped).
In the gauntlet, the model sees each enemy and builds a counter. The 27B beat 12/15, the Uncensored 12/15 and Flash-Next 13/17. Flash-Next ran an earlier version of the gauntlet with two more enemies, so treat that row as close, not ranked. Thinking cost: the 27B's arena army took 39K thinking tokens in 4 minutes. Flash-Next's took 73K in 9 minutes. 5. The SVG test is also a fact checkPrompt: find out which card local-AI hobbyists run and which current open model fits it, then draw the card lifting the model, labelled with a quant and a size that fit. All three picked the RTX 3090. I checked every label against what each session actually fetched.
All three passed 8/8 format checks. The 27B looked at its render twice and Flash-Next three times, where the rule allows one look. 6. Video editing, voxel and designVideo editing: the model gets a raw 132-second take with fillers, a retake and a swear. It never sees the footage, only transcription and silence-detection tools, and then edits through FableCut's tools. There are two cases, each scored by hand out of 10:
Voxel (Wawel Castle in three.js), ranked by eye:
Flash-Next also used the fewest thinking tokens there: 73K, against 101K for the 27B. Design (an animated explainer board in my design system), ranked by eye: Flash-Next first, and the two 27Bs shared second. All three passed 7/7 hard rules. 7. Rube Goldberg: only one machineThe setup is a fixed level: a ball on a ledge, a cup on the floor and a wall in between. The model writes a parts list (no code), and a 2D physics engine runs it. It can run and look as often as it likes within 90 minutes. The score is automatic: does the ball itself end in the cup? Flash-Next: yes, at 12.4s. It made 49 simulator runs and used 40 parts (35 of them moved). The ball travelled 1,672 px. It used 224K thinking tokens and compacted its context 8 times. 27B and Uncensored: no machine. Both spent about 111K tokens thinking in their first answer, reached the per-answer limit and stopped before writing a single part. Everyone got the same rules and one attempt. A rule that let a model continue after hitting the limit might change this, and I haven't tested that yet. 8. CAPTCHA: local models in a real browserI used Open CaptchaWorld (20 CAPTCHA types, two of each). The model only sees screenshots and only acts with the mouse and keyboard. The site's own checker marks the first answer, and it must arrive within 7 minutes.
With its own 20-minute limit, Flash-Next solved 23/40. The paper reports 93.3% for humans and 40% for the best agent on its full set, which isn't the same 40 puzzles. With one run each, a three-puzzle gap is not a strong signal. Which one should you run?
ResourcesConfigs, Docker setup and reports The test harness is still private while it's changing. Full video with the battles, the castles and the ball run: https://youtu.be/VOtfja_Toj4 I abused AI to help write this up and to check it against the report. Every number above comes from a saved run. Which test do you find most interesting and maybe you have some other creative ideas how to test models? [link] [comments] |
More from r/LocalLLaMA
-
The AI race: China's open-weight models trail US closed models by just 7 months
Oct 14
-
Anthropic Executive Says Open-Source AI Models Are Higher Risk
Oct 14
-
ComfyUI v0.40.0 released
Oct 14
-
So this is how they do it --> RTX 5090, once a consumer gaming card, is now on smugglers’ menu alongside cigarettes
Oct 14
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.