Experience report - Qwen 3.8 Flash Next on memory rich, GPU poor setup

(not written by Claude, all errors and crappy text are result of too little coffee on a Sunday morning ;)

Our home server is a 2018 Thinkstation P520, bought for about 600eur in 2023.

It's been upgraded with a 2TB Samsung 980 Pro NVMe, a Xeon W-2145 and a 12GB 3060 - total cost about ~1k all in. Not nothing, but not a crazy amount of cash for all the capability it provides. 256GB ECC DDR4 at 2666mhz, quad channel at about 80GB/s.

Qwen 3.6 35b a3b Q4_K_M was the daily driver, on builds of llama.cpp with intel MKL extensions at compile time. It's not the smartest model, but just about good enough for doing basic tasks. The quant does lobotomise it, but on this setup larger quants radically impact the throughput.

Qwen 3.6 35b a3b Q4_K_M Resident: ~20GB of RAM Prefill: ~400tps Gen: 30-50tps CTX: 128k VRAM: ~10.5GB.

Flash next is a completely different beast and even though it's a much bigger model, the throughput and prefill hold up reasonably well. The sheer size of the model is what conditions performance in this setup, not unsurprisingly.

Qwen 3.8 Flash Next UD-Q4_K_XL Resident: ~110GB of RAM Prefill: ~200tps Gen: 12-15tps CTX: 65k VRAM: ~10.5GB.

It's slow, and low context, but the output is night and day better than the 35b.

A couple of interesting things popped up: - 35b speeds are very tolerant to the box being busy on other tasks while working, losing little performance. To be expected as more of the model fits on GPU - Flash next perf falls to pieces if anything else is being done on the box (even running opencode) with speeds dropping to 3-5 tps. Memory is being absolutely hammered and is extremely sensitive to contention. It's only really usable if I run opencode from another box. - synthetic, random content benchmarks gave completely wrong answers on performance. Make sure you use contexts that are realistic to measure MoE models. This tripped me up while tuning the server, and only popped up when moving to opencode to really try it out.

Below for info are the two llama.cpp server invocations. I chose to let Flash next have unbounded thinking to get full quality, and 35b is limited for speed. It does make it dumber for sure. I use the uncensored 35b as an experiment in making it faster, less time hand wringing over permissions, which seems to work.

Flash next: llama-server \ --model /mnt/storage/models/qwen38-flash-next/UD-Q4_K_XL/Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf \ --mmproj /mnt/storage/models/qwen38-flash-next/mmproj-F16.gguf \ --no-mmproj-offload \ --alias qwen3.8-flash-next \ --host 0.0.0.0 \ --port 8080 \ --n-gpu-layers 999 \ --n-cpu-moe 48 \ --batch-size 2048 \ --ubatch-size 2048 \ --load-mode none \ --threads 16 \ --threads-batch 16 \ --threads-http 2 \ --ctx-size 65536 \ -ctk f16 \ -ctv f16 \ --flash-attn on \ --cache-reuse 256 \ --temp 0.7 \ --top-p 0.80 \ --top-k 20 \ --min-p 0.0 \ --presence-penalty 1.5 \ --repeat-penalty 1.0 \ --jinja \ --reasoning-format deepseek \ --parallel 1 \ --slots \ --slot-save-path /mnt/storage/models/.cache/slots \ --metrics \ --log-timestamps \ --timeout 600

35b: ``` llama-server \ --model /mnt/storage/models/qwen36/Qwen3.6-35B-A3B-uncensored-heretic-Native-MTP-Preserved-Q4_K_M.gguf \ --mmproj /mnt/storage/models/qwen36/mmproj-Qwen3.6-35B-A3B-Abliterated-Heretic.gguf \ --no-mmproj-offload \ --no-mmap \ --alias qwen3.6-35b-a3b-mtp \ --host 0.0.0.0 \ --port 8080 \ --ctx-size 131744 \ --batch-size 1024 \ --ubatch-size 512 \ --threads 8 \ --threads-batch 8 \ --threads-http 4 \ --n-gpu-layers 999 \ --n-cpu-moe 26 \ -ctk q4_0 \ -ctv q4_0 \ --flash-attn on \ --cache-reuse 256 \ --spec-type draft-mtp \ --spec-draft-n-max 1 \ --temp 0.6 \ --top-p 0.95 \ --top-k 20 \ --min-p 0.0 \ --presence-penalty 0.0 \ --repeat-penalty 1.0 \ --reasoning-budget 1200 \ --reasoning-budget-message '

Considering the limited time by the user, I have to give the solution based on the thinking directly now. ' \ --jinja \ --reasoning-format deepseek \ --parallel 1 \ --slots \ --slot-save-path /mnt/storage/models/.cache/slots \ --metrics \ --log-timestamps \ --timeout 600 ```

Flash Next quality is excellent for "fire and forget" tasks that don't need interactivity. Being able to swap for different kinds of tasks is the key, with the weights on NVMe to keep loading time reasonable.

No MTP on Flash Next yet, but looking forward to trying it out.

Anyone else running flash next on ancient boxes? I'd be curious how it compares to other constrained setups.