Qwen3.8-Flash-Next 177B NVFP4(119GiB): SSD streaming at 9-10 tok/s on one 16 GB RTX 5060 Ti + 32 GB RAM

We built an inference engine for MoE models that don't fit in VRAM + RAM. Most of the model stays on the SSD, and experts are read as tokens need them.

This started as a proof of concept, and poc worked, we are getting 9-10 tok/s decode on Qwen3.8-Flash-Next NVFP4 (9.06 on the benchmark turn, 10.4 on the best turn).

This is just the start. With better SSD streaming, we expect v2 to reach ~14-15 tok/s decode.

Model: Qwen3.8-Flash-Next, 176.9B params, NVFP4 GGUF (119 GiB): https://huggingface.co/CompiledThoughts/Qwen3.8-Flash-Next-NVFP4-Q8_0

Machine: RTX 5060 Ti 16 GB, Ryzen 7 9700X, 32 GB DDR5, Gen5 NVMe SSD 1Tb, Windows 11

Where the 119 GiB lives

part of the file	size	where
dense weights (attention, shared experts, LM head)	4.4 GiB	VRAM
token embedding table	0.6 GiB	RAM, one row read per token
hottest routed experts	8.8 GiB	VRAM
next-hottest routed experts	6.0 GiB	pinned RAM
remaining routed experts	48.5 GiB	SSD, streamed on demand
n-gram table	50.7 GiB	SSD, 16 rows read per token

So 20 GiB is in memory and 99 GiB stays on the SSD: 48.5 GiB of routed experts, streamed as the router picks them, and the 50.7 GiB hashed n-gram table (looking forward to qwen4 ngram).

Speed

• Decode: 9.06 tok/s on the benchmark turn, 10.4 on the best turn at xhigh effort
• Prefill: 49.2 tok/s on a 5.5k-token prompt
• llama.cpp on the same machine: 4.9 tok/s average decode

How it works

• VRAM holds the dense weights and the hottest experts (GCLOCK eviction), pinned RAM the next tier (read over PCIe), and the rest come off the NVMe on 8 read threads.
• Lookahead prefetch guesses the next layer's experts and starts their reads early.
• NVFP4 matmuls run FP4 x FP4 on the tensor cores, with no unpacking first.
• About 270 MiB is read from the SSD per token, and roughly 75% of expert lookups hit memory.
• Each token uses 480 experts (10 in each of 48 layers). About 377 of them are already in VRAM or RAM; the other ~103 are read from the SSD, about 270 MiB per token. This hit hit-rate is what allowed us to reach 9tps.
• It only reads from the SSD and almost no writes so ssd should have minimal wear due to writes. But we saw SSD hit 70C during long runs.

Also supported: Qwen3.6-35B-A3B NVFP4. It fits in VRAM + RAM . You can also run it via ssd streaming and it was the learning curve for this work. On the same machine with tuned config it hits: 47.3 tok/s decode at ~4k context, 591 tok/s prefill.

Serving: an OpenAI-compatible server that renders the model's own chat template, with tool calls (still buggy and tested with Cline), the reasoning split out from the answer, and a built-in chat page.

Limits of v1: RTX 50-series / Blackwell (sm_120) only, tested on Windows 11 and WSL2 only, greedy decoding only.

Links

• Repo: https://github.com/compiledthoughts/Inferred-Thoughts
• Qwen3.8-Flash-Next 177B NVFP4: https://huggingface.co/CompiledThoughts/Qwen3.8-Flash-Next-NVFP4-Q8_0
• Qwen3.6-35B-A3B NVFP4: https://huggingface.co/CompiledThoughts/Qwen3.6-35B-A3B-NVFP4-Q8_0-it
• All models: https://huggingface.co/CompiledThoughts

Questions and feedback welcome, especially from anyone running big MoEs on small cards.