specs

• hardware: M4 Max 128GB Studio
• inference engine: llama.cpp (qwen4exp branch)
• judge: claude-opus-4-6

AtomicChat/Qwen3.8-Flash-Next-GGUF

Qwen3.8-Flash-Next is a great model I benched in my previous post, but it is very tight, since all n-grams / PLE are loaded along with the experts, taking 106GB, leaving very little room for K/V, context, etc. Offloading PLE to SSD currently slows down prefill from 600 t/s to 180 t/s on oMLX.

u/erikdhoward suggested to try the Atomic Chat quant which I did not know anything about. I tried it, and it is... really good.

AtomicChat's quant uses llama.cpp mmap (through GGUF shard layout vs. in the runtime) and keeps the PLE table (n-grams) pageable backed by a file. Because of this the same model that took 106GB, now takes 65GB (starts from 55GB) in RAM. And since PLE is pageable the prefill is actually not that bad, cold start is about 500 t/s.

oMLX "right behind you!"

Qwen3.8-Flash-Next just came out, and there are many open PRs in oMLX to address the size and performance, including this one that makes PLE offload SSD cold prefill almost 3 times faster 🎉