Qwen3.8-27B NVFP4 with vision + 451K token KV-cache on one RTX 5090 (power limited to 400W) at 120 tokens/s average

Hello,

So I've been trying lots of combinations in that never-ending landscape of options and settings.

I wanted a proper quant of 3.8 27B running as fast as possible on my 5090 at 400W, with vision and with as much KV-cache as possible and with concurrency enabled (aiming at 3 parallel sessions).

tl;dr: I'm using this setup and it works very well, very fast and is accurate in coding sessions.

---

vLLM is the obvious choice, even if not user-friendly at first, once it runs, it runs well.

I tried fancy DSpark / DFlash2 but the cost in context size wasn't worth the marginal gains vs MTP=3.

Here is a full NVFP4 setup (model and cache):

Model: https://huggingface.co/gittensor-model-hub/Qwen3.8-27B-NVFP4-RTX5090

Context: 196K per session / 451K global KV-cache (NVFP4)

Speed: conservative average at 120 tps.

(tests from llama-benchy 0.4.0)

Full write-up with how to setup vLLM and every gotchas: https://gist.github.com/co-l/c2aeaf40b53fcacfe9dd3293be75f23a (+ more benchmarks, including a real coding session, in line with numbers above).

Note: env is Linux (Bazzite of all distros lol), with UI deactivated to allow 0.98 mem use.