Qwen3.8-Flash-Next on six 3090s with vLLM: 7 users at 262k, 6.5k t/s prefill, 57 t/s decode solo / 206 t/s aggregate with all 7 generating. Fixes for the upstream included.

Write-up, serve command and raw data: https://github.com/lukolszewski/vllm

This shows vLLM fixes and setup that let you put PLE on SSD and save 108GB of system RAM, use int8 kv and squeeze more users in your VRAM without the performace hit of fp8. Speeds in comparison to my earlier llama-multigpu performance fixes https://www.reddit.com/r/LocalLLaMA/comments/1x023z4/qwen38flashnext_on_6x3090_6x4090_without_nvlink/

Upstream vLLM does not start this model on that machine at all: the model refuses --pipeline-parallel-size > 1, TP has to be 1/2/4/8 so six cards are useless for it, and a 24 GB card cannot hold it at TP=1. Putting the 102 GB n-gram (PLE) table on SSD has just showed up as PR on upstream today (keeping it in RAM is the typical setup), and fp8 KV crashes on Ampere.

Mashine is a consumer PC, no epyc/threadripper, no P2P, still pretty usable (6x3090, 7950X, 187 GB RAM, four cards behind Thunderbolt), GPTQ-INT4 checkpoint (nvfp4 was slower), one streaming request per cell, tokens/s:

Things that did not pan out: NVFP4 checkpoints load with no changes but are 10-12 % slower than GPTQ on these cards, both go through Marlin weight-only, there is nothing for FP4 to accelerate on Ampere. And per-user decode under pipeline parallelism falls with the user count (57 -> 44 -> 38 -> 29 at 1/3/5/7 users) while llama.cpp's grouped decode stays nearly flat (34 -> 30 at 5); the sums are similar at five users. So: vLLM for prefill, single-user speed and seat count; llama.cpp if you want every user to see the same speed (I posted a similar set of fixes for llama earlier).

The AMD side exists too: u/alanoo81 ran the model at PP=3 on Radeon V620s earlier and that is where the PP idea came from; on current upstream only the guard removal is still needed and the branch keeps a partition check instead. If you are on AMD start there.

Caveats: one model family, NVIDIA only, tested on exactly one machine (plus kernel unit tests on a 5060 Ti). Batched outputs are not byte-identical to solo runs, vLLM's usual batch nondeterminism. Load to healthy is 4-5 minutes. As with the llama.cpp work, the code was written with an LLM and validated by measurement and output checks (needle tests, kernel outputs against oracles), not by review; each change is one commit, take what you need. This is a fixing branch, not a fork I intend to maintain.