Prismix
r/LocalLLaMA · · 1 min read

Qwen3.8-Flash-Next NVFP4 Day-3 support for 4xV100

#model-release

Mirrored from r/LocalLLaMA for archival readability. Support the source by reading on the original site.

Like Read original ↗
Qwen3.8-Flash-Next NVFP4 Day-3 support for 4xV100

RadixArk/Qwen3.8-Flash-Next-NVFP4 is now supported in SGLang-V100. 4 V100 32GB running full context.

A little more than 50 GB ngram offloaded to system RAM. Prefill around 4000tks flat and decode around 60 tks all the way to the end of the 256k context.

Prompt TTFT no MTP TTFT MTP ITL no MTP ITL MTP Prefill no MTP Prefill MTP Output no MTP Output MTP
10k 2,154 2,307 16.75 12.33 4,655 4,354 59.93 81.39
30k 6,395 6,747 16.83 15.22 4,696 4,454 59.64 65.97
50k 11,381 11,497 16.89 18.17 4,397 4,353 59.45 55.26
70k 15,272 16,015 16.97 15.50 4,586 4,374 59.17 64.76
90k 19,830 20,777 17.00 18.92 4,541 4,334 59.04 53.06
110k 24,733 25,856 17.09 12.34 4,449 4,256 58.74 81.33
130k 29,553 30,950 17.14 12.19 4,400 4,202 58.57 82.33
150k 34,736 36,319 17.23 16.32 4,320 4,131 58.26 61.53
170k 39,705 41,747 17.36 17.42 4,283 4,073 57.83 57.64
190k 45,259 47,513 17.45 17.20 4,199 4,000 57.53 58.37
210k 51,017 53,538 17.59 16.10 4,117 3,923 57.07 62.36
230k 57,273 60,011 17.61 14.62 4,017 3,833 57.00 68.65
250k 63,451 66,587 17.66 16.04 3,941 3,755 56.85 62.58

Concurrency 8192->1024

Concurrency Output no MTP (tok/s) Output MTP (tok/s) MTP Difference TTFT no MTP (ms) TTFT MTP (ms) ITL no MTP (ms) ITL MTP (ms) MTP Accept Length
1 55.12 74.82 +35.7% 1,552.28 1,637.65 16.62 11.76 3.375
4 137.26 149.25 +8.7% 4,856.90 5,049.17 24.39 20.04 3.089
8 172.98 194.13 +12.2% 8,000.80 8,329.74 38.43 28.94 3.130
16 239.36 183.87 -23.2% 14,063.34 14,524.25 53.12 55.64 3.048

https://github.com/haohervchb/sglang-V100

submitted by /u/Primary_Exchange21
[link] [comments]

Discussion (0)

Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.

Sign in →

No comments yet. Sign in and be the first to say something.

More from r/LocalLLaMA