Qwen3.8-Flash-Next turns 4xR9700 into a local AI powerhouse! 120 t/s TG and 12k t/s PP single request with optimized vLLM

If you own 4xR9700 and were waiting for the model to make them shine, then I have some good news for you!

It's running at 80-120 tokens/second for generation and 12k token/second prefill for a single request, using tcclaviger's MXFP4-FP8 quant and custom vLLM image docker.io/tcclaviger/vllm:DevQwenNextFlash optimized for R9700.

Here is the full command:

podman run --rm -it \ --init \ --network host \ -v /models:/models:ro \ -v ~/.vllm-cache:/cache \ -e VLLM_PLE_CPU_OFFLOAD=1 \ -e VLLM_ROCM_USE_AITER=0 \ -e ROCR_VISIBLE_DEVICES=0,1,2,3 \ -e VLLM_CACHE_ROOT=/cache/vllm \ -e TORCHINDUCTOR_CACHE_DIR=/cache/inductor \ -e TRITON_CACHE_DIR=/cache/triton \ --device /dev/kfd \ --device /dev/dri \ --group-add keep-groups \ --annotation run.oci.keep_original_groups=1 \ --security-opt label=disable \ --security-opt seccomp=unconfined \ --shm-size 8g \ docker.io/tcclaviger/vllm:DevQwenNextFlash \ /models/tcclaviger/Qwen3.8-Flash-Next-MXFP4-FP8 \ --served-model-name Qwen3.8-Flash-Next \ --tensor-parallel-size 4 \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder \ --enable-auto-tool-choice \ --max-num-seqs 16 \ --enable-prefix-caching \ --enable-chunked-prefill \ --kv-cache-dtype fp8 \ --max-num-batched-tokens 4096 \ --gpu-memory-utilization 0.96 \ --mm-processor-cache-gb 4.0 \ --override-generation-config '{"max_tokens": 65536, "temperature": 1.0, "top_p": 0.95, "top_k": 40, "presence_penalty": 1}' \ --speculative-config '{"method": "mtp", "num_speculative_tokens": 4}' \ --compilation-config '{"cudagraph_capture_sizes": [5,10,15,20,25,30,35,40], "max_cudagraph_capture_size": 40}' \ --host 0.0.0.0 \ --port 8080