(NInfer Fork) I wanted to have a 1M context Qwen-3.8 27B, tp2, dual 5090s

Hey! I forked NInfer (a from-scratch C++20/CUDA inference engine for Qwen models) and added two things: tensor-parallel across two GPUs, and YaRN ×4 rope scaling. Together they let Qwen3.8-27B NVFP4 run a 1,048,576-token context on two consumer 5090s — 27.4 GB per card, no NVLink.

Numbers (single stream, 500 W per GPU cap):

• Decode at 653k context: 119 tok/s with MTP speculative decoding, 57 without. vLLM on the same prompts: 42 tok/s.
• Why: past its native 262k window, vLLM's MTP acceptance drops to exactly zero (0 of 1,533 drafts accepted) — it keeps paying for the drafter. NInfer's acceptance stays ~55–60% out to 1M.
• Prefill is where vLLM wins: 1.2–1.3× faster. A full 1M prompt takes ~18 minutes on NInfer. That's the untuned part.
• At 1M: 48 tok/s decode, ~100 with MTP.
• vLLM's fp8 KV cache tops out at ~759k tokens on this hardware; NInfer's INT8 KV fits 1,048,576 in less memory.

Two GPUs are also just faster than one: 75 vs 54 tok/s at 250k, because weights and KV traffic halve per card and the ~128 cross-GPU reductions per token cost only ~0.2 ms under CUDA graphs.

Fork

Performance