Hi all,

Of course I had to jump on Qwen3.8-27B and get it running on the RTX 3090 port of NInfer.

For anyone unfamiliar with it, NInfer is a small, dedicated C++/CUDA inference runtime built around fused kernels rather than a huge general-purpose serving stack. The 3090 fork specifically targets Ampere GPU's (edit: just added early support for the 4090 also)

And Qwen3.8-27B does pretty damn well on a single 24 GB 3090. As far as I know, this is currently (among) the fastest throughput, single and batched for this model on this hardware. Prefill is quick too.

Qwen3.8-27B - RTX 3090

Qwen3.8-27B is now validated from C1 through C8.

One of the more interesting additions is ReplaySSM, which cuts the memory overhead of speculative decoding enough that we can keep the faster MTP3 mode enabled even at C8.

These are sustained runs, not short bursts. Every request generated 1,024 output tokens, with CUDA Graphs enabled.

So the interesting part for me isn't only the ~70 tok/s single-stream result. A single old 3090 can push 165 tok/s decode across eight concurrent requests, while staying inside 24 GB VRAM and keeping speculative decoding active. Btw I did not have time to run this headless so there is still a 10 - 15% extra in the tank, if you can run it without desktop.

Qwen3-35B-A3B still works great too. I've measured around 260 tok/s single-stream, and over 400 tok/s on highly repetitive workloads. (Still to measure concurrency well for this model, this could go over 600 tokens per second I guess?)

Huge props to Neroued for building the original NInfer project incl ReplaySSM:
https://github.com/Neroued/ninfer

The 3090 port is basically my attempt to see how far we can push these models on hardware a lot of us already have sitting under our desks. PR's very much appreaciated, we much push it beyond 100 tokens a sec, at least.

Would love to hear results from anyone else running Qwen3.8-27B on a 3090, especially llama.cpp / SGLang / vLLM comparisons.

edit:

- Addition: man this model is certainly token hungry. 170K context window now support now validated in INT8, and I added Rotorquant RK8v4, which safely gives 248K. Maybe a bit more.

- Early RTX 4090 support is running. Still needs optimalisation: