r/LocalLLaMA · · 1 min read

5090: Windows or Linux for Qwen3.8.27b

Mirrored from r/LocalLLaMA for archival readability. Support the source by reading on the original site.

I've got a dedicated AI rig sitting here with a RTX 5090 and 96GB RAM and for the past few years have been using Windows 11 and primarily LM Studio, but have also used vLLM, llama.cpp and Ollama.

With Qwen3.8.27b I want to get the most out of this model. I get the feeling from the community that LM Studio on Windows might not be the best way to go.

If you had this rig sitting on your desk and wanted the best environment to host Qwen3.8-27b, what would you use for it? (Intel i9-14900K, RTX 5090, 96GB DDR5)

Windows 11 + which inference server
Linux + which OS (Ubuntu?) & inference server

I'm happy to replace Windows 11 with Linux if that's the best way to get the maximum performance out of this model. I'm not tied to Win11 if this is holding me up. I don't use it for anything else.

UPDATE:

After testing different KV cache settings with qwen/Qwen3.8-27B-Q6_K.gguf on my 5090 (Windows, LM Studio -- the t/s as reported via LM Studio built-in chat) with default MTP (0-3, 0.75):

  • Q8_0 @ 128K: 28.9/31.5 GB VRAM, 88–92 t/s
  • Q4_0 @ 128K: 26.8 GB, ~87 t/s
  • Q4_0 @ 256K: 30.3 GB, ~88 t/s
  • Q5_0 @ 256K: 29.9 GB, ~75 t/s
  • Q8_0 @ 256K: 31.2 GB, 73 t/s first run, then 52 t/s

So for now, Q4_0 @ 256K looks like the sweet spot: full context, still ~88 t/s, and enough VRAM headroom that I'm not riding the absolute limit.

submitted by /u/toolman10
[link] [comments]

Discussion (0)

Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.

Sign in →

No comments yet. Sign in and be the first to say something.

More from r/LocalLLaMA