How I got Qwen 3.8 27b running at ~75t/s decode on 16GB RTX 5080

Hi all,

I have recently been experimenting with different LLM set ups and after everyone was raving about how good Qwen 3.8 27b was, I was inspired to try and get it deploying.

After some battling with settings I've managed to get it running at an average of 75t/s, sometimes seeing 100+t/s with good MTP acceptance.

I've managed to do this using:
Qwen3.8-27B-i1-IQ4_XS-GGUF-Smaller - jrell/Qwen3.8-27B-i1-IQ4_XS-GGUF-Smaller · Hugging Face

"A custom hybrid quantization of the Qwen3.8-27B base model, specifically designed to fit Multi-Token Prediction (MTP) and long context into a strict 16GB VRAM hardware budget (like consumer RTX 4080 / RTX 5080 cards)."

These are my llama.cpp settings:

$llamaPath = "C:\Tools\llama-cuda2\llama-server.exe" $modelPath = "D:\models\Qwen3.8-27B-i1-IQ4_XS-GGUF-Smaller.gguf" $chatTemplatePath = Join-Path (Split-Path $modelPath -Parent) "chat_template.jinja" $llamaArgs = @( "-m", $modelPath, "-a", "qwen3.8-27b", "-ngl", "99", "-c", "85000", "-np", "1", "-b", "512", "-ub", "512", "-fa", "on", "-t", "8", "-tb", "8", "-ctk", "q4_0", "-ctv", "q4_0", "--reasoning-preserve", "--reasoning-effort", "medium", "--temp", "1", "--top-p", "0.95", "--top-k", "20", "--min-p", "0.0", "--spec-type", "draft-mtp", "--spec-draft-n-max", "3", "--repeat-penalty", "1.0", "--presence-penalty", "0.0", "--jinja", "--chat-template-file", $chatTemplatePath, "--host", "0.0.0.0", "--port", "8080" ) & $llamaPath 

Here's an example of a recent run:

10.17.039.668 I slot launch_slot_: id 0 | task 10227 | processing task, is_child = 0 10.20.936.381 I slot print_timing: id 0 | task 10227 | n_gen = 239, tg = 78.94 t/s, tg_3s = 79.26 t/s 10.23.953.739 I slot print_timing: id 0 | task 10227 | n_gen = 424, tg = 70.12 t/s, tg_3s = 61.31 t/s 10.26.978.406 I slot print_timing: id 0 | task 10227 | n_gen = 642, tg = 70.77 t/s, tg_3s = 72.07 t/s 10.30.010.411 I slot print_timing: id 0 | task 10227 | n_gen = 853, tg = 70.48 t/s, tg_3s = 69.59 t/s 10.33.032.673 I slot print_timing: id 0 | task 10227 | n_gen = 1082, tg = 71.54 t/s, tg_3s = 75.77 t/s 10.36.038.373 I slot print_timing: id 0 | task 10227 | n_gen = 1327, tg = 73.19 t/s, tg_3s = 81.51 t/s 10.39.043.415 I slot print_timing: id 0 | task 10227 | n_gen = 1532, tg = 72.48 t/s, tg_3s = 68.22 t/s 10.42.075.239 I slot print_timing: id 0 | task 10227 | n_gen = 1819, tg = 75.27 t/s, tg_3s = 94.66 t/s 10.45.082.345 I slot print_timing: id 0 | task 10227 | n_gen = 2005, tg = 73.78 t/s, tg_3s = 61.85 t/s 10.48.100.133 I slot print_timing: id 0 | task 10227 | n_gen = 2210, tg = 73.20 t/s, tg_3s = 67.93 t/s 10.51.129.760 I slot print_timing: id 0 | task 10227 | n_gen = 2436, tg = 73.32 t/s, tg_3s = 74.60 t/s 10.54.152.649 I slot print_timing: id 0 | task 10227 | n_gen = 2646, tg = 73.00 t/s, tg_3s = 69.47 t/s 10.57.172.073 I slot print_timing: id 0 | task 10227 | n_gen = 2847, tg = 72.51 t/s, tg_3s = 66.57 t/s 11.00.182.456 I slot print_timing: id 0 | task 10227 | n_gen = 3117, tg = 73.73 t/s, tg_3s = 89.69 t/s 11.03.195.203 I slot print_timing: id 0 | task 10227 | n_gen = 3338, tg = 73.71 t/s, tg_3s = 73.36 t/s 11.06.202.960 I slot print_timing: id 0 | task 10227 | n_gen = 3545, tg = 73.40 t/s, tg_3s = 68.82 t/s 11.09.219.971 I slot print_timing: id 0 | task 10227 | n_gen = 3764, tg = 73.35 t/s, tg_3s = 72.59 t/s 11.12.237.385 I slot print_timing: id 0 | task 10227 | n_gen = 3979, tg = 73.24 t/s, tg_3s = 71.25 t/s 11.15.246.902 I slot print_timing: id 0 | task 10227 | n_gen = 4170, tg = 72.72 t/s, tg_3s = 63.47 t/s 11.18.268.286 I slot print_timing: id 0 | task 10227 | n_gen = 4370, tg = 72.40 t/s, tg_3s = 66.19 t/s 11.21.299.694 I slot print_timing: id 0 | task 10227 | n_gen = 4587, tg = 72.36 t/s, tg_3s = 71.58 t/s 11.24.331.662 I slot print_timing: id 0 | task 10227 | n_gen = 4815, tg = 72.49 t/s, tg_3s = 75.20 t/s 11.27.359.294 I slot print_timing: id 0 | task 10227 | n_gen = 5116, tg = 73.66 t/s, tg_3s = 99.42 t/s 11.30.365.338 I slot print_timing: id 0 | task 10227 | n_gen = 5309, tg = 73.27 t/s, tg_3s = 64.20 t/s 11.33.366.835 I slot print_timing: id 0 | task 10227 | n_gen = 5573, tg = 73.85 t/s, tg_3s = 87.96 t/s 11.36.390.528 I slot print_timing: id 0 | task 10227 | n_gen = 5835, tg = 74.35 t/s, tg_3s = 86.65 t/s 11.39.398.044 I slot print_timing: id 0 | task 10227 | n_gen = 6058, tg = 74.34 t/s, tg_3s = 74.15 t/s 11.42.424.685 I slot print_timing: id 0 | task 10227 | n_gen = 6355, tg = 75.19 t/s, tg_3s = 98.13 t/s 11.45.441.398 I slot print_timing: id 0 | task 10227 | n_gen = 6571, tg = 75.07 t/s, tg_3s = 71.60 t/s 11.48.465.392 I slot print_timing: id 0 | task 10227 | n_gen = 6789, tg = 74.97 t/s, tg_3s = 72.09 t/s 11.51.466.866 I slot print_timing: id 0 | task 10227 | n_gen = 6973, tg = 74.53 t/s, tg_3s = 61.30 t/s 11.54.494.631 I slot print_timing: id 0 | task 10227 | n_gen = 7163, tg = 74.16 t/s, tg_3s = 62.75 t/s 11.57.525.053 I slot print_timing: id 0 | task 10227 | n_gen = 7371, tg = 73.99 t/s, tg_3s = 68.64 t/s 12.00.558.694 I slot print_timing: id 0 | task 10227 | n_gen = 7606, tg = 74.10 t/s, tg_3s = 77.46 t/s 12.03.580.920 I slot print_timing: id 0 | task 10227 | n_gen = 7917, tg = 74.92 t/s, tg_3s = 102.90 t/s 12.06.603.979 I slot print_timing: id 0 | task 10227 | n_gen = 8189, tg = 75.34 t/s, tg_3s = 89.98 t/s 12.09.618.569 I slot print_timing: id 0 | task 10227 | n_gen = 8384, tg = 75.05 t/s, tg_3s = 64.69 t/s 12.12.636.851 I slot print_timing: id 0 | task 10227 | n_gen = 8593, tg = 74.90 t/s, tg_3s = 69.24 t/s 12.15.640.585 I slot print_timing: id 0 | task 10227 | n_gen = 8777, tg = 74.55 t/s, tg_3s = 61.26 t/s 12.18.673.135 I slot print_timing: id 0 | task 10227 | n_gen = 8998, tg = 74.51 t/s, tg_3s = 72.88 t/s 12.21.679.116 I slot print_timing: id 0 | task 10227 | n_gen = 9340, tg = 75.46 t/s, tg_3s = 113.77 t/s 12.24.704.678 I slot print_timing: id 0 | task 10227 | n_gen = 9647, tg = 76.08 t/s, tg_3s = 101.47 t/s 12.27.738.435 I slot print_timing: id 0 | task 10227 | n_gen = 9996, tg = 76.99 t/s, tg_3s = 115.04 t/s 12.30.743.188 I slot print_timing: id 0 | task 10227 | n_gen = 10291, tg = 77.47 t/s, tg_3s = 98.18 t/s