Getting Qwen3.8-27B with decent speed on my 4080 with 16Gb card

I saw that Q2 is actually very good and produce real good results: https://youtu.be/WNMnbba35VI?is=UNokHqdY4bA5kDgw

and I also saw how dflash2 make its running at generating >60 t/s with a 120k context lenght.

https://youtu.be/RBlRTUwJMI4?is=LCtTHgkaiWnGfLv9

And I like what its doing!!

Heres how to set it up (ai wrote this)

DFlash2 speculative decoding on 16GB VRAM — 1.72x faster (setup guide)

DFlash2 pairs a small drafter model with your big model. The drafter guesses a block of tokens in parallel, the target verifies them in one pass. Lossless — identical output quality, just faster.

Result on RTX 4080 16GB with Qwen3.8-27B: 50.6 → 86.7 tok/s (1.72x), costs 561MB VRAM.

---

1. Build llama.cpp from the PR branch

DFlash2 is not in master (only DFlash 1 is). It lives in unmerged PR #27342. If you use a release build you'll get wrong number of tensors; expected 81, got 58 on every drafter quant.

git clone --depth 1 https://github.com/ggml-org/llama.cpp.git llama.cpp_new

cd llama.cpp_new

git remote add zlab https://github.com/z-lab/llama.cpp-fork.git

git fetch --depth 1 zlab dflash2

git checkout FETCH_HEAD

Verify before building:

grep -c "LLM_KV_DFLASH_SELECTOR_RANK" src/llama-arch.cpp # must print 1

Build into a separate directory from any existing llama.cpp.

1. Get the models

- Target: your Qwen3.8-27B GGUF (mine: Q2_K_P, 9.94GB)

- Drafter: HermiHg/Qwen3.8-27B-DFlash2-Q2_K_S-MIX-GGUF (561MB)

Drafters are model-pair-specific — trained for one exact target.

1. Run it

llama-server.exe \

-m Qwen3.8-27B-Q2_K_P.gguf \

--model-draft Qwen3.8-27B-DFlash2-Q2_K_S-MIX.gguf \

--spec-type draft-dflash \

--spec-draft-n-max 3 \

-ngl 99 -c 122880 -fa on \

--cache-type-k q4_0 --cache-type-v q4_0 \

--host 0.0.0.0 --port 8080 --jinja

--spec-type draft-dflash is required — without it the drafter loads but never engages. Note it's --model-draft (not --draft-model) and -ngl (not --ngl).

---

Throughput

Use Q2_K_S-MIX at n_max=3. It's faster than the official Q4_K_M at half the size. n_max=5 is worse across the board — acceptance drops faster than parallelism gains.

Context vs speed

4k

• KV: q4_0

• tok/s: 86.7

• Speedup: 1.72x

120k

• KV: q4_0

• tok/s: 66.1

• Speedup: 1.31x

Max stable context with the drafter is 122880. Above that you hit a PR bug (not OOM):

- 131072 without drafter → works

- 131072 with drafter → invalid vector subscript

- 126976 → flaky (loads with --no-warmup, crashes with warmup)