I might have found the perfect config parameters for qwen 3.8 27b
Mirrored from r/LocalLLaMA for archival readability. Support the source by reading on the original site.
Hello everyone, tried so hard to optimize my config and finally I simply get up to 70 t/s with q6 variant. And wanted to share with you guys so that other people with the same setup can enjoy. Please check out and see if that improves your performance in any kind of way.
Also huge thanks to qwen and unsloth teams.
"C:\Users\dsdt\llama\llama-server.exe" ^ -m "C:\Users\dsdt\models\Qwen3.8-27B-UD-Q6_K.gguf" ^ --mmproj "C:\Users\dsdt\models\mmproj-BF16.gguf" ^ --jinja ^ --chat-template-kwargs "{\"reasoning_effort\":\"medium\"}" ^ --reasoning on ^ --reasoning-preserve ^ -c 100000 ^ --split-mode tensor ^ --flash-attn on ^ --cache-type-k q8_0 ^ --cache-type-v q8_0 ^ --spec-type draft-mtp,ngram-mod ^ --spec-draft-n-max 2 ^ --spec-ngram-mod-n-match 24 ^ --spec-ngram-mod-n-min 24 ^ --spec-ngram-mod-n-max 86 ^ -t 8 ^ --batch-size 8869 ^ --ubatch-size 531 ^ -ngl 105 ^ -np 1 ^ --fit off ^ --temp 1.0 ^ --top-p 0.95 ^ --top-k 20 ^ --min-p 0.00 ^ --presence-penalty 0.0 ^ --host 0.0.0.0 ^ --port 8080 - System & Model Configuration
| Parameter | Value |
|---|---|
| Model | Qwen3.8-27B-UD-Q6_K.gguf |
| Context Length | 100,000 tokens |
| GPUs | 2 × RTX 5060 Ti (Total VRAM: 32 GB) |
| Vision | Enabled (VRAM allocated, not used during generation) |
| Task ID / Slot | Task 2536 / Slot 0 |
- Core Performance Metrics
| Metric | Value |
|---|---|
| Prompt Processing | 646.62 ms / 27 tokens → 41.76 t/s (23.95 ms/token) |
| Generation (Eval) | 126,199.16 ms / 8,624 tokens → 68.33 t/s (14.64 ms/token) |
| Overall Throughput | 68.33 t/s (prompt overhead negligible) |
| Total Time | 126,845.77 ms (~2 min 6.8 s) |
| Total Tokens Processed | 8,651 (27 prompt + 8,624 generated) |
| Final Tokens Count (with cache) | 8,973 (clean stop, no truncation) |
| Graphs Reused | 5,569 (excellent computational graph reuse) |
- Generation Speed Stability (Short-term)
| Window | Observed Speed Range |
|---|---|
| 3-second rolling window (tg_3s) | ~50 – 79 t/s (early fluctuations) |
| Steady-state (later stages) | ~62 – 72 t/s |
| Overall average (tg) | Converged to stable 68.5 – 69.5 t/s |
- Speculative Decoding Efficiency
| Metric | Value |
|---|---|
| Draft Acceptance Rate | 80.04% (5,510 accepted / 6,884 drafted) |
| Average Draft Length | 2.77 tokens per forward pass of the target model |
[link] [comments]
More from r/LocalLLaMA
-
Unpopular opinion Qwen 3.8 is hard to understand
Aug 30
-
Qwen3.8-Flash-Next turns 4xR9700 into a local AI powerhouse! 120 t/s TG and 12k t/s PP single request with optimized vLLM
Aug 30
-
Oh so that's where my PCIe lanes went...
Aug 30
-
Uncensored Multi-Model Releases, LongCat-Flash-Lite-Sparse with MTPs and LSAs, Qwen3.8-27B with MTPs, Qwen3.5-122B-A10B with MTPs, Qwen3-Coder-Next and Laguna-S2.1 with Vision, All Available in GGUF Format! Bonus: Links to my llama.cpp Fork for LongCat-Flash-Lite Support and…
Aug 30
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.