What's the best setup for Qwen3.8 27b for a 16 gig VRAM?
Mirrored from r/LocalLLaMA for archival readability. Support the source by reading on the original site.
Hello guys!
I have been experimenting with qwen 3.8 for a long time and I hadn't been able to get reasonable speed. I am on a 5060Ti 16 GB, and although this gpu can game, I am aware that AI demands more than 16 GB.
I am on a Fedora 44, AMD Ryzen 9600x and 16 GB system ram (16 GB system ram and 16 GB vram, totaling to 32 GB) and I would like to use llamacpp, although I would use any other tool if I could if it meant faster speed.
I am looking for a large context. Atleast 128k context. The first question is, what quantization to pick? In my experience Q3 UD was satisfying, but I am looking for uncensored model. In my experience, MTP has never lived up to its hype for me (and I don't know why!?), which is why I am thoroughly lost on making a good setup after an honest week of experimentation, which is why I have resorted to ask here as a last resort.
Update : Found a model, thanks to u/_wortkarg_
link : https://huggingface.co/RentedNoodle/Qwen3.8-27B-GSQ-RCO-IQ3_XXS-Uncensored
command (A better command would be appreciated and updated accordingly):
~/llama.cpp/build/bin/llama-server \ --model ~/Documents/Models/Qwen3.8-27B-GSQ-RCO-IQ3_XXS-mtp.gguf \ --alias "llamacpp" --host 0.0.0.0 --port 8001 \ -ngl 99 --flash-attn on --ctx-size 131072 \ --cache-type-k q4_0 --cache-type-v q4_0 \ --parallel 1 --batch-size 512 --ubatch-size 256 \ --no-warmup --jinja \ --spec-type draft-mtp,ngram-mod --spec-draft-n-max 2 \ --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 I am hitting at about 35 t/s+ speed with this one.
[link] [comments]
More from r/LocalLLaMA
-
The AI race: China's open-weight models trail US closed models by just 7 months
Oct 14
-
Anthropic Executive Says Open-Source AI Models Are Higher Risk
Oct 14
-
ComfyUI v0.40.0 released
Oct 14
-
So this is how they do it --> RTX 5090, once a consumer gaming card, is now on smugglers’ menu alongside cigarettes
Oct 14
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.