I managed to get Qwen3.8-2.4T-A95B running locally with llama.cpp on mu PC just for fun, cause why not.

I was using the Unsloth Qwen3.8-2.4T-A95B-UD-Q1_0 GGUF quantization. The full GGUF is about 397 GiB. The model uses 512 routed experts, with 10 active per token.

My hardware:

• CPU: AMD Ryzen 9 9950X3D 16 cores / 32 threads
• GPU 0: NVIDIA RTX 5090 32 GB VRAM
• GPU 1: NVIDIA RTX 5060Ti 16 GB VRAM
• RAM: 128 GB DDR5 5600 CL36
• Swap: 350 GB
• OS: Arch Linux
• llama.cpp: Unsloth build 10360

The interesting part is that I also managed to get the model’s native MTP speculative decoding working.

After testing several configurations, the best settings I found were:

• --tensor-split 4,1
• --n-cpu-moe 91
• MTP n_max = 3
• MTP p_min = 0.5
• MTP routed experts from block 92 forced to CPU

The block 92 expert tensors had to stay on CPU because otherwise MTP would run out of VRAM.

Performance

For a controlled 32-token generation test:

Prompt tokens: 16 Generated tokens: 32 Prompt speed: 0.8299 tok/s Generation speed: 0.8033 tok/s Decode time: 39.84 s Total wall time: 59.14 s MTP draft tokens: 21 MTP accepted: 19 MTP acceptance: 90.48% 

Without MTP, the same test gave:

Generation speed: 0.7750 tok/s Decode time: 41.29 s Total wall time: 60.81 s 

So my final MTP configuration gives roughly:

+3.64% generation throughput -2.74% total wall time 

Not a huge speedup, but considering this is a 2.45T parameter model running locally on consumer hardware, I’ll take it.

I also tested different MTP settings:

n_max=2 -> too conservative n_max=3 -> best n_max=4 -> much slower p_min=0.0 -> too many bad speculative tokens p_min=0.5 -> best result p_min=0.7 -> too aggressive 

Interestingly, p_min=0.7 reached 100% draft acceptance, but it generated too few useful speculative tokens and ended up slower than p_min=0.5.

Final llama-server command

MODEL="/home/marcin/Pobrane/Qwen3.8-2.4T-A95B-UD-Q1_0-00001-of-00010.gguf" CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=0,1 LLAMA_ATTN_ROT_DISABLE=1 \ /home/marcin/Pobrane/unsloth-qwen38-runtime/llama.cpp/build/bin/llama-server \ --model "$MODEL" \ --alias Qwen3.8-2.4T-A95B \ --device CUDA0,CUDA1 \ --main-gpu 0 \ --split-mode layer \ --tensor-split 4,1 \ --n-gpu-layers all \ --n-cpu-moe 91 \ -ot 'blk\.92\.ffn_(down|gate|up)_exps\.weight=CPU' \ --fit on \ --fit-target 2048,1536 \ --no-op-offload \ --ctx-size 512 \ --flash-attn on \ --cache-type-k q8_0 \ --cache-type-v q8_0 \ --kv-unified \ --batch-size 128 \ --ubatch-size 32 \ --threads 16 \ --threads-batch 16 \ --parallel 1 \ --spec-type draft-mtp \ --spec-draft-device CUDA0 \ --spec-draft-ngl all \ --spec-draft-cpu-moe \ --spec-draft-n-max 3 \ --spec-draft-p-min 0.5 \ --temp 0 \ --top-k 0 \ --top-p 0.95 \ --min-p 0.0 \ --cache-ram 0 \ --ctx-checkpoints 0 \ --load-mode mmap \ --no-warmup \ --no-ui \ --host 127.0.0.1 \ --port 8200 \ --metrics 

With MTP enabled, VRAM usage after generation is roughly:

RTX 5090: 29.6 GB used / ~2.6 GB free RTX 5060 Ti: 12.3 GB used / ~3.6 GB free 

One important info: 0.80 tok/s is from a controlled 32-token test, not a large multi-prompt benchmark, so I wouldn’t present it as a universal average for every workload. Longer prompts, context size, output patterns and expert routing can change the result.

Still, seeing a 2.4T MoE model actually generating locally on a 5090 + 5060 Ti at around 0.8 tok/s is pretty wild.