Is it possible to run it with a combined memory setup: 16 GB VRAM + 64 GB RAM + SSD for offloading n-grams?
Mirrored from r/LocalLLaMA for archival readability. Support the source by reading on the original site.
Hardware: rtx 5080 16 gb vram; 64 gb ram ddr5 6000hz; ssd with unlimited memory; ryzen 7 9800 x3d.
OS: Windows 11
Software: I’d prefer llama.cpp, but it’s not a strict requirement; I’ll use whatever you suggest, as long as it works on Windows.
My attempts to run it with llama.cpp:
llama-server ^ -m "F:.lmstudio\models\unsloth\Qwen-Next\Qwen3.8-Flash-Next-UD-IQ3_XXS-00001-of-00003.gguf" ^ -c 10000 ^ --n-gpu-layers 999 ^ -b 512 ^ -ub 512 ^ --fit off ^ --parallel 1 ^ --jinja ^ --flash-attn auto ^ --load-mode mmap ^ --no-host ^ --override-tensor "per_layer_token_embd.weight=CPU" and 2nd attemtp:
llama-server ^ -m "F:.lmstudio\models\AtomicChat\Qwen3.8-Flash-Next-GGUF\Qwen3.8-Flash-Next-AD-4.27bpw-Q4_K_M-M64-00001-of-00033.gguf" ^ -c 10000 ^ -ngl 99 ^ -b 512 ^ -ub 512 ^ --fit off ^ --parallel 1 ^ --jinja ^ -fa on and i got 6 t/sec, its just unusable
[link] [comments]
More from r/LocalLLaMA
-
an unscientific qwen 3.8 flash next and glm 5.3 flash comparison
Aug 30
-
Ran Qwen3.8-Flash-Next (79 GB, 2-bit) at 350K ctx for 3.5 hours on a 128 GB M5 Max — speed vs context depth, 100 turns, one graph
Aug 30
-
Nemotron-3.5-Lightning at 11.77 GiB, a 16 GB option for a model that didn't have one
Aug 29
-
Humaneval benchmark for Deepseek V4 Flash 0731 vs GLM5.3 Flash on 2x DGX Spark setup
Aug 29
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.