r/LocalLLaMA · · 1 min read

Qwen Flash next on 64GB RAM unified iGPU anyone ? (non-mac)

Mirrored from r/LocalLLaMA for archival readability. Support the source by reading on the original site.

I've seen people reporting running it with 12GB VRAM + 64 GB RAM. Also with 64 GB RAM unified in Macs, but I was wondering if it's possible with any inference backend to run it for example on a 64 GB RAM minipc+ iGPU (780m in my case).
I'm currently running 125B Ling 3.0 flash at Q2 quants with llama.cpp (vulkan), its relatively usable, so I was wondering if a similar quant of Qwen Flash Next with the ngrams offloaded to SSD could work (even at low token/s). As far as I know this can't be done with llama.cpp now. Other inference engines does not seem to work with vulkan.

EDIT: Thanks everyone! It's working with the Q2 quant Qwen3.8-Flash-Next-GSQ-RCO-GGUF using llama.cpp with -lm mmap --lazy-mode on

submitted by /u/SomeITGuyLA
[link] [comments]

Discussion (0)

Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.

Sign in →

No comments yet. Sign in and be the first to say something.

More from r/LocalLLaMA