OMG! If you have a Mac with 64GB, try Qwen3.8-Flash-Next-oQ4e-mtp with oMLX!
Mirrored from r/LocalLLaMA for archival readability. Support the source by reading on the original site.
I'm genuinely shocked! I was able to run Qwen3.8-Flash-Next-oQ4e-mtp on M3Max 64GB with oMLX!
Even a couple of weeks ago, I wasn't able to get it to run. I just tried the latest commit for fun, and it worked! It feels like some kind of sorcery to be able to run a 100GB model with 58GB allocated to GPU!
I can even run up to 130k context window!
Here are stats after running a short session with PI.
- Requests: 13
- Total Prefill Tokens: 321,309
- Cached Tokens: 292,209
- Cache Efficiency: 90.9%
- Prompt Processing (excl. cached): 140.0 tok/s
- Token Generation: 15.8 tok/s
Here are the settings I used on oMLX:
- Memory guard: Aggressive
- Hot Cache Limit (In-Memory Cache): 2GB
- Lightning MTP: on
- MoE Expert Offload: on
- RESIDENT EXPERTS: 50%
[link] [comments]
More from r/LocalLLaMA
-
The AI race: China's open-weight models trail US closed models by just 7 months
Oct 14
-
Anthropic Executive Says Open-Source AI Models Are Higher Risk
Oct 14
-
ComfyUI v0.40.0 released
Oct 14
-
So this is how they do it --> RTX 5090, once a consumer gaming card, is now on smugglers’ menu alongside cigarettes
Oct 14
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.