Deepseek v4 flash Q2 on a single 4090 😅

It freaking worked lol🔥

Deepseek-v4-flash-0731 @UnslothAI 's IXQ2/Q3 checkpoint on one single RTX4090 with just 64 GB of RAM at usable token rate without dspark. All kernels running on Blaze (my custom developed ML compiler + inference engine) - no llama.cpp or vllm in the picture.

The setup keeps heavily utilized experts in RAM with CPU (a trick from this guy )

Current tps is around 8 tps with slight expert miss causing a disk read which lowers it to 5 tps momentarily. With dspark and perhaps more RAM, it can probably hit double digits. Prefill is also WIP.

I'll publish something on this stack soon on my substack : 😊