Ling-3.0 (BailingMoE3) lands in llama.cpp mainline - Quick benchmarks on Intel Arc B580

Finally llama.cpp now officially supports Ling-3.0! (Starting from build b10472+)

If you want to run them locally, bartowski has already released the GGUF imatrix quantizations for both models:
- Ling-3.0-tiny (8B)
- Ling-3.0-flash (127B)

After quite a while, PR #26608 has officially been merged into master! There are still a few minor details left to iron out, but BailingMoE3 support is now fully official in llama.cpp.

For my B580 on Linux I used:

./llama-bench -m Ling-3.0-tiny-Q8_0.gguf -ngl 99 -fa on --cache-type-k q8_0 --cache-type-v q8_0 -p 16384 -n 128 -n 32 -r 1 

And these were the results:

| model | size | params | backend | ngl | type_k | type_v | fa | test | t/s | | ------------------------------ | ---------: | ---------: | ---------- | --: | -----: | -----: | --: | --------------: | -------------------: | | bailingmoe3 7.9B.A1.3B Q8_0 | 7.83 GiB | 7.89 B | Vulkan | 99 | q8_0 | q8_0 | 1 | pp16384 | 120.76 ± 0.00 | | bailingmoe3 7.9B.A1.3B Q8_0 | 7.83 GiB | 7.89 B | Vulkan | 99 | q8_0 | q8_0 | 1 | tg128 | 114.24 ± 0.00 | | bailingmoe3 7.9B.A1.3B Q8_0 | 7.83 GiB | 7.89 B | Vulkan | 99 | q8_0 | q8_0 | 1 | tg32 | 114.78 ± 0.00 | build: 9d77fa172 (10488) 

I also ran a test with a larger context:

./llama-bench -m Ling-3.0-tiny-Q8_0.gguf -ngl 99 -fa on --cache-type-k q8_0 --cache-type-v q8_0 -p 32768 -n 32 -r 1 

Plaintext

| model | size | params | backend | ngl | type_k | type_v | fa | test | t/s | | ------------------------------ | ---------: | ---------: | ---------- | --: | -----: | -----: | --: | --------------: | -------------------: | | bailingmoe3 7.9B.A1.3B Q8_0 | 7.83 GiB | 7.89 B | Vulkan | 99 | q8_0 | q8_0 | 1 | pp32768 | 62.53 ± 0.00 | | bailingmoe3 7.9B.A1.3B Q8_0 | 7.83 GiB | 7.89 B | Vulkan | 99 | q8_0 | q8_0 | 1 | tg32 | 110.49 ± 0.00 | build: 9d77fa172 (10488) 

Overall, the speed is quite good. I can actually run the full 128K context within my 12GB of VRAM using -c 131072 -ngl 99 -fa 1 --cache-type-k q8_0 --cache-type-v q8_0. While I still need to test if it can be pushed even further since a few warnings pop up, it is entirely feasible and remarkably fast for a local setup.