Prismix
r/LocalLLaMA · · 3 min read

llama.cpp Open PRs list - CPU/RAM/Disk/Hybrid Related - Better for CPU-only & Hybrid inference

Mirrored from r/LocalLLaMA for archival readability. Support the source by reading on the original site.

Like Read original ↗
llama.cpp Open PRs list - CPU/RAM/Disk/Hybrid Related - Better for CPU-only & Hybrid inference

Folks! We're just 50 PRs away from more faster inference. Hopefully by end of year.

Experts!, please chip in there.

List of Open/Ongoing PRs(and also Discussions) related to CPU/RAM/Disk/Hybrid:

  1. [Discussion] RFC: MoE expert cache, VRAM caching of hot CPU-resident experts with hybrid hit/miss execution #24528
  2. AVX2: Speed up large batch size prompt processing of IQ models #27402
  3. llama: add Maple 20B-A1B ternary MoE architecture (CPU)- #27000
  4. ggml-cpu: tiled mul_mat for k-quants- #27851
  5. ggml-cpu: add AVX-512 and VNNI paths for Q5_K/Q6_K dot products- #27590
  6. ggml-cpu: add x86 VNNI Q2_0 dot product -- 3x speed improvement for VNNI-compatible CPUs- #26348
  7. llama: add pshard runtime for plan switching and streamed weights- #22692
  8. CPU Optimizations - Prefill, Tokenization, and Token Generation- #27032
  9. llama : stream MoE routed experts from disk - #25294
  10. ggml : speed up batch-1 CPU decode, align large allocations- #27478
  11. misc : prevent RAM peaking at model loading stage- #27483
  12. recurrent : support equal splits for recurrent-state rollback- #25004
  13. ggml-cpu : add AVX2 vec_dot kernel for STQ1_0- #27377
  14. --numa mirror: mirror model weights to every Numa node in the system- #16000
  15. CPU flash-attn: support quantized K/V in the tiled prefill kernel- #26948
  16. ggml-cpu/amx: fix block_q8_K VNNI quantization and enable VNNI path- #27024
  17. server : add /slots endpoint action=clone_to (KV clone between slots)- #26204
  18. ggml : fuse soft_max sweeps into fewer passes- #26468
  19. ggml-cpu : add STQ1_0 ternary quantization with ARM NEON vec_dot kernel- #22836
  20. llama-hot-experts: pin hottest MoE experts in RAM via --pin-hot-experts- #26414
  21. llama : add --lazy-experts for MoE models larger than RAM- #26003
  22. ggml : vectorize rms_norm reduce and fuse the scale write- #26486
  23. MoE disk offloading for Metal- #23440
  24. ggml-cpu: Added RVV VLEN=1024 vector dot product (vec_dot) kernels for quantized types.- #25397
  25. ggml-cpu: detect AVX-VNNI in MSVC native builds- #25346
  26. ggml-cpu: replace cyclic chunk distribution with atomic work-stealing- #25048
  27. Improve performance of ggml_gemv_q4_K_8x8_q8_K for +12-23% tok/s on AVX-VNNI systems- #23309
  28. ggml-cpu: Optimized Arm NEON cpu q1_0 dot (with plain/DP/I8MM)- #23358
  29. ggml-cpu: ARM Repack kernels for Q1_0- #23492
  30. ggml-cpu: add wasm simd path for iq4_nl_q8_0- #24058
  31. ggml-cpu: optimize ggml_gemm_q4_K_8x8_q8_K interleaving/staging for AVX-512 (and AVX2)- #22525
  32. ggml/cpu: skip zero-scale blocks in TQ1_0 and TQ2_0 vec_dot kernels- #23439
  33. ggml-cpu:Optimized risc-v cpu nvfp4- #23402
  34. ggml-cpu : fix riscv xtheadvector builds and add a q1_0 vec dot kernel- #23009
  35. Q5_0 - Block Interleaving Implementation for x86 SIMD (AVX512/AVX2)- #22250
  36. ggml-cpu: optimize q8 quantization on x86 SIMD- #22331
  37. Optimize reduction stage of dot product of q4_L/q5_K to q8_K on AVX2- #22181
  38. ggml: introduce GGML_NUMA_MIGRATE to optimize cross NUMA op computation - #14232
  39. ggml-cpu: improve --n-cpu-moe TG performance- #20596
  40. ggml : add CPU backend reference implementation (wip)- #16004
  41. ggml: optimize ggml_vec_dot_mxfp4_q8_0 dot product on ARM SVE- #19171
  42. Q6_K - Block Interleaving Implementation for x86 SIMD (AVX512/AVX2)- #19706
  43. ggml-cpu: optimize q4_0_q8_0 scales using Zvfhmin- #19196
  44. ggml-cpu: add q4_0 repack support for wasm- #18858
  45. Improving inference speed for the repack buffer type on NUMA architectures- #18698
  46. ggml: optimized runtime for x86 cpu backend and Q4_K quantized weights paired with Q8_K activations - #18495
  47. CPU SIMD and pipeline optimizations across vec/mmq/ops/kv-cache/repack - #17113
  48. ggml-cpu: optimise rms_norm op- #16650

PRs related to New Quant types:

  1. Add ROCmFP4 CPU quantization support- #24185
  2. ggml: add support for MXFP8 CPU- #26157
  3. ggml: Add initial MXFP6 CPU implementation- #22671
  4. ggml : add E4M3 (fp8) CPU quantization type- #25336

(Just had some extra time, so went through almost entire Open PRs of llama.cpp. For Poor GPU Club mainly)

submitted by /u/pmttyji
[link] [comments]

Discussion (0)

Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.

Sign in →

No comments yet. Sign in and be the first to say something.

More from r/LocalLLaMA