llama.cpp releases
500 articles archived · Visit source ↗ · RSS
-
llama.cpp releases dev-tools 1mo ago
b10037
quant : allow using manual tensor types with --pure ( #25716 ) macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu s390x (CPU) Ubuntu x64 (Vulkan) Ubuntu…
32 -
b10036
opencl: disable FA and MoE weights repack to work around compiler issues for Adreno 850 GPU ( #25745 ) opencl: workaround for A850 compiler compat opencl: fix DX compiler version parsing and cleanup Co-authored-by: Li He lih@qti.qualcomm.com macOS/iOS: macOS Apple Silicon…
7 -
b10035
cuda: extract Q1_0 elements via __byte_perm ( #25628 ) macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu s390x (CPU) Ubuntu x64 (Vulkan) Ubuntu arm64…
18 -
b10034
opencl: exclude some moe kernels on Adreno a7x ( #25698 ) opencl: exclude Adreno A7x from using Adreno MoE kernels Some compilers for A7x devices miscompile the repack kernels, corrupting the weights and causing MoE models to generate garbage output opencl: exclude A6x and…
22 -
b10032
cuda : CUDA GGML_OP_LIGHTNING_INDEXER implementation (generic vector kernel + wmma kernel) ( #25545 ) cuda : CUDA GGML_OP_LIGHTNING_INDEXER implementation (generic vector kernel + wmma kernel) chore : remove indentation of #pragma unroll cuda : remove unnecessary kernel template…
37 -
b10031
tokenize : drop --stdin mutual-exclusion check ( #25672 ) match cli and completion, which don't enforce it macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU)…
13 -
b10025
cuda : relax tensor contiguity requirements for quantized concat ( #25678 ) cuda : relax tensor contiguity requirements for quantized concat tests : add test cases for non-contiguous quantized concat ggml : relax contiguity requirements for quantized concat Co-authored-by:…
22 -
b10021
DeepseekV4: reduce graph splits ( #25702 ) macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu s390x (CPU) Ubuntu x64 (Vulkan) Ubuntu arm64 (Vulkan)…
9 -
b10020
sycl : fix get_rows Q2_K, Q4_K, Q5_K ( #25656 ) macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu s390x (CPU) Ubuntu x64 (Vulkan) Ubuntu arm64 (Vulkan)…
35 -
b10019
sycl : support kernel type fp16 for conv2d_dw ( #25653 ) macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu s390x (CPU) Ubuntu x64 (Vulkan) Ubuntu arm64…
21 -
b10018
sycl : implement xielu op ( #25550 ) macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu s390x (CPU) Ubuntu x64 (Vulkan) Ubuntu arm64 (Vulkan) Ubuntu x64…
34 -
b10017
sycl: Increase minimum buffer size for USM system allocations ( #25525 ) Raise the threshold for minimum buffer size from 1 GiB to 4 GiB, based on real-world experiments of overcommitting device memory with model weights larger than available VRAM, for example Qwen3.5-35B-A3B-Q8…
28 -
b10016
[SYCL] Flash Attention with XMX engine via oneDNN ( #25222 ) [SYCL] F16 (default) Flash Attention with XMX engine via oneDNN graph API; Qwen3.6-27b-Q8_0 prefill speed up x1.21 at p=512 and x4.26 at p=80k [SYCL] Address review on FA oneDNN path. Result: llama-bench---pp512; 32%…
8 -
b10015
opencl: do not use clCreateBufferWithProperties when targeting CL 2.x ( #25673 ) macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu s390x (CPU) Ubuntu…
6 -
b10012
hexagon: fix hmx-queue signal enum-narrowing problem ( #25677 ) macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu s390x (CPU) Ubuntu x64 (Vulkan)…
15 -
b10011
server : refactor prompt cache state ownership ( #25649 ) server : clear checkpoints upon prompt clear server : move the prompt state data to the server_prompt_cache Assisted-by: pi:llama.cpp/Qwen3.6-27B server : handle batched slot being cleared macOS/iOS: macOS Apple Silicon…
38 -
b10010
server: add --cors-* options ( #25655 ) server: add --cors-* options add special "localhost" value add tests fix test add link to PR macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64…
33 -
b10007
opencl: fix a dp4a bug for devices where cl_khr_integer_dot_product is unavailable ( #25639 ) opencl: do not fail backend init on devices without cl_khr_integer_dot_product opencl: do not call dp4 kernels when dp is unavailable Co-authored-by: Li He lih@qti.qualcomm.com…
10 -
b10005
DeepseekV4: fix seq_rm ( #25588 ) DeepseekV4: fix seq_rm implement proper seq_cp create actual update context macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64…
25 -
b10004
vulkan/cpu: Support f16 as SET_ROWS src. ( #25432 ) vulkan/cpu: Support f16 as SET_ROWS src. This adds full support for f16 SET_ROWS (equivalent to f32) to vulkan and CPU backends, and adds more backend tests. Set DenormPreserve 16 when supported, to try to fix failures on Intel…
6 -
b10003
tokenize : align usage by using common args ( #25516 ) Migrate the tokenize tool to common_params_parse, replacing its hand-rolled argv parsing, Windows UTF-8 handling and file reading with the shared common helpers. Expose the model-sourcing flags (-m, -mu, -dr, -hf, -hff,…
16 -
b10002
ggml : add a set of functions for checking contiguity of inner tensor dimensions ( #25650 ) Co-authored-by: Stanisław Szymczyk sszymczy@gmail.com macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework…
27 -
b10001
tests: export-graph-ops: exit gracefully when called w/o arguments ( #25619 ) Fixes a segfault when test-export-graph-ops is called without any arguments. macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS…
10 -
b10000
ggml: uniformize im2col dst_type for all conv ops ( #23660 ) ggml: uniformize im2col dst_type for all conv ops Update ggml/src/ggml.c Co-authored-by: Georgi Gerganov ggerganov@gmail.com ggml : uniformize im2col casting logic across all conv ops fix : allow im2col_f16 to accept…
4 -
b9999
kleidiai : add SME2 f32 kernel ( #24414 ) kleidiai : add SME2 f32 kernel enable dynamic scheduling for SME2 f32 kernel macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu…
30 -
b9996
arg: Flush log before exiting after usage() ( #25504 ) Under certain conditions, it's possible for messages emitted via LOG() to get lost before exit, apparently because they are emitted by another thread. common_params_print_usage() uses printf directly, and is not affected.…
36 -
b9995
sycl: set fattn_vec_nthreads to 256 for Battlemage ( #25205 ) Currently detects lunarlake + battlemage / xe2 and sets the value to 256. Keeps default at 128, Intel's ARC Alchemist's prefered value. macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI…
15 -
b9994
metal : add Q2_0 support ( #25419 ) macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu s390x (CPU) Ubuntu x64 (Vulkan) Ubuntu arm64 (Vulkan) Ubuntu x64…
29 -
b9993
model: add Hy3 (hy_v3) support with MTP speculative decoding ( #25395 ) model: add Hy3 (hy_v3) architecture support Adds Tencent Hunyuan 3 (HF architecture HYV3ForCausalLM, GGUF arch hy_v3): a MoE decoder stack with per-head Q/K RMSNorm, a sigmoid router with expert selection…
29 -
b9992
CUDA: refactor MMQ kernel configuration ( #24127 ) CUDA: refactor MMQ kernel configuration fix Blackwell config remove legacy code macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64…
4 -
b9990
spec: add Minimax2 eagle3 support Fix nullptr in minimax2 EAGLE3 minor : add newline Co-authored-by: Georgi Gerganov ggerganov@gmail.com macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu…
27 -
b9988
tests: Harmonize header use ( #25616 ) tests: Harmonize the use of private ggml includes tests: In test-backend-ops, use quoted includes As with all other tests. This is to ensure that the build uses shipped headers over possibly system-installed ones. macOS/iOS: macOS Apple…
17 -
b9987
gguf : add tensor shape accessor ( #24405 ) gguf : add tensor shape accessors gguf : return tensor shape as const int64_t * gguf : remove n_dims accessor, keep only gguf_get_tensor_ne macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED…
33 -
b9986
chat : fix reasoning leak with force-opened bare templates ( #24674 ) chat : fix reasoning leak with force-opened bare templates The reasoning start tag inferred from prior turns can carry trailing whitespace (e.g. \n) while a force-open template prefills a bare . Trim the tag…
9 -
b9985
sycl: add fused top-k MoE ( #25217 ) sycl: add fused top-k MoE sycl: address review: GGML_SYCL_ENABLE_FUSION env, move fusion dispatch to topk-moe sycl: print GGML_SYCL_ENABLE_FUSION at startup like other env vars Co-Authored-By: Claude Fable 5 noreply@anthropic.com…
8 -
b9984
sycl: add Q2_K to DMMV reorder path ( #25064 ) Signed-off-by: Todd Malsbary todd.malsbary@intel.com macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu…
30 -
b9982
server: honour per-request reasoning_budget_tokens in chat completions ( #23116 ) server: honour per-request reasoning_budget_tokens in chat completions The reasoning-budget block in oaicompat_chat_params_parse read only the server-level default (opt.reasoning_budget, typically…
4 -
b9981
vendor : update cpp-httplib to 0.50.1 ( #25576 ) macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu s390x (CPU) Ubuntu x64 (Vulkan) Ubuntu arm64…
13 -
b9980
server: Don't consider models with --no-mmproj-auto as multimodal ( #25590 ) If mmproj is explicitly disabled via the model preset or command-line parameters then the model won't be able to handle image/audio inputs and this shouldn't be declared as supported input modality on…
25 -
b9979
mtmd: fix silent prompt truncation on embedded NUL ( #25548 ) mtmd: fix silent prompt truncation on embedded NUL mtmd_input_text carried the prompt as a bare const char* with no length, so a NUL byte in message content cut the prompt at the tokenizer boundary and dropped every…
32 -
b9978
server : evict checkpoints within min-step of each other ( #25472 ) macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu s390x (CPU) Ubuntu x64 (Vulkan)…
33 -
b9977
server : fix image blocks in tool_result being dropped during Anthropic OpenAI conversion ( #22536 ) server : fix image blocks in tool_result being dropped during Anthropic→OpenAI conversion server_chat_convert_anthropic_to_oai() silently discarded image blocks inside Anthropic…
28 -
b9975
gguf : reject empty metadata keys ( #24917 ) macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu s390x (CPU) Ubuntu x64 (Vulkan) Ubuntu arm64 (Vulkan)…
21 -
b9974
cuda: Don't crash when querying memory on device with no free memory. ( #25157 ) If a Cuda device has no or limited available memory, the actual call to cudaMemGetInfo() itself can cause a fatal crash due to a cuda out of memory error (there is not enough memory to actually…
38 -
b9973
DeepseekV4: clear cache only for seq rather than full ( #25521 ) macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu s390x (CPU) Ubuntu x64 (Vulkan)…
15 -
b9972
server: allow stream for exec_shell_command ( #25526 ) init stream add stream for shell tool add test nits update docs macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu…
27 -
b9971
server: refactor server_stream ( #25541 ) server: refactoring, remove spipe from server_http_res wip remove non-thread-safe rd.stop() call move server_res_spipe nits improve server_stream_create_spipe server-stream: update dev docs for the improved API Co-authored-by: Pascal…
28 -
b9970
ggml : add GGML_OP_LIGHTNING_INDEXER that implements DeepSeek V3.2/V4 lightning indexer ( #24231 ) ggml : add GGML_OP_LIGHTNING_INDEXER that implements DeepSeek V3.2/V4 lightning indexer ggml : remove scale parameters from lightning indexer OP, add f16 mask parameter tests : add…
14 -
b9969
Vulkan: route large matmuls to medium tile on Adreno ( #24877 ) [Vulkan] Fixes llama-cli breaking over longer promts sizes The llama-cli was breaking for longer promts sizes for q4_0 quantized networks. Causing due to insufficient shared memory. Removed the un-used Adreno device…
10 -
b9968
opencl: add int8 dp4 dense and MoE prefill optimization for Adreno GPUs ( #25537 ) opencl: add int8 dp4 dense and moe GEMM opencl: refactor Co-authored-by: Li He lih@qti.qualcomm.com macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED…
30