llama.cpp releases · · 1 min read

b11413: vulkan: sparse flash attention for quantized K/V (#29639)

Mirrored from llama.cpp releases for archival readability. Support the source by reading on the original site.

  • vulkan: sparse flash attention for quantized K/V

Assisted-by: Claude

  • vulkan: single-scan sparse FA index compaction

The compaction ran one workgroup per mask row and walked the row in
BLOCK_SIZE chunks, with a workgroup scan per chunk. For decode that is
one workgroup doing KV/1024 barrier-bound iterations, so at 128k cells
it cost more than the sparse attention it feeds.

Split the row into contiguous segments instead: one per subgroup with
ballot counting over coalesced loads, or one per thread without
subgroups. A single scan over the segment counts then gives each
segment its output offset. The index list stays ascending.

Discussion (0)

Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.

Sign in →

No comments yet. Sign in and be the first to say something.

More from llama.cpp releases