llama.cpp releases · · 1 min read

b11422: cuda: use the vector lightning indexer kernel on MUSA (#29990)

Mirrored from llama.cpp releases for archival readability. Support the source by reading on the original site.

  • cuda: stage the lightning indexer queries in head passes for MUSA

MUSA archs 21 and 22 cap static shared memory at 28 KB, and the tile
kernel staged the queries of all four heads next to the key tile for
33 KB. The queries are now staged in passes of
LIGHTNING_INDEXER_TILE_HEADS_PER_PASS heads: two on MUSA for 25 KB,
four elsewhere where the single pass folds to the previous kernel.

  • cuda: use the vector lightning indexer kernel on MUSA

Address review from am17an: the tile kernel stays off MUSA, whose archs
21 and 22 cap static shared memory at 28 KB, below the 33 KB the tile
needs, so MUSA keeps the vector kernel it ran before. This replaces the
head passes, CUDA and ROCm run the merged kernel unchanged.

Discussion (0)

Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.

Sign in →

No comments yet. Sign in and be the first to say something.

More from llama.cpp releases