b11612: unicode : add opt-in NFC normalization for Unigram tokenizers (#30390)
Mirrored from llama.cpp releases for archival readability. Support the source by reading on the original site.
- unicode : add opt-in NFC normalization for Unigram tokenizers
Note: used help from Codex, Sol 6.1 Medium
-
Fixing initial bot comments + linter error (using existing out instead of print)
-
removed test_nfc_tokenizer
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.