v0.40.0-rc1: mlx: match publisher tokenizer semantics (#18779)
Mirrored from Ollama releases for archival readability. Support the source by reading on the original site.
- mlx: match publisher tokenizer semantics
Honor pretokenizer stage order, split behavior, Unicode boundaries,
added-token normalization, and ranked BPE merges. Handle empty added
tokens and empty Metaspace input consistently.
Add shared Go/Python reference cases using published tokenizers, pulling
missing models directly and failing on errors, plus focused regressions
for configuration precedence, byte fallback, and parallel encoding.
- address comments
Discussion (0)
Sign in to join the discussion. Free account, 30 seconds — email code or GitHub.
Sign in →No comments yet. Sign in and be the first to say something.