0
Fork 0
mirror of https://github.com/zml/zml.git synced 2026-11-16 19:11:09 +00:00
zml/third_party/iree
Benjamin Rabier d4b413f5f9
zml/tokenizer: support Mistral's numeric lookahead (#997)
Mistral Large 4's Tekken tokenizer uses positive lookahead to split
number runs into groups of three from the right. IREE's regex compiler
rejects this pattern, preventing the tokenizer from loading.

Patch IREE to recognize the exact isolated Split pattern, match complete
number runs with its DFA, and emit the original groups using Unicode
codepoint counts. Preserve progress when output fills during streaming
and finalization. Register the patch in ZML's IREE repository rule.

Treat RESOURCE_EXHAUSTED as normal encoder backpressure instead of
logging errors before retrying. Add regressions for numeric token IDs,
Unicode numbers, byte-at-a-time input, and output overflow.
History 2026-10-29 14:10:32 +01:00
..
BUILD.bazel zml/tokenizer: add IREE tokenizer (#416) 2026-04-07 16:42:55 +02:00
fix-added-token-matching.patch third_party/iree: match all added tokens (#591) 2026-06-23 09:16:21 +02:00
fix-gemma4-encode.patch zml/tokenizer/iree: Improve IREE tokenizer with upstream fix (#482) 2026-04-20 11:15:37 +02:00
fix-normalizer-lifetime-and-boundaries.patch zml/tokenizer: add normalizer support (#847) 2026-09-23 16:57:04 +02:00
futex-public-api-header.patch workspace: bump to bazel 9.1.1 (#601) 2026-06-25 19:37:51 +02:00
laguna-tokenizer-compat.patch zml: ground work to support Poolside's Laguna models (#669) 2026-07-20 13:31:54 +02:00
match-hf-tokenizer.patch zml/tokenizer: Have matching tokenizer output (#623) 2026-07-02 12:26:27 +02:00
mistral-right-aligned-numbers.patch zml/tokenizer: support Mistral's numeric lookahead (#997) 2026-10-29 14:10:32 +01:00
repo.bzl zml/tokenizer: support Mistral's numeric lookahead (#997) 2026-10-29 14:10:32 +01:00
tokenizer-only.patch third_party/iree: bump iree tokenizer to 4d4e97d00f099a21f38eeff26f82a6d9e3643a11 (#624) 2026-07-02 15:46:02 +02:00