mirror of
https://github.com/zml/zml.git
synced 2026-11-16 19:11:09 +00:00
Mistral Large 4's Tekken tokenizer uses positive lookahead to split number runs into groups of three from the right. IREE's regex compiler rejects this pattern, preventing the tokenizer from loading. Patch IREE to recognize the exact isolated Split pattern, match complete number runs with its DFA, and emit the original groups using Unicode codepoint counts. Preserve progress when output fills during streaming and finalization. Register the patch in ZML's IREE repository rule. Treat RESOURCE_EXHAUSTED as normal encoder backpressure instead of logging errors before retrying. Add regressions for numeric token IDs, Unicode numbers, byte-at-a-time input, and output overflow. |
History
|
|
|---|---|---|
| .. | ||
| BUILD.bazel | ||
| fix-added-token-matching.patch | ||
| fix-gemma4-encode.patch | ||
| fix-normalizer-lifetime-and-boundaries.patch | ||
| futex-public-api-header.patch | ||
| laguna-tokenizer-compat.patch | ||
| match-hf-tokenizer.patch | ||
| mistral-right-aligned-numbers.patch | ||
| repo.bzl | ||
| tokenizer-only.patch | ||