0
Fork 0
mirror of https://github.com/zml/zml.git synced 2026-11-16 19:11:09 +00:00
Commit graph

9 commits

Author SHA1 Message Date
Benjamin Rabier
d4b413f5f9
zml/tokenizer: support Mistral's numeric lookahead (#997)
Mistral Large 4's Tekken tokenizer uses positive lookahead to split
number runs into groups of three from the right. IREE's regex compiler
rejects this pattern, preventing the tokenizer from loading.

Patch IREE to recognize the exact isolated Split pattern, match complete
number runs with its DFA, and emit the original groups using Unicode
codepoint counts. Preserve progress when output fills during streaming
and finalization. Register the patch in ZML's IREE repository rule.

Treat RESOURCE_EXHAUSTED as normal encoder backpressure instead of
logging errors before retrying. Add regressions for numeric token IDs,
Unicode numbers, byte-at-a-time input, and output overflow.
2026-10-29 14:10:32 +01:00
Damien Mehala
11d49d8bc5
zml/tokenizer: add normalizer support (#847)
Support normalization from HF JSON and typed normalizer sequences.

Changes:
   - Patch IREE memory ownership and chunk-boundary bugs
   - Add regression tests for IREE bugs.
2026-09-23 16:57:04 +02:00
Corentin Godeau
d57d851b63
zml: ground work to support Poolside's Laguna models (#669)
This PR contains two main changes:
* A patch for the tokenizer to support the tokenizer config file
* A temporary workaround to speed-up loading of the model when executing
arbitrary executable at load time
2026-07-20 13:31:54 +02:00
Corentin Godeau
51db372455
third_party/iree: bump iree tokenizer to 4d4e97d00f099a21f38eeff26f82a6d9e3643a11 (#624) 2026-07-02 15:46:02 +02:00
Corentin Godeau
001dbbbde8
zml/tokenizer: Have matching tokenizer output (#623)
The IREE tokenizer was not producing the same output as the HF
tokenizer, potentially degrading the performance of the models. I put a
clanker to the task (because the tokenizer is already full clanked).
Outputs are now matching without any performance drop.

I also fixed an edge case issue on the `Writer` of the tokenizer.
2026-07-02 12:26:27 +02:00
Corentin Kerisit
aa90f92832
workspace: bump to bazel 9.1.1 (#601)
A few missing loads
Remove all flipped flags or deperacted flags from .bazelrc

Important:
- Keep autoload rules_cc and rules_python because XLA still has
transitive deps that are missing loads, PR in progress.
2026-06-25 19:37:51 +02:00
elogir
f586cbfc3d
third_party/iree: match all added tokens (#591)
IREE's tokenizer only registered added tokens for pre-segmentation
matching if they were flagged special or normalized. Tokens that are
neither (e.g. <think> was dropped)

The fix is to match all added tokens
2026-06-23 09:16:21 +02:00
Benjamin Rabier
df86cdfd96
zml/tokenizer/iree: Improve IREE tokenizer with upstream fix (#482)
I've reworked the Tokenizer API to actually work with all tokenizers and
not just IREE and moved the `Writer` part into the IREE tokenizer. I'm
now avoiding intermediate copies as much as possible now.

I'm also more defensive now to avoid the same infinite loop and put more
effort into providing appropriate output buffers to IREE. It fails with
empty output buffers and may return a `ResourceExhausted` if it's too
small. Also added a bunch of tests.

IREE tokenizer itself had a bug which I fixed here
https://github.com/iree-org/iree/pull/24150. In the meantime, I'm
pointing bazel to our fork.
2026-04-20 11:15:37 +02:00
Benjamin Rabier
448084908a
zml/tokenizer: add IREE tokenizer (#416)
Adding the [IREE
tokenizer](https://github.com/iree-org/iree/blob/main/runtime/src/iree/tokenizer/README.md).Being
made from the ground up with streaming (de-)tokenization in mind, I've
extended the APIs to expose a `std.Io.Writer` for the tokenization.

I've quickly benchmarked IREE's tokenizer vs HuggingFace's one with a
request I retrieved from a coding agent (opencode) in the middle of a
session. The request body was 360KB encoded in 95K tokens. I also did
another test with a similarly sized request with the same results as
shown hereafter.

I've tested two cases:

- oneshot: the template is rendered all at once and only then we encode
the whole string.
- stream: the template rendering and encoding are pipelined to avoid the
intermediate allocation.

There is a slight difference in the generated tokens between HuggingFace
and IREE. Some tokens like `introdu` were encoded as `i`, `n`, `t`, etc.
Decoding them returns the same text. It's very close in terms of total
token count:

- hf: 95779
- iree/stream: 95705
- iree/oneshot: 95587

| Model | IREE avg (oneshot / stream) ms | HuggingFace avg (ms) | Uplift
vs HF (oneshot / stream) |
|---|---:|---:|---|
| liquidai/LFM2.5-1.2B-Instruct | 8.5 / 7.3 | 40.5 | 4.8x / 5.5x |
| Qwen/Qwen3.5-9B | 6.0 / 8.2 | 59.4 | 9.9x / 7.2x |
| mistralai/Ministral-3-3B-Reasoning-2512 | 7.4 / 6.9 | 59.2 | 8x / 8.6x
|


It's unclear why the Qwen's chat template is somehow slower in the
stream case, maybe because there are more writes. Adding non-empty
buffer in the interface helps a bit, but tends to slow down mistral
case. So well

---------

Co-authored-by: Corentin Kerisit <corentin.kerisit@gmail.com>
2026-04-07 16:42:55 +02:00