0
Fork 0
mirror of https://github.com/zml/zml.git synced 2026-11-16 19:11:09 +00:00
zml/third_party/iree/match-hf-tokenizer.patch
Corentin Godeau 001dbbbde8
zml/tokenizer: Have matching tokenizer output (#623)
The IREE tokenizer was not producing the same output as the HF
tokenizer, potentially degrading the performance of the models. I put a
clanker to the task (because the tokenizer is already full clanked).
Outputs are now matching without any performance drop.

I also fixed an edge case issue on the `Writer` of the tokenizer.
2026-07-02 12:26:27 +02:00

1643 lines
74 KiB