mirror of
https://github.com/zml/zml.git
synced 2026-11-16 19:11:09 +00:00
The IREE tokenizer was not producing the same output as the HF tokenizer, potentially degrading the performance of the models. I put a clanker to the task (because the tokenizer is already full clanked). Outputs are now matching without any performance drop. I also fixed an edge case issue on the `Writer` of the tokenizer.
1643 lines
74 KiB
1643 lines
74 KiB