This website requires JavaScript.
About
Status
₊˚⊹Discord
Explore
Register
Sign in
Reference in a new issue
Blame
Copy permalink
Reference in a new issue
Copy permalink
DKB
/
zml
Star
0
Fork
You've already forked zml
0
mirror of
https://github.com/zml/zml.git
synced
2026-11-18 18:57:29 +00:00
Code
Issues
23
Releases
1
Packages
Activity
raphael/decoder-buffer
zml
/
third_party
/
iree
/
tokenizer-only.patch
50 lines
1.8 KiB
Raw
Permalink
Normal View
History
zml/tokenizer: add IREE tokenizer (#416) Adding the [IREE tokenizer](https://github.com/iree-org/iree/blob/main/runtime/src/iree/tokenizer/README.md).Being made from the ground up with streaming (de-)tokenization in mind, I've extended the APIs to expose a `std.Io.Writer` for the tokenization. I've quickly benchmarked IREE's tokenizer vs HuggingFace's one with a request I retrieved from a coding agent (opencode) in the middle of a session. The request body was 360KB encoded in 95K tokens. I also did another test with a similarly sized request with the same results as shown hereafter. I've tested two cases: - oneshot: the template is rendered all at once and only then we encode the whole string. - stream: the template rendering and encoding are pipelined to avoid the intermediate allocation. There is a slight difference in the generated tokens between HuggingFace and IREE. Some tokens like `introdu` were encoded as `i`, `n`, `t`, etc. Decoding them returns the same text. It's very close in terms of total token count: - hf: 95779 - iree/stream: 95705 - iree/oneshot: 95587 | Model | IREE avg (oneshot / stream) ms | HuggingFace avg (ms) | Uplift vs HF (oneshot / stream) | |---|---:|---:|---| | liquidai/LFM2.5-1.2B-Instruct | 8.5 / 7.3 | 40.5 | 4.8x / 5.5x | | Qwen/Qwen3.5-9B | 6.0 / 8.2 | 59.4 | 9.9x / 7.2x | | mistralai/Ministral-3-3B-Reasoning-2512 | 7.4 / 6.9 | 59.2 | 8x / 8.6x | It's unclear why the Qwen's chat template is somehow slower in the stream case, maybe because there are more writes. Adding non-empty buffer in the interface helps a bit, but tends to slow down mistral case. So well --------- Co-authored-by: Corentin Kerisit <corentin.kerisit@gmail.com>
2026-04-07 16:42:55 +02:00
third_party/iree: bump iree tokenizer to 4d4e97d00f099a21f38eeff26f82a6d9e3643a11 (#624)
2026-07-02 15:46:02 +02:00
zml/tokenizer: add IREE tokenizer (#416) Adding the [IREE tokenizer](https://github.com/iree-org/iree/blob/main/runtime/src/iree/tokenizer/README.md).Being made from the ground up with streaming (de-)tokenization in mind, I've extended the APIs to expose a `std.Io.Writer` for the tokenization. I've quickly benchmarked IREE's tokenizer vs HuggingFace's one with a request I retrieved from a coding agent (opencode) in the middle of a session. The request body was 360KB encoded in 95K tokens. I also did another test with a similarly sized request with the same results as shown hereafter. I've tested two cases: - oneshot: the template is rendered all at once and only then we encode the whole string. - stream: the template rendering and encoding are pipelined to avoid the intermediate allocation. There is a slight difference in the generated tokens between HuggingFace and IREE. Some tokens like `introdu` were encoded as `i`, `n`, `t`, etc. Decoding them returns the same text. It's very close in terms of total token count: - hf: 95779 - iree/stream: 95705 - iree/oneshot: 95587 | Model | IREE avg (oneshot / stream) ms | HuggingFace avg (ms) | Uplift vs HF (oneshot / stream) | |---|---:|---:|---| | liquidai/LFM2.5-1.2B-Instruct | 8.5 / 7.3 | 40.5 | 4.8x / 5.5x | | Qwen/Qwen3.5-9B | 6.0 / 8.2 | 59.4 | 9.9x / 7.2x | | mistralai/Ministral-3-3B-Reasoning-2512 | 7.4 / 6.9 | 59.2 | 8x / 8.6x | It's unclear why the Qwen's chat template is somehow slower in the stream case, maybe because there are more writes. Adding non-empty buffer in the interface helps a bit, but tends to slow down mistral case. So well --------- Co-authored-by: Corentin Kerisit <corentin.kerisit@gmail.com>
2026-04-07 16:42:55 +02:00
third_party/iree: bump iree tokenizer to 4d4e97d00f099a21f38eeff26f82a6d9e3643a11 (#624)
2026-07-02 15:46:02 +02:00
zml/tokenizer: add IREE tokenizer (#416) Adding the [IREE tokenizer](https://github.com/iree-org/iree/blob/main/runtime/src/iree/tokenizer/README.md).Being made from the ground up with streaming (de-)tokenization in mind, I've extended the APIs to expose a `std.Io.Writer` for the tokenization. I've quickly benchmarked IREE's tokenizer vs HuggingFace's one with a request I retrieved from a coding agent (opencode) in the middle of a session. The request body was 360KB encoded in 95K tokens. I also did another test with a similarly sized request with the same results as shown hereafter. I've tested two cases: - oneshot: the template is rendered all at once and only then we encode the whole string. - stream: the template rendering and encoding are pipelined to avoid the intermediate allocation. There is a slight difference in the generated tokens between HuggingFace and IREE. Some tokens like `introdu` were encoded as `i`, `n`, `t`, etc. Decoding them returns the same text. It's very close in terms of total token count: - hf: 95779 - iree/stream: 95705 - iree/oneshot: 95587 | Model | IREE avg (oneshot / stream) ms | HuggingFace avg (ms) | Uplift vs HF (oneshot / stream) | |---|---:|---:|---| | liquidai/LFM2.5-1.2B-Instruct | 8.5 / 7.3 | 40.5 | 4.8x / 5.5x | | Qwen/Qwen3.5-9B | 6.0 / 8.2 | 59.4 | 9.9x / 7.2x | | mistralai/Ministral-3-3B-Reasoning-2512 | 7.4 / 6.9 | 59.2 | 8x / 8.6x | It's unclear why the Qwen's chat template is somehow slower in the stream case, maybe because there are more writes. Adding non-empty buffer in the interface helps a bit, but tends to slow down mistral case. So well --------- Co-authored-by: Corentin Kerisit <corentin.kerisit@gmail.com>
2026-04-07 16:42:55 +02:00
third_party/iree: bump iree tokenizer to 4d4e97d00f099a21f38eeff26f82a6d9e3643a11 (#624)
2026-07-02 15:46:02 +02:00
zml/tokenizer: add IREE tokenizer (#416) Adding the [IREE tokenizer](https://github.com/iree-org/iree/blob/main/runtime/src/iree/tokenizer/README.md).Being made from the ground up with streaming (de-)tokenization in mind, I've extended the APIs to expose a `std.Io.Writer` for the tokenization. I've quickly benchmarked IREE's tokenizer vs HuggingFace's one with a request I retrieved from a coding agent (opencode) in the middle of a session. The request body was 360KB encoded in 95K tokens. I also did another test with a similarly sized request with the same results as shown hereafter. I've tested two cases: - oneshot: the template is rendered all at once and only then we encode the whole string. - stream: the template rendering and encoding are pipelined to avoid the intermediate allocation. There is a slight difference in the generated tokens between HuggingFace and IREE. Some tokens like `introdu` were encoded as `i`, `n`, `t`, etc. Decoding them returns the same text. It's very close in terms of total token count: - hf: 95779 - iree/stream: 95705 - iree/oneshot: 95587 | Model | IREE avg (oneshot / stream) ms | HuggingFace avg (ms) | Uplift vs HF (oneshot / stream) | |---|---:|---:|---| | liquidai/LFM2.5-1.2B-Instruct | 8.5 / 7.3 | 40.5 | 4.8x / 5.5x | | Qwen/Qwen3.5-9B | 6.0 / 8.2 | 59.4 | 9.9x / 7.2x | | mistralai/Ministral-3-3B-Reasoning-2512 | 7.4 / 6.9 | 59.2 | 8x / 8.6x | It's unclear why the Qwen's chat template is somehow slower in the stream case, maybe because there are more writes. Adding non-empty buffer in the interface helps a bit, but tends to slow down mistral case. So well --------- Co-authored-by: Corentin Kerisit <corentin.kerisit@gmail.com>
2026-04-07 16:42:55 +02:00
third_party/iree: bump iree tokenizer to 4d4e97d00f099a21f38eeff26f82a6d9e3643a11 (#624)
2026-07-02 15:46:02 +02:00
zml/tokenizer: add IREE tokenizer (#416) Adding the [IREE tokenizer](https://github.com/iree-org/iree/blob/main/runtime/src/iree/tokenizer/README.md).Being made from the ground up with streaming (de-)tokenization in mind, I've extended the APIs to expose a `std.Io.Writer` for the tokenization. I've quickly benchmarked IREE's tokenizer vs HuggingFace's one with a request I retrieved from a coding agent (opencode) in the middle of a session. The request body was 360KB encoded in 95K tokens. I also did another test with a similarly sized request with the same results as shown hereafter. I've tested two cases: - oneshot: the template is rendered all at once and only then we encode the whole string. - stream: the template rendering and encoding are pipelined to avoid the intermediate allocation. There is a slight difference in the generated tokens between HuggingFace and IREE. Some tokens like `introdu` were encoded as `i`, `n`, `t`, etc. Decoding them returns the same text. It's very close in terms of total token count: - hf: 95779 - iree/stream: 95705 - iree/oneshot: 95587 | Model | IREE avg (oneshot / stream) ms | HuggingFace avg (ms) | Uplift vs HF (oneshot / stream) | |---|---:|---:|---| | liquidai/LFM2.5-1.2B-Instruct | 8.5 / 7.3 | 40.5 | 4.8x / 5.5x | | Qwen/Qwen3.5-9B | 6.0 / 8.2 | 59.4 | 9.9x / 7.2x | | mistralai/Ministral-3-3B-Reasoning-2512 | 7.4 / 6.9 | 59.2 | 8x / 8.6x | It's unclear why the Qwen's chat template is somehow slower in the stream case, maybe because there are more writes. Adding non-empty buffer in the interface helps a bit, but tends to slow down mistral case. So well --------- Co-authored-by: Corentin Kerisit <corentin.kerisit@gmail.com>
2026-04-07 16:42:55 +02:00