mirror of
https://github.com/zml/zml.git
synced 2026-11-16 19:11:09 +00:00
I've reworked the Tokenizer API to actually work with all tokenizers and not just IREE and moved the `Writer` part into the IREE tokenizer. I'm now avoiding intermediate copies as much as possible now. I'm also more defensive now to avoid the same infinite loop and put more effort into providing appropriate output buffers to IREE. It fails with empty output buffers and may return a `ResourceExhausted` if it's too small. Also added a bunch of tests. IREE tokenizer itself had a bug which I fixed here https://github.com/iree-org/iree/pull/24150. In the meantime, I'm pointing bazel to our fork.
149 lines
7.3 KiB
149 lines
7.3 KiB