0
Fork 0
mirror of https://github.com/zml/zml.git synced 2026-11-16 19:11:09 +00:00

Compare commits

..

2 commits

Author SHA1 Message Date
Raphael Nahoum
b5de0f1670 Add comments 2026-11-09 16:22:30 +01:00
Raphael Nahoum
9a9ed49c9d zml/tokenizer: expose the decoder's maximum output length
IREE writes a token's text whole or not at all, so `feedOne` needs a
buffer that fits any token. `maxOutputLength` returns the bound IREE
uses for its own decode buffer: the vocabulary's longest token plus 4
bytes.
2026-11-09 13:24:56 +01:00
100 changed files with 5720 additions and 7589 deletions