0
Fork 0
mirror of https://github.com/zml/zml.git synced 2026-11-16 19:11:09 +00:00

Compare commits

...

1 commit

Author SHA1 Message Date
Raphael Nahoum
46bd32de2f zml/tokenizer: grow the decoder output to fit any token
IREE writes a token's text whole or not at all, so `feedOne` asserted
whenever the caller's buffer was smaller than the token, e.g. Laguna's
415-byte whitespace tokens in a 128-byte buffer. The decoder now owns
its output buffer and grows it the way `decode` already did, so
`feedOne` and `finalize` no longer take a buffer.
2026-11-09 10:56:20 +01:00
6 changed files with 33 additions and 49 deletions