0
Fork 0
mirror of https://github.com/zml/zml.git synced 2026-11-16 19:11:09 +00:00
zml/stdx
Guillaume Wenzek f1d0c64db3
Restore tinyllama models (#173)
This is a follow up on #162 

* restore homemade tokenizer, integrate it with the new zml/tokenizer
library. This will only be used for working with tinyllama, I could also
make it work to support GGUF if needed (couple lines of code).
* this allows us to use the original tinyllama files from karpathy repo
which generates actual sentences. This is important when working on
modeling because subtle bugs can degrade the output quality, which is
not detectable if the output is crap to begin with.
* fix casing of zml/tokenizer api: fromFile, tokenToId. Let's stick to
the Zig style guide. Function written in Rust follows rust convention,
but the Zig bindings should follow Zig convention.
* make tokenizer/main.zig a proper tokenizer test tool reusing
llama/test_tokenizer.zig

~~use native_binary instead of cc_binary~~
Reverted because apparently native_binary break the runfiles and
therefore the cuda sandbox
History 2025-02-03 12:06:33 +01:00
..
BUILD.bazel Restore tinyllama models (#173) 2025-02-03 12:06:33 +01:00
debug.zig zml: add stdx, rework async signature inference (#89) 2024-11-15 17:12:07 +01:00
flags.zig Restore tinyllama models (#173) 2025-02-03 12:06:33 +01:00
io.zig async: rework async runtime to use coroutines and cross thread signaling (#100) 2024-11-22 17:56:22 +01:00
json.zig zml/tokenizer: use HuggingFace Tokenizers and SentencePiece & Llama improvements (#162) 2025-01-24 18:25:57 +01:00
math.zig zml: hotfixes for #89 (#91) 2024-11-15 18:03:00 +01:00
meta.zig async: vendor zigcoro and unify APIs (#152) 2025-01-14 19:33:16 +01:00
queue.zig async: vendor zigcoro and unify APIs (#152) 2025-01-14 19:33:16 +01:00
signature.zig async: vendor zigcoro and unify APIs (#152) 2025-01-14 19:33:16 +01:00
stdx.zig Restore tinyllama models (#173) 2025-02-03 12:06:33 +01:00
time.zig zml/tokenizer: use HuggingFace Tokenizers and SentencePiece & Llama improvements (#162) 2025-01-24 18:25:57 +01:00