mirror of
https://github.com/zml/zml.git
synced 2026-11-16 19:11:09 +00:00
This is a follow up on #162 * restore homemade tokenizer, integrate it with the new zml/tokenizer library. This will only be used for working with tinyllama, I could also make it work to support GGUF if needed (couple lines of code). * this allows us to use the original tinyllama files from karpathy repo which generates actual sentences. This is important when working on modeling because subtle bugs can degrade the output quality, which is not detectable if the output is crap to begin with. * fix casing of zml/tokenizer api: fromFile, tokenToId. Let's stick to the Zig style guide. Function written in Rust follows rust convention, but the Zig bindings should follow Zig convention. * make tokenizer/main.zig a proper tokenizer test tool reusing llama/test_tokenizer.zig ~~use native_binary instead of cc_binary~~ Reverted because apparently native_binary break the runfiles and therefore the cuda sandbox
582 lines
21 KiB
582 lines
21 KiB