This website requires JavaScript.
About
Status
₊˚⊹Discord
Explore
Register
Sign in
Reference in a new issue
Blame
Copy permalink
Reference in a new issue
Copy permalink
DKB
/
zml
Star
0
Fork
You've already forked zml
0
mirror of
https://github.com/zml/zml.git
synced
2026-11-16 19:11:09 +00:00
Code
Issues
23
Releases
1
Packages
Activity
gw/dev2
zml
/
stdx
/
flags.zig
582 lines
21 KiB
Raw
Permalink
Normal View
History
zml: initial release
2024-09-17 11:11:18 +02:00
Restore tinyllama models (#173) This is a follow up on #162 * restore homemade tokenizer, integrate it with the new zml/tokenizer library. This will only be used for working with tinyllama, I could also make it work to support GGUF if needed (couple lines of code). * this allows us to use the original tinyllama files from karpathy repo which generates actual sentences. This is important when working on modeling because subtle bugs can degrade the output quality, which is not detectable if the output is crap to begin with. * fix casing of zml/tokenizer api: fromFile, tokenToId. Let's stick to the Zig style guide. Function written in Rust follows rust convention, but the Zig bindings should follow Zig convention. * make tokenizer/main.zig a proper tokenizer test tool reusing llama/test_tokenizer.zig ~~use native_binary instead of cc_binary~~ Reverted because apparently native_binary break the runfiles and therefore the cuda sandbox
2025-02-03 12:06:33 +01:00
zml: initial release
2024-09-17 11:11:18 +02:00