This website requires JavaScript.
About
Status
₊˚⊹Discord
Explore
Register
Sign in
Reference in a new issue
Blame
Copy permalink
Reference in a new issue
Copy permalink
DKB
/
zml
Star
0
Fork
You've already forked zml
0
mirror of
https://github.com/zml/zml.git
synced
2026-11-16 19:11:09 +00:00
Code
Issues
23
Releases
1
Packages
Activity
gw/flat_bufferized
zml
/
stdx
/
json.zig
153 lines
6 KiB
Raw
Permalink
Normal View
History
zml/tokenizer: use HuggingFace Tokenizers and SentencePiece & Llama improvements (#162) # HF tokenizers & Llama improvements 1. Added HuggingFace tokenizer and SentencePiece bindings, removed old tokenizers 2. Improved Llama example ## 1. zml/tokenizer: add HuggingFace tokenizers binding, remove old tokenizers `zml.Tokenizer` now uses the new HuggingFace tokenizer bindings when loading a `tokenizer.json`. To move fast, we dropped the `tinyllama`, `gguf`, and `sentencepiece` tokenizers. ## 2. examples/llama Changes to the llama example: - config.json support -> less cmdline flags - switched to [zig-clap](https://github.com/Hejsil/zig-clap) for cmdline args parsing - using HuggingFace tokenizer - re-worked and simplified the code - single token prediction - removed esoteric models from BUILD.bazel - default container now uses Llama-3.2-1B-Instruct --------- Co-authored-by: Steeve Morin <steeve@zml.ai> Co-authored-by: Corentin Godeau <corentin.godeau@zml.ai> Co-authored-by: Corentin Kerisit <corentin.kerisit@gmail.com>
2025-01-24 18:25:57 +01:00
zml: add some buffer / hostbuffer helpers (#232) * buffer/hostbuffer utils * more precise f32 -> bf16 conversion * visit should not read irrelevant fields * HostBuffer.data -> HostBuffer.bytes() * zml.aio loadBuffers: type inference * store expected input shapes in the executable * zml.meta.visit support std.BoundedArray * stdx.json.TaggedUnion * protection against undefined float hyperparams * avoid meta.TagPayload which can be quite slow * improve logging
2025-05-20 15:08:35 +02:00
zml/tokenizer: use HuggingFace Tokenizers and SentencePiece & Llama improvements (#162) # HF tokenizers & Llama improvements 1. Added HuggingFace tokenizer and SentencePiece bindings, removed old tokenizers 2. Improved Llama example ## 1. zml/tokenizer: add HuggingFace tokenizers binding, remove old tokenizers `zml.Tokenizer` now uses the new HuggingFace tokenizer bindings when loading a `tokenizer.json`. To move fast, we dropped the `tinyllama`, `gguf`, and `sentencepiece` tokenizers. ## 2. examples/llama Changes to the llama example: - config.json support -> less cmdline flags - switched to [zig-clap](https://github.com/Hejsil/zig-clap) for cmdline args parsing - using HuggingFace tokenizer - re-worked and simplified the code - single token prediction - removed esoteric models from BUILD.bazel - default container now uses Llama-3.2-1B-Instruct --------- Co-authored-by: Steeve Morin <steeve@zml.ai> Co-authored-by: Corentin Godeau <corentin.godeau@zml.ai> Co-authored-by: Corentin Kerisit <corentin.kerisit@gmail.com>
2025-01-24 18:25:57 +01:00
zml: add some buffer / hostbuffer helpers (#232) * buffer/hostbuffer utils * more precise f32 -> bf16 conversion * visit should not read irrelevant fields * HostBuffer.data -> HostBuffer.bytes() * zml.aio loadBuffers: type inference * store expected input shapes in the executable * zml.meta.visit support std.BoundedArray * stdx.json.TaggedUnion * protection against undefined float hyperparams * avoid meta.TagPayload which can be quite slow * improve logging
2025-05-20 15:08:35 +02:00
zml/tokenizer: use HuggingFace Tokenizers and SentencePiece & Llama improvements (#162) # HF tokenizers & Llama improvements 1. Added HuggingFace tokenizer and SentencePiece bindings, removed old tokenizers 2. Improved Llama example ## 1. zml/tokenizer: add HuggingFace tokenizers binding, remove old tokenizers `zml.Tokenizer` now uses the new HuggingFace tokenizer bindings when loading a `tokenizer.json`. To move fast, we dropped the `tinyllama`, `gguf`, and `sentencepiece` tokenizers. ## 2. examples/llama Changes to the llama example: - config.json support -> less cmdline flags - switched to [zig-clap](https://github.com/Hejsil/zig-clap) for cmdline args parsing - using HuggingFace tokenizer - re-worked and simplified the code - single token prediction - removed esoteric models from BUILD.bazel - default container now uses Llama-3.2-1B-Instruct --------- Co-authored-by: Steeve Morin <steeve@zml.ai> Co-authored-by: Corentin Godeau <corentin.godeau@zml.ai> Co-authored-by: Corentin Kerisit <corentin.kerisit@gmail.com>
2025-01-24 18:25:57 +01:00
zml: add some buffer / hostbuffer helpers (#232) * buffer/hostbuffer utils * more precise f32 -> bf16 conversion * visit should not read irrelevant fields * HostBuffer.data -> HostBuffer.bytes() * zml.aio loadBuffers: type inference * store expected input shapes in the executable * zml.meta.visit support std.BoundedArray * stdx.json.TaggedUnion * protection against undefined float hyperparams * avoid meta.TagPayload which can be quite slow * improve logging
2025-05-20 15:08:35 +02:00
zml/tokenizer: use HuggingFace Tokenizers and SentencePiece & Llama improvements (#162) # HF tokenizers & Llama improvements 1. Added HuggingFace tokenizer and SentencePiece bindings, removed old tokenizers 2. Improved Llama example ## 1. zml/tokenizer: add HuggingFace tokenizers binding, remove old tokenizers `zml.Tokenizer` now uses the new HuggingFace tokenizer bindings when loading a `tokenizer.json`. To move fast, we dropped the `tinyllama`, `gguf`, and `sentencepiece` tokenizers. ## 2. examples/llama Changes to the llama example: - config.json support -> less cmdline flags - switched to [zig-clap](https://github.com/Hejsil/zig-clap) for cmdline args parsing - using HuggingFace tokenizer - re-worked and simplified the code - single token prediction - removed esoteric models from BUILD.bazel - default container now uses Llama-3.2-1B-Instruct --------- Co-authored-by: Steeve Morin <steeve@zml.ai> Co-authored-by: Corentin Godeau <corentin.godeau@zml.ai> Co-authored-by: Corentin Kerisit <corentin.kerisit@gmail.com>
2025-01-24 18:25:57 +01:00
zml: add some buffer / hostbuffer helpers (#232) * buffer/hostbuffer utils * more precise f32 -> bf16 conversion * visit should not read irrelevant fields * HostBuffer.data -> HostBuffer.bytes() * zml.aio loadBuffers: type inference * store expected input shapes in the executable * zml.meta.visit support std.BoundedArray * stdx.json.TaggedUnion * protection against undefined float hyperparams * avoid meta.TagPayload which can be quite slow * improve logging
2025-05-20 15:08:35 +02:00
zml/tokenizer: use HuggingFace Tokenizers and SentencePiece & Llama improvements (#162) # HF tokenizers & Llama improvements 1. Added HuggingFace tokenizer and SentencePiece bindings, removed old tokenizers 2. Improved Llama example ## 1. zml/tokenizer: add HuggingFace tokenizers binding, remove old tokenizers `zml.Tokenizer` now uses the new HuggingFace tokenizer bindings when loading a `tokenizer.json`. To move fast, we dropped the `tinyllama`, `gguf`, and `sentencepiece` tokenizers. ## 2. examples/llama Changes to the llama example: - config.json support -> less cmdline flags - switched to [zig-clap](https://github.com/Hejsil/zig-clap) for cmdline args parsing - using HuggingFace tokenizer - re-worked and simplified the code - single token prediction - removed esoteric models from BUILD.bazel - default container now uses Llama-3.2-1B-Instruct --------- Co-authored-by: Steeve Morin <steeve@zml.ai> Co-authored-by: Corentin Godeau <corentin.godeau@zml.ai> Co-authored-by: Corentin Kerisit <corentin.kerisit@gmail.com>
2025-01-24 18:25:57 +01:00
zml: add some buffer / hostbuffer helpers (#232) * buffer/hostbuffer utils * more precise f32 -> bf16 conversion * visit should not read irrelevant fields * HostBuffer.data -> HostBuffer.bytes() * zml.aio loadBuffers: type inference * store expected input shapes in the executable * zml.meta.visit support std.BoundedArray * stdx.json.TaggedUnion * protection against undefined float hyperparams * avoid meta.TagPayload which can be quite slow * improve logging
2025-05-20 15:08:35 +02:00
zml/tokenizer: use HuggingFace Tokenizers and SentencePiece & Llama improvements (#162) # HF tokenizers & Llama improvements 1. Added HuggingFace tokenizer and SentencePiece bindings, removed old tokenizers 2. Improved Llama example ## 1. zml/tokenizer: add HuggingFace tokenizers binding, remove old tokenizers `zml.Tokenizer` now uses the new HuggingFace tokenizer bindings when loading a `tokenizer.json`. To move fast, we dropped the `tinyllama`, `gguf`, and `sentencepiece` tokenizers. ## 2. examples/llama Changes to the llama example: - config.json support -> less cmdline flags - switched to [zig-clap](https://github.com/Hejsil/zig-clap) for cmdline args parsing - using HuggingFace tokenizer - re-worked and simplified the code - single token prediction - removed esoteric models from BUILD.bazel - default container now uses Llama-3.2-1B-Instruct --------- Co-authored-by: Steeve Morin <steeve@zml.ai> Co-authored-by: Corentin Godeau <corentin.godeau@zml.ai> Co-authored-by: Corentin Kerisit <corentin.kerisit@gmail.com>
2025-01-24 18:25:57 +01:00
zig 0.14.0 (#189) Co-authored-by: Hugo Mano <hugo@zml.ai>
2025-03-14 09:49:23 +01:00
zml/tokenizer: use HuggingFace Tokenizers and SentencePiece & Llama improvements (#162) # HF tokenizers & Llama improvements 1. Added HuggingFace tokenizer and SentencePiece bindings, removed old tokenizers 2. Improved Llama example ## 1. zml/tokenizer: add HuggingFace tokenizers binding, remove old tokenizers `zml.Tokenizer` now uses the new HuggingFace tokenizer bindings when loading a `tokenizer.json`. To move fast, we dropped the `tinyllama`, `gguf`, and `sentencepiece` tokenizers. ## 2. examples/llama Changes to the llama example: - config.json support -> less cmdline flags - switched to [zig-clap](https://github.com/Hejsil/zig-clap) for cmdline args parsing - using HuggingFace tokenizer - re-worked and simplified the code - single token prediction - removed esoteric models from BUILD.bazel - default container now uses Llama-3.2-1B-Instruct --------- Co-authored-by: Steeve Morin <steeve@zml.ai> Co-authored-by: Corentin Godeau <corentin.godeau@zml.ai> Co-authored-by: Corentin Kerisit <corentin.kerisit@gmail.com>
2025-01-24 18:25:57 +01:00
zml: add some buffer / hostbuffer helpers (#232) * buffer/hostbuffer utils * more precise f32 -> bf16 conversion * visit should not read irrelevant fields * HostBuffer.data -> HostBuffer.bytes() * zml.aio loadBuffers: type inference * store expected input shapes in the executable * zml.meta.visit support std.BoundedArray * stdx.json.TaggedUnion * protection against undefined float hyperparams * avoid meta.TagPayload which can be quite slow * improve logging
2025-05-20 15:08:35 +02:00
zml/tokenizer: use HuggingFace Tokenizers and SentencePiece & Llama improvements (#162) # HF tokenizers & Llama improvements 1. Added HuggingFace tokenizer and SentencePiece bindings, removed old tokenizers 2. Improved Llama example ## 1. zml/tokenizer: add HuggingFace tokenizers binding, remove old tokenizers `zml.Tokenizer` now uses the new HuggingFace tokenizer bindings when loading a `tokenizer.json`. To move fast, we dropped the `tinyllama`, `gguf`, and `sentencepiece` tokenizers. ## 2. examples/llama Changes to the llama example: - config.json support -> less cmdline flags - switched to [zig-clap](https://github.com/Hejsil/zig-clap) for cmdline args parsing - using HuggingFace tokenizer - re-worked and simplified the code - single token prediction - removed esoteric models from BUILD.bazel - default container now uses Llama-3.2-1B-Instruct --------- Co-authored-by: Steeve Morin <steeve@zml.ai> Co-authored-by: Corentin Godeau <corentin.godeau@zml.ai> Co-authored-by: Corentin Kerisit <corentin.kerisit@gmail.com>
2025-01-24 18:25:57 +01:00