This website requires JavaScript.
About
Status
₊˚⊹Discord
Explore
Register
Sign in
Reference in a new issue
Blame
Copy permalink
Reference in a new issue
Copy permalink
DKB
/
zml
Star
0
Fork
You've already forked zml
0
mirror of
https://github.com/zml/zml.git
synced
2026-11-16 19:11:09 +00:00
Code
Issues
23
Releases
1
Packages
Activity
raphael/decoder-buffer
zml
/
stdx
/
time.zig
43 lines
998 B
Raw
Permalink
Normal View
History
zml/tokenizer: use HuggingFace Tokenizers and SentencePiece & Llama improvements (#162) # HF tokenizers & Llama improvements 1. Added HuggingFace tokenizer and SentencePiece bindings, removed old tokenizers 2. Improved Llama example ## 1. zml/tokenizer: add HuggingFace tokenizers binding, remove old tokenizers `zml.Tokenizer` now uses the new HuggingFace tokenizer bindings when loading a `tokenizer.json`. To move fast, we dropped the `tinyllama`, `gguf`, and `sentencepiece` tokenizers. ## 2. examples/llama Changes to the llama example: - config.json support -> less cmdline flags - switched to [zig-clap](https://github.com/Hejsil/zig-clap) for cmdline args parsing - using HuggingFace tokenizer - re-worked and simplified the code - single token prediction - removed esoteric models from BUILD.bazel - default container now uses Llama-3.2-1B-Instruct --------- Co-authored-by: Steeve Morin <steeve@zml.ai> Co-authored-by: Corentin Godeau <corentin.godeau@zml.ai> Co-authored-by: Corentin Kerisit <corentin.kerisit@gmail.com>
2025-01-24 18:25:57 +01:00
workspace: various fixes and bumps (#229) This PR add various fixes and bumps in the workspace, notably to prevent dual builds for LLVM tools.
2025-04-22 16:12:26 +02:00
zml/tokenizer: use HuggingFace Tokenizers and SentencePiece & Llama improvements (#162) # HF tokenizers & Llama improvements 1. Added HuggingFace tokenizer and SentencePiece bindings, removed old tokenizers 2. Improved Llama example ## 1. zml/tokenizer: add HuggingFace tokenizers binding, remove old tokenizers `zml.Tokenizer` now uses the new HuggingFace tokenizer bindings when loading a `tokenizer.json`. To move fast, we dropped the `tinyllama`, `gguf`, and `sentencepiece` tokenizers. ## 2. examples/llama Changes to the llama example: - config.json support -> less cmdline flags - switched to [zig-clap](https://github.com/Hejsil/zig-clap) for cmdline args parsing - using HuggingFace tokenizer - re-worked and simplified the code - single token prediction - removed esoteric models from BUILD.bazel - default container now uses Llama-3.2-1B-Instruct --------- Co-authored-by: Steeve Morin <steeve@zml.ai> Co-authored-by: Corentin Godeau <corentin.godeau@zml.ai> Co-authored-by: Corentin Kerisit <corentin.kerisit@gmail.com>
2025-01-24 18:25:57 +01:00
workspace: add debug/release configurations and various stdx (#234) This PR bumps rules_zig to add command line to the zig compilation in order to enable debug and release configs from the bazel command line (and bazelrc). Most notably, this is useful for passing `-mcpu` flags to Zig (especially for SIMD support like AVX512). While we're there, add `stdx.time.Duration.hz()` and `stdx.math.roundeven()`.
2025-05-19 21:26:28 +02:00
stdx: add time.hzFloat
2026-01-05 09:07:18 +00:00
Small fix in stdx
2025-12-15 16:10:37 +01:00
workspace: zig 0.15 (#316) Much of the work was done by @gwenzek Sadly we drop the torch loader in the process, to be added back later ! --------- Co-authored-by: Guillaume Wenzek <wzk@zml.ai>
2025-08-27 14:11:22 +02:00
zml/tokenizer: use HuggingFace Tokenizers and SentencePiece & Llama improvements (#162) # HF tokenizers & Llama improvements 1. Added HuggingFace tokenizer and SentencePiece bindings, removed old tokenizers 2. Improved Llama example ## 1. zml/tokenizer: add HuggingFace tokenizers binding, remove old tokenizers `zml.Tokenizer` now uses the new HuggingFace tokenizer bindings when loading a `tokenizer.json`. To move fast, we dropped the `tinyllama`, `gguf`, and `sentencepiece` tokenizers. ## 2. examples/llama Changes to the llama example: - config.json support -> less cmdline flags - switched to [zig-clap](https://github.com/Hejsil/zig-clap) for cmdline args parsing - using HuggingFace tokenizer - re-worked and simplified the code - single token prediction - removed esoteric models from BUILD.bazel - default container now uses Llama-3.2-1B-Instruct --------- Co-authored-by: Steeve Morin <steeve@zml.ai> Co-authored-by: Corentin Godeau <corentin.godeau@zml.ai> Co-authored-by: Corentin Kerisit <corentin.kerisit@gmail.com>
2025-01-24 18:25:57 +01:00
workspace: zig 0.15 (#316) Much of the work was done by @gwenzek Sadly we drop the torch loader in the process, to be added back later ! --------- Co-authored-by: Guillaume Wenzek <wzk@zml.ai>
2025-08-27 14:11:22 +02:00
zml/tokenizer: use HuggingFace Tokenizers and SentencePiece & Llama improvements (#162) # HF tokenizers & Llama improvements 1. Added HuggingFace tokenizer and SentencePiece bindings, removed old tokenizers 2. Improved Llama example ## 1. zml/tokenizer: add HuggingFace tokenizers binding, remove old tokenizers `zml.Tokenizer` now uses the new HuggingFace tokenizer bindings when loading a `tokenizer.json`. To move fast, we dropped the `tinyllama`, `gguf`, and `sentencepiece` tokenizers. ## 2. examples/llama Changes to the llama example: - config.json support -> less cmdline flags - switched to [zig-clap](https://github.com/Hejsil/zig-clap) for cmdline args parsing - using HuggingFace tokenizer - re-worked and simplified the code - single token prediction - removed esoteric models from BUILD.bazel - default container now uses Llama-3.2-1B-Instruct --------- Co-authored-by: Steeve Morin <steeve@zml.ai> Co-authored-by: Corentin Godeau <corentin.godeau@zml.ai> Co-authored-by: Corentin Kerisit <corentin.kerisit@gmail.com>
2025-01-24 18:25:57 +01:00
workspace: various fixes and bumps (#229) This PR add various fixes and bumps in the workspace, notably to prevent dual builds for LLVM tools.
2025-04-22 16:12:26 +02:00
zml/tokenizer: use HuggingFace Tokenizers and SentencePiece & Llama improvements (#162) # HF tokenizers & Llama improvements 1. Added HuggingFace tokenizer and SentencePiece bindings, removed old tokenizers 2. Improved Llama example ## 1. zml/tokenizer: add HuggingFace tokenizers binding, remove old tokenizers `zml.Tokenizer` now uses the new HuggingFace tokenizer bindings when loading a `tokenizer.json`. To move fast, we dropped the `tinyllama`, `gguf`, and `sentencepiece` tokenizers. ## 2. examples/llama Changes to the llama example: - config.json support -> less cmdline flags - switched to [zig-clap](https://github.com/Hejsil/zig-clap) for cmdline args parsing - using HuggingFace tokenizer - re-worked and simplified the code - single token prediction - removed esoteric models from BUILD.bazel - default container now uses Llama-3.2-1B-Instruct --------- Co-authored-by: Steeve Morin <steeve@zml.ai> Co-authored-by: Corentin Godeau <corentin.godeau@zml.ai> Co-authored-by: Corentin Kerisit <corentin.kerisit@gmail.com>
2025-01-24 18:25:57 +01:00