This website requires JavaScript.
About
Status
₊˚⊹Discord
Explore
Register
Sign in
Reference in a new issue
Blame
Copy permalink
Reference in a new issue
Copy permalink
DKB
/
zml
Star
0
Fork
You've already forked zml
0
mirror of
https://github.com/zml/zml.git
synced
2026-11-16 19:11:09 +00:00
Code
Issues
23
Releases
1
Packages
Activity
steeve/zig_binary
zml
/
stdx
/
time.zig
42 lines
941 B
Raw
Permalink
Normal View
History
zml/tokenizer: use HuggingFace Tokenizers and SentencePiece & Llama improvements (#162) # HF tokenizers & Llama improvements 1. Added HuggingFace tokenizer and SentencePiece bindings, removed old tokenizers 2. Improved Llama example ## 1. zml/tokenizer: add HuggingFace tokenizers binding, remove old tokenizers `zml.Tokenizer` now uses the new HuggingFace tokenizer bindings when loading a `tokenizer.json`. To move fast, we dropped the `tinyllama`, `gguf`, and `sentencepiece` tokenizers. ## 2. examples/llama Changes to the llama example: - config.json support -> less cmdline flags - switched to [zig-clap](https://github.com/Hejsil/zig-clap) for cmdline args parsing - using HuggingFace tokenizer - re-worked and simplified the code - single token prediction - removed esoteric models from BUILD.bazel - default container now uses Llama-3.2-1B-Instruct --------- Co-authored-by: Steeve Morin <steeve@zml.ai> Co-authored-by: Corentin Godeau <corentin.godeau@zml.ai> Co-authored-by: Corentin Kerisit <corentin.kerisit@gmail.com>
2025-01-24 18:25:57 +01:00
workspace: various fixes and bumps (#229) This PR add various fixes and bumps in the workspace, notably to prevent dual builds for LLVM tools.
2025-04-22 16:12:26 +02:00
zml/tokenizer: use HuggingFace Tokenizers and SentencePiece & Llama improvements (#162) # HF tokenizers & Llama improvements 1. Added HuggingFace tokenizer and SentencePiece bindings, removed old tokenizers 2. Improved Llama example ## 1. zml/tokenizer: add HuggingFace tokenizers binding, remove old tokenizers `zml.Tokenizer` now uses the new HuggingFace tokenizer bindings when loading a `tokenizer.json`. To move fast, we dropped the `tinyllama`, `gguf`, and `sentencepiece` tokenizers. ## 2. examples/llama Changes to the llama example: - config.json support -> less cmdline flags - switched to [zig-clap](https://github.com/Hejsil/zig-clap) for cmdline args parsing - using HuggingFace tokenizer - re-worked and simplified the code - single token prediction - removed esoteric models from BUILD.bazel - default container now uses Llama-3.2-1B-Instruct --------- Co-authored-by: Steeve Morin <steeve@zml.ai> Co-authored-by: Corentin Godeau <corentin.godeau@zml.ai> Co-authored-by: Corentin Kerisit <corentin.kerisit@gmail.com>
2025-01-24 18:25:57 +01:00
workspace: add debug/release configurations and various stdx (#234) This PR bumps rules_zig to add command line to the zig compilation in order to enable debug and release configs from the bazel command line (and bazelrc). Most notably, this is useful for passing `-mcpu` flags to Zig (especially for SIMD support like AVX512). While we're there, add `stdx.time.Duration.hz()` and `stdx.math.roundeven()`.
2025-05-19 21:26:28 +02:00
zml/tokenizer: use HuggingFace Tokenizers and SentencePiece & Llama improvements (#162) # HF tokenizers & Llama improvements 1. Added HuggingFace tokenizer and SentencePiece bindings, removed old tokenizers 2. Improved Llama example ## 1. zml/tokenizer: add HuggingFace tokenizers binding, remove old tokenizers `zml.Tokenizer` now uses the new HuggingFace tokenizer bindings when loading a `tokenizer.json`. To move fast, we dropped the `tinyllama`, `gguf`, and `sentencepiece` tokenizers. ## 2. examples/llama Changes to the llama example: - config.json support -> less cmdline flags - switched to [zig-clap](https://github.com/Hejsil/zig-clap) for cmdline args parsing - using HuggingFace tokenizer - re-worked and simplified the code - single token prediction - removed esoteric models from BUILD.bazel - default container now uses Llama-3.2-1B-Instruct --------- Co-authored-by: Steeve Morin <steeve@zml.ai> Co-authored-by: Corentin Godeau <corentin.godeau@zml.ai> Co-authored-by: Corentin Kerisit <corentin.kerisit@gmail.com>
2025-01-24 18:25:57 +01:00
workspace: various fixes and bumps (#229) This PR add various fixes and bumps in the workspace, notably to prevent dual builds for LLVM tools.
2025-04-22 16:12:26 +02:00
zml/tokenizer: use HuggingFace Tokenizers and SentencePiece & Llama improvements (#162) # HF tokenizers & Llama improvements 1. Added HuggingFace tokenizer and SentencePiece bindings, removed old tokenizers 2. Improved Llama example ## 1. zml/tokenizer: add HuggingFace tokenizers binding, remove old tokenizers `zml.Tokenizer` now uses the new HuggingFace tokenizer bindings when loading a `tokenizer.json`. To move fast, we dropped the `tinyllama`, `gguf`, and `sentencepiece` tokenizers. ## 2. examples/llama Changes to the llama example: - config.json support -> less cmdline flags - switched to [zig-clap](https://github.com/Hejsil/zig-clap) for cmdline args parsing - using HuggingFace tokenizer - re-worked and simplified the code - single token prediction - removed esoteric models from BUILD.bazel - default container now uses Llama-3.2-1B-Instruct --------- Co-authored-by: Steeve Morin <steeve@zml.ai> Co-authored-by: Corentin Godeau <corentin.godeau@zml.ai> Co-authored-by: Corentin Kerisit <corentin.kerisit@gmail.com>
2025-01-24 18:25:57 +01:00