0
Fork 0
mirror of https://github.com/zml/zml.git synced 2026-11-16 19:11:09 +00:00
zml/third_party
Rene Schallner f41e023e22
zml/tokenizer: use HuggingFace Tokenizers and SentencePiece & Llama improvements (#162)
# HF tokenizers & Llama improvements

1. Added HuggingFace tokenizer and SentencePiece bindings, removed old
tokenizers
2. Improved Llama example

## 1. zml/tokenizer: add HuggingFace tokenizers binding, remove old
tokenizers

`zml.Tokenizer` now uses the new HuggingFace tokenizer bindings when
loading a `tokenizer.json`.

To move fast, we dropped the `tinyllama`, `gguf`, and `sentencepiece`
tokenizers.

## 2. examples/llama

Changes to the llama example:

- config.json support -> less cmdline flags
- switched to [zig-clap](https://github.com/Hejsil/zig-clap) for cmdline
args parsing
- using HuggingFace tokenizer
- re-worked and simplified the code
    - single token prediction
- removed esoteric models from BUILD.bazel
- default container now uses Llama-3.2-1B-Instruct

---------

Co-authored-by: Steeve Morin <steeve@zml.ai>
Co-authored-by: Corentin Godeau <corentin.godeau@zml.ai>
Co-authored-by: Corentin Kerisit <corentin.kerisit@gmail.com>
History 2025-01-24 18:25:57 +01:00
..
com_google_sentencepiece zml/tokenizer: use HuggingFace Tokenizers and SentencePiece & Llama improvements (#162) 2025-01-24 18:25:57 +01:00
modules workspace: upgrade to Bazel 8 (#143) 2025-01-23 18:36:57 +01:00
org_swig_swig zml/tokenizer: use HuggingFace Tokenizers and SentencePiece & Llama improvements (#162) 2025-01-24 18:25:57 +01:00
zls workspace: rework code completion using a custom build runner (#131) 2024-12-16 13:20:27 +01:00
bazel_registry.json zml: initial release 2024-09-17 11:11:18 +02:00
non_module_deps.bzl zml/tokenizer: use HuggingFace Tokenizers and SentencePiece & Llama improvements (#162) 2025-01-24 18:25:57 +01:00