0
Fork 0
mirror of https://github.com/zml/zml.git synced 2026-11-16 19:11:09 +00:00
zml/docs
Rene Schallner f41e023e22
zml/tokenizer: use HuggingFace Tokenizers and SentencePiece & Llama improvements (#162)
# HF tokenizers & Llama improvements

1. Added HuggingFace tokenizer and SentencePiece bindings, removed old
tokenizers
2. Improved Llama example

## 1. zml/tokenizer: add HuggingFace tokenizers binding, remove old
tokenizers

`zml.Tokenizer` now uses the new HuggingFace tokenizer bindings when
loading a `tokenizer.json`.

To move fast, we dropped the `tinyllama`, `gguf`, and `sentencepiece`
tokenizers.

## 2. examples/llama

Changes to the llama example:

- config.json support -> less cmdline flags
- switched to [zig-clap](https://github.com/Hejsil/zig-clap) for cmdline
args parsing
- using HuggingFace tokenizer
- re-worked and simplified the code
    - single token prediction
- removed esoteric models from BUILD.bazel
- default container now uses Llama-3.2-1B-Instruct

---------

Co-authored-by: Steeve Morin <steeve@zml.ai>
Co-authored-by: Corentin Godeau <corentin.godeau@zml.ai>
Co-authored-by: Corentin Kerisit <corentin.kerisit@gmail.com>
History 2025-01-24 18:25:57 +01:00
..
howtos pjrt/zml: expose client creation flags (#129) 2024-12-11 13:38:04 +01:00
learn docs: move image in concepts.md (#128) 2024-12-09 19:48:10 +01:00
misc zml: initial release 2024-09-17 11:11:18 +02:00
tutorials zml/tokenizer: use HuggingFace Tokenizers and SentencePiece & Llama improvements (#162) 2025-01-24 18:25:57 +01:00
huggingface-access-token.md workspace: be a little more precise regarding the huggingface token (#45) 2024-10-06 14:34:38 -07:00
README.md [Docs] : fix typos (#60) 2024-10-22 17:38:05 +02:00

Greetings, Programs!

If you want to run our example models, please head over to Getting Started.

Ready to write some code? Try starting with your first model in ZML, or familiarize yourself with the high-level ZML concepts first.

MENU:

Tutorials

How to ?

Learn more...

  • ZML Concepts : Tensors, Models, Executables, etc. explained

Contribute