mirror of
https://github.com/zml/zml.git
synced 2026-11-16 19:11:09 +00:00
IREE writes a token's text whole or not at all, so `feedOne` asserted whenever the caller's buffer was smaller than the token, e.g. Laguna's 415-byte whitespace tokens in a 128-byte buffer. The decoder now owns its output buffer and grows it the way `decode` already did, so `feedOne` and `finalize` no longer take a buffer. |
History
|
|
|---|---|---|
| .. | ||
| models | ||
| BUILD.bazel | ||
| chat.zig | ||
| main.zig | ||
| models.zig | ||
| README.md | ||
LLM
//examples/llm runs interactive or one-shot text generation from a model repository from HuggingFace.
We support the following models, automatically detected from the model_type in the config.json:
- Llama 3.1
- Qwen 3.5
- LFM 2.5
Run
To load a model from HuggingFace directly:
# CPU
bazel run //examples/llm -- --model=hf://meta-llama/Llama-3.1-8B-Instruct
# CUDA
bazel run //examples/llm --@zml//platforms:cuda=true -- --model=hf://meta-llama/Llama-3.1-8B-Instruct
# ROCm
bazel run //examples/llm --@zml//platforms:rocm=true -- --model=hf://meta-llama/Llama-3.1-8B-Instruct
From a local directory:
bazel run //examples/llm --@zml//platforms:cuda=true -- --model=/var/models/meta-llama/Llama-3.1-8B-Instruct/
For a single non-interactive prompt:
bazel run //examples/llm --@zml//platforms:cuda=true -- --model=hf://meta-llama/Llama-3.1-8B-Instruct --prompt="Write a haiku about Zig"
Options
--model=<path>: Required. Model repository to load. This can be a local path or a huggingface/S3 URI such ashf://...ors3://....--prompt=<string>: Optional. Runs a single prompt instead of opening the interactive chat loop.--seqlen=<number>: Optional. Maximum sequence length. Defaults to2048.--backend=<stablehlo|cute|nki|metal>: Optional. Attention backend.cuteuses CuTe flash attention on CUDA GPUs with compute capability 8.0 or newer. If omitted, the program auto-selects one for the current platform, includingcuteon supported CUDA GPUs.