0
Fork 0
mirror of https://github.com/zml/zml.git synced 2026-11-16 19:11:09 +00:00
zml/examples/llm
Raphael Nahoum 46bd32de2f zml/tokenizer: grow the decoder output to fit any token
IREE writes a token's text whole or not at all, so `feedOne` asserted
whenever the caller's buffer was smaller than the token, e.g. Laguna's
415-byte whitespace tokens in a 128-byte buffer. The decoder now owns
its output buffer and grows it the way `decode` already did, so
`feedOne` and `finalize` no longer take a buffer.
History 2026-11-09 10:56:20 +01:00
..
models zml/tokenizer: grow the decoder output to fit any token 2026-11-09 10:56:20 +01:00
BUILD.bazel bazel: make repo-wide cquery work across host and linux target analysis (#465) 2026-04-17 11:48:09 +02:00
chat.zig zml/exe: TypedExe (#697) 2026-08-18 16:36:38 +02:00
main.zig zml: re-structure attention kernels (#926) 2026-10-12 20:31:17 +02:00
models.zig zml/Tensor: explicit sharding in Tensor code (#868) 2026-10-06 15:37:22 +02:00
README.md zml: re-structure attention kernels (#926) 2026-10-12 20:31:17 +02:00

LLM

//examples/llm runs interactive or one-shot text generation from a model repository from HuggingFace.

We support the following models, automatically detected from the model_type in the config.json:

  • Llama 3.1
  • Qwen 3.5
  • LFM 2.5

Run

To load a model from HuggingFace directly:

# CPU
bazel run //examples/llm -- --model=hf://meta-llama/Llama-3.1-8B-Instruct
# CUDA
bazel run //examples/llm --@zml//platforms:cuda=true -- --model=hf://meta-llama/Llama-3.1-8B-Instruct
# ROCm
bazel run //examples/llm --@zml//platforms:rocm=true -- --model=hf://meta-llama/Llama-3.1-8B-Instruct

From a local directory:

bazel run //examples/llm --@zml//platforms:cuda=true -- --model=/var/models/meta-llama/Llama-3.1-8B-Instruct/

For a single non-interactive prompt:

bazel run //examples/llm --@zml//platforms:cuda=true -- --model=hf://meta-llama/Llama-3.1-8B-Instruct --prompt="Write a haiku about Zig"

Options

  • --model=<path>: Required. Model repository to load. This can be a local path or a huggingface/S3 URI such as hf://... or s3://....
  • --prompt=<string>: Optional. Runs a single prompt instead of opening the interactive chat loop.
  • --seqlen=<number>: Optional. Maximum sequence length. Defaults to 2048.
  • --backend=<stablehlo|cute|nki|metal>: Optional. Attention backend. cute uses CuTe flash attention on CUDA GPUs with compute capability 8.0 or newer. If omitted, the program auto-selects one for the current platform, including cute on supported CUDA GPUs.