0
Fork 0
mirror of https://github.com/zml/zml.git synced 2026-11-16 19:11:09 +00:00
zml/examples/llm
Hugo Mano 1042cba3ff
zml/kernels/moe: make mesh and partitioning explicit (#1028)
MOE API currently relies on input sharding metadata or
partitioning choices made inside backend kernels. Ordinary graph
operations can leave that metadata unspecified, making the resulting
sharding depend on the backend and preceding operations.

Move the manual computation boundary into the shared
`zml.kernels.moe.forwardMoe` API using `manualComputation2`. 
Callers provide a logical mesh and typed input/output partitioning specs.
History 2026-11-09 15:26:03 +01:00
..
models zml/kernels/moe: make mesh and partitioning explicit (#1028) 2026-11-09 15:26:03 +01:00
BUILD.bazel bazel: make repo-wide cquery work across host and linux target analysis (#465) 2026-04-17 11:48:09 +02:00
chat.zig zml/exe: TypedExe (#697) 2026-08-18 16:36:38 +02:00
main.zig zml: re-structure attention kernels (#926) 2026-10-12 20:31:17 +02:00
models.zig zml/Tensor: explicit sharding in Tensor code (#868) 2026-10-06 15:37:22 +02:00
README.md zml: re-structure attention kernels (#926) 2026-10-12 20:31:17 +02:00

LLM

//examples/llm runs interactive or one-shot text generation from a model repository from HuggingFace.

We support the following models, automatically detected from the model_type in the config.json:

  • Llama 3.1
  • Qwen 3.5
  • LFM 2.5

Run

To load a model from HuggingFace directly:

# CPU
bazel run //examples/llm -- --model=hf://meta-llama/Llama-3.1-8B-Instruct
# CUDA
bazel run //examples/llm --@zml//platforms:cuda=true -- --model=hf://meta-llama/Llama-3.1-8B-Instruct
# ROCm
bazel run //examples/llm --@zml//platforms:rocm=true -- --model=hf://meta-llama/Llama-3.1-8B-Instruct

From a local directory:

bazel run //examples/llm --@zml//platforms:cuda=true -- --model=/var/models/meta-llama/Llama-3.1-8B-Instruct/

For a single non-interactive prompt:

bazel run //examples/llm --@zml//platforms:cuda=true -- --model=hf://meta-llama/Llama-3.1-8B-Instruct --prompt="Write a haiku about Zig"

Options

  • --model=<path>: Required. Model repository to load. This can be a local path or a huggingface/S3 URI such as hf://... or s3://....
  • --prompt=<string>: Optional. Runs a single prompt instead of opening the interactive chat loop.
  • --seqlen=<number>: Optional. Maximum sequence length. Defaults to 2048.
  • --backend=<stablehlo|cute|nki|metal>: Optional. Attention backend. cute uses CuTe flash attention on CUDA GPUs with compute capability 8.0 or newer. If omitted, the program auto-selects one for the current platform, including cute on supported CUDA GPUs.