mirror of
https://github.com/zml/zml.git
synced 2026-11-16 19:11:09 +00:00
MOE API currently relies on input sharding metadata or partitioning choices made inside backend kernels. Ordinary graph operations can leave that metadata unspecified, making the resulting sharding depend on the backend and preceding operations. Move the manual computation boundary into the shared `zml.kernels.moe.forwardMoe` API using `manualComputation2`. Callers provide a logical mesh and typed input/output partitioning specs. |
History
|
|
|---|---|---|
| .. | ||
| models | ||
| BUILD.bazel | ||
| chat.zig | ||
| main.zig | ||
| models.zig | ||
| README.md | ||
LLM
//examples/llm runs interactive or one-shot text generation from a model repository from HuggingFace.
We support the following models, automatically detected from the model_type in the config.json:
- Llama 3.1
- Qwen 3.5
- LFM 2.5
Run
To load a model from HuggingFace directly:
# CPU
bazel run //examples/llm -- --model=hf://meta-llama/Llama-3.1-8B-Instruct
# CUDA
bazel run //examples/llm --@zml//platforms:cuda=true -- --model=hf://meta-llama/Llama-3.1-8B-Instruct
# ROCm
bazel run //examples/llm --@zml//platforms:rocm=true -- --model=hf://meta-llama/Llama-3.1-8B-Instruct
From a local directory:
bazel run //examples/llm --@zml//platforms:cuda=true -- --model=/var/models/meta-llama/Llama-3.1-8B-Instruct/
For a single non-interactive prompt:
bazel run //examples/llm --@zml//platforms:cuda=true -- --model=hf://meta-llama/Llama-3.1-8B-Instruct --prompt="Write a haiku about Zig"
Options
--model=<path>: Required. Model repository to load. This can be a local path or a huggingface/S3 URI such ashf://...ors3://....--prompt=<string>: Optional. Runs a single prompt instead of opening the interactive chat loop.--seqlen=<number>: Optional. Maximum sequence length. Defaults to2048.--backend=<stablehlo|cute|nki|metal>: Optional. Attention backend.cuteuses CuTe flash attention on CUDA GPUs with compute capability 8.0 or newer. If omitted, the program auto-selects one for the current platform, includingcuteon supported CUDA GPUs.