mirror of
https://github.com/zml/zml.git
synced 2026-11-16 19:11:09 +00:00
This PR introduces a restructuration of the kernels in ZML. This is done
to simplify and speedup bringup on new platforms.
From now on each "kernel" will have its own directory under
`zml/kernels`. "Kernel" is in quote because we need to think about it
being more of an "operation in a graph" than an actual kernel. For
example, `moe` will eventually be part of this `kernels` directory
despite being an operation potentially launching multiple actual kernels
on some hardware.
Also it's probable that in the future, this `kernels` directory will be
put outside of zml or, at least, properly separated so that we don't
have circular imports in the codebase. Currently, we have
`@import("../../../zml.zig")` in the kernel code and this is a recipe
for disaster.
On top of the reorganization, it also adds a level of indirection for
attention. Where previously `attention` was a generic term, we now have
`attention` and `flash_attention` because `flash_attention` is an
implementation of `attention`.
Here is a detailed summary of the changes introduced by this PR:
* Kernels reorganization. `flash_attention` and `paged_attention` are
now separated. Each folder has a subfolder for each backend we support.
`flash_attention` is also now a `BackendType` of `zml.attention` with
its own implementations.
* Removed `cuda_fa2` and `cuda_fa3` backend because they were obsolete.
They have been replaced by a `cute` backend that has been generated
automatically following the operation documentation, showing that this
approach of scoping kernels per DSL actually scales.
* Removed `attnd` support for now
1.5 KiB
1.5 KiB
LLM
//examples/llm runs interactive or one-shot text generation from a model repository from HuggingFace.
We support the following models, automatically detected from the model_type in the config.json:
- Llama 3.1
- Qwen 3.5
- LFM 2.5
Run
To load a model from HuggingFace directly:
# CPU
bazel run //examples/llm -- --model=hf://meta-llama/Llama-3.1-8B-Instruct
# CUDA
bazel run //examples/llm --@zml//platforms:cuda=true -- --model=hf://meta-llama/Llama-3.1-8B-Instruct
# ROCm
bazel run //examples/llm --@zml//platforms:rocm=true -- --model=hf://meta-llama/Llama-3.1-8B-Instruct
From a local directory:
bazel run //examples/llm --@zml//platforms:cuda=true -- --model=/var/models/meta-llama/Llama-3.1-8B-Instruct/
For a single non-interactive prompt:
bazel run //examples/llm --@zml//platforms:cuda=true -- --model=hf://meta-llama/Llama-3.1-8B-Instruct --prompt="Write a haiku about Zig"
Options
--model=<path>: Required. Model repository to load. This can be a local path or a huggingface/S3 URI such ashf://...ors3://....--prompt=<string>: Optional. Runs a single prompt instead of opening the interactive chat loop.--seqlen=<number>: Optional. Maximum sequence length. Defaults to2048.--backend=<stablehlo|cute|nki|metal>: Optional. Attention backend.cuteuses CuTe flash attention on CUDA GPUs with compute capability 8.0 or newer. If omitted, the program auto-selects one for the current platform, includingcuteon supported CUDA GPUs.