Add a [zio](https://github.com/lalinsky/zio) dependency.
We require ZML to be aware of which `std.Io` implementation it's running
because we use `threadlocal` to store the current `zml.Compiler`
instance when compiling a graph (to make the DSL pleasant to use).
This is our only `threadlocal` usage but it is one too much. So we must
make it compatible with `zio.TaskLocal`. Not a big deal, but this
requires opt-in from the user.
And so `zio` support can be opt-in enabled by setting the build flag
(default is `std`):
```
--@zml//zml:io_impl=zio
```
Sadly there is no `std.Io` response to `threadlocal` (or task local).
Hopefully this will be fixed in the future.
Note that this creates problem if using `std.Io.Evented`.
# Bump intel oneapi and rules_distroless
The Debian package set is now also exposed separately through the
rules_distroless APT extension:
Now use `rules_distroless` to install the oneAPI packages in a
distroless container image.
So the platforms/oneapi/packages.lock.json file is now generated from
the platforms/oneapi/packages.yaml file, and the packages.yaml file is
now the source of truth for the oneAPI packages.
The MODULE.bazel contain rules_distroless patch
`third_party/rules_distroless/accept-nospace-version-constraints.patch`
because the rules_distroless APT extension does not support version
constraints with spaces, e.g. `>= 2023.2.0` is not supported, but
`>=2023.2.0` is supported.
A PR has been submitted to rules_distroless to fix this issue, but until
it is merged and released, the patch is needed in zml
https://github.com/bazel-contrib/rules_distroless/pull/229.
`_ROOT_MODULE_DIRECT_DEPS` does not need anymore to list the individual
oneAPI and Ubuntu package repositories.
`_ROOT_PACKAGES` is now in `_UBUNTU_PACKAGES` because all the package
share the same `_UBUNTU_STRIP_PREFIX`
## Tests
```
bazel run @apt_oneapi//:lock
```
```
ONEAPI_DEVICE_SELECTOR=level_zero:0 \
bazel run \
--config=release \
--@zml//platforms:cpu=false \
--@zml//platforms:oneapi=true \
//examples/llm -- \
--model=/var/models/meta-llama/Llama-3.1-8B-Instruct \
--topk=4 \
--prompt="Tell me a story about evariste galois"
```
__-> decode 26.248s · 34.9tok/s__
```
ONEAPI_DEVICE_SELECTOR=level_zero:0,1 \
bazel run \
--config=release \
--@zml//platforms:cpu=false \
--@zml//platforms:oneapi=true \
//examples/llm -- \
--model=/var/models/meta-llama/Llama-3.1-8B-Instruct \
--topk=4 \
--prompt="Tell me a story about evariste galois"
```
__-> decode 13.017s · 56.6tok/s__
A few missing loads
Remove all flipped flags or deperacted flags from .bazelrc
Important:
- Keep autoload rules_cc and rules_python because XLA still has
transitive deps that are missing loads, PR in progress.
Updates to latest XLA which added support for bazel 8.x workspace and
fixed a bunch of things that made bazel 9 impossible.
3 problematic changes with this bump:
- PJRT headers were again incompatible with translate-c so a patch is
made and PR has been done in XLA
- PJRT introduced a proper type fo PJRT_Error so I used the InnerMixin
pattern for those.
- //tsl/platform/default package force evaluates Label("@cuda_nvcc")
even for CPU builds and we don't have this repo. I commented for now
until I find a solution on XLA side.
Bumped a few deps necessary for this and additional llvm deps that are
now evaluated as part of the llvm-project graph on XLA side (llvm_zlib
and llvm_zstd).
Next will be to investigate bazel 9 for real.
Add experimental Intel GPU backend
Add oneAPI as a ZML platform target with Bazel flags, module extension
repositories, and PJRT runtime packaging for Linux Intel GPUs.
- package libpjrt_oneapi with oneAPI runtime libraries, Level Zero
loader,
libsycl, and a zmlxoneapi dlopen shim for versioned shared libraries
- register the oneAPI platform loader and detect available render
devices
before loading the packaged PJRT plugin from runfiles
- thread the oneAPI target through platform selection, sharding,
placement,
buffers, execution, memory, IO, tensor behavior, and XLA compile flags
- document oneAPI prerequisites and example LLM usage
- update Bazel module lockfile entries for the new runtime dependencies
---------
Co-authored-by: Steeve Morin <steeve@zml.ai>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Steeve Morin <steeve@9960x-b70x2.tailb57cfe.ts.net>
This PR bumps rules_zig back to upstream/main where we now have all the
necessary
features to enable zig local actions while the reste on remote.
It also adds the necessary configurable to actually enable this.
Additionally, since rules_zig now ships ZLS support, we remove zls
entirely from zml.
### How the Bazel resolution works
Bazel resolves execution platforms and toolchains during analysis for
each
target that requires toolchains. If a rule defines exe, resolution
happens separately for the default target execution group and for each
`exec_group`;
actions assigned to an exec group use that group’s selected execution
platform
and toolchains.
Execution platforms come from an ordered set. In this repository’s
remote
configuration, the relevant order is:
1. `//:rbe_platform`, supplied by
`--extra_execution_platforms=@zml//:rbe_platform`
2. `//bazel/zig_toolchain_config:local_execution_platform`, registered
by this PR
3. the host platform, included automatically by Bazel
Earlier platforms are selected when they can satisfy the requested
toolchains.
For each target or exec group, Bazel takes the required toolchain types,
the
target platform, the ordered execution platforms, and the ordered
registered
toolchains. It filters execution platforms by execution constraints,
then tries
to find a matching toolchain for every required toolchain type. A
toolchain
matches when:
- its `exec_compatible_with` constraints are present on the candidate
execution
platform
- its `target_compatible_with` constraints are present on the target
platform
- its `target_settings` match the current configuration
Any execution platform missing a required toolchain is rejected. The
first
remaining execution platform wins, and the first matching toolchain for
each type
is selected.
### How this makes Zig local-only
This PR defines a custom execution constraint:
- `//bazel/zig_toolchain_config:local_zig_execution`
It then defines a host-derived execution platform:
- `//bazel/zig_toolchain_config:local_execution_platform`
That platform inherits the host OS/CPU constraints from
`@platforms//host`, adds
`local_zig_execution`, and sets:
```starlark
exec_properties = {
"no-remote-exec": "1",
}
```
The zig.toolchain(...) module extension call now passes:
extra_exec_compatible_with =
["//bazel/zig_toolchain_config:local_zig_execution"]
rules_zig appends that extra execution constraint to every generated Zig
SDK
toolchain. As a result, when rules_zig resolves
@rules_zig//zig:toolchain_type
for its zig execution group:
- the remote execution platform is considered first, but it does not
have
local_zig_execution, so it cannot match the Zig SDK toolchain
- the custom local execution platform has both the host OS/CPU
constraints and
local_zig_execution, so it matches
- the selected platform carries no-remote-exec, so Zig actions execute
locally
even in remote builds
Other toolchains are not constrained by local_zig_execution, so they can
continue
to resolve to the remote execution platform first. This keeps the build
remote by
default while carving out Zig actions as local-only.
---------
Co-authored-by: Steeve Morin <steeve@zml.ai>
List of changes:
* Removed `Context`
* Simplified metadata to not contain scratch buffers. They are created
uninitialized on the fly instead (thanks to #535)
* Added helper for memory movement (host_pinned -> device) (thanks to
#534)
* FA2/FA3 don't read the `max_seqlen_q` from host anymore. This was
error prone. Instead rely on a fixed value for now. (It's error prone
because it's baked in the command buffer and can result in wrong value
if the value changes but the command buffer is not recorded again).
### Overview
Context: Modern models utilize a MLP called Mixture of Experts (MoE). In
a MoE, the MLP is organized into "experts," where specific sub-tensors
are specialized to process particular linguistic patterns or elements.
During inference, the hidden states of each token are routed to the most
relevant experts, ensuring that matrix multiplications (matmul) are
performed only on the active parameters.
Problem: Due to the sparse and dynamic nature of MoE where token
assignment is determined at runtime via routing, treating an MoE layer
as a standard matmul results in significant performance degradation.
Standard kernels are not optimized for the irregular memory access
patterns in sparse routing.
Approach: MoE is implemented as a primitive in the ZML operation set.
The generic MoE Backend is designed to support various specialized
kernels tailored to specific hardware and weight precisions. The primary
implementation currently features a Triton MoE backend optimized for
NVIDIA Hopper architecture using BF16 and FP8 weights. This
implementation is utilized in the Qwen 3.5 MoE model.
### Features
**Generic MoE Backend (zml/moe/moe.zig)**
- Integrate specialized MoE backends (e.g., Triton).
- Select the optimal backend based on hardware and precision.
- Implements essential utilities and structures for registration,
parameter loading, and metadata management.
- The forwardMoe() function acts as the entry point to invoke the
appropriate backend.
**Triton MoE Backend (zml/moe/triton.zig)**
- Generates TTIR (Triton Intermediate Representation) at compile time.
- Leverages Triton's custom calls exposed via XLA.
Execution Flow:
1. Select the optimal configuration for the fused MoE kernel.
2. Determine token distribution: use naive assignment if the
distribution is sparse; otherwise, sort and align tokens for efficiency.
3. Quantization: quantize activations if using FP8 weights.
4. Fused Kernel: execute the fused MoE kernel for the gate and
up-projection layers.
5. Post-Processing: split the output and apply the activation function.
6. Quantization : re-quantize activations for the next stage if required
by the FP8 path.
7. Fused Kernel: execute the fused MoE kernel for the down-projection
layer.
8. Aggregation: sum the Top-K results for each token.
Qwen 3.5 MoE
- Fully utilizes the Triton MoE backend.
- BF16 weights implementation (also tested with FP8 but not implemented
here)
---------
Co-authored-by: loupicaaa <louis@zml.ai>
Co-authored-by: Raphael NAHOUM <raphael@zml.ai>
Drops the Python build-time dependency for Triton and Mosaic TPU
kernels. ZML now emits TTIR and Mosaic TPU IR directly from Zig, against
the same MLIR context as the rest of the compiler. So no more JAX,
Triton at build time and a lighter sandbox.
Split into two commits:
- **`c72f44884` — Add Triton and Mosaic Zig DSL.** Additive: IR builders
(`kernels/triton/`, `kernels/mosaic_tpu/`), upstream dialect bindings
(`ttir`, `mosaic_tpu`, plus `arith`/`scf`/`math`/`memref`/`vector`/
`affine`/`cf`), and the high-level
`zml.kernel.{triton,mosaic_tpu}.Kernel(Config, spec)` declarative form.
Comes with author guides in each `README.md`.
- **`cb61e529f` — Remove Python and replace with Zig kernels.** Migrates
the two production kernels (`unified_attention` for Triton,
`ragged_paged` for Mosaic TPU) and deletes everything Python related in
the sandbox and what was generating the IR.
---------
Co-authored-by: Steeve Morin <steeve@zml.ai>
I'm doing a pass on the example code we have.
* llama/model.zig, qwen/model.zig: change layer_index from being a
tensor to being an integer.
It's clearer this way what is happening. Note that lfm doesn't have the
issue because the layer index is passed everywhere.
* tokenizePrompt: the code was doing unnecessary copies of data, I added
some helper to tokenizer.encoder to streamline the code by removing the
extra copies and allocations.
* `.stdout().writer() -> .stdout().writerStreaming()` this one is weird
but apparently using stdout by position isn't always allowed, in
particular when running the exe from within SublimeText.
* runDecode -> show case bake
Update Zig compiler to 0.16.0-dev.3133+5ec8e45f3 and adjust code
for breaking standard library changes:
- Replace ArrayListUnmanaged/ArrayList init with .empty
- Switch to unmanaged AutoArrayHashMap/StringArrayHashMap
- Use debug.GeneralPurposeAllocator (renamed upstream)
- Replace std.Io.File.ReadStreamingError with error.EndOfStream
- Refactor meta.visit to use structured visitor pattern
- Fix alignment allocation in mem.bufferizeInner
Also update third-party dependencies: flashattn 0.0.6-rc8, arocc,
uucode, libvaxis, and zigimg patches.
Co-authored-by: Kevin Barre <kevin@zml.ai>
This PR introduces a Triton paged attention backend, unlocking ROCm
support.
It relies on a Python sandbox to generate the TTIR.
The major change is that the KVCache now needs to be per-layer for paged
attention. The `layer_index` is now superfluous and will probably be
removed in the future.
Adding the [IREE
tokenizer](https://github.com/iree-org/iree/blob/main/runtime/src/iree/tokenizer/README.md).Being
made from the ground up with streaming (de-)tokenization in mind, I've
extended the APIs to expose a `std.Io.Writer` for the tokenization.
I've quickly benchmarked IREE's tokenizer vs HuggingFace's one with a
request I retrieved from a coding agent (opencode) in the middle of a
session. The request body was 360KB encoded in 95K tokens. I also did
another test with a similarly sized request with the same results as
shown hereafter.
I've tested two cases:
- oneshot: the template is rendered all at once and only then we encode
the whole string.
- stream: the template rendering and encoding are pipelined to avoid the
intermediate allocation.
There is a slight difference in the generated tokens between HuggingFace
and IREE. Some tokens like `introdu` were encoded as `i`, `n`, `t`, etc.
Decoding them returns the same text. It's very close in terms of total
token count:
- hf: 95779
- iree/stream: 95705
- iree/oneshot: 95587
| Model | IREE avg (oneshot / stream) ms | HuggingFace avg (ms) | Uplift
vs HF (oneshot / stream) |
|---|---:|---:|---|
| liquidai/LFM2.5-1.2B-Instruct | 8.5 / 7.3 | 40.5 | 4.8x / 5.5x |
| Qwen/Qwen3.5-9B | 6.0 / 8.2 | 59.4 | 9.9x / 7.2x |
| mistralai/Ministral-3-3B-Reasoning-2512 | 7.4 / 6.9 | 59.2 | 8x / 8.6x
|
It's unclear why the Qwen's chat template is somehow slower in the
stream case, maybe because there are more writes. Adding non-empty
buffer in the interface helps a bit, but tends to slow down mistral
case. So well
---------
Co-authored-by: Corentin Kerisit <corentin.kerisit@gmail.com>
Fix ROCm runtime hermeticity in the packaged LLM sandbox by patching transitive ROCm libraries to use zmlxrocm_dlopen and by adding missing libdw/liblzma/libbz2 runtime deps for distroless execution. Also update the ROCm packaging to 7.2.1 and refresh the pinned libpjrt_rocm
Move profiling behind a zml-owned primitive instead of exposing PJRT
details in user code. Add the XLA/TSL Bazel deps needed to convert
collected XSpace protobufs into Perfetto traces with XLA/TLS APIs
instead of having a custom fallible converter.