Update XLA plugins to release 202610151602.227.1.e215eb02db73
for CPU, CUDA, ROCm, Metal, and Furiosa.
Updated CUDA to 13.4.3 and ROCm with TheRock 10.1.
Now we use CUDA compatibility libraries and NCCL 2.32.3 through NVIDIA's
APT repositories. Starting with CUDA 13.4's redistribution manifest
omits the generic cuda_compat package
(https://docs.nvidia.com/deploy/cuda-compatibility/why-cuda-compatibility.html).
So we resolve it through NVIDIA's APT repositories instead.
These repositories expose flat package indexes at separate x86_64
and SBSA URLs, so we extend rules_distroless to support flat sources.
For NCCL we leverage the same APT source workflow, replacing
separate cryptic wheel URLs.
Even if `exec_compatible_with = ["@platforms//os:macos"],` is removed
from `platforms/metal/BUILD.bazel`, fetching the pinned Metal toolchain
on Linux currently fails:
```
ERROR: /home/oboulant/.cache/bazel/_bazel_oboulant/4ea653214b23f6728bdb9f5b40de2f50/external/zml+/bazel/http_aea_archive.bzl:20:13: An error occurred during the fetch of repository 'zml++http_aea_archive+metal_toolchain':
Traceback (most recent call last):
File "/home/oboulant/.cache/bazel/_bazel_oboulant/4ea653214b23f6728bdb9f5b40de2f50/external/zml+/bazel/http_aea_archive.bzl", line 43, column 19, in _http_aea_archive_impl
_require_macos(rctx)
File "/home/oboulant/.cache/bazel/_bazel_oboulant/4ea653214b23f6728bdb9f5b40de2f50/external/zml+/bazel/http_aea_archive.bzl", line 20, column 13, in _require_macos
fail("http_aea_archive is macOS-only; current repository OS is %r" % rctx.os.name)
Error in fail: http_aea_archive is macOS-only; current repository OS is "linux"
```
Add checksum-pinned `ipsw` and `7-Zip` binaries for Linux x86_64.
`bazel/http_aea_archive.bzl` now selects the right binaries according to
running host. It preserves the existing macOS extraction path.
Linux ARM64 remains uncovered.
Validation:
```bash
bazel build \
# selects compiler binaries from the extracted external repository \
# filegroup defined in platforms/metal/metal_toolchain.BUILD.bazel \
@metal_toolchain//:metal_compile_tools \
# selects the compiler’s runtime files \
# filegroup also defined in platforms/metal/metal_toolchain.BUILD.bazel \
@metal_toolchain//:metal_compile_runtime \
# copies compiler binaries and runtime files into the layout consumed by ZML \
//platforms/metal:metal_toolchain \
# selects macOS ARM64 as the build target \
--platforms=//platforms:macos_arm64 \
--lockfile_mode=off
```
---------
Co-authored-by: Corentin Kerisit <corentin.kerisit@gmail.com>
This enables runtime selection of the std.Io runtime. This also fixes
the compilation hangs by using zio.blockInPlace for CPU bound ops.
This may require some more `zio.blockInPlace`, so don't merge until
we're sure.
Add a [zio](https://github.com/lalinsky/zio) dependency.
We require ZML to be aware of which `std.Io` implementation it's running
because we use `threadlocal` to store the current `zml.Compiler`
instance when compiling a graph (to make the DSL pleasant to use).
This is our only `threadlocal` usage but it is one too much. So we must
make it compatible with `zio.TaskLocal`. Not a big deal, but this
requires opt-in from the user.
And so `zio` support can be opt-in enabled by setting the build flag
(default is `std`):
```
--@zml//zml:io_impl=zio
```
Sadly there is no `std.Io` response to `threadlocal` (or task local).
Hopefully this will be fixed in the future.
Note that this creates problem if using `std.Io.Evented`.
Adds a Zig front end for NVIDIA's CuTe DSL, shaped after `cutlass.cute`,
plus the `__gpu$xla.gpu.cute` custom call that compiles and launches it
on CUDA.
- `mlir/dialects/cute_ir`: bindings for the `cute` and `cute_nvgpu`
dialects
- `zml.ops.cute`: emits the custom call with the textual module, `grid`
and `block`.
- `kernels/cute` + `zml.kernel.cute.Kernel`: builder, values, tensors
with coordinate `get`/`set`, thread/block indices, static smem, sync,
`scf`. See `kernels/cute/README.md`.
- bump `xprof-nightly`
---------
Co-authored-by: Steeve Morin <steeve@zml.ai>
Adds a Zig front end for the FlyDSL dialect, so AMD GPU kernels can be
written in Zig alongside the existing Mosaic, Triton and tile ones, and
dispatched the same way.
Types and layouts are built through the dialect's own C entry points
rather than by printing and reparsing assembly. That holds in both
directions: host-side layout literals become attributes directly, and
anything the dialect computes is read back through typed accessors.
Launch geometry is derived from the device rather than hard-coded, so
the same kernels run on both AMD architecture families and the tests
stay architecture agnostic. Everything builds and tests on any host, no
GPU needed.
Also carries one unrelated cleanup, dropping a needless null terminator
from the Triton and tile kernel strings.
A third kernel language beside Triton and Mosaic TPU. It emits NVIDIA's
cuda_tile dialect and reaches the GPU through XLA's
`__gpu$xla.gpu.cuda_tile` custom call.
- `third_party/cuda_tile`: cuda-tile v13.3.3
- `mlir/dialects/cuda_tile`: every type and enum through the dialect's C
API, one constructor per op (all 100, in Ops.td order), and a small C
shim for the TileView interface the C API does not expose.
- `kernels/cuda_tile`: views, tokens, for/if/loop, reduce/scan (single
and variadic), atomics, MMA, every elementwise op.
- `zml.kernel.cuda_tile.Kernel` and `ops.cudaTile`: the custom call,
with a grid and no num_warps/num_stages (those are optimization_hints in
the IR).
This PR add the support of nvfp4 x nvfp4 moe backend on sm120 / sm100
(natively supported nvfp4 hardware). It :
- retrieves the released artifact cutlass moe flashinfer :
https://github.com/zml/flashinfer/releases/tag/cutlass-moe-v0.1.0
- loads the .so
- implements the nvfp4 x nvfp4 backend (added to existing triton and
mosaic tpu).
# Bump intel oneapi and rules_distroless
The Debian package set is now also exposed separately through the
rules_distroless APT extension:
Now use `rules_distroless` to install the oneAPI packages in a
distroless container image.
So the platforms/oneapi/packages.lock.json file is now generated from
the platforms/oneapi/packages.yaml file, and the packages.yaml file is
now the source of truth for the oneAPI packages.
The MODULE.bazel contain rules_distroless patch
`third_party/rules_distroless/accept-nospace-version-constraints.patch`
because the rules_distroless APT extension does not support version
constraints with spaces, e.g. `>= 2023.2.0` is not supported, but
`>=2023.2.0` is supported.
A PR has been submitted to rules_distroless to fix this issue, but until
it is merged and released, the patch is needed in zml
https://github.com/bazel-contrib/rules_distroless/pull/229.
`_ROOT_MODULE_DIRECT_DEPS` does not need anymore to list the individual
oneAPI and Ubuntu package repositories.
`_ROOT_PACKAGES` is now in `_UBUNTU_PACKAGES` because all the package
share the same `_UBUNTU_STRIP_PREFIX`
## Tests
```
bazel run @apt_oneapi//:lock
```
```
ONEAPI_DEVICE_SELECTOR=level_zero:0 \
bazel run \
--config=release \
--@zml//platforms:cpu=false \
--@zml//platforms:oneapi=true \
//examples/llm -- \
--model=/var/models/meta-llama/Llama-3.1-8B-Instruct \
--topk=4 \
--prompt="Tell me a story about evariste galois"
```
__-> decode 26.248s · 34.9tok/s__
```
ONEAPI_DEVICE_SELECTOR=level_zero:0,1 \
bazel run \
--config=release \
--@zml//platforms:cpu=false \
--@zml//platforms:oneapi=true \
//examples/llm -- \
--model=/var/models/meta-llama/Llama-3.1-8B-Instruct \
--topk=4 \
--prompt="Tell me a story about evariste galois"
```
__-> decode 13.017s · 56.6tok/s__
Updates to latest XLA which added support for bazel 8.x workspace and
fixed a bunch of things that made bazel 9 impossible.
3 problematic changes with this bump:
- PJRT headers were again incompatible with translate-c so a patch is
made and PR has been done in XLA
- PJRT introduced a proper type fo PJRT_Error so I used the InnerMixin
pattern for those.
- //tsl/platform/default package force evaluates Label("@cuda_nvcc")
even for CPU builds and we don't have this repo. I commented for now
until I find a solution on XLA side.
Bumped a few deps necessary for this and additional llvm deps that are
now evaluated as part of the llvm-project graph on XLA side (llvm_zlib
and llvm_zstd).
Next will be to investigate bazel 9 for real.
**Summary**
This PR update to Neuron v2.30, introduces the Neuron NKI stack into ZML and wires it into the
LLM attention path.
The main goal is to make Neuron attention run through NKI kernels.
It adds the compiler/runtime plumbing needed to embed NKI kernels as
Neuron custom-native calls, then uses that path for SDPA-style
prefill/decode attention. Paged Attention will come later.
On inf2.8xlarge it runs :
- 25 tok/s for Llama 3.1 8B BF16
- 63.5 tok/s for LFM2.5-1.2B-Instruct BF16
- 19 tok/s for Qwen3.5-9B BF16
Add experimental Intel GPU backend
Add oneAPI as a ZML platform target with Bazel flags, module extension
repositories, and PJRT runtime packaging for Linux Intel GPUs.
- package libpjrt_oneapi with oneAPI runtime libraries, Level Zero
loader,
libsycl, and a zmlxoneapi dlopen shim for versioned shared libraries
- register the oneAPI platform loader and detect available render
devices
before loading the packaged PJRT plugin from runfiles
- thread the oneAPI target through platform selection, sharding,
placement,
buffers, execution, memory, IO, tensor behavior, and XLA compile flags
- document oneAPI prerequisites and example LLM usage
- update Bazel module lockfile entries for the new runtime dependencies
---------
Co-authored-by: Steeve Morin <steeve@zml.ai>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Steeve Morin <steeve@9960x-b70x2.tailb57cfe.ts.net>
This PR bumps rules_zig back to upstream/main where we now have all the
necessary
features to enable zig local actions while the reste on remote.
It also adds the necessary configurable to actually enable this.
Additionally, since rules_zig now ships ZLS support, we remove zls
entirely from zml.
### How the Bazel resolution works
Bazel resolves execution platforms and toolchains during analysis for
each
target that requires toolchains. If a rule defines exe, resolution
happens separately for the default target execution group and for each
`exec_group`;
actions assigned to an exec group use that group’s selected execution
platform
and toolchains.
Execution platforms come from an ordered set. In this repository’s
remote
configuration, the relevant order is:
1. `//:rbe_platform`, supplied by
`--extra_execution_platforms=@zml//:rbe_platform`
2. `//bazel/zig_toolchain_config:local_execution_platform`, registered
by this PR
3. the host platform, included automatically by Bazel
Earlier platforms are selected when they can satisfy the requested
toolchains.
For each target or exec group, Bazel takes the required toolchain types,
the
target platform, the ordered execution platforms, and the ordered
registered
toolchains. It filters execution platforms by execution constraints,
then tries
to find a matching toolchain for every required toolchain type. A
toolchain
matches when:
- its `exec_compatible_with` constraints are present on the candidate
execution
platform
- its `target_compatible_with` constraints are present on the target
platform
- its `target_settings` match the current configuration
Any execution platform missing a required toolchain is rejected. The
first
remaining execution platform wins, and the first matching toolchain for
each type
is selected.
### How this makes Zig local-only
This PR defines a custom execution constraint:
- `//bazel/zig_toolchain_config:local_zig_execution`
It then defines a host-derived execution platform:
- `//bazel/zig_toolchain_config:local_execution_platform`
That platform inherits the host OS/CPU constraints from
`@platforms//host`, adds
`local_zig_execution`, and sets:
```starlark
exec_properties = {
"no-remote-exec": "1",
}
```
The zig.toolchain(...) module extension call now passes:
extra_exec_compatible_with =
["//bazel/zig_toolchain_config:local_zig_execution"]
rules_zig appends that extra execution constraint to every generated Zig
SDK
toolchain. As a result, when rules_zig resolves
@rules_zig//zig:toolchain_type
for its zig execution group:
- the remote execution platform is considered first, but it does not
have
local_zig_execution, so it cannot match the Zig SDK toolchain
- the custom local execution platform has both the host OS/CPU
constraints and
local_zig_execution, so it matches
- the selected platform carries no-remote-exec, so Zig actions execute
locally
even in remote builds
Other toolchains are not constrained by local_zig_execution, so they can
continue
to resolve to the remote execution platform first. This keeps the build
remote by
default while carving out Zig actions as local-only.
---------
Co-authored-by: Steeve Morin <steeve@zml.ai>
This PR modernizes ZML profiling in four areas:
- replaces the old C++ `xspace_to_perfetto` tool with a native Zig
exporter built on upb
- integrates trace export directly into `zml/profiler.zig`
- introduces a scoped tracing API in `zml.tracer` and wires it through
examples
- adds platform-specific annotation support and Bazel-native profiler
wrappers for CUDA and ROCm
The result is a simpler profiling flow, better large-trace behavior, and
a more consistent experience across Linux CUDA, Linux ROCm, and macOS.
Drops the Python build-time dependency for Triton and Mosaic TPU
kernels. ZML now emits TTIR and Mosaic TPU IR directly from Zig, against
the same MLIR context as the rest of the compiler. So no more JAX,
Triton at build time and a lighter sandbox.
Split into two commits:
- **`c72f44884` — Add Triton and Mosaic Zig DSL.** Additive: IR builders
(`kernels/triton/`, `kernels/mosaic_tpu/`), upstream dialect bindings
(`ttir`, `mosaic_tpu`, plus `arith`/`scf`/`math`/`memref`/`vector`/
`affine`/`cf`), and the high-level
`zml.kernel.{triton,mosaic_tpu}.Kernel(Config, spec)` declarative form.
Comes with author guides in each `README.md`.
- **`cb61e529f` — Remove Python and replace with Zig kernels.** Migrates
the two production kernels (`unified_attention` for Triton,
`ragged_paged` for Mosaic TPU) and deletes everything Python related in
the sandbox and what was generating the IR.
---------
Co-authored-by: Steeve Morin <steeve@zml.ai>
Bump rules_zig to latest zig_toolchain changes (for bootstrap)
Bump ZLS to 0.16.0
Amend ZLS completion rules and runners to match rules_zig new providers
This PR fixes several Bazel analysis regressions so both of these
commands work reliably:
- `bazel cquery //...`
- `bazel cquery --platforms=@zml//platforms:linux_amd64 //...`
After recent changes and the rebase, repo-wide cquery was failing for
three separate reasons:
- ZLS toolchains were over-constrained by target platform: updated ZLS
toolchain registration to constrain only the exec platform.
This avoids cross-target analysis failures when host-side completion
tooling is analyzed under a foreign target platform.
- OCI packaging targets were analyzed on host platforms where their base
images do not apply: marked Linux OCI image targets as Linux x86_64
compatible only. Also switched the Debian base image pull from Docker
Hub to the GCR mirror to avoid CI fetch failures when resolving
packaging images manifests due to rate limiting
- Triton Python deps were fetched for the host Python platform instead
of the Linux wheel platform the lockfile was generated for: restored
Linux x86_64 wheel resolution for `triton_py_deps` in `MODULE.bazel`.
This prevents macOS host analysis from attempting to fetch a
`triton==3.6.0` wheel for an unsupported host platform.
Also added a step in ci to prevent regression
Update Zig compiler to 0.16.0-dev.3133+5ec8e45f3 and adjust code
for breaking standard library changes:
- Replace ArrayListUnmanaged/ArrayList init with .empty
- Switch to unmanaged AutoArrayHashMap/StringArrayHashMap
- Use debug.GeneralPurposeAllocator (renamed upstream)
- Replace std.Io.File.ReadStreamingError with error.EndOfStream
- Refactor meta.visit to use structured visitor pattern
- Fix alignment allocation in mem.bufferizeInner
Also update third-party dependencies: flashattn 0.0.6-rc8, arocc,
uucode, libvaxis, and zigimg patches.
Co-authored-by: Kevin Barre <kevin@zml.ai>
This PR introduces a Triton paged attention backend, unlocking ROCm
support.
It relies on a Python sandbox to generate the TTIR.
The major change is that the KVCache now needs to be per-layer for paged
attention. The `layer_index` is now superfluous and will probably be
removed in the future.
Adding the [IREE
tokenizer](https://github.com/iree-org/iree/blob/main/runtime/src/iree/tokenizer/README.md).Being
made from the ground up with streaming (de-)tokenization in mind, I've
extended the APIs to expose a `std.Io.Writer` for the tokenization.
I've quickly benchmarked IREE's tokenizer vs HuggingFace's one with a
request I retrieved from a coding agent (opencode) in the middle of a
session. The request body was 360KB encoded in 95K tokens. I also did
another test with a similarly sized request with the same results as
shown hereafter.
I've tested two cases:
- oneshot: the template is rendered all at once and only then we encode
the whole string.
- stream: the template rendering and encoding are pipelined to avoid the
intermediate allocation.
There is a slight difference in the generated tokens between HuggingFace
and IREE. Some tokens like `introdu` were encoded as `i`, `n`, `t`, etc.
Decoding them returns the same text. It's very close in terms of total
token count:
- hf: 95779
- iree/stream: 95705
- iree/oneshot: 95587
| Model | IREE avg (oneshot / stream) ms | HuggingFace avg (ms) | Uplift
vs HF (oneshot / stream) |
|---|---:|---:|---|
| liquidai/LFM2.5-1.2B-Instruct | 8.5 / 7.3 | 40.5 | 4.8x / 5.5x |
| Qwen/Qwen3.5-9B | 6.0 / 8.2 | 59.4 | 9.9x / 7.2x |
| mistralai/Ministral-3-3B-Reasoning-2512 | 7.4 / 6.9 | 59.2 | 8x / 8.6x
|
It's unclear why the Qwen's chat template is somehow slower in the
stream case, maybe because there are more writes. Adding non-empty
buffer in the interface helps a bit, but tends to slow down mistral
case. So well
---------
Co-authored-by: Corentin Kerisit <corentin.kerisit@gmail.com>
This PR introduces **zml-smi**, a tool to monitor metrics of any
platform supported by ZML, featuring but not limited to:
- real time `--top` mode
- per process tracking
- responsive layout
---------
Co-authored-by: Steeve Morin <steeve@zml.ai>