Follow-up to https://github.com/zml/zml/pull/888:
We introduce a new `.standard` entry to `AttentionMask` enum, that
accepts both flags `causal` and `sliding_window`. This makes the
interface more flexible, to allow cases with both non-causal _and_
sliding_window options.
In Buffer code we already have explicit sharding, but not for tensor.
I'm adding a `zml.Sharding` to `.withPartitioning` to make it explicit
which mesh we are working with.
To simplify the migration, `.withPartitioning` in Tensor code can use
`.model` syntax to lookup a sharding by name.
This replace the previous logic of `selectSharding(shape)` which would
look up a sharding that matches the axes of the given partitioning which
could yield surprising behavior.
This also modifies how sharding are stored in zml.Shape.
Before it was storing a list of axis names + a tag (2 * 8 * 64bits), now
I'm storing a zml.Sharding (64 bits) and a compact PartitionArray (32
bits) that references the logical axis of the sharding by offset instead
of by name.
There is an impact on the `TensorRegistry` because it also needs to be
aware of the possible shardings, instead of delegating the matching
partition spec -> sharding to loading. This means error will be detected
sooner.
`store.createTensor` accepts `@enumLiteral()` to lookup the relevant
sharding.
This also simplifies `zml.Buffer` because the full sharding is now
stored in the shape, so it doesn't need to be passed independently. It
also means that it will propagate from the tensor who are graph inputs,
to the buffers that are eg inputs.
`Placement` logic also got a bit simpler.
Replace `is_causal` and `sliding_window` fields in `AttentionOptions`
with a tagged union. This prepares the API for a subsequent commit that
will introduce sparse attention mask. A tagged union makes these
mutually exclusive behaviour explicit.
This PR removes a lot of slop in the MoE part of ZML, as well as some
global improvements.
Among the improvements:
* Unify activation functions and parameters across backends
* Unify activation computation across backends
* Replace error bubbling with explicit asserts (because we have no
meanignful way to handle them)
* Add a StableHLO MoE backend
Record donated input indices during compilation and destroy their
PJRT_Buffer handles once execution is enqueued for `zml.Exe.Runner`.
I'm not doing this automatically in `call()` because it's a bit
invasive. Not all the code uses the pattern of overwriting an input
buffer like `run(.{cache}, .{&cache})`, but most do. So I'm cleaning it
up on easy path and leave `call` as is for finer control.
Removing callOpts & runOpts to simplify the API and changing `io` to be
required for API clarity. I'm replacing `zml.testing.autoCall` with
`Exe.eval` which solves most needs in a simpler way IMHO
DirectShardWriter.init treated retrieveBuffer errors as unreachable and
leaked its transfer manager and buffer when a later step failed.
DirectMemoryWriter.init counted a shard writer as initialized even when
its init failed, so its errdefer deinitialized an undefined writer.
Allows placing all the threads/memory on the right numa nodes. It's
faster for a device to access memory on the same numa node and overall
the it's better to have all threads on the same numa node if all devices
are on the same one.
NVPTXAddressSpace is NVPTX specific. In LLVM, addressSpace is defined
for each target (nvptx, amdgpu)
The pointerType is general in llvm dialect by accepting an anytype.
- Implement the flashmla kernel for sm100 using the cute dsl in zig.
- Extend the cute dsl with specialized sm100 (tcgen05) ops.
- Introduces a new cute backend which calls the cute kernel.
- Uses a new cache for MLA (latent or quantized).
Even if `exec_compatible_with = ["@platforms//os:macos"],` is removed
from `platforms/metal/BUILD.bazel`, fetching the pinned Metal toolchain
on Linux currently fails:
```
ERROR: /home/oboulant/.cache/bazel/_bazel_oboulant/4ea653214b23f6728bdb9f5b40de2f50/external/zml+/bazel/http_aea_archive.bzl:20:13: An error occurred during the fetch of repository 'zml++http_aea_archive+metal_toolchain':
Traceback (most recent call last):
File "/home/oboulant/.cache/bazel/_bazel_oboulant/4ea653214b23f6728bdb9f5b40de2f50/external/zml+/bazel/http_aea_archive.bzl", line 43, column 19, in _http_aea_archive_impl
_require_macos(rctx)
File "/home/oboulant/.cache/bazel/_bazel_oboulant/4ea653214b23f6728bdb9f5b40de2f50/external/zml+/bazel/http_aea_archive.bzl", line 20, column 13, in _require_macos
fail("http_aea_archive is macOS-only; current repository OS is %r" % rctx.os.name)
Error in fail: http_aea_archive is macOS-only; current repository OS is "linux"
```
Add checksum-pinned `ipsw` and `7-Zip` binaries for Linux x86_64.
`bazel/http_aea_archive.bzl` now selects the right binaries according to
running host. It preserves the existing macOS extraction path.
Linux ARM64 remains uncovered.
Validation:
```bash
bazel build \
# selects compiler binaries from the extracted external repository \
# filegroup defined in platforms/metal/metal_toolchain.BUILD.bazel \
@metal_toolchain//:metal_compile_tools \
# selects the compiler’s runtime files \
# filegroup also defined in platforms/metal/metal_toolchain.BUILD.bazel \
@metal_toolchain//:metal_compile_runtime \
# copies compiler binaries and runtime files into the layout consumed by ZML \
//platforms/metal:metal_toolchain \
# selects macOS ARM64 as the build target \
--platforms=//platforms:macos_arm64 \
--lockfile_mode=off
```
---------
Co-authored-by: Corentin Kerisit <corentin.kerisit@gmail.com>
The error it fixes:
```bash
bazel build //llmd:upload_sentry_symbols_linux_amd64 //llmd:upload_sentry_symbols_linux_arm64 --config=remote
...
ERROR: no such package '@@zml++cuda_packages+llvm-libunwind1_linux_arm64//': java.io.IOException: Error downloading [https://apt.llvm.org/bookworm/pool/main/l/llvm-toolchain-23/llvm-libunwind1_23.1.2~++20260916013112+21a77e7bb6bf-1~exp1~20260916133123.74_arm64.deb] to /Users/oboulant/Library/Caches/bazel/_bazel_oboulant/50730f1e2ab11dae9d7f564452062ab5/external/zml++cuda_packages+llvm-libunwind1_linux_arm64/.downloaded.llvm-libunwind1_23.1.2~++20260916013112+21a77e7bb6bf-1~exp1~20260916133123.74_arm64.deb: GET returned 404 Not Found
...
```
The `platforms/cuda/packages.lock.json` was updated with `bazel run
@apt_cuda//:lock`.
This PR adds support for models downloaded via `hf download` without the
`--local-dir` flag.
This is done by keeping the given path instead of getting it from the
`std.Io.File`. The problem with `File.realpath` is that in the case of
models downloaded from HF, the files points to blobs whose name are hex
strings, thus losing the required file extension. Unfortunately, we rely
on the file extension to detect the kind of repository the path
represents.
I also added `//examples/io:playground` to the `//:completion` target so
that we have completion when working on it.
The changed code needs to round up the scale to the next power of two.
The math is:
```
(bits + 0x7fffff) & 0xff800000
```
Since 0x7fffff is \(2^{23}-1\):
- When \(m=0\), adding it fills the fraction bits without carrying into
the exponent.
- When \(m>0\), the addition carries exactly once into the exponent.
- The mask then clears all 23 fraction bits.
Too many problems for now in XLA:
- https://github.com/zml/zml/pull/863
- a bf16 dot failed
We have fixes incoming for those, but unclear whether it's enough
Refactoring of the MoE selection and add repacking contract required by
the model inits.
This solve a design problem in the model init :
- Before, backend selection required the resulting MoE after repacking
(dtype), and repacking required to know the selected backend.
- Now, backend selection is based on raw weights loaded from HF
(quantization scheme and dtype). Then it induces the packing.
Add useful structs for packing that are specific to MoE
(`ExpertsLayout`).
Introduces the CuTe mxfp4 moe backend. The kernels use the cute dsl and
are optimized (and so at the moment constraint to its shapes).
Introduces the cute launch host func and so the associated cuda dialect
in zml.
The CuTe dialects no longer build on a git checkout with a private
overlay.
- Every operation, type, attribute and enum of the compiler's cute and
cute_nvgpu dialects is typed, with its documentation, operand
constraints, traits and result inference. Nothing is kept as an opaque
payload, and the compiler's IR round-trips through it unchanged.
- The definitions are generated following `REGENERATING.md`, from the
release's Python bindings and by probing the compiler.
- The C API and the Zig builders follow: atom types take their
parameters instead of a text payload, constrained integers are one type,
and pointers take an address space enum.
Support normalization from HF JSON and typed normalizer sequences.
Changes:
- Patch IREE memory ownership and chunk-boundary bugs
- Add regression tests for IREE bugs.
Add explicit debug info in ZML.
Model author can now add `pushLocation` to mark specific sections of the
IR
---------
Co-authored-by: Guillaume Wenzek <gwenzek@users.noreply.github.com>
Update CPU, CUDA, Metal, and ROCm plugin references to release
202609221707.75.1.0cf3c5a71e94.
Default ROCm to one HIP graph stream and one hardware queue per device
to reduce synchronization overhead and throughput variability. Disable
rocprofiler queue interposition by default, while preserving explicit
environment overrides.
Restrict collective command-buffer capture to all-reduce and all-gather.
Other collective types execute outside HIP graphs.
With SKIP_TEMP, captured NCCL collectives can retain stale physical
mappings if VMM reclaims their backing under memory pressure, causing
silent corruption.
Enable user buffers to assign color 1 and receive the reclaim exemption:
https://github.com/openxla/xla/pull/46029