0
Fork 0
mirror of https://github.com/zml/zml.git synced 2026-11-16 19:11:09 +00:00
Commit graph

148 commits

Author SHA1 Message Date
Raphael Nahoum
5b079127bb
workspace: bump ZML plugins (#1034)
Bump plugins to
[202611060925.355.1.2ca211fa5fbd](2026110609.355.1.2ca211fa5fbd)
2026-11-06 10:56:05 +01:00
Raphael Nahoum
3285b7e0f4
zml: FP8 block activations and paged attention query padding on Furiosa and bump plugins (#1033)
- Quantize block FP8 activations to E4M3FN on Furiosa, so scaled dots
run natively. Other block sizes keep BF16 activations and XLA's
fallback.

- Pad paged attention query groups to a power of two with zero queries,
as TCC fails other group sizes. Marked as a workaround.

- Bump to
[202611051630.351.1.48d539621544](2026110516.351.1.48d539621544)
2026-11-05 17:38:31 +01:00
Hugo Mano
47f50667d8
zml/attention: declare manual partitioning in the shared API instead of kernels (#1016)
Move manual regions out of flash attention kernels and use
`manualComputation2` for every shared attention backend.

Also fully replicate the embedding executable outputs in Qwen and Llama.
Partial graph constraints leave their batch and sequence dimensions
open, which is invalid at executable boundaries.

Fix two further blockers exposed by the Qwen run that we forgot to move
recently: softplus comparisons must use the input dtype and ROCm DNN
runfiles must include the MIOpen kernel archives.
2026-11-05 13:00:00 +01:00
Louis Lechevalier
69a6b2bb22
workspace: bump ZML plugins (#1031) 2026-11-05 11:44:39 +01:00
Raphael Nahoum
ca39742f14
zml/platform: bump plugins and preallocate Metal memory again (#1001)
- Turns preallocation back on by default on Metal, as before #979: not
preallocating seems to cause memory issues we haven't looked into yet.
- Bumps the PJRT plugins to `202610290951.309.1.785550677e75`.
2026-10-29 12:52:21 +01:00
Raphael Nahoum
70f1bd4af4
dsl/metal: add the Metal DSL (#989)
Adds a Zig DSL for Metal kernels over the AIR dialect, and moves the
Metal paged attention kernels onto it.

- `mlir/dialects/air`: a `thread` address space. Each op has an
`llvmBuilder` that calls its `air.*` builtin with Apple's attributes,
and the dialect's translation interface builds the `!air.kernel`
metadata, so lowering to LLVM IR is MLIR's own translation. The verifier
rejects what Apple's compiler can't take: bool, scalable and n-D
vectors, buffers of non-Metal scalars, scalar f32 converts, and
`air.kernel` on a declaration.
- `dsl/metal`: an MSL-shaped builder. It emits `air` plus the
`arith`/`cf`/`vector`/`llvm` IR that clang produces for MSL, and covers
MSL's math, SIMD-group, barrier and `simdgroup_matrix` functions.
- `zml.kernel.metal` / `ops.metal`: wrap a kernel into a
`__gpu$xla.gpu.air` custom call. XLA lowers it, compiles it to a
metallib and launches it.
2026-10-29 12:52:20 +01:00
Damien Mehala
47e692c300
platforms/oneapi: bump to IGC 2.41.15 (#996)
Bump to resolve a compiler bug encountered on LFM3.

LFM3 broadcasts one sigmoid gate per attention head. This generated code
of sigmoid introduced a BF16 conversion and conditional select. The
compiler then lower it to a scalar computation which had a
register-aligment bug. The fix introduced in 2.41.11
<5d92530a07>
enforces alignment.
2026-10-29 11:15:39 +01:00
Raphael Nahoum
3736f3dc29
dsl: unify the kernel DSLs (#991)
Makes kernels written in the DSLs read alike, without changing what they
emit.

- Every DSL target is `//dsl/<name>:builder`, and `dsl/common` is
imported as common.
- Scopes without values end with `close()`.
- The scf DSLs (triton, mosaic, cute, fly) share
`openFor`/`openIf`/`openIfElse`/`openWhile` in `dsl/common`. cuda_tile
keeps its own scopes with the same shapes.
- Scopes take tuples or arrays, and result types can be dtypes (.f32),
Values, or MLIR types.
- Typed constants are `constant(dtype, value)`, alongside constIndex and
constantLike.
- Comparisons and bitcasts have the same names in every DSL: Fly gains
`lt`/`eq`/…, and every DSL spells it `value.bitCast(dtype)`.
- One block stack and one `common.verify`, which logs the module that
fails to verify.
- READMEs updated.
2026-10-27 17:53:31 +01:00
Damien Mehala
aa30110932
platforms/rocm: default rocprofiler log level to error (#992)
By default rocprofiler log level is `warning`, which can produce
unnecessary log output. Now, we override this log level to error, unless
`ROCPROFILER_LOG_LEVEL` environment variable is already set.
2026-10-27 13:01:40 +01:00
Kevin Barre
8f40b8b09f
zml/kernels: add TCL paged attention and MoE backends for Furiosa (#973)
zml: add TCL kernels and Furiosa attention and MoE backends

Introduce a typed TCL kernel DSL and MLIR bindings, and select TCL
automatically for Furiosa paged attention and BF16 SwiGLU MoE.

- Support native kernel calls, fusion, in-place outputs, device loops,
  compiler configuration, and chip-aware layouts.
- Implement standard paged attention for decode, prefill, and mixed
  batches, including causal/non-causal masks, sliding windows, custom
  scales, and per-head sinks.
- Use BF16 contractions with FP32 normalization and accumulation.
  Exclude unused KV entries before contractions and return zero for
  inactive or fully masked queries.
- Reuse prepared metadata for the optimized decode path and process
  longer contexts in page-aligned chunks.
- Add activation-stationary and blockwise fused-expert kernels.
- Fix StableHLO attention metadata for variable batch sizes and add
  host-reference, emission, and hardware regression coverage.
- Bump the Furiosa sandbox to 202610221403.283.1.d9f4f68759e1.

---------

Co-authored-by: Raphael Nahoum <raphael@zml.ai>
2026-10-26 15:47:48 +01:00
Raphael Nahoum
3f60e626e4
mlir/dialects: split into one package per dialect and unify them (#982)
Splits the monolithic mlir/dialects library into one Bazel package per
dialect, then makes every package follow the same layout, C API and Zig
API. Each package owns its Zig bindings, its C API when upstream doesn't
provide one, and its tests. Dialect registration moves out of the core
mlir package, so a target only pulls in the dialects it actually uses.
The conventions are documented in `mlir/dialects/README.md`, with a
porting checklist and conformance checks.

Along the way:

- cute_ir is flattened, and its Cuda, CuteNVGPU and NVVM sub-dialects
become their own packages
- fly's ROCDL extension moves to fly_rocdl
- new packages for ROCDL and UB
- kernels/ is renamed to dsl/, since it holds the DSLs
- every package registers through its own registerDialect, and each DSL
through registerDialects, instead of dialect name strings
- the C and C++ follow the LLVM and MLIR C API conventions, formatted
with a new mlir/.clang-format
- BUILD files share one template
- accessors are named as nouns like in the rest of ZML
- each package tests itself in its own test.zig, including that every
bound operation is registered and that every wrapper method compiles

Fixes:

- memref atomic_rmw emitted the wrong kind, because the AtomicRMWKind
values didn't match the pinned LLVM
- the pinned Triton and Mosaic no longer have tt.cat and tpu.repeat, so
the DSL's cat and repeat now lower the way tl.cat and pltpu.repeat do
- the ROCDL bindings are completed: buffer address spaces, vector types,
waves_per_eu range

Breaking changes:

- register dialects through their package instead of
DialectHandle.fromString, and mlir.registerFuncExtensions becomes
func.registerExtensions
- accessors drop the get prefix, so getValue() becomes value()
- ttir.cat and mosaic_tpu.repeat are removed, and stablehlo pad and
mosaic_tpu sort take their options before the location
- zml.kernel.fly.rocdl becomes fly_rocdl
- Mesh.sdyMeshAttr takes a context and returns an attribute instead of
text
- func.func no longer prints public, its default visibility

---------

Co-authored-by: Steeve Morin <steeve@zml.ai>
2026-10-26 11:41:05 +01:00
Raphael Nahoum
79bcc6af03
workspace: bump plugins (#985)
Update the metal plugin to improve performance under load
2026-10-23 17:08:20 +02:00
Louis Lechevalier
2e7871a802
zml/platforms: bump ZML plugins (#984) 2026-10-23 15:30:55 +02:00
Tristan Gautié
e81b0ecdc9
llmd/tpu/ragged_paged_attn: fix for sequences with empty kv (#938)
Drafter scheduling in speculative decoding
(https://github.com/zml/monorepo/pull/685) can create empty slots with
empty kv content in full-attn layers of the drafter.

This fixes this issue by aligning the TPU logic with other platforms (eg
Triton) by avoiding to request DMA transfers of KV buffers for invalid
slots.
2026-10-22 18:07:07 +02:00
Raphael Nahoum
5d85f0f521
platforms: bump plugins and stop preallocating by default on Metal (#979)
Metal does not need to pre-allocate memory
2026-10-22 14:25:28 +02:00
Hugo Mano
6cd00a0d00
ci: add supported platforms with bumped and audited deps (#967) 2026-10-20 14:06:49 +02:00
Louis Lechevalier
94ead7094e
zml/platforms: bump ZML plugins and enable all reduce collective in cmd buffer (#959)
This enable Lamport all reduce collective if compute capabilities matched on CUDA and ROCm
2026-10-19 15:18:23 +02:00
Damien Mehala
2d516eeb78
zml/platforms: rollback to CUDA 13.3.0 (#955)
CUDA 13.4.2 ships PTAX 13.4.92, which miscompiles a valid atomic CAS in
some LLM models, causing crashses. The generated code uses a different
byte offset for the load and atomic update, so the two operations access
different memory locations.
2026-10-16 18:46:10 +02:00
Damien Mehala
8329631da1
platforms/cuda: fix redist json checksum (#949) 2026-10-16 15:22:13 +02:00
Benjamin Rabier
c36b3f8f1e
platforms: Bump plugins for reduced Thunk Prepare/Initialize overhead (#946)
For DeepSeek 4.1-Flash on 4 GB300
```
|                                    |       current |     new |
|------------------------------------|--------------:|--------:|
| Start of XlaModule → cuGraphLaunch | 1,008 µs      | 132 µs  |
| ...of which on CPU                 | 896 µs        | 70 µs   |
| Prepare + Initialize               | ~790 µs       | ~2 µs   |
```
2026-10-16 13:43:03 +02:00
Hugo Mano
bc9d6c2c8d
workspace: bump ZML plugins and CUDA/ROCm runtimes (#942)
Update XLA plugins to release 202610151602.227.1.e215eb02db73
for CPU, CUDA, ROCm, Metal, and Furiosa.

Updated CUDA to 13.4.3 and ROCm with TheRock 10.1.

Now we use CUDA compatibility libraries and NCCL 2.32.3 through NVIDIA's
APT repositories. Starting with CUDA 13.4's redistribution manifest
omits the generic cuda_compat package
(https://docs.nvidia.com/deploy/cuda-compatibility/why-cuda-compatibility.html).
So we resolve it through NVIDIA's APT repositories instead.
These repositories expose flat package indexes at separate x86_64
and SBSA URLs, so we extend rules_distroless to support flat sources.

For NCCL we leverage the same APT source workflow, replacing
separate cryptic wheel URLs.
2026-10-16 10:31:34 +02:00
Raphael Nahoum
e385d9c272
workspace: bump plugins to fix Metal GDN overflow (#940)
Metal GDN used to overflow on state indexes since the rework.
2026-10-15 14:45:20 +02:00
Raphael Nahoum
cddc9cf607
zml/io: load Furiosa weights through pinned host memory (#928)
Staging blocks are uninitialized `.host_pinned` buffers, and tensors are
sent with the direct writer. The plugin backs `.host_pinned` with SDK
pinned (dma-buf) memory, which the SDK recognizes by address and
transfers without pinning pages on every request.

Pinned memory comes from `/dev/dma_heap/system`, so the loading user
needs access to it.
2026-10-14 19:32:13 +02:00
Hugo Mano
016376f2f4
workspace: remove flashattn (#939) 2026-10-14 18:29:02 +02:00
Hugo Mano
82e800dc5c
workspace: bump ZML plugins (#937)
This introduce the new tiling path support.
2026-10-14 15:18:04 +02:00
Kevin Barre
a5c40b7614
zml/platform/furiosa: rename environment variables from to FURIOSA_* (#921) 2026-10-13 19:54:27 +02:00
Guillaume Wenzek
00a0af4a55
zig 0.17 (second attempt) (#923)
Bump for zig 0.17

Deps:
- uucode bumped to
1fb73433bb
- zigimg bumped to
734fa919fb
- libvaxis bumped to
14b7684272
- zio bumped to:
417757935e
- rules_zig upgraded to [version
0.17](https://github.com/hermeticbuild/rules_zig/releases/tag/v0.17.0)
- zls is using a commit from master, but they don't have yet a tagged
release so Codex did a bit of packaging. I confirmed it correctly
autocompletes on my machine
2026-10-13 17:11:38 +02:00
Corentin Godeau
bd0886c3e5
zml: re-structure attention kernels (#926)
This PR introduces a restructuration of the kernels in ZML. This is done
to simplify and speedup bringup on new platforms.

From now on each "kernel" will have its own directory under
`zml/kernels`. "Kernel" is in quote because we need to think about it
being more of an "operation in a graph" than an actual kernel. For
example, `moe` will eventually be part of this `kernels` directory
despite being an operation potentially launching multiple actual kernels
on some hardware.

Also it's probable that in the future, this `kernels` directory will be
put outside of zml or, at least, properly separated so that we don't
have circular imports in the codebase. Currently, we have
`@import("../../../zml.zig")` in the kernel code and this is a recipe
for disaster.

On top of the reorganization, it also adds a level of indirection for
attention. Where previously `attention` was a generic term, we now have
`attention` and `flash_attention` because `flash_attention` is an
implementation of `attention`.

Here is a detailed summary of the changes introduced by this PR:
* Kernels reorganization. `flash_attention` and `paged_attention` are
now separated. Each folder has a subfolder for each backend we support.
`flash_attention` is also now a `BackendType` of `zml.attention` with
its own implementations.
* Removed `cuda_fa2` and `cuda_fa3` backend because they were obsolete.
They have been replaced by a `cute` backend that has been generated
automatically following the operation documentation, showing that this
approach of scoping kernels per DSL actually scales.
* Removed `attnd` support for now
2026-10-12 20:31:17 +02:00
Corentin Godeau
c0992073fd
workspace: bump other plugins (#922) 2026-10-09 17:36:28 +02:00
Louis Lechevalier
8173a5b7f6
platforms/cuda: Bump zml CUDA plugin (#918) 2026-10-09 17:19:04 +02:00
Kevin Barre
74562d518d
zml/platform: Add furiosa platform (#871)
Co-authored-by: Raphael Nahoum <raphael@zml.ai>
2026-10-09 12:21:16 +02:00
Tristan Gautié
734ec46230
workspace: bump ZML plugins to 202610081055.168.1.56e99704ada4 (#915) 2026-10-08 14:22:40 +02:00
Olivier Boulant
daab01572f
platforms: switch LLVM libunwind packages from 23 to 22 (#907)
`apt.llvm.org` removed some links, breaking downloads from our lockfiles
and resulting in 404.

Switch CPU/CUDA to `llvm-toolchain-bookworm-22` and ROCm to
`llvm-toolchain-jammy-22`, and regenerate the lockfiles.

All test passed on:
- macOS
- 9985wx-5090x4
- gb300-2* 
- mi300

* See reverted commit
[f7eb1cb](https://github.com/zml/zml/pull/907/commits/f7eb1cbe3b715da465bd4e9cba8a2b071499b045)
@hugomano

---

It is a follow up from https://github.com/zml/zml/pull/875
2026-10-07 19:13:33 +02:00
Hugo Mano
b148f0b919
workspace: bump XLA + LLVM + IRs (#903) 2026-10-07 16:33:44 +02:00
Guillaume Wenzek
303ff3e6a4
Revert "zig 0.17 (#693)" (#909)
This reverts commit 5a18757368.

Our internal repos has a few zig patches, I need to also upgrade them
before we ugprade ZML
2026-10-07 11:34:51 +02:00
Guillaume Wenzek
5a18757368
zig 0.17 (#693)
Bump for zig 0.17

Deps:
- uucode bumped to
1fb73433bb
- zio bumped to:
417757935e
- rules_zig upgraded to [version
0.17](https://github.com/hermeticbuild/rules_zig/releases/tag/v0.17.0)

There is one Codex generated patch:
- zigimg: minor fixes to allow zig 0.17 to compile it, until they
officially support
2026-10-07 11:08:06 +02:00
Guillaume Wenzek
b2f0e03043
zml/Tensor: explicit sharding in Tensor code (#868)
In Buffer code we already have explicit sharding, but not for tensor.

I'm adding a `zml.Sharding` to `.withPartitioning` to make it explicit
which mesh we are working with.
To simplify the migration, `.withPartitioning` in Tensor code can use
`.model` syntax to lookup a sharding by name.
This replace the previous logic of `selectSharding(shape)` which would
look up a sharding that matches the axes of the given partitioning which
could yield surprising behavior.

This also modifies how sharding are stored in zml.Shape.
Before it was storing a list of axis names + a tag (2 * 8 * 64bits), now
I'm storing a zml.Sharding (64 bits) and a compact PartitionArray (32
bits) that references the logical axis of the sharding by offset instead
of by name.

There is an impact on the `TensorRegistry` because it also needs to be
aware of the possible shardings, instead of delegating the matching
partition spec -> sharding to loading. This means error will be detected
sooner.

`store.createTensor` accepts `@enumLiteral()` to lookup the relevant
sharding.

This also simplifies `zml.Buffer` because the full sharding is now
stored in the shape, so it doesn't need to be passed independently. It
also means that it will propagate from the tensor who are graph inputs,
to the buffers that are eg inputs.

`Placement` logic also got a bit simpler.
2026-10-06 15:37:22 +02:00
Benjamin Rabier
773fb8f2f3
platforms/oneapi: update latest pjrt artifact (#883) 2026-10-02 22:30:34 +02:00
Olivier Boulant
8cbd3bd38c
platform/metal: support toolchain extraction on Linux x86_64 (#880)
Even if `exec_compatible_with = ["@platforms//os:macos"],` is removed
from `platforms/metal/BUILD.bazel`, fetching the pinned Metal toolchain
on Linux currently fails:
```
ERROR: /home/oboulant/.cache/bazel/_bazel_oboulant/4ea653214b23f6728bdb9f5b40de2f50/external/zml+/bazel/http_aea_archive.bzl:20:13: An error occurred during the fetch of repository 'zml++http_aea_archive+metal_toolchain':
   Traceback (most recent call last):
        File "/home/oboulant/.cache/bazel/_bazel_oboulant/4ea653214b23f6728bdb9f5b40de2f50/external/zml+/bazel/http_aea_archive.bzl", line 43, column 19, in _http_aea_archive_impl
                _require_macos(rctx)
        File "/home/oboulant/.cache/bazel/_bazel_oboulant/4ea653214b23f6728bdb9f5b40de2f50/external/zml+/bazel/http_aea_archive.bzl", line 20, column 13, in _require_macos
                fail("http_aea_archive is macOS-only; current repository OS is %r" % rctx.os.name)
Error in fail: http_aea_archive is macOS-only; current repository OS is "linux"
```

Add checksum-pinned `ipsw` and `7-Zip` binaries for Linux x86_64.
`bazel/http_aea_archive.bzl` now selects the right binaries according to
running host. It preserves the existing macOS extraction path.
Linux ARM64 remains uncovered.

Validation:

```bash
bazel  build \
  # selects compiler binaries from the extracted external repository \
  # filegroup defined in platforms/metal/metal_toolchain.BUILD.bazel \
  @metal_toolchain//:metal_compile_tools \
  # selects the compiler’s runtime files \
  # filegroup also defined in platforms/metal/metal_toolchain.BUILD.bazel \
  @metal_toolchain//:metal_compile_runtime \
  # copies compiler binaries and runtime files into the layout consumed by ZML \
  //platforms/metal:metal_toolchain \
  # selects macOS ARM64 as the build target \
  --platforms=//platforms:macos_arm64 \
  --lockfile_mode=off
```

---------

Co-authored-by: Corentin Kerisit <corentin.kerisit@gmail.com>
2026-10-01 18:41:45 +02:00
Olivier Boulant
676f26a4e6
platforms/cuda: fix libunwind downloads blocking Linux ARM64 build (#875)
The error it fixes:
```bash
 bazel build //llmd:upload_sentry_symbols_linux_amd64 //llmd:upload_sentry_symbols_linux_arm64 --config=remote
...
ERROR: no such package '@@zml++cuda_packages+llvm-libunwind1_linux_arm64//': java.io.IOException: Error downloading [https://apt.llvm.org/bookworm/pool/main/l/llvm-toolchain-23/llvm-libunwind1_23.1.2~++20260916013112+21a77e7bb6bf-1~exp1~20260916133123.74_arm64.deb] to /Users/oboulant/Library/Caches/bazel/_bazel_oboulant/50730f1e2ab11dae9d7f564452062ab5/external/zml++cuda_packages+llvm-libunwind1_linux_arm64/.downloaded.llvm-libunwind1_23.1.2~++20260916013112+21a77e7bb6bf-1~exp1~20260916133123.74_arm64.deb: GET returned 404 Not Found
...
```

The `platforms/cuda/packages.lock.json` was updated with `bazel run
@apt_cuda//:lock`.
2026-09-30 22:28:09 +02:00
Benjamin Rabier
4d6c61f595
platforms/oneapi: update pjrt plugin (#877) 2026-09-30 19:24:34 +02:00
Raphael Nahoum
a34428e05c
workspace: bump plugins (#876) 2026-09-30 18:24:24 +02:00
Raphael Nahoum
c6530f08d3
workspace: bump plugins (#857) 2026-09-25 13:52:26 +02:00
Steeve Morin
9b7d8fb58c bazel: rework zig_options to be config_settting compatible
zig_option can now take a `flag` dependency to expose existing flags or
`bool`, `int or `string` for config_setting flags
2026-09-24 22:21:44 +02:00
Benjamin Rabier
cabcdd8f59
platforms/oneapi: Use latest pjrt artifact (#855) 2026-09-24 20:13:04 +02:00
Damien Mehala
90f914ad07
platforms/cuda: use info level instead of warn (#827) 2026-09-24 17:57:54 +02:00
Raphael Nahoum
778a5ab3c4
zml/compiler: enable experimental all-reduce kernels on CUDA and bump plugins (#852)
Enable XLA's experimental all-reduce kernels on CUDA (custom one-shot /
two-shot kernels instead of NCCL)
2026-09-24 17:56:20 +02:00
Raphael Nahoum
1e045ec017
mlir/dialects/cute_ir: carry the DSL compiler's CuTe dialects (#849)
The CuTe dialects no longer build on a git checkout with a private
overlay.

- Every operation, type, attribute and enum of the compiler's cute and
cute_nvgpu dialects is typed, with its documentation, operand
constraints, traits and result inference. Nothing is kept as an opaque
payload, and the compiler's IR round-trips through it unchanged.
- The definitions are generated following `REGENERATING.md`, from the
release's Python bindings and by probing the compiler.
- The C API and the Zig builders follow: atom types take their
parameters instead of a text payload, constrained integers are one type,
and pointers take an address space enum.
2026-09-23 17:54:37 +02:00
Hugo Mano
ca9a342377
platforms: update plugins and tune ROCm graph execution (#848)
Update CPU, CUDA, Metal, and ROCm plugin references to release
202609221707.75.1.0cf3c5a71e94.

Default ROCm to one HIP graph stream and one hardware queue per device
to reduce synchronization overhead and throughput variability. Disable
rocprofiler queue interposition by default, while preserving explicit
environment overrides.

Restrict collective command-buffer capture to all-reduce and all-gather.
Other collective types execute outside HIP graphs.
2026-09-22 21:05:06 +02:00
Steeve Morin
fc0eb7d878
platforms: bump plugins to 202609181631.57.1.f601577e1ab1 (#837)
Notably, adds LTO
2026-09-18 20:00:56 +02:00