0
Fork 0
mirror of https://github.com/zml/zml.git synced 2026-11-16 19:11:09 +00:00
Commit graph

80 commits

Author SHA1 Message Date
Raphael Nahoum
5b079127bb
workspace: bump ZML plugins (#1034)
Bump plugins to
[202611060925.355.1.2ca211fa5fbd](2026110609.355.1.2ca211fa5fbd)
2026-11-06 10:56:05 +01:00
Raphael Nahoum
3285b7e0f4
zml: FP8 block activations and paged attention query padding on Furiosa and bump plugins (#1033)
- Quantize block FP8 activations to E4M3FN on Furiosa, so scaled dots
run natively. Other block sizes keep BF16 activations and XLA's
fallback.

- Pad paged attention query groups to a power of two with zero queries,
as TCC fails other group sizes. Marked as a workaround.

- Bump to
[202611051630.351.1.48d539621544](2026110516.351.1.48d539621544)
2026-11-05 17:38:31 +01:00
Louis Lechevalier
69a6b2bb22
workspace: bump ZML plugins (#1031) 2026-11-05 11:44:39 +01:00
Raphael Nahoum
ca39742f14
zml/platform: bump plugins and preallocate Metal memory again (#1001)
- Turns preallocation back on by default on Metal, as before #979: not
preallocating seems to cause memory issues we haven't looked into yet.
- Bumps the PJRT plugins to `202610290951.309.1.785550677e75`.
2026-10-29 12:52:21 +01:00
Raphael Nahoum
70f1bd4af4
dsl/metal: add the Metal DSL (#989)
Adds a Zig DSL for Metal kernels over the AIR dialect, and moves the
Metal paged attention kernels onto it.

- `mlir/dialects/air`: a `thread` address space. Each op has an
`llvmBuilder` that calls its `air.*` builtin with Apple's attributes,
and the dialect's translation interface builds the `!air.kernel`
metadata, so lowering to LLVM IR is MLIR's own translation. The verifier
rejects what Apple's compiler can't take: bool, scalable and n-D
vectors, buffers of non-Metal scalars, scalar f32 converts, and
`air.kernel` on a declaration.
- `dsl/metal`: an MSL-shaped builder. It emits `air` plus the
`arith`/`cf`/`vector`/`llvm` IR that clang produces for MSL, and covers
MSL's math, SIMD-group, barrier and `simdgroup_matrix` functions.
- `zml.kernel.metal` / `ops.metal`: wrap a kernel into a
`__gpu$xla.gpu.air` custom call. XLA lowers it, compiles it to a
metallib and launches it.
2026-10-29 12:52:20 +01:00
Raphael Nahoum
79bcc6af03
workspace: bump plugins (#985)
Update the metal plugin to improve performance under load
2026-10-23 17:08:20 +02:00
Louis Lechevalier
2e7871a802
zml/platforms: bump ZML plugins (#984) 2026-10-23 15:30:55 +02:00
Raphael Nahoum
5d85f0f521
platforms: bump plugins and stop preallocating by default on Metal (#979)
Metal does not need to pre-allocate memory
2026-10-22 14:25:28 +02:00
Louis Lechevalier
94ead7094e
zml/platforms: bump ZML plugins and enable all reduce collective in cmd buffer (#959)
This enable Lamport all reduce collective if compute capabilities matched on CUDA and ROCm
2026-10-19 15:18:23 +02:00
Damien Mehala
2d516eeb78
zml/platforms: rollback to CUDA 13.3.0 (#955)
CUDA 13.4.2 ships PTAX 13.4.92, which miscompiles a valid atomic CAS in
some LLM models, causing crashses. The generated code uses a different
byte offset for the load and atomic update, so the two operations access
different memory locations.
2026-10-16 18:46:10 +02:00
Damien Mehala
8329631da1
platforms/cuda: fix redist json checksum (#949) 2026-10-16 15:22:13 +02:00
Benjamin Rabier
c36b3f8f1e
platforms: Bump plugins for reduced Thunk Prepare/Initialize overhead (#946)
For DeepSeek 4.1-Flash on 4 GB300
```
|                                    |       current |     new |
|------------------------------------|--------------:|--------:|
| Start of XlaModule → cuGraphLaunch | 1,008 µs      | 132 µs  |
| ...of which on CPU                 | 896 µs        | 70 µs   |
| Prepare + Initialize               | ~790 µs       | ~2 µs   |
```
2026-10-16 13:43:03 +02:00
Hugo Mano
bc9d6c2c8d
workspace: bump ZML plugins and CUDA/ROCm runtimes (#942)
Update XLA plugins to release 202610151602.227.1.e215eb02db73
for CPU, CUDA, ROCm, Metal, and Furiosa.

Updated CUDA to 13.4.3 and ROCm with TheRock 10.1.

Now we use CUDA compatibility libraries and NCCL 2.32.3 through NVIDIA's
APT repositories. Starting with CUDA 13.4's redistribution manifest
omits the generic cuda_compat package
(https://docs.nvidia.com/deploy/cuda-compatibility/why-cuda-compatibility.html).
So we resolve it through NVIDIA's APT repositories instead.
These repositories expose flat package indexes at separate x86_64
and SBSA URLs, so we extend rules_distroless to support flat sources.

For NCCL we leverage the same APT source workflow, replacing
separate cryptic wheel URLs.
2026-10-16 10:31:34 +02:00
Raphael Nahoum
e385d9c272
workspace: bump plugins to fix Metal GDN overflow (#940)
Metal GDN used to overflow on state indexes since the rework.
2026-10-15 14:45:20 +02:00
Raphael Nahoum
cddc9cf607
zml/io: load Furiosa weights through pinned host memory (#928)
Staging blocks are uninitialized `.host_pinned` buffers, and tensors are
sent with the direct writer. The plugin backs `.host_pinned` with SDK
pinned (dma-buf) memory, which the SDK recognizes by address and
transfers without pinning pages on every request.

Pinned memory comes from `/dev/dma_heap/system`, so the loading user
needs access to it.
2026-10-14 19:32:13 +02:00
Hugo Mano
016376f2f4
workspace: remove flashattn (#939) 2026-10-14 18:29:02 +02:00
Hugo Mano
82e800dc5c
workspace: bump ZML plugins (#937)
This introduce the new tiling path support.
2026-10-14 15:18:04 +02:00
Guillaume Wenzek
00a0af4a55
zig 0.17 (second attempt) (#923)
Bump for zig 0.17

Deps:
- uucode bumped to
1fb73433bb
- zigimg bumped to
734fa919fb
- libvaxis bumped to
14b7684272
- zio bumped to:
417757935e
- rules_zig upgraded to [version
0.17](https://github.com/hermeticbuild/rules_zig/releases/tag/v0.17.0)
- zls is using a commit from master, but they don't have yet a tagged
release so Codex did a bit of packaging. I confirmed it correctly
autocompletes on my machine
2026-10-13 17:11:38 +02:00
Louis Lechevalier
8173a5b7f6
platforms/cuda: Bump zml CUDA plugin (#918) 2026-10-09 17:19:04 +02:00
Tristan Gautié
734ec46230
workspace: bump ZML plugins to 202610081055.168.1.56e99704ada4 (#915) 2026-10-08 14:22:40 +02:00
Olivier Boulant
daab01572f
platforms: switch LLVM libunwind packages from 23 to 22 (#907)
`apt.llvm.org` removed some links, breaking downloads from our lockfiles
and resulting in 404.

Switch CPU/CUDA to `llvm-toolchain-bookworm-22` and ROCm to
`llvm-toolchain-jammy-22`, and regenerate the lockfiles.

All test passed on:
- macOS
- 9985wx-5090x4
- gb300-2* 
- mi300

* See reverted commit
[f7eb1cb](https://github.com/zml/zml/pull/907/commits/f7eb1cbe3b715da465bd4e9cba8a2b071499b045)
@hugomano

---

It is a follow up from https://github.com/zml/zml/pull/875
2026-10-07 19:13:33 +02:00
Hugo Mano
b148f0b919
workspace: bump XLA + LLVM + IRs (#903) 2026-10-07 16:33:44 +02:00
Guillaume Wenzek
303ff3e6a4
Revert "zig 0.17 (#693)" (#909)
This reverts commit 5a18757368.

Our internal repos has a few zig patches, I need to also upgrade them
before we ugprade ZML
2026-10-07 11:34:51 +02:00
Guillaume Wenzek
5a18757368
zig 0.17 (#693)
Bump for zig 0.17

Deps:
- uucode bumped to
1fb73433bb
- zio bumped to:
417757935e
- rules_zig upgraded to [version
0.17](https://github.com/hermeticbuild/rules_zig/releases/tag/v0.17.0)

There is one Codex generated patch:
- zigimg: minor fixes to allow zig 0.17 to compile it, until they
officially support
2026-10-07 11:08:06 +02:00
Guillaume Wenzek
b2f0e03043
zml/Tensor: explicit sharding in Tensor code (#868)
In Buffer code we already have explicit sharding, but not for tensor.

I'm adding a `zml.Sharding` to `.withPartitioning` to make it explicit
which mesh we are working with.
To simplify the migration, `.withPartitioning` in Tensor code can use
`.model` syntax to lookup a sharding by name.
This replace the previous logic of `selectSharding(shape)` which would
look up a sharding that matches the axes of the given partitioning which
could yield surprising behavior.

This also modifies how sharding are stored in zml.Shape.
Before it was storing a list of axis names + a tag (2 * 8 * 64bits), now
I'm storing a zml.Sharding (64 bits) and a compact PartitionArray (32
bits) that references the logical axis of the sharding by offset instead
of by name.

There is an impact on the `TensorRegistry` because it also needs to be
aware of the possible shardings, instead of delegating the matching
partition spec -> sharding to loading. This means error will be detected
sooner.

`store.createTensor` accepts `@enumLiteral()` to lookup the relevant
sharding.

This also simplifies `zml.Buffer` because the full sharding is now
stored in the shape, so it doesn't need to be passed independently. It
also means that it will propagate from the tensor who are graph inputs,
to the buffers that are eg inputs.

`Placement` logic also got a bit simpler.
2026-10-06 15:37:22 +02:00
Olivier Boulant
676f26a4e6
platforms/cuda: fix libunwind downloads blocking Linux ARM64 build (#875)
The error it fixes:
```bash
 bazel build //llmd:upload_sentry_symbols_linux_amd64 //llmd:upload_sentry_symbols_linux_arm64 --config=remote
...
ERROR: no such package '@@zml++cuda_packages+llvm-libunwind1_linux_arm64//': java.io.IOException: Error downloading [https://apt.llvm.org/bookworm/pool/main/l/llvm-toolchain-23/llvm-libunwind1_23.1.2~++20260916013112+21a77e7bb6bf-1~exp1~20260916133123.74_arm64.deb] to /Users/oboulant/Library/Caches/bazel/_bazel_oboulant/50730f1e2ab11dae9d7f564452062ab5/external/zml++cuda_packages+llvm-libunwind1_linux_arm64/.downloaded.llvm-libunwind1_23.1.2~++20260916013112+21a77e7bb6bf-1~exp1~20260916133123.74_arm64.deb: GET returned 404 Not Found
...
```

The `platforms/cuda/packages.lock.json` was updated with `bazel run
@apt_cuda//:lock`.
2026-09-30 22:28:09 +02:00
Raphael Nahoum
a34428e05c
workspace: bump plugins (#876) 2026-09-30 18:24:24 +02:00
Raphael Nahoum
c6530f08d3
workspace: bump plugins (#857) 2026-09-25 13:52:26 +02:00
Steeve Morin
9b7d8fb58c bazel: rework zig_options to be config_settting compatible
zig_option can now take a `flag` dependency to expose existing flags or
`bool`, `int or `string` for config_setting flags
2026-09-24 22:21:44 +02:00
Damien Mehala
90f914ad07
platforms/cuda: use info level instead of warn (#827) 2026-09-24 17:57:54 +02:00
Raphael Nahoum
778a5ab3c4
zml/compiler: enable experimental all-reduce kernels on CUDA and bump plugins (#852)
Enable XLA's experimental all-reduce kernels on CUDA (custom one-shot /
two-shot kernels instead of NCCL)
2026-09-24 17:56:20 +02:00
Raphael Nahoum
1e045ec017
mlir/dialects/cute_ir: carry the DSL compiler's CuTe dialects (#849)
The CuTe dialects no longer build on a git checkout with a private
overlay.

- Every operation, type, attribute and enum of the compiler's cute and
cute_nvgpu dialects is typed, with its documentation, operand
constraints, traits and result inference. Nothing is kept as an opaque
payload, and the compiler's IR round-trips through it unchanged.
- The definitions are generated following `REGENERATING.md`, from the
release's Python bindings and by probing the compiler.
- The C API and the Zig builders follow: atom types take their
parameters instead of a text payload, constrained integers are one type,
and pointers take an address space enum.
2026-09-23 17:54:37 +02:00
Hugo Mano
ca9a342377
platforms: update plugins and tune ROCm graph execution (#848)
Update CPU, CUDA, Metal, and ROCm plugin references to release
202609221707.75.1.0cf3c5a71e94.

Default ROCm to one HIP graph stream and one hardware queue per device
to reduce synchronization overhead and throughput variability. Disable
rocprofiler queue interposition by default, while preserving explicit
environment overrides.

Restrict collective command-buffer capture to all-reduce and all-gather.
Other collective types execute outside HIP graphs.
2026-09-22 21:05:06 +02:00
Steeve Morin
fc0eb7d878
platforms: bump plugins to 202609181631.57.1.f601577e1ab1 (#837)
Notably, adds LTO
2026-09-18 20:00:56 +02:00
Steeve Morin
df3a43d29f
bazel: add zig_options rule (#820)
Introduce `zig_option` and `zig_options` rules to generate the bazel
equivalent of option modules plugged on bazel flags.

On `string` flags, if the flag has a `allowed_values`, the `string` is
converted into a `enum` in zig code.

Also, port platforms to this and remove the ugly `defines` hack to
comptime test for platforms enablement.

In the process, port `arocc` to the `zig_options`.
2026-09-16 21:50:22 +02:00
Benjamin Rabier
51235e7eb6
zml: block32 FP8 with E8M0 scales, and an explicit Linear output dtype (#811)
Two things, the second one falling out of the first.

**`fp8_block32`**: E4M3FN weights with one scale per 32x32 tile, the
layout
DeepSeek V4.1 ships. `quantizeBlockFp8` takes a block size and a scale
dtype;
the E8M0 path floors amax at 1e-4 and rounds the exponent up, so the
activation
side is MXFP8: one E8M0 scale per row per 32 contracted values. That
pairing is
what XLA's fp8_block_gemv arm needs to hand both scales to
`tt.dot_scaled`
(`Fp8BlockGemvScaleInMma` requires E8M0 on *both* sides); F32 activation
scales
would quietly fall to the per-block multiply. The Triton MoE path gets
the same
scheme at `block_size_k = 32`.

`Linear.forward` also stops rounding the activation to BF16 before
quantizing,
so amax is computed on the input as given.

**An explicit output dtype** on `scaledDot` and `Linear.forward`.
Inference
cannot answer what an FP8 x FP8 product should return: the accumulation
is F32
either way, so BF16 or F32 back is the caller's choice. `forwardScaled`
now
hands the caller's type to `scaledDot` rather than rounding through BF16
and
converting after. Narrow outputs are rejected.

Separately, `Linear.forward` converts the weight and bias to the
activation
dtype. `dotGeneral` asserts both operands share one, which broke the
moment a
model raised its activation to F32 against a BF16 weight; the wrappers
this API
replaced converted for exactly that reason.

**Breaking:** `Linear.forward` takes a second argument and `scaledDot` a
fifth.

---------

Co-authored-by: Raphael Nahoum <raphael@zml.ai>
2026-09-16 18:13:08 +02:00
Hugo Mano
4adb31b4d6
platforms: bump plugins for CPU, ROCm, CUDA and Metal (#812)
Bump to build `202609161226.43.1.b992d049cec2`
2026-09-16 14:53:44 +02:00
Raphael Nahoum
94257d3747
kernels/cute: Zig CuTe DSL and XLA custom call and bump xprof (#810)
Adds a Zig front end for NVIDIA's CuTe DSL, shaped after `cutlass.cute`,
plus the `__gpu$xla.gpu.cute` custom call that compiles and launches it
on CUDA.

- `mlir/dialects/cute_ir`: bindings for the `cute` and `cute_nvgpu`
dialects
- `zml.ops.cute`: emits the custom call with the textual module, `grid`
and `block`.
- `kernels/cute` + `zml.kernel.cute.Kernel`: builder, values, tensors
with coordinate `get`/`set`, thread/block indices, static smem, sync,
`scf`. See `kernels/cute/README.md`.
- bump `xprof-nightly`

---------

Co-authored-by: Steeve Morin <steeve@zml.ai>
2026-09-16 14:02:46 +02:00
Corentin Godeau
91c8b31502
platforms: bump plugins for CPU, ROCm, CUDA and Metal (#809)
Bump to `202609151311.37.1.f1b6432d55a0`
2026-09-15 16:07:39 +02:00
Hugo Mano
0ebb0c21a9 platforms: bump plugins for CPU, ROCm, CUDA and Metal (#788) 2026-09-10 14:34:40 +00:00
Raphael Nahoum
12b299186a
platforms/cuda: bump the plugin and cuDNN to 9.24.0 (#774) 2026-09-08 14:34:30 +02:00
Damien Mehala
e1e983c802
zml/moe: support Flashinfer cutlass MoE for sm 10.3 (#740)
Update flashinfer cutlass moe to retrieve sm103 compiled .so. 
Unpack forwardMoe abi by removing the Nvfp4Scales type.

---------

Co-authored-by: Louis Lechevalier <louis@zml.ai>
Co-authored-by: Corentin Godeau <corentin@zml.ai>
2026-09-01 16:11:57 +02:00
Louis Lechevalier
533afb8915
zml/moe: flashinfer nvfp4 x nvfp4 moe backend (#707)
This PR add the support of nvfp4 x nvfp4 moe backend on sm120 / sm100
(natively supported nvfp4 hardware). It :
- retrieves the released artifact cutlass moe flashinfer :
https://github.com/zml/flashinfer/releases/tag/cutlass-moe-v0.1.0
- loads the .so
- implements the nvfp4 x nvfp4 backend (added to existing triton and
mosaic tpu).
2026-08-20 11:51:32 +02:00
Kevin Barre
039fc9516a
zml/cuda : update PJRT CUDA asset SHA256 checksums for amd64 and arm64 (#700) 2026-07-31 20:08:08 +02:00
Raphael Nahoum
14a0a8f507
zml: start of quantization support (#690)
Add MLIR composites, `scaledDot`, and a Linear path for NVFP4 weights
(unpack, optional activation quant on Blackwell, global scales). Skip
fused multi-source tensors in the generic loader so pack/`loadExecute`
can fill them (may be used later for fused QKV etc).
2026-07-31 11:02:07 +02:00
Louis Lechevalier
f4282a47ae
platforms/cuda: Update libpjrt cuda (#684)
Update of libpjrt cuda after rebase and patches update
2026-07-28 15:57:04 +02:00
Louis Lechevalier
a35fac997e
platforms/cuda: Bump CUDA to 13.3.0 (#672)
Bump CUDA to 13.3.0 and CUDNN to 9.22.
2026-07-20 20:57:19 +02:00
Hugo Mano
833f42029d
platforms: bump cpu + cuda plugins and align xla (#626)
Related PR: https://github.com/zml/monorepo/pull/304
2026-07-03 14:22:23 +02:00
Louis Lechevalier
0e12e4c998
platforms/cuda: update pjrt plugin (#617)
related: https://github.com/zml/pjrt-artifacts/pull/91/changes
From 130 to 200 tok/s for Gemma 4 26B A4B on GH200
2026-06-30 17:31:14 +02:00
Hugo Mano
13cee57bf2
zml/profiling: unify profiling flow and tracing (#443)
This PR modernizes ZML profiling in four areas:

- replaces the old C++ `xspace_to_perfetto` tool with a native Zig
exporter built on upb
- integrates trace export directly into `zml/profiler.zig`
- introduces a scoped tracing API in `zml.tracer` and wires it through
examples
- adds platform-specific annotation support and Bazel-native profiler
wrappers for CUDA and ROCm

The result is a simpler profiling flow, better large-trace behavior, and
a more consistent experience across Linux CUDA, Linux ROCm, and macOS.
2026-05-11 09:52:11 +02:00