0
Fork 0
mirror of https://github.com/zml/zml.git synced 2026-11-16 19:11:09 +00:00
Commit graph

1,078 commits

Author SHA1 Message Date
Steeve Morin
63d8204be2
zml: add Platform.io_impl to enable threaded or zio at runtime (#841)
This enables runtime selection of the std.Io runtime. This also fixes
the compilation hangs by using zio.blockInPlace for CPU bound ops.

This may require some more `zio.blockInPlace`, so don't merge until
we're sure.
2026-09-21 18:59:29 +02:00
Steeve Morin
fc0eb7d878
platforms: bump plugins to 202609181631.57.1.f601577e1ab1 (#837)
Notably, adds LTO
2026-09-18 20:00:56 +02:00
Corentin Godeau
c5d476d5e8
zml/testing: fix environment init (#836) 2026-09-18 18:44:39 +02:00
Corentin Godeau
8bae3485f5
zml/Compiler: enable optimizations flags (#834)
* `xla_gpu_experimental_use_collective_kernels=CONCURRENT` allows
parallel execution
* `xla_gpu_experimental_use_collective_kernels=SKIP_TEMP` avoids graph
updates
* `xla_gpu_experimental_use_collective_kernels` disables custom
collectives while we are looking for a fix
* `xla_gpu_enable_command_buffer` adds collectives to the graphs
2026-09-18 17:22:07 +02:00
Corentin Godeau
a6c4464dba
zml/platforms: use vmm by default on CUDA and ROCm (#833)
So that we benefit from `SKIP_TEMP`
2026-09-18 17:22:07 +02:00
Corentin Godeau
00d5fcfb00
kernels/triton/builder: add explicit cast to avoid compilation error (#831) 2026-09-18 16:34:20 +02:00
Olivier Boulant
1f0c465f9c
bazel: fix runfiles initializer retention on macOS (#829) 2026-09-18 16:27:00 +02:00
Hugo Mano
86e55c0c91
zml/mem: use UninitializedBufferAllocator for ROCm (#832)
It's faster than using the default pageable memory allocator with the
latest version of the plugin,
2026-09-18 16:22:21 +02:00
Benjamin Rabier
243dcda29b
zml/attention/triton: expose tokenToSequence (#818) 2026-09-18 10:33:00 +02:00
Corentin Godeau
ebb9813157
zml/platform: disable async dispatch on cuda (#828)
This has a negative impact on single graph forward
2026-09-18 10:17:10 +02:00
Steeve Morin
b31dca5b12
bazel: make lto work again with bazel runfiles init (#822)
in LTO, the linker would garbage collect that exported symbol since it
wouldn't be used inside the final binary

forcing it undefined via a link flag prevents it from being removed.
2026-09-17 23:52:43 +02:00
Hugo Mano
5b48183654
zml/quantization: recognize FNUZ block-32 FP8 (#826)
Allow block-32 scales to accompany E4M3FNUZ weights after load-time
normalization on CDNA3. The tensor layout and quantization scheme stay
unchanged, so existing Linear paths can use native FP8 conversion
without materializing software-decoded FN weights.
2026-09-17 12:48:01 +02:00
Hugo Mano
cf7c93c81b
zml/io: use direct memory writer on ROCm (#825) 2026-09-17 12:47:28 +02:00
Guillaume Wenzek
f09060e5bf
//tools/xprof: bump to xprof 2.32.2 (#823)
I found a non-nightly version that seems to work well for us.
Previous nightly versions we were using is not hosted anymore on
Pypi.org
2026-09-17 10:02:57 +02:00
Steeve Morin
d60a11e2a2
zml: add support for zio (#821)
Add a [zio](https://github.com/lalinsky/zio) dependency.

We require ZML to be aware of which `std.Io` implementation it's running
because we use `threadlocal` to store the current `zml.Compiler`
instance when compiling a graph (to make the DSL pleasant to use).

This is our only `threadlocal` usage but it is one too much. So we must
make it compatible with `zio.TaskLocal`. Not a big deal, but this
requires opt-in from the user.

And so `zio` support can be opt-in enabled by setting the build flag
(default is `std`):
```
--@zml//zml:io_impl=zio
```

Sadly there is no `std.Io` response to `threadlocal` (or task local).
Hopefully this will be fixed in the future.

Note that this creates problem if using `std.Io.Evented`.
2026-09-16 22:27:34 +02:00
Steeve Morin
6de80971d2 platforms/oneapi: add missing header in oneapi target 2026-09-16 21:53:11 +02:00
Steeve Morin
df3a43d29f
bazel: add zig_options rule (#820)
Introduce `zig_option` and `zig_options` rules to generate the bazel
equivalent of option modules plugged on bazel flags.

On `string` flags, if the flag has a `allowed_values`, the `string` is
converted into a `enum` in zig code.

Also, port platforms to this and remove the ugly `defines` hack to
comptime test for platforms enablement.

In the process, port `arocc` to the `zig_options`.
2026-09-16 21:50:22 +02:00
Benjamin Rabier
51235e7eb6
zml: block32 FP8 with E8M0 scales, and an explicit Linear output dtype (#811)
Two things, the second one falling out of the first.

**`fp8_block32`**: E4M3FN weights with one scale per 32x32 tile, the
layout
DeepSeek V4.1 ships. `quantizeBlockFp8` takes a block size and a scale
dtype;
the E8M0 path floors amax at 1e-4 and rounds the exponent up, so the
activation
side is MXFP8: one E8M0 scale per row per 32 contracted values. That
pairing is
what XLA's fp8_block_gemv arm needs to hand both scales to
`tt.dot_scaled`
(`Fp8BlockGemvScaleInMma` requires E8M0 on *both* sides); F32 activation
scales
would quietly fall to the per-block multiply. The Triton MoE path gets
the same
scheme at `block_size_k = 32`.

`Linear.forward` also stops rounding the activation to BF16 before
quantizing,
so amax is computed on the input as given.

**An explicit output dtype** on `scaledDot` and `Linear.forward`.
Inference
cannot answer what an FP8 x FP8 product should return: the accumulation
is F32
either way, so BF16 or F32 back is the caller's choice. `forwardScaled`
now
hands the caller's type to `scaledDot` rather than rounding through BF16
and
converting after. Narrow outputs are rejected.

Separately, `Linear.forward` converts the weight and bias to the
activation
dtype. `dotGeneral` asserts both operands share one, which broke the
moment a
model raised its activation to F32 against a BF16 weight; the wrappers
this API
replaced converted for exactly that reason.

**Breaking:** `Linear.forward` takes a second argument and `scaledDot` a
fifth.

---------

Co-authored-by: Raphael Nahoum <raphael@zml.ai>
2026-09-16 18:13:08 +02:00
Corentin Godeau
78a89679fa
zml/buffer: introduce simple createView (#817)
`Buffer.createView` can be used to create a view from a host pinned
buffer and use it as if it was one the device. This avoids XLA
limitations of doing operations directly on host pinned buffers.
2026-09-16 17:57:41 +02:00
Corentin Godeau
ae9ce333ed
zml/io: introduce support of memory in the loader (#815)
This only works on OneAPI and Cuda for now.

For other platform we would need a writer that allocates host pinned
uninitialized buffers for the shards and write directly to them.
2026-09-16 17:57:40 +02:00
Corentin Godeau
7adc862cbd
pjrt: rename ambiguous field of CreateViewOfDeviceBufferArgs (#814) 2026-09-16 17:57:40 +02:00
Raphael Nahoum
ace231388d
platforms/cpu: fix sandbox lib path after plugin layout change (#819)
The CPU plugin archives now ship the shared library under lib/, and
copy_to_directory outputs to sandbox/ instead of sandbox/lib. Look up
the sandbox directory in runfiles and join lib/ when building the
library path, matching metal, rocm and cuda.
2026-09-16 17:48:02 +02:00
Hugo Mano
4adb31b4d6
platforms: bump plugins for CPU, ROCm, CUDA and Metal (#812)
Bump to build `202609161226.43.1.b992d049cec2`
2026-09-16 14:53:44 +02:00
Raphael Nahoum
94257d3747
kernels/cute: Zig CuTe DSL and XLA custom call and bump xprof (#810)
Adds a Zig front end for NVIDIA's CuTe DSL, shaped after `cutlass.cute`,
plus the `__gpu$xla.gpu.cute` custom call that compiles and launches it
on CUDA.

- `mlir/dialects/cute_ir`: bindings for the `cute` and `cute_nvgpu`
dialects
- `zml.ops.cute`: emits the custom call with the textual module, `grid`
and `block`.
- `kernels/cute` + `zml.kernel.cute.Kernel`: builder, values, tensors
with coordinate `get`/`set`, thread/block indices, static smem, sync,
`scf`. See `kernels/cute/README.md`.
- bump `xprof-nightly`

---------

Co-authored-by: Steeve Morin <steeve@zml.ai>
2026-09-16 14:02:46 +02:00
Corentin Godeau
91c8b31502
platforms: bump plugins for CPU, ROCm, CUDA and Metal (#809)
Bump to `202609151311.37.1.f1b6432d55a0`
2026-09-15 16:07:39 +02:00
Corentin Godeau
550f2bbee0
zml/Compiler: allocate inputs/outputs for func.func (#806)
When we have a lot of inputs/outputs, it can overflow the defaults we
put in the `func.func` helper.
2026-09-14 18:19:03 +02:00
Hugo Mano
fdb79caa05
kernels/fly: MOE + MLA (#805)
Add native FlyDSL kernels for MXFP4 MoE and BF16 sparse MLA on `gfx942`,
with SSA builder operations for matrix loops
2026-09-14 15:42:56 +02:00
Raphael Nahoum
b78c0f03d7
kernels/fly: FlyDSL Zig DSL (#799)
Adds a Zig front end for the FlyDSL dialect, so AMD GPU kernels can be
written in Zig alongside the existing Mosaic, Triton and tile ones, and
dispatched the same way.

Types and layouts are built through the dialect's own C entry points
rather than by printing and reparsing assembly. That holds in both
directions: host-side layout literals become attributes directly, and
anything the dialect computes is read back through typed accessors.

Launch geometry is derived from the device rather than hard-coded, so
the same kernels run on both AMD architecture families and the tests
stay architecture agnostic. Everything builds and tests on any host, no
GPU needed.

Also carries one unrelated cleanup, dropping a needless null terminator
from the Triton and tile kernel strings.
2026-09-14 09:57:36 +02:00
Corentin Godeau
4788ccc284
zml/moe: add triton_mxfp4 backend (#801)
This is a kernel that is quite specific to Deepseek V4 Flash

Ideally, we would want to unify the triton kernels.
2026-09-11 18:28:51 +02:00
Alhussein Jamil
ecb824e859
bin/zml-smi: keep NVML polling on one thread so --top does not freeze (#766)
## Summary
- `zml-smi --top` was stalling under load because each GPU and the
process list hit NVML on their own thread. Those calls serialize on the
driver RM lock, so one slow ioctl froze the whole TUI.
- All NVIDIA queries now run on a single poller.
`nvmlDeviceGetPcieThroughput` (it blocks while sampling) is only taken
on the first tick and every fourth tick after that; link speed/width
stay on the hot path.
- Process-query sizes above 4096 are ignored, and a failed per-process
util sample no longer skips the rest of that GPU.

---------

Co-authored-by: alhussein jamil <alhussein@zml.ai>
2026-09-11 16:29:29 +02:00
Benjamin Rabier
f75f5bd27e
zml/Compiler: Allow providing a Progress.Node (#798) 2026-09-11 13:07:45 +02:00
Benjamin Rabier
7204aaacbf
zml/moe: Merge MXFP4 & MXFP8 paths (#795) 2026-09-11 12:56:35 +02:00
Benjamin Rabier
46bc6df684
zml/moe: proper FP8 scaling and bias in triton experts (#793)
- The current Fp8 scaling wasn't properly finished. We didn't use the
scales at all.
- Adding a `quantize_input` option that GLM5.3-Flash uses
- `applyActivation` wasn't properly using the threshold for relu
- The bias was silently ignored for the MoE.
2026-09-11 12:55:59 +02:00
Corentin Godeau
432609ed22
zml/triton: wire TMA and global scratch memory size in Triton kernels (#797)
XLA supports TMA and scratch memory in the triton custom call but we
never wired it.

It is now solved.
2026-09-11 11:21:20 +02:00
Benjamin Rabier
28ea0571e6
zml/moe/triton: remove makeFusedMoeConfig and use clearer names (#792)
`makeFusedMoeConfig` was just adding extra complexity and we used `a`,
`b` and `c` everywhere. I think it's fine in the triton kernel itself,
but outside of it's just confusing IMHO.
I removed a bunch of useless parameters also that just end up returning
an `Unsupported` error.
2026-09-10 19:19:34 +02:00
Hugo Mano
0ebb0c21a9 platforms: bump plugins for CPU, ROCm, CUDA and Metal (#788) 2026-09-10 14:34:40 +00:00
Benjamin Rabier
662bc8ea5d
zml/moe: Simplify triton API and config loader (#790) 2026-09-10 09:50:57 +02:00
Benjamin Rabier
f6af7f3e52
zml/moe: remove unused metadata (#789) 2026-09-09 19:07:33 +02:00
Damien Mehala
0f7b3bb4b0
zml/moe: enforce padded_num_experts as a power of two (#786)
Fix an issue when `padded_num_experts` computed doesn't land as a power of two. Notably for DeepSeek-v4 Pro with its 384 experts, divided by 32 warp is not of power of two, leading to a crash in the triton which expects a power of two number of experts.
2026-09-09 15:17:32 +02:00
Benjamin Rabier
41fce4f2fc
zml/nn: Add zml.nn.forwardQuantized (#783) 2026-09-09 09:32:43 +02:00
Benjamin Rabier
06074d81f6
zml: support block-128 FP8 quantized input and f32 scale & add rocm computeCapability (#782)
Imitating `platform.cuda.computeCapability` as fp8 support is native for
e4m3fnuz on gfx942 but e4m3fn for everything else that has hardware
support AFAIK.
2026-09-09 09:32:26 +02:00
Guillaume Wenzek
2e5150eb0f
stdx/json: {Union,TaggedUnion}Helpers (#770) 2026-09-09 09:01:10 +02:00
Benjamin Rabier
afb3559f7f
zml/moe: apply activation thresholds (#781) 2026-09-08 15:56:58 +02:00
Raphael Nahoum
77e41879d3
zml/kernel: add a CUDA Tile IR (cuda_tile) Zig DSL (#779)
A third kernel language beside Triton and Mosaic TPU. It emits NVIDIA's
cuda_tile dialect and reaches the GPU through XLA's
`__gpu$xla.gpu.cuda_tile` custom call.

- `third_party/cuda_tile`: cuda-tile v13.3.3
- `mlir/dialects/cuda_tile`: every type and enum through the dialect's C
API, one constructor per op (all 100, in Ops.td order), and a small C
shim for the TileView interface the C API does not expose.
- `kernels/cuda_tile`: views, tokens, for/if/loop, reduce/scan (single
and variadic), atomics, MMA, every elementwise op.
- `zml.kernel.cuda_tile.Kernel` and `ops.cudaTile`: the custom call,
with a grid and no num_warps/num_stages (those are optimization_hints in
the IR).
2026-09-08 14:57:40 +02:00
Raphael Nahoum
9c92ff6203
zml/platform: one ComputeCapability for every CUDA gate (#777)
Every CUDA gate used to parse the compute capability string itself. They
now share `platform.cuda.computeCapability`, which returns null off CUDA
and offers `.is` / `.atLeast` / `.sm`.

Also folds the NVFP4 scale clamp into its reduce to remove one launch
per projection, and is equivalent.
2026-09-08 14:42:32 +02:00
Raphael Nahoum
520f8d63c1
zml/Compiler: set the fork's flag defaults per target (#775)
The fork keeps upstream's flag defaults, so zml turns on what each
target actually wants:
- the Tile IR scaled-dot backend, subchannel dequantize fusion and
Triton multi-output fusion on CUDA
- Metal AIR fast math off, since it lets the AIR compiler assume no NaNs
and is_finite(NaN) comes back true.
2026-09-08 14:39:08 +02:00
Raphael Nahoum
12b299186a
platforms/cuda: bump the plugin and cuDNN to 9.24.0 (#774) 2026-09-08 14:34:30 +02:00
Benjamin Rabier
7f5c678a58
kernels: Add support for e4m3fnuz dtype (#780) 2026-09-08 13:52:52 +02:00
Raphael Nahoum
1fd10b6528
platforms/metal: update plugin (#773)
improve gated delta net and rebase on latest XLA
2026-09-07 17:02:57 +02:00
Tristan Gautié
053577d82d
zml/buffer: switch uninitialized buffer log to debug level (#771) 2026-09-07 09:37:39 +02:00