Add helpers for TMA gather/prefetch, weight-stationary MMA,
shared-memory descriptors, packed FP32 arithmetic, async cluster stores,
and register limits.
Make CTA groups and barrier scopes explicit across TMA, block-scaled
MMA, TMEM, and synchronization, including two-CTA operations and
multicast completion. Consolidate named CTA barriers and update the MoE
and paged-attention callers.
Cover instruction operands, descriptor encoding, and MLIR round-trips in
builder tests.
Adds a Zig DSL for Metal kernels over the AIR dialect, and moves the
Metal paged attention kernels onto it.
- `mlir/dialects/air`: a `thread` address space. Each op has an
`llvmBuilder` that calls its `air.*` builtin with Apple's attributes,
and the dialect's translation interface builds the `!air.kernel`
metadata, so lowering to LLVM IR is MLIR's own translation. The verifier
rejects what Apple's compiler can't take: bool, scalable and n-D
vectors, buffers of non-Metal scalars, scalar f32 converts, and
`air.kernel` on a declaration.
- `dsl/metal`: an MSL-shaped builder. It emits `air` plus the
`arith`/`cf`/`vector`/`llvm` IR that clang produces for MSL, and covers
MSL's math, SIMD-group, barrier and `simdgroup_matrix` functions.
- `zml.kernel.metal` / `ops.metal`: wrap a kernel into a
`__gpu$xla.gpu.air` custom call. XLA lowers it, compiles it to a
metallib and launches it.
The DSLs and the kernels no longer make ops or set dialect attributes by
name: each dialect package gets the builder or helper they were missing,
and every consumer goes through it. This now fix ZML core files as well
Adds Apple's AIR dialect, for writing Metal kernels in MLIR.
- `mlir/dialects/air`: the AIR that Apple's Metal compiler emits for
MSL. Each op is named after its `air.*` intrinsic without the type
suffix (`fast_exp`, `max.u`, `simd_shuffle_xor`, `wg.barrier`,
`simdgroup_matrix_8x8_load`, ...), with fastmath flags on float ops.
Kernel arguments are described the way that compiler describes them:
builtins, buffers with access/noalias, and `air.arg_name`.
- `llvm`, `arith`: bind array and named struct types, linkage, loop
annotations, and integer overflow flags.
Splits the monolithic mlir/dialects library into one Bazel package per
dialect, then makes every package follow the same layout, C API and Zig
API. Each package owns its Zig bindings, its C API when upstream doesn't
provide one, and its tests. Dialect registration moves out of the core
mlir package, so a target only pulls in the dialects it actually uses.
The conventions are documented in `mlir/dialects/README.md`, with a
porting checklist and conformance checks.
Along the way:
- cute_ir is flattened, and its Cuda, CuteNVGPU and NVVM sub-dialects
become their own packages
- fly's ROCDL extension moves to fly_rocdl
- new packages for ROCDL and UB
- kernels/ is renamed to dsl/, since it holds the DSLs
- every package registers through its own registerDialect, and each DSL
through registerDialects, instead of dialect name strings
- the C and C++ follow the LLVM and MLIR C API conventions, formatted
with a new mlir/.clang-format
- BUILD files share one template
- accessors are named as nouns like in the rest of ZML
- each package tests itself in its own test.zig, including that every
bound operation is registered and that every wrapper method compiles
Fixes:
- memref atomic_rmw emitted the wrong kind, because the AtomicRMWKind
values didn't match the pinned LLVM
- the pinned Triton and Mosaic no longer have tt.cat and tpu.repeat, so
the DSL's cat and repeat now lower the way tl.cat and pltpu.repeat do
- the ROCDL bindings are completed: buffer address spaces, vector types,
waves_per_eu range
Breaking changes:
- register dialects through their package instead of
DialectHandle.fromString, and mlir.registerFuncExtensions becomes
func.registerExtensions
- accessors drop the get prefix, so getValue() becomes value()
- ttir.cat and mosaic_tpu.repeat are removed, and stablehlo pad and
mosaic_tpu sort take their options before the location
- zml.kernel.fly.rocdl becomes fly_rocdl
- Mesh.sdyMeshAttr takes a context and returns an attribute instead of
text
- func.func no longer prints public, its default visibility
---------
Co-authored-by: Steeve Morin <steeve@zml.ai>
here I'm promoting a pattern I would like to extend to more of ZML:
function that takes `anytype` argument should do as little work as
possible and called a fully typed function.
I kept `manualComputation` signature to not break existing code,
I moved the code to `zml/ops/manual_computation.zig` to better
encapsulate the required helper functions.
(and `zml/ops.zig` is pretty big atm)
Similarly `Sharding.Partitioning.parse` now calls `const logical =
comptime Logical.parse(partitioning);` then `logical.applyUseAll(tags,
mesh, default_value);` which is fully typed.
I'm also documenting `manualComputation` better
NVPTXAddressSpace is NVPTX specific. In LLVM, addressSpace is defined
for each target (nvptx, amdgpu)
The pointerType is general in llvm dialect by accepting an anytype.
- Implement the flashmla kernel for sm100 using the cute dsl in zig.
- Extend the cute dsl with specialized sm100 (tcgen05) ops.
- Introduces a new cute backend which calls the cute kernel.
- Uses a new cache for MLA (latent or quantized).
Introduces the CuTe mxfp4 moe backend. The kernels use the cute dsl and
are optimized (and so at the moment constraint to its shapes).
Introduces the cute launch host func and so the associated cuda dialect
in zml.
The CuTe dialects no longer build on a git checkout with a private
overlay.
- Every operation, type, attribute and enum of the compiler's cute and
cute_nvgpu dialects is typed, with its documentation, operand
constraints, traits and result inference. Nothing is kept as an opaque
payload, and the compiler's IR round-trips through it unchanged.
- The definitions are generated following `REGENERATING.md`, from the
release's Python bindings and by probing the compiler.
- The C API and the Zig builders follow: atom types take their
parameters instead of a text payload, constrained integers are one type,
and pointers take an address space enum.
Add explicit debug info in ZML.
Model author can now add `pushLocation` to mark specific sections of the
IR
---------
Co-authored-by: Guillaume Wenzek <gwenzek@users.noreply.github.com>
Adds a Zig front end for NVIDIA's CuTe DSL, shaped after `cutlass.cute`,
plus the `__gpu$xla.gpu.cute` custom call that compiles and launches it
on CUDA.
- `mlir/dialects/cute_ir`: bindings for the `cute` and `cute_nvgpu`
dialects
- `zml.ops.cute`: emits the custom call with the textual module, `grid`
and `block`.
- `kernels/cute` + `zml.kernel.cute.Kernel`: builder, values, tensors
with coordinate `get`/`set`, thread/block indices, static smem, sync,
`scf`. See `kernels/cute/README.md`.
- bump `xprof-nightly`
---------
Co-authored-by: Steeve Morin <steeve@zml.ai>
Adds a Zig front end for the FlyDSL dialect, so AMD GPU kernels can be
written in Zig alongside the existing Mosaic, Triton and tile ones, and
dispatched the same way.
Types and layouts are built through the dialect's own C entry points
rather than by printing and reparsing assembly. That holds in both
directions: host-side layout literals become attributes directly, and
anything the dialect computes is read back through typed accessors.
Launch geometry is derived from the device rather than hard-coded, so
the same kernels run on both AMD architecture families and the tests
stay architecture agnostic. Everything builds and tests on any host, no
GPU needed.
Also carries one unrelated cleanup, dropping a needless null terminator
from the Triton and tile kernel strings.
A third kernel language beside Triton and Mosaic TPU. It emits NVIDIA's
cuda_tile dialect and reaches the GPU through XLA's
`__gpu$xla.gpu.cuda_tile` custom call.
- `third_party/cuda_tile`: cuda-tile v13.3.3
- `mlir/dialects/cuda_tile`: every type and enum through the dialect's C
API, one constructor per op (all 100, in Ops.td order), and a small C
shim for the TileView interface the C API does not expose.
- `kernels/cuda_tile`: views, tokens, for/if/loop, reduce/scan (single
and variadic), atomics, MMA, every elementwise op.
- `zml.kernel.cuda_tile.Kernel` and `ops.cudaTile`: the custom call,
with a grid and no num_warps/num_stages (those are optimization_hints in
the IR).
Add MLIR composites, `scaledDot`, and a Linear path for NVFP4 weights
(unpack, optional activation quant on Blackwell, global scales). Skip
fused multi-source tensors in the generic loader so pack/`loadExecute`
can fill them (may be used later for fused QKV etc).
Implement paged_attention in pure stablehlo.
introduces `zml.ops.if` and `zml.ops.if2`
The first one is similar to `zml.ops.while` and requires a full struct
with two functions. it's a bit heavy to use, but it follows the
stablehlo API, and if one of the branch has side effects it will
correctly evaluate the side effect only in that branch.
`zml.ops.if2` is a simpler version that doesn't require to create
intermediary struct, but as a consequence, if you have side effect they
will always evaluate, and it relies on the compiler to correctly avoid
duplicated compute by moving relevant compute inside the branches.
(open to feedback about if/if2)
**Summary**
This PR update to Neuron v2.30, introduces the Neuron NKI stack into ZML and wires it into the
LLM attention path.
The main goal is to make Neuron attention run through NKI kernels.
It adds the compiler/runtime plumbing needed to embed NKI kernels as
Neuron custom-native calls, then uses that path for SDPA-style
prefill/decode attention. Paged Attention will come later.
On inf2.8xlarge it runs :
- 25 tok/s for Llama 3.1 8B BF16
- 63.5 tok/s for LFM2.5-1.2B-Instruct BF16
- 19 tok/s for Qwen3.5-9B BF16
This PR introduces the sharding as expert parallelism for MoE :
- Introduce a .experts logical sharding (required to express an other
semantic than TP (.model))
- Wrap the whole MoE impl into a manualComputation
- Implement a manual stableHlo all reduce since shardy can not infer
the correct communication after a manualComputation (blackbox)
- Partition qwen 3.5 moe example with TP for the rest of the model (non
MoE)
---------
Co-authored-by: loupicaaa <louis@zml.ai>
This PR changes the following:
- Name-based `output_operand_aliases` for MosaicTPU and Triton DSLs
- New `kernels/mosaic_tpu/pipeline.zig`, a Zig port of `pltpu.emit_pipeline`
- Some fixes to MosaicTPU
This PR exposes the XLA-specific custom calls that allow to create
uninitialized tensors (thus not paying the cost of a kernel to
essentially do a memset).
The `stablehlo` part of it was already done, so I added a wrapper in
`tensor.zig`
This PR introduces the following:
- Wire up the LLVM dialect so the Triton builder can emit
`llvm.intr.assume` so we can match
[`triton.language.assume`](https://triton-lang.org/main/python-api/generated/triton.language.assume.html#triton.language.assume)
- Thread `tt.num_stages` through `scf.for` for pipelining hints
- Add an inline-return mode to `ReturnIfScope`. Not necessary but better
for IR matching with Python reference
- Build DSL kernels IR in a new MLIR context
Also fixes i1 extension to use `extui` instead of `extsi`.
Drops the Python build-time dependency for Triton and Mosaic TPU
kernels. ZML now emits TTIR and Mosaic TPU IR directly from Zig, against
the same MLIR context as the rest of the compiler. So no more JAX,
Triton at build time and a lighter sandbox.
Split into two commits:
- **`c72f44884` — Add Triton and Mosaic Zig DSL.** Additive: IR builders
(`kernels/triton/`, `kernels/mosaic_tpu/`), upstream dialect bindings
(`ttir`, `mosaic_tpu`, plus `arith`/`scf`/`math`/`memref`/`vector`/
`affine`/`cf`), and the high-level
`zml.kernel.{triton,mosaic_tpu}.Kernel(Config, spec)` declarative form.
Comes with author guides in each `README.md`.
- **`cb61e529f` — Remove Python and replace with Zig kernels.** Migrates
the two production kernels (`unified_attention` for Triton,
`ragged_paged` for Mosaic TPU) and deletes everything Python related in
the sandbox and what was generating the IR.
---------
Co-authored-by: Steeve Morin <steeve@zml.ai>
When looking at the new Sharding code I was triggered by the high number
of "try".
I then realized most of the try where traced back to `axes = try
.init(0)` which can never fail.
So I removed BoundedArray.init (aka Andrew mode)
It's generally unsafe because it makes `slice()` return undefined.;
I replaced`try init(0)` by `.empty`
I replaced `try init(n)` by `.fromSlice` or with `.{ .buffer =
undefined, .len = n }` to explicitly show the unsafeness
This leads to widespread changes because it means that most of the
sharding operations can't fail anymore.
Today, creating a custom call is a bit tedious and error prone. The
primary goal of this PR is to only accept structs as
input/output/attributes and rely more strongly on the names everywhere
when creating as custom call. I've migrated the `fa2` custom to provide
an example. Can migrate the rest.
- Custom call relies on the position of the input/outputs/attributes to
be in the correct order to work. The `output_operand_aliases` also maps
the outputs to the input field position.
- If the custom call is "sharding aware", as in it knows it works on a
single partition on multi devices setup, one must use
`zml.ops.manualComputation` which is a bit tedious to set up.
- `flashattn.zig` has a bunch of logic to register said custom call but
also to map inputs into something more usable which can be shared.
- both `zml.ops.customCall` and `zml.ops.manualComputation` tend to
accept to many different types IMHO which makes it a bit hard to follow.
But well, to be fair not sure how much simpler my code is. I do
constrain myself to only accepting simple structs for the generic types.
I've re-used the existing `manualComputationInternal` and
`customCallInteral` for this new API without touching them. The only
exception is the `output_operand_alises` where I needed a bit more
flexibility.
This pull request includes several improvements and bug fixes across the
codebase, focusing on memory handling, correctness in tensor testing,
and API consistency. The most significant changes are grouped below:
### Tensor Testing and Memory Management Improvements
* Fixed a bug in `zml/testing.zig`'s `testlayer` by updating the
callback signature to take a mutable context and increment the index
correctly, ensuring tensors are correctly initialized.
* Improved memory management for expected output slices in `testlayer`
by switching to `zml.Slice.alloc` for allocation, using `readSliceAll`,
and freeing only the data instead of the whole slice, removing the
potential alignment problems.
* Enhanced test logging to include the tolerance value in success
messages for better diagnostics.
### API and Code Quality Enhancements
* Increased the evaluation branch quota in `MapRestrict` to support more
complex compile-time type mapping, preventing quota exceeded errors.
* Updated the `fromCStruct` function in `pjrt/pjrt.zig` to accept a
`meta.Struct`.
### StableHLO Output Operand Alias Handling
* Improved handling of output operand aliases in
`mlir/dialects/stablehlo/stablehlo.zig::custom_call` by passing
`output_tuple_indices` for multi-result functions, increasing
correctness in custom call attribute construction.