0
Fork 0
mirror of https://github.com/zml/zml.git synced 2026-11-16 19:11:09 +00:00
Commit graph

1,078 commits

Author SHA1 Message Date
Raphael Nahoum
46bd32de2f zml/tokenizer: grow the decoder output to fit any token
IREE writes a token's text whole or not at all, so `feedOne` asserted
whenever the caller's buffer was smaller than the token, e.g. Laguna's
415-byte whitespace tokens in a 128-byte buffer. The decoder now owns
its output buffer and grows it the way `decode` already did, so
`feedOne` and `finalize` no longer take a buffer.
2026-11-09 10:56:20 +01:00
Corentin Godeau
59a888f7e2
third_party/zls: allow all 0.17 zig versions (#1036) 2026-11-06 15:14:38 +01:00
Raphael Nahoum
5b079127bb
workspace: bump ZML plugins (#1034)
Bump plugins to
[202611060925.355.1.2ca211fa5fbd](2026110609.355.1.2ca211fa5fbd)
2026-11-06 10:56:05 +01:00
PierreLgol
488c21c899
workspace: add Helix editor configuration for Zig (#1029)
adding basic support for the helix editor, to automatically work
2026-11-06 10:29:31 +01:00
erwanvivien
837e45022a
zml/kernels/paged_attention: fix stablehlo backend slot_id increment (#1027)
If sequences were not 16-aligned we would have sequence leaks. Now we
increase slot_id by query_start_len and pad when needed
2026-11-06 09:57:09 +01:00
Corentin Godeau
630d96726b
zml/io: add consolidated version of safetensors to defaults (#1035)
Some models entrypoints are exposed as
`consolidated.safetensors.index.json` or `consolidated.safetensors`.
2026-11-05 19:34:17 +01:00
Raphael Nahoum
3285b7e0f4
zml: FP8 block activations and paged attention query padding on Furiosa and bump plugins (#1033)
- Quantize block FP8 activations to E4M3FN on Furiosa, so scaled dots
run natively. Other block sizes keep BF16 activations and XLA's
fallback.

- Pad paged attention query groups to a power of two with zero queries,
as TCC fails other group sizes. Marked as a workaround.

- Bump to
[202611051630.351.1.48d539621544](2026110516.351.1.48d539621544)
2026-11-05 17:38:31 +01:00
Corentin Godeau
5decef1fca
zml/kernels/paged_attention: enable cute by default when available (#1017) 2026-11-05 14:17:44 +01:00
Hugo Mano
2438b9c364
zml/kernels/paged_attention: make mesh and partitioning explicit (#1024)
Paged attention currently relies on input sharding metadata or
partitioning choices made inside backend kernels. Ordinary graph
operations can leave that metadata unspecified, making the resulting
sharding depend on the backend and preceding operations.

Move the manual computation boundary into the shared `pagedAttention`
API using `manualComputation2`. Callers provide a logical mesh and typed
input/output partitioning specs.
2026-11-05 13:19:58 +01:00
Hugo Mano
47f50667d8
zml/attention: declare manual partitioning in the shared API instead of kernels (#1016)
Move manual regions out of flash attention kernels and use
`manualComputation2` for every shared attention backend.

Also fully replicate the embedding executable outputs in Qwen and Llama.
Partial graph constraints leave their batch and sequence dimensions
open, which is invalid at executable boundaries.

Fix two further blockers exposed by the Qwen run that we forgot to move
recently: softplus comparisons must use the input dtype and ROCm DNN
runfiles must include the MIOpen kernel archives.
2026-11-05 13:00:00 +01:00
Guillaume Wenzek
4baec25924
kernels/moe: remove confusing bitcast (#1030)
the bitcast discard partitioning info
the API issue is tracked by https://github.com/zml/zml/issues/914
2026-11-05 11:53:31 +01:00
Louis Lechevalier
69a6b2bb22
workspace: bump ZML plugins (#1031) 2026-11-05 11:44:39 +01:00
Damien Mehala
8898051c7b
zml/kernels: fix Triton split-K (#1012)
The split-K/3D path writes partial results only for active queries, but
the reduction kernel processes inactive queries in the padded batch.
These queries have no valid results and cause invalid
computation/outputs.

This fix zeroes the ouput for inactive queries and return before before
accessing sequence metadata.

Resolves #1006
2026-11-04 20:48:56 +01:00
Raphael Nahoum
5764202e5f
zml/kernels/paged_attention/tcl: follow sequence lengths and support several chips (#1026)
- **Work follows sequence lengths:** decode's compute no longer runs
over the block table's whole width, and flash prefill loops only over
the chunks in use.

- **One plan per call:** each call picks straight decode, the hybrid
loop or flash attention, following a calibration sweep based on what
compiles. The kernels might change once the compiler evolves.

- **Several chips:** each chip runs the kernels over its own heads
inside a manual computation.
2026-11-04 18:52:05 +01:00
Guillaume Wenzek
db119b2d35
zml.Shape: improve parseTags (#1023)
avoid `stdx.BoundedArray` for tags, use `[]const Tag`, this allows to
simplify some logic and more explicitly separate comptime work from
runtime work.
It reduces binary sizes, preventing big functions to be duplicated
because of different tags.
2026-11-04 16:04:19 +01:00
Guillaume Wenzek
8992fbe434
zml.Shape.assertTags (#1022)
we often use `withTags` to document the current shape, but it could also
hide bug by overriding previous tags.
I introduce `assertTags` that should be favored when possible
2026-11-04 16:04:18 +01:00
Guillaume Wenzek
17ba49cce8
zml.ops.manualComputation: fix error message (#1021)
use `.format` instead of default Zig formatting
2026-11-04 16:04:18 +01:00
Guillaume Wenzek
350c6c1606
zml.Mesh: axis/hasAxis helpers (#1019) 2026-11-04 16:04:17 +01:00
Guillaume Wenzek
4707e58774
zml.Compiler: allow to propagate tensor store names (#1018)
The name can appear in the IR and thanks to #1003 it shows in the web
view of the model making all compute blocks pretty obvious.

The screenshot below without names would be quite confusing, 
but now we can see this is the QKV of first layer of a Llama model.
<img width="2211" height="1208" alt="image"
src="https://github.com/user-attachments/assets/1917edb0-a772-4a1d-9aa7-ad5aaf7b5260"
/>
2026-11-04 16:04:17 +01:00
erwanvivien
194175b36d
zml/kernels/flash_attn: add prefill flash_attention kernel and add Causal (#1000)
Prefill kernel is based on libflashattn.so from zml/flash-attention
2026-11-04 14:35:55 +01:00
Hugo Mano
6a1698f270
zml/ops: add manualComputation2 with explicit partitioning (#1013)
Accept a logical mesh and typed input/output partition specs at manual
computation boundaries.

Will replace `zml.ops.manualComputation` in a future PR.
2026-11-03 18:22:59 +01:00
Hugo Mano
4c355c61b7
zml/sharding: keep unconstrained graph results open (#1004)
Ordinary graph results should have an unspecified layout until
constrained or inferred by the partitioner.

Previously, we could report replication simply because a tensor had no
mesh, even when its stored partition specs were open. Other paths
preserved selected operations’ sharding metadata, making the frontend’s
interpretation depend on how a tensor was produced.

This change gives those results a consistent open state while requiring
concrete layouts at executable boundaries.

Shardy explicitly defines the intended semantics: [“no sharding
attribute on a tensor is equivalent to a fully open tensor
sharding.”](https://openxla.org/shardy/sharding_representation#open)

Open dimensions remain available for further partitioning; closed
dimensions fix the specified layout.
Replication is therefore a constraint, whereas an open layout lets the
partitioner choose either sharding or replication:

This distinction also applies to unused mesh axes. Marking an axis
explicitly replicated prevents propagation from using it, even when a
tensor dimension is open. For example, `[{?}, {}]` must leave unused
axes available; adding replicated={"data", "model"} removes those
choices.

Keeping intermediates open lets the partitioner use constraints from
producers and consumers, with operation-specific rules for
transformations such as dot and reshape.
2026-11-03 18:11:15 +01:00
Corentin Kerisit
439ba6c0a6
bazel: register runfiles initialization from C (#1005)
ZML initializes runfiles before `main` so its loaders can locate
packaged libraries and data.

When using `-fno-llvm -fincremental` and linking with the incremental
ELF linker, declaring the registration pointer in Zig produces incorrect
`.init_array` data.

See https://codeberg.org/ziglang/zig/issues/37453

Use a C constructor on Linux and macOS so the C compiler emits the
platform-specific startup registration, while Zig exports the existing
runfiles initialization function directly.
We can revert when https://codeberg.org/ziglang/zig/issues/37453 is
fixed.

With this fix, we bypass the wrong behavior in incremental ELF linker
and get closer to ZML compliance with zig incremental using their
incremental linker.

The registration is shared by the `use_cc_common_link=true` and `false`
paths. Keep the explicit linker retention so the constructor is included
even when its object is in a static archive.
2026-11-03 14:45:23 +01:00
Benjamin Rabier
86ab457d0b
dsl/cute: add SM100 primitives and CTA-group support (#1007)
Add helpers for TMA gather/prefetch, weight-stationary MMA,
shared-memory descriptors, packed FP32 arithmetic, async cluster stores,
and register limits.

Make CTA groups and barrier scopes explicit across TMA, block-scaled
MMA, TMEM, and synchronization, including two-CTA operations and
multicast completion. Consolidate named CTA barriers and update the MoE
and paged-attention callers.

Cover instruction operands, descriptor encoding, and MLIR round-trips in
builder tests.
2026-11-03 10:26:31 +01:00
Louis Lechevalier
2532589572
zml/moe: CuTe MoE nvfp4 on sm103 (#981)
This PR port the CuTe MoE for nvfp4 x nvfp4 from cutlass flashinfer into
zml. It uses the CuTe zig DSL. It is the first PR of a serie that aim to
replace the covering of flashinfer cutlass in ZML. The final goal is to
remove the flashinfer cutlass binding. It begins by removing the binding
of the bf16 kernel on sm90 (never used, we have triton).
2026-11-02 14:46:41 +01:00
Benjamin Rabier
d4b413f5f9
zml/tokenizer: support Mistral's numeric lookahead (#997)
Mistral Large 4's Tekken tokenizer uses positive lookahead to split
number runs into groups of three from the right. IREE's regex compiler
rejects this pattern, preventing the tokenizer from loading.

Patch IREE to recognize the exact isolated Split pattern, match complete
number runs with its DFA, and emit the original groups using Unicode
codepoint counts. Preserve progress when output fills during streaming
and finalization. Register the patch in ZML's IREE repository rule.

Treat RESOURCE_EXHAUSTED as normal encoder backpressure instead of
logging errors before retrying. Add regressions for numeric token IDs,
Unicode numbers, byte-at-a-time input, and output overflow.
2026-10-29 14:10:32 +01:00
Raphael Nahoum
ca39742f14
zml/platform: bump plugins and preallocate Metal memory again (#1001)
- Turns preallocation back on by default on Metal, as before #979: not
preallocating seems to cause memory issues we haven't looked into yet.
- Bumps the PJRT plugins to `202610290951.309.1.785550677e75`.
2026-10-29 12:52:21 +01:00
Raphael Nahoum
e4e031b2c5
zml/kernels/paged_attention/metal: port the kernels to the Metal DSL (#999)
Moves Metal paged attention from the `zml$paged_attn` custom call to
Metal DSL ports of the same kernels.

- `fa_vec_paged`, with a split-K reduce, handles causal bf16 decode.
`paged_attention_tiled` and `zero_padding_rows` handle everything else.
- The shared paged attention test now runs on Metal. Triton counts as
available only on CUDA, ROCm and oneAPI. StableHLO is the reference on
Metal for the causal case.
2026-10-29 12:52:20 +01:00
Raphael Nahoum
70f1bd4af4
dsl/metal: add the Metal DSL (#989)
Adds a Zig DSL for Metal kernels over the AIR dialect, and moves the
Metal paged attention kernels onto it.

- `mlir/dialects/air`: a `thread` address space. Each op has an
`llvmBuilder` that calls its `air.*` builtin with Apple's attributes,
and the dialect's translation interface builds the `!air.kernel`
metadata, so lowering to LLVM IR is MLIR's own translation. The verifier
rejects what Apple's compiler can't take: bool, scalable and n-D
vectors, buffers of non-Metal scalars, scalar f32 converts, and
`air.kernel` on a declaration.
- `dsl/metal`: an MSL-shaped builder. It emits `air` plus the
`arith`/`cf`/`vector`/`llvm` IR that clang produces for MSL, and covers
MSL's math, SIMD-group, barrier and `simdgroup_matrix` functions.
- `zml.kernel.metal` / `ops.metal`: wrap a kernel into a
`__gpu$xla.gpu.air` custom call. XLA lowers it, compiles it to a
metallib and launches it.
2026-10-29 12:52:20 +01:00
Damien Mehala
b725ff2481
zml/Compiler: add XLA flags to improve generated kernels for ROCm (#998)
Enable two flags for ROCm:
- multi output fusion: allows to search for fusion opportunity
generating different output.
- latency hiding scheduler: overlap async communication with
computation.
2026-10-29 11:16:03 +01:00
Damien Mehala
47e692c300
platforms/oneapi: bump to IGC 2.41.15 (#996)
Bump to resolve a compiler bug encountered on LFM3.

LFM3 broadcasts one sigmoid gate per attention head. This generated code
of sigmoid introduced a BF16 conversion and conditional select. The
compiler then lower it to a scalar computation which had a
register-aligment bug. The fix introduced in 2.41.11
<5d92530a07>
enforces alignment.
2026-10-29 11:15:39 +01:00
Raphael Nahoum
b37ba31500
mlir/dialects: build every remaining op and attribute through the dialect packages (#993)
The DSLs and the kernels no longer make ops or set dialect attributes by
name: each dialect package gets the builder or helper they were missing,
and every consumer goes through it. This now fix ZML core files as well
2026-10-27 17:53:31 +01:00
Raphael Nahoum
3736f3dc29
dsl: unify the kernel DSLs (#991)
Makes kernels written in the DSLs read alike, without changing what they
emit.

- Every DSL target is `//dsl/<name>:builder`, and `dsl/common` is
imported as common.
- Scopes without values end with `close()`.
- The scf DSLs (triton, mosaic, cute, fly) share
`openFor`/`openIf`/`openIfElse`/`openWhile` in `dsl/common`. cuda_tile
keeps its own scopes with the same shapes.
- Scopes take tuples or arrays, and result types can be dtypes (.f32),
Values, or MLIR types.
- Typed constants are `constant(dtype, value)`, alongside constIndex and
constantLike.
- Comparisons and bitcasts have the same names in every DSL: Fly gains
`lt`/`eq`/…, and every DSL spells it `value.bitCast(dtype)`.
- One block stack and one `common.verify`, which logs the module that
fails to verify.
- READMEs updated.
2026-10-27 17:53:31 +01:00
Damien Mehala
aa30110932
platforms/rocm: default rocprofiler log level to error (#992)
By default rocprofiler log level is `warning`, which can produce
unnecessary log output. Now, we override this log level to error, unless
`ROCPROFILER_LOG_LEVEL` environment variable is already set.
2026-10-27 13:01:40 +01:00
Kevin Barre
8f40b8b09f
zml/kernels: add TCL paged attention and MoE backends for Furiosa (#973)
zml: add TCL kernels and Furiosa attention and MoE backends

Introduce a typed TCL kernel DSL and MLIR bindings, and select TCL
automatically for Furiosa paged attention and BF16 SwiGLU MoE.

- Support native kernel calls, fusion, in-place outputs, device loops,
  compiler configuration, and chip-aware layouts.
- Implement standard paged attention for decode, prefill, and mixed
  batches, including causal/non-causal masks, sliding windows, custom
  scales, and per-head sinks.
- Use BF16 contractions with FP32 normalization and accumulation.
  Exclude unused KV entries before contractions and return zero for
  inactive or fully masked queries.
- Reuse prepared metadata for the optimized decode path and process
  longer contexts in page-aligned chunks.
- Add activation-stationary and blockwise fused-expert kernels.
- Fix StableHLO attention metadata for variable batch sizes and add
  host-reference, emission, and hardware regression coverage.
- Bump the Furiosa sandbox to 202610221403.283.1.d9f4f68759e1.

---------

Co-authored-by: Raphael Nahoum <raphael@zml.ai>
2026-10-26 15:47:48 +01:00
Raphael Nahoum
b71982fda7
mlir/dialects: add AIR dialect (#988)
Adds Apple's AIR dialect, for writing Metal kernels in MLIR.

- `mlir/dialects/air`: the AIR that Apple's Metal compiler emits for
MSL. Each op is named after its `air.*` intrinsic without the type
suffix (`fast_exp`, `max.u`, `simd_shuffle_xor`, `wg.barrier`,
`simdgroup_matrix_8x8_load`, ...), with fastmath flags on float ops.
Kernel arguments are described the way that compiler describes them:
builtins, buffers with access/noalias, and `air.arg_name`.
- `llvm`, `arith`: bind array and named struct types, linkage, loop
annotations, and integer overflow flags.
2026-10-26 11:41:06 +01:00
Raphael Nahoum
c74d5ff637
mlir/dialects: add TCL dialect and DSL (#986)
Support Furiosa's TCL dialect and add the builder as well to write
custom tcl kernels
2026-10-26 11:41:06 +01:00
Raphael Nahoum
3f60e626e4
mlir/dialects: split into one package per dialect and unify them (#982)
Splits the monolithic mlir/dialects library into one Bazel package per
dialect, then makes every package follow the same layout, C API and Zig
API. Each package owns its Zig bindings, its C API when upstream doesn't
provide one, and its tests. Dialect registration moves out of the core
mlir package, so a target only pulls in the dialects it actually uses.
The conventions are documented in `mlir/dialects/README.md`, with a
porting checklist and conformance checks.

Along the way:

- cute_ir is flattened, and its Cuda, CuteNVGPU and NVVM sub-dialects
become their own packages
- fly's ROCDL extension moves to fly_rocdl
- new packages for ROCDL and UB
- kernels/ is renamed to dsl/, since it holds the DSLs
- every package registers through its own registerDialect, and each DSL
through registerDialects, instead of dialect name strings
- the C and C++ follow the LLVM and MLIR C API conventions, formatted
with a new mlir/.clang-format
- BUILD files share one template
- accessors are named as nouns like in the rest of ZML
- each package tests itself in its own test.zig, including that every
bound operation is registered and that every wrapper method compiles

Fixes:

- memref atomic_rmw emitted the wrong kind, because the AtomicRMWKind
values didn't match the pinned LLVM
- the pinned Triton and Mosaic no longer have tt.cat and tpu.repeat, so
the DSL's cat and repeat now lower the way tl.cat and pltpu.repeat do
- the ROCDL bindings are completed: buffer address spaces, vector types,
waves_per_eu range

Breaking changes:

- register dialects through their package instead of
DialectHandle.fromString, and mlir.registerFuncExtensions becomes
func.registerExtensions
- accessors drop the get prefix, so getValue() becomes value()
- ttir.cat and mosaic_tpu.repeat are removed, and stablehlo pad and
mosaic_tpu sort take their options before the location
- zml.kernel.fly.rocdl becomes fly_rocdl
- Mesh.sdyMeshAttr takes a context and returns an attribute instead of
text
- func.func no longer prints public, its default visibility

---------

Co-authored-by: Steeve Morin <steeve@zml.ai>
2026-10-26 11:41:05 +01:00
Raphael Nahoum
79bcc6af03
workspace: bump plugins (#985)
Update the metal plugin to improve performance under load
2026-10-23 17:08:20 +02:00
Louis Lechevalier
2e7871a802
zml/platforms: bump ZML plugins (#984) 2026-10-23 15:30:55 +02:00
Tristan Gautié
e81b0ecdc9
llmd/tpu/ragged_paged_attn: fix for sequences with empty kv (#938)
Drafter scheduling in speculative decoding
(https://github.com/zml/monorepo/pull/685) can create empty slots with
empty kv content in full-attn layers of the drafter.

This fixes this issue by aligning the TPU logic with other platforms (eg
Triton) by avoiding to request DMA transfers of KV buffers for invalid
slots.
2026-10-22 18:07:07 +02:00
Damien Mehala
fcdef99e1f
zml/kernels: remove clamp to CuTe DSL (#983)
Seems like this is not properly lowered to CuTe DSL. Revert and
investigate later why.
2026-10-22 18:00:35 +02:00
Tristan Gautié
29b67dd10e
zml/manual_computation: translate output shapes to local before calling the callback (#980)
We fix a bug introduced in https://github.com/zml/zml/pull/925: the
output shapes passed to the local callback were left in their global
state and not translated any more to the local state.
2026-10-22 15:45:17 +02:00
Raphael Nahoum
5d85f0f521
platforms: bump plugins and stop preallocating by default on Metal (#979)
Metal does not need to pre-allocate memory
2026-10-22 14:25:28 +02:00
Damien Mehala
9658a71ec0
zml/kernels: relax MoE Fly restrictions (#978)
Remove hardcoded hidden and intermediate hidden dimensions check.
2026-10-22 13:00:00 +02:00
Corentin Godeau
93c0fd5ddb
zml/kernels/paged_attention/cute: add sm10x specific kernels (#974)
This PR adds a `sm10x` specific implementation of paged attention in
`cute`.

It replaces the gemma 4 specific introduced in #969
2026-10-22 10:48:09 +02:00
Corentin Godeau
be49a71bff
zml/kernels/paged_attention/cute: add sm120 specific kernels (#972)
This PR adds a `sm120` specific implementation of paged attention in
`cute`
2026-10-22 10:48:08 +02:00
Corentin Godeau
ddb520cf39
zml/kernels/paged_attention/cute: add SM100 specialization for gemma 4 (#969)
This PR adds a `sm10x` specific implementation for the head dim used in
Gemma 4
2026-10-22 10:48:08 +02:00
Damien Mehala
a1c9049f44
zml/kernels: relax CuTe MoE conditions (#976)
Changes:
  - introduce `clamp` function to CuTe builder
  - make swiglu limit field optional
  - add 16 and 32 reduction tile
2026-10-22 10:38:37 +02:00
Damien Mehala
b168417cce
zml/kernels: rework sparse GQA (#945)
Move sparse GQA into an standalone kernel rather than maintaining it as
a separate branch of the unified attention kernel. This separation
improves maintainability and keeps the unified attention kernel more
manageable, as previous performance optimizations had significantly
increased its complexity.

To reviewer: there's remaining work to merge the oneapi kernel into
unified attention and clean the kernels/kernels_sparse import.
2026-10-21 13:20:31 +02:00