- Select TCL for plain E4M3FN block-128 experts with matching BF16 or
F32 scales. Preserve rejection of unsupported quantization.
- Quantize activations per block and apply checkpoint scales during
native projections. Keep BF16 materialization and F32 down and
routing accumulation.
- Reuse local expert routing and AllReduce for expert parallelism.
Choose expert-stationary for small FP8 token counts and use the
existing activation-stationary and blockwise recipes for larger
counts.
- Share FP8 and BF16 dispatch preparation while retaining the token
padding and BF16 SiLU rounding workarounds.
Refactor `zml/safetensors.zig`
* Drop metadata parsing
* refactor code to be more "object oriented", code is owned by the
`TensorRegistry`
* implement a new format `model.safetensors.full_index.json` that is
similar to `index.json` but actually has data inside it:
```
{
"metadata": {"num_tensors":96085,"total_size":510286023000},
"weight_map": {
"aligner.w1.bias": {"filename":"model-00001-of-00048.safetensors","shape":[5120],"data_offsets":[27384,37624],"dtype":"BF16"},
"aligner.w1.weight": {"filename":"model-00001-of-00048.safetensors","shape":[5120,9216],"data_offsets":[37624,94409464],"dtype":"BF16"},
"aligner.w2.bias": {"filename":"model-00001-of-00048.safetensors","shape":[5120],"data_offsets":[94409464,94419704],"dtype":"BF16"},
"aligner.w2.weight": {"filename":"model-00001-of-00048.safetensors","shape":[5120,5120],"data_offsets":[94419704,146848504],"dtype":"BF16"},
"vision.blocks.0.attn.wo.bias": {"filename":"model-00001-of-00048.safetensors","shape":[1024],"data_offsets":[146848504,146850552],"dtype":"BF16"},
"vision.blocks.0.attn.wo.weight": {"filename":"model-00001-of-00048.safetensors","shape":[1024,1024],"data_offsets":[146850552,148947704],"dtype":"BF16"},
"vision.blocks.0.attn.wqkv.bias": {"filename":"model-00001-of-00048.safetensors","shape":[3072],"data_offsets":[148947704,148953848],"dtype":"BF16"},
...
```
I followed conventions of `model.saftensors.index.json`
Replaces the `zml$flash_attn` custom call with Metal DSL kernels over
contiguous K/V:
- `fa_vec` + `fa_vec_reduce` for bf16 decode, at most 256 threads per
threadgroup.
- `fa_tiled` for prefill and f16.
- Batched queries work (the old path panicked). Other dtypes and head
dims fall back to sdpa.
Bumps the plugins to
[202611101651.363.1.d46c7471c419](2026111016.363.1.d46c7471c419)
Fixes Metal decode on M1/M2: `fa_vec_paged` launched up to 512 threads
per threadgroup, over the 384 those GPUs accept.
- A threadgroup holds up to 4 query heads of one KV head times position
splits, at most 256 threads.
- The split count follows the PJRT core count and the number of rows.
- Head dims 32 and 64 now use the vector kernel instead of the tiled
one.
llmd decode on M5 Max vs master: no change at short context, +4–27% at
8K, +12–74% at 24K, bs16 unchanged.
Head-dim-64 models: 2–7.6× at long context.
Should fix https://github.com/zml/zml/issues/1037 but needs some testing
on a real M1/M2
This PR enable the MegaMoe kernel support in llmd by :
- Making the symmetric workspace an arg of CuteOps
- Adding cuda.grid_constant arg to the DSL builder, enabling GPU kernels
to receive TMA tensormap
- Adapting the existing mxfp4 kernel (non mega) to the new shared
swizzling `interleaved_8`
`DirectMemoryWriter.drain` only wrote the first slice of data and
ignored the splat count. The `std.Io.Writer` contract is to write every
slice in order, then repeat the last one splat times.
- A one-byte splat advanced one byte per drain call.
- After a partial write, `splatBytesAll` passes an empty first slice
followed by the pattern. The old drain returned 0 for that forever, so
the fill never finished.
`drain` now writes every slice and repeats the last one splat times.
* Stopped using `zml.ops.CustomCall` because it creates cycles
impossible to break.
* renamed some `cutlass_flashinfer` to `flashinfer_cutlass` for
consistency
I didn't replace `CustomCall` because flashinfer_cutlass is the last
user and will be removed soon.
`zml/kernels/moe.zig` is not imported by backend implementation anymore.
I introduced `options.zig` to contain the shared options. I also
introduced `weights.zig` to expose the unpacking helpers.
MOE API currently relies on input sharding metadata or
partitioning choices made inside backend kernels. Ordinary graph
operations can leave that metadata unspecified, making the resulting
sharding depend on the backend and preceding operations.
Move the manual computation boundary into the shared
`zml.kernels.moe.forwardMoe` API using `manualComputation2`.
Callers provide a logical mesh and typed input/output partitioning specs.
- Quantize block FP8 activations to E4M3FN on Furiosa, so scaled dots
run natively. Other block sizes keep BF16 activations and XLA's
fallback.
- Pad paged attention query groups to a power of two with zero queries,
as TCC fails other group sizes. Marked as a workaround.
- Bump to
[202611051630.351.1.48d539621544](2026110516.351.1.48d539621544)
Paged attention currently relies on input sharding metadata or
partitioning choices made inside backend kernels. Ordinary graph
operations can leave that metadata unspecified, making the resulting
sharding depend on the backend and preceding operations.
Move the manual computation boundary into the shared `pagedAttention`
API using `manualComputation2`. Callers provide a logical mesh and typed
input/output partitioning specs.
Move manual regions out of flash attention kernels and use
`manualComputation2` for every shared attention backend.
Also fully replicate the embedding executable outputs in Qwen and Llama.
Partial graph constraints leave their batch and sequence dimensions
open, which is invalid at executable boundaries.
Fix two further blockers exposed by the Qwen run that we forgot to move
recently: softplus comparisons must use the input dtype and ROCm DNN
runfiles must include the MIOpen kernel archives.
The split-K/3D path writes partial results only for active queries, but
the reduction kernel processes inactive queries in the padded batch.
These queries have no valid results and cause invalid
computation/outputs.
This fix zeroes the ouput for inactive queries and return before before
accessing sequence metadata.
Resolves#1006
- **Work follows sequence lengths:** decode's compute no longer runs
over the block table's whole width, and flash prefill loops only over
the chunks in use.
- **One plan per call:** each call picks straight decode, the hybrid
loop or flash attention, following a calibration sweep based on what
compiles. The kernels might change once the compiler evolves.
- **Several chips:** each chip runs the kernels over its own heads
inside a manual computation.
avoid `stdx.BoundedArray` for tags, use `[]const Tag`, this allows to
simplify some logic and more explicitly separate comptime work from
runtime work.
It reduces binary sizes, preventing big functions to be duplicated
because of different tags.
we often use `withTags` to document the current shape, but it could also
hide bug by overriding previous tags.
I introduce `assertTags` that should be favored when possible
The name can appear in the IR and thanks to #1003 it shows in the web
view of the model making all compute blocks pretty obvious.
The screenshot below without names would be quite confusing,
but now we can see this is the QKV of first layer of a Llama model.
<img width="2211" height="1208" alt="image"
src="https://github.com/user-attachments/assets/1917edb0-a772-4a1d-9aa7-ad5aaf7b5260"
/>
Accept a logical mesh and typed input/output partition specs at manual
computation boundaries.
Will replace `zml.ops.manualComputation` in a future PR.
Ordinary graph results should have an unspecified layout until
constrained or inferred by the partitioner.
Previously, we could report replication simply because a tensor had no
mesh, even when its stored partition specs were open. Other paths
preserved selected operations’ sharding metadata, making the frontend’s
interpretation depend on how a tensor was produced.
This change gives those results a consistent open state while requiring
concrete layouts at executable boundaries.
Shardy explicitly defines the intended semantics: [“no sharding
attribute on a tensor is equivalent to a fully open tensor
sharding.”](https://openxla.org/shardy/sharding_representation#open)
Open dimensions remain available for further partitioning; closed
dimensions fix the specified layout.
Replication is therefore a constraint, whereas an open layout lets the
partitioner choose either sharding or replication:
This distinction also applies to unused mesh axes. Marking an axis
explicitly replicated prevents propagation from using it, even when a
tensor dimension is open. For example, `[{?}, {}]` must leave unused
axes available; adding replicated={"data", "model"} removes those
choices.
Keeping intermediates open lets the partitioner use constraints from
producers and consumers, with operation-specific rules for
transformations such as dot and reshape.
ZML initializes runfiles before `main` so its loaders can locate
packaged libraries and data.
When using `-fno-llvm -fincremental` and linking with the incremental
ELF linker, declaring the registration pointer in Zig produces incorrect
`.init_array` data.
See https://codeberg.org/ziglang/zig/issues/37453
Use a C constructor on Linux and macOS so the C compiler emits the
platform-specific startup registration, while Zig exports the existing
runfiles initialization function directly.
We can revert when https://codeberg.org/ziglang/zig/issues/37453 is
fixed.
With this fix, we bypass the wrong behavior in incremental ELF linker
and get closer to ZML compliance with zig incremental using their
incremental linker.
The registration is shared by the `use_cc_common_link=true` and `false`
paths. Keep the explicit linker retention so the constructor is included
even when its object is in a static archive.
Add helpers for TMA gather/prefetch, weight-stationary MMA,
shared-memory descriptors, packed FP32 arithmetic, async cluster stores,
and register limits.
Make CTA groups and barrier scopes explicit across TMA, block-scaled
MMA, TMEM, and synchronization, including two-CTA operations and
multicast completion. Consolidate named CTA barriers and update the MoE
and paged-attention callers.
Cover instruction operands, descriptor encoding, and MLIR round-trips in
builder tests.
This PR port the CuTe MoE for nvfp4 x nvfp4 from cutlass flashinfer into
zml. It uses the CuTe zig DSL. It is the first PR of a serie that aim to
replace the covering of flashinfer cutlass in ZML. The final goal is to
remove the flashinfer cutlass binding. It begins by removing the binding
of the bf16 kernel on sm90 (never used, we have triton).
Mistral Large 4's Tekken tokenizer uses positive lookahead to split
number runs into groups of three from the right. IREE's regex compiler
rejects this pattern, preventing the tokenizer from loading.
Patch IREE to recognize the exact isolated Split pattern, match complete
number runs with its DFA, and emit the original groups using Unicode
codepoint counts. Preserve progress when output fills during streaming
and finalization. Register the patch in ZML's IREE repository rule.
Treat RESOURCE_EXHAUSTED as normal encoder backpressure instead of
logging errors before retrying. Add regressions for numeric token IDs,
Unicode numbers, byte-at-a-time input, and output overflow.
- Turns preallocation back on by default on Metal, as before #979: not
preallocating seems to cause memory issues we haven't looked into yet.
- Bumps the PJRT plugins to `202610290951.309.1.785550677e75`.
Moves Metal paged attention from the `zml$paged_attn` custom call to
Metal DSL ports of the same kernels.
- `fa_vec_paged`, with a split-K reduce, handles causal bf16 decode.
`paged_attention_tiled` and `zero_padding_rows` handle everything else.
- The shared paged attention test now runs on Metal. Triton counts as
available only on CUDA, ROCm and oneAPI. StableHLO is the reference on
Metal for the causal case.
Adds a Zig DSL for Metal kernels over the AIR dialect, and moves the
Metal paged attention kernels onto it.
- `mlir/dialects/air`: a `thread` address space. Each op has an
`llvmBuilder` that calls its `air.*` builtin with Apple's attributes,
and the dialect's translation interface builds the `!air.kernel`
metadata, so lowering to LLVM IR is MLIR's own translation. The verifier
rejects what Apple's compiler can't take: bool, scalable and n-D
vectors, buffers of non-Metal scalars, scalar f32 converts, and
`air.kernel` on a declaration.
- `dsl/metal`: an MSL-shaped builder. It emits `air` plus the
`arith`/`cf`/`vector`/`llvm` IR that clang produces for MSL, and covers
MSL's math, SIMD-group, barrier and `simdgroup_matrix` functions.
- `zml.kernel.metal` / `ops.metal`: wrap a kernel into a
`__gpu$xla.gpu.air` custom call. XLA lowers it, compiles it to a
metallib and launches it.
Enable two flags for ROCm:
- multi output fusion: allows to search for fusion opportunity
generating different output.
- latency hiding scheduler: overlap async communication with
computation.
Bump to resolve a compiler bug encountered on LFM3.
LFM3 broadcasts one sigmoid gate per attention head. This generated code
of sigmoid introduced a BF16 conversion and conditional select. The
compiler then lower it to a scalar computation which had a
register-aligment bug. The fix introduced in 2.41.11
<5d92530a07>
enforces alignment.
The DSLs and the kernels no longer make ops or set dialect attributes by
name: each dialect package gets the builder or helper they were missing,
and every consumer goes through it. This now fix ZML core files as well
Makes kernels written in the DSLs read alike, without changing what they
emit.
- Every DSL target is `//dsl/<name>:builder`, and `dsl/common` is
imported as common.
- Scopes without values end with `close()`.
- The scf DSLs (triton, mosaic, cute, fly) share
`openFor`/`openIf`/`openIfElse`/`openWhile` in `dsl/common`. cuda_tile
keeps its own scopes with the same shapes.
- Scopes take tuples or arrays, and result types can be dtypes (.f32),
Values, or MLIR types.
- Typed constants are `constant(dtype, value)`, alongside constIndex and
constantLike.
- Comparisons and bitcasts have the same names in every DSL: Fly gains
`lt`/`eq`/…, and every DSL spells it `value.bitCast(dtype)`.
- One block stack and one `common.verify`, which logs the module that
fails to verify.
- READMEs updated.
By default rocprofiler log level is `warning`, which can produce
unnecessary log output. Now, we override this log level to error, unless
`ROCPROFILER_LOG_LEVEL` environment variable is already set.
zml: add TCL kernels and Furiosa attention and MoE backends
Introduce a typed TCL kernel DSL and MLIR bindings, and select TCL
automatically for Furiosa paged attention and BF16 SwiGLU MoE.
- Support native kernel calls, fusion, in-place outputs, device loops,
compiler configuration, and chip-aware layouts.
- Implement standard paged attention for decode, prefill, and mixed
batches, including causal/non-causal masks, sliding windows, custom
scales, and per-head sinks.
- Use BF16 contractions with FP32 normalization and accumulation.
Exclude unused KV entries before contractions and return zero for
inactive or fully masked queries.
- Reuse prepared metadata for the optimized decode path and process
longer contexts in page-aligned chunks.
- Add activation-stationary and blockwise fused-expert kernels.
- Fix StableHLO attention metadata for variable batch sizes and add
host-reference, emission, and hardware regression coverage.
- Bump the Furiosa sandbox to 202610221403.283.1.d9f4f68759e1.
---------
Co-authored-by: Raphael Nahoum <raphael@zml.ai>