0
Fork 0
mirror of https://github.com/zml/zml.git synced 2026-11-16 19:11:09 +00:00

Compare commits

..

47 commits

Author SHA1 Message Date
erwanvivien
1a26a051a5
zml/tensor: return a distinct tensor from toMemory on single-memory platforms (#1087)
On cpu, neuron and metal, `toMemory` returned self. Donating a result to
a device buffer and its `toMemory` copy to a host buffer then called
`reuseBuffer` twice on the same tensor id: the second donation replaced
the first, and compilation panicked because both outputs aliased the
host buffer. This broke llmd async scheduling on Metal and CPU.

Return the same value under a new tensor id instead.
2026-11-16 14:17:19 +01:00
Raphael Nahoum
8fcff8c462
workspace: bump ZML plugins (#1082)
Bump plugins to

[202611132003.375.1.7d765f57a407](2026111320.375.1.7d765f57a407)

oneAPI: weight-only FP8 and FP4 (NVFP4, MXFP4) scaled dots now run in a
Triton kernel that reads the
weights in their stored format.
2026-11-13 22:27:25 +01:00
Raphael Nahoum
f65072a740
platforms/furiosa: use all RNGD chips by default (#1083) 2026-11-13 22:09:27 +01:00
Benjamin Rabier
9c9161bac6
platforms/oneapi: update pjrt plugin and bundle libmkl_def (#1070)
Update PJRT OneAPI plugin with fixes for Triton
2026-11-13 17:59:02 +01:00
Kevin Barre
d2a69b9718
zml/moe: [refacto] consolidate Furiosa TCL kernel setup (#1066)
Separate dispatch from kernel generation. Use one factory to select BF16
and block-128 FP8 kernels by format and schedule, and reuse the shared
ActivationKind.
2026-11-13 17:34:09 +01:00
Kevin Barre
4c8f520251
zml/moe: support BF16 GEGLU-tanh on Furiosa (#1063)
Pass the requested activation to both BF16 TCL MoE kernels and compute
tanh GELU on the gate before multiplying by the up projection. This lets
Gemma 4 compile and run with its required activation.

Verified both kernel paths against a CPU reference and confirmed
coherent Gemma 4 generation on two and four RNGD cards.
2026-11-13 17:34:09 +01:00
Kevin Barre
3ecf1ff68b
zml/moe: mask nonlocal BF16 routes on Furiosa (#1062)
Replace nonlocal expert IDs with an in-range placeholder and zero their
routing weights before dispatch. This prevents the activation-stationary
and blockwise kernels from using -1 in gathers and scatters.

Verified BF16 outputs against the CPU reference and coherent
Qwen3.5-35B-A3B generation on two and four RNGD cards.
2026-11-13 17:34:08 +01:00
erwanvivien
4af281a0be
stdx/flags: cleanup flag_name using mem.replaceScalar (#1068)
Just porting 5253ea1362 from #1065
2026-11-13 15:38:46 +01:00
Damien Mehala
a0533da7b8
zml/kernel: introduce packages per DSL (#1061)
Split `kernel.zig` into one package per DSL
2026-11-13 14:13:10 +01:00
Hugo Mano
8c41b89801
zml/ops: manualComputation v2 (#1045)
Last PR for manualComputation v2 which promote as default and only one
API for manual computation. Custom calls now require to be wrapped in a
manual computation too.
2026-11-13 13:52:58 +01:00
Damien Mehala
2d45ef0804
zml/platform: fix zml$print leak (#1060) 2026-11-13 13:52:07 +01:00
Corentin Godeau
dc829cfb7d
[Modularization] zml/kernels/moe: replace options.zig with api.zig (#1059)
Also moved `LocalRouting` to `api.zig`
2026-11-13 10:50:15 +01:00
Kevin Barre
c40fa52d2d
zml/moe [Platform:tcl]: support block-128 FP8 SwiGLU experts (#1053)
- Select TCL for plain E4M3FN block-128 experts with matching BF16 or
  F32 scales. Preserve rejection of unsupported quantization.

- Quantize activations per block and apply checkpoint scales during
  native projections. Keep BF16 materialization and F32 down and
  routing accumulation.

- Reuse local expert routing and AllReduce for expert parallelism.
  Choose expert-stationary for small FP8 token counts and use the
  existing activation-stationary and blockwise recipes for larger
  counts.

- Share FP8 and BF16 dispatch preparation while retaining the token
  padding and BF16 SiLU rounding workarounds.
2026-11-12 13:47:48 +01:00
Guillaume Wenzek
8ebed54029
zml: refactor safetensors (#1048)
Refactor `zml/safetensors.zig`

* Drop metadata parsing
* refactor code to be more "object oriented", code is owned by the
`TensorRegistry`
* implement a new format `model.safetensors.full_index.json` that is
similar to `index.json` but actually has data inside it:

```
{
"metadata": {"num_tensors":96085,"total_size":510286023000},
"weight_map": {
  "aligner.w1.bias": {"filename":"model-00001-of-00048.safetensors","shape":[5120],"data_offsets":[27384,37624],"dtype":"BF16"},
  "aligner.w1.weight": {"filename":"model-00001-of-00048.safetensors","shape":[5120,9216],"data_offsets":[37624,94409464],"dtype":"BF16"},
  "aligner.w2.bias": {"filename":"model-00001-of-00048.safetensors","shape":[5120],"data_offsets":[94409464,94419704],"dtype":"BF16"},
  "aligner.w2.weight": {"filename":"model-00001-of-00048.safetensors","shape":[5120,5120],"data_offsets":[94419704,146848504],"dtype":"BF16"},
  "vision.blocks.0.attn.wo.bias": {"filename":"model-00001-of-00048.safetensors","shape":[1024],"data_offsets":[146848504,146850552],"dtype":"BF16"},
  "vision.blocks.0.attn.wo.weight": {"filename":"model-00001-of-00048.safetensors","shape":[1024,1024],"data_offsets":[146850552,148947704],"dtype":"BF16"},
  "vision.blocks.0.attn.wqkv.bias": {"filename":"model-00001-of-00048.safetensors","shape":[3072],"data_offsets":[148947704,148953848],"dtype":"BF16"},
 ...
 ```
 
 I followed conventions of `model.saftensors.index.json`
2026-11-12 13:12:25 +01:00
Raphael Nahoum
2b604ebfc6
zml/kernels/flash_attention/metal: port the kernels to the Metal DSL (#1054)
Replaces the `zml$flash_attn` custom call with Metal DSL kernels over
contiguous K/V:

- `fa_vec` + `fa_vec_reduce` for bf16 decode, at most 256 threads per
threadgroup.
- `fa_tiled` for prefill and f16.
- Batched queries work (the old path panicked). Other dtypes and head
dims fall back to sdpa.

Bumps the plugins to
[202611101651.363.1.d46c7471c419](2026111016.363.1.d46c7471c419)
2026-11-10 18:51:08 +01:00
Raphael Nahoum
9f8da93eaf
zml/kernels/paged_attention/metal: fit decode in 256 threads and split it over the GPU cores (#1046)
Fixes Metal decode on M1/M2: `fa_vec_paged` launched up to 512 threads
per threadgroup, over the 384 those GPUs accept.

- A threadgroup holds up to 4 query heads of one KV head times position
splits, at most 256 threads.
- The split count follows the PJRT core count and the number of rows.
- Head dims 32 and 64 now use the vector kernel instead of the tiled
one.

llmd decode on M5 Max vs master: no change at short context, +4–27% at
8K, +12–74% at 24K, bs16 unchanged.
Head-dim-64 models: 2–7.6× at long context.

Should fix https://github.com/zml/zml/issues/1037 but needs some testing
on a real M1/M2
2026-11-10 18:51:08 +01:00
Guillaume Wenzek
6ee4f84ea7
zml: Exe.memoryStats (#1052) 2026-11-10 14:24:41 +01:00
Louis Lechevalier
b61eede864
zml/kernels: Enable MegaMoe support in llmd (#1032)
This PR enable the MegaMoe kernel support in llmd by :
- Making the symmetric workspace an arg of CuteOps
- Adding cuda.grid_constant arg to the DSL builder, enabling GPU kernels
to receive TMA tensormap
- Adapting the existing mxfp4 kernel (non mega) to the new shared
swizzling `interleaved_8`
2026-11-10 14:23:27 +01:00
Raphael Nahoum
7b40b1b0f8
zml/io: honor slices and splat in DirectMemoryWriter.drain (#1050)
`DirectMemoryWriter.drain` only wrote the first slice of data and
ignored the splat count. The `std.Io.Writer` contract is to write every
slice in order, then repeat the last one splat times.

- A one-byte splat advanced one byte per drain call.
- After a partial write, `splatBytesAll` passes an empty first slice
followed by the pattern. The old drain returned 0 for that forever, so
the fill never finished.

`drain` now writes every slice and repeats the last one splat times.
2026-11-09 20:48:45 +01:00
Corentin Godeau
b23dbe334a
[Modularization] zml/kernels/moe: break cycle between flashinfer_cutlass and platform (#1042)
* Stopped using `zml.ops.CustomCall` because it creates cycles
impossible to break.
* renamed some `cutlass_flashinfer` to `flashinfer_cutlass` for
consistency

I didn't replace `CustomCall` because flashinfer_cutlass is the last
user and will be removed soon.
2026-11-09 19:24:38 +01:00
Corentin Godeau
2c57959202
[Modularization] zml/kernels/moe: break cycle (#1041)
`zml/kernels/moe.zig` is not imported by backend implementation anymore.

I introduced `options.zig` to contain the shared options. I also
introduced `weights.zig` to expose the unpacking helpers.
2026-11-09 19:24:38 +01:00
Corentin Godeau
b73de969eb
[Modularization] zml: break cycle in zml/kernels/paged_attention (#1039) 2026-11-09 19:24:37 +01:00
Corentin Godeau
7fbd15eee4
[Modularization] zml: break cycle in zml/attention (#1038)
Break cycle between `zml/attention/attention.zig` and
`zml/kernels/flash_attention/flash_attention.zig`
2026-11-09 19:24:37 +01:00
Hugo Mano
1042cba3ff
zml/kernels/moe: make mesh and partitioning explicit (#1028)
MOE API currently relies on input sharding metadata or
partitioning choices made inside backend kernels. Ordinary graph
operations can leave that metadata unspecified, making the resulting
sharding depend on the backend and preceding operations.

Move the manual computation boundary into the shared
`zml.kernels.moe.forwardMoe` API using `manualComputation2`. 
Callers provide a logical mesh and typed input/output partitioning specs.
2026-11-09 15:26:03 +01:00
Corentin Godeau
59a888f7e2
third_party/zls: allow all 0.17 zig versions (#1036) 2026-11-06 15:14:38 +01:00
Raphael Nahoum
5b079127bb
workspace: bump ZML plugins (#1034)
Bump plugins to
[202611060925.355.1.2ca211fa5fbd](2026110609.355.1.2ca211fa5fbd)
2026-11-06 10:56:05 +01:00
PierreLgol
488c21c899
workspace: add Helix editor configuration for Zig (#1029)
adding basic support for the helix editor, to automatically work
2026-11-06 10:29:31 +01:00
erwanvivien
837e45022a
zml/kernels/paged_attention: fix stablehlo backend slot_id increment (#1027)
If sequences were not 16-aligned we would have sequence leaks. Now we
increase slot_id by query_start_len and pad when needed
2026-11-06 09:57:09 +01:00
Corentin Godeau
630d96726b
zml/io: add consolidated version of safetensors to defaults (#1035)
Some models entrypoints are exposed as
`consolidated.safetensors.index.json` or `consolidated.safetensors`.
2026-11-05 19:34:17 +01:00
Raphael Nahoum
3285b7e0f4
zml: FP8 block activations and paged attention query padding on Furiosa and bump plugins (#1033)
- Quantize block FP8 activations to E4M3FN on Furiosa, so scaled dots
run natively. Other block sizes keep BF16 activations and XLA's
fallback.

- Pad paged attention query groups to a power of two with zero queries,
as TCC fails other group sizes. Marked as a workaround.

- Bump to
[202611051630.351.1.48d539621544](2026110516.351.1.48d539621544)
2026-11-05 17:38:31 +01:00
Corentin Godeau
5decef1fca
zml/kernels/paged_attention: enable cute by default when available (#1017) 2026-11-05 14:17:44 +01:00
Hugo Mano
2438b9c364
zml/kernels/paged_attention: make mesh and partitioning explicit (#1024)
Paged attention currently relies on input sharding metadata or
partitioning choices made inside backend kernels. Ordinary graph
operations can leave that metadata unspecified, making the resulting
sharding depend on the backend and preceding operations.

Move the manual computation boundary into the shared `pagedAttention`
API using `manualComputation2`. Callers provide a logical mesh and typed
input/output partitioning specs.
2026-11-05 13:19:58 +01:00
Hugo Mano
47f50667d8
zml/attention: declare manual partitioning in the shared API instead of kernels (#1016)
Move manual regions out of flash attention kernels and use
`manualComputation2` for every shared attention backend.

Also fully replicate the embedding executable outputs in Qwen and Llama.
Partial graph constraints leave their batch and sequence dimensions
open, which is invalid at executable boundaries.

Fix two further blockers exposed by the Qwen run that we forgot to move
recently: softplus comparisons must use the input dtype and ROCm DNN
runfiles must include the MIOpen kernel archives.
2026-11-05 13:00:00 +01:00
Guillaume Wenzek
4baec25924
kernels/moe: remove confusing bitcast (#1030)
the bitcast discard partitioning info
the API issue is tracked by https://github.com/zml/zml/issues/914
2026-11-05 11:53:31 +01:00
Louis Lechevalier
69a6b2bb22
workspace: bump ZML plugins (#1031) 2026-11-05 11:44:39 +01:00
Damien Mehala
8898051c7b
zml/kernels: fix Triton split-K (#1012)
The split-K/3D path writes partial results only for active queries, but
the reduction kernel processes inactive queries in the padded batch.
These queries have no valid results and cause invalid
computation/outputs.

This fix zeroes the ouput for inactive queries and return before before
accessing sequence metadata.

Resolves #1006
2026-11-04 20:48:56 +01:00
Raphael Nahoum
5764202e5f
zml/kernels/paged_attention/tcl: follow sequence lengths and support several chips (#1026)
- **Work follows sequence lengths:** decode's compute no longer runs
over the block table's whole width, and flash prefill loops only over
the chunks in use.

- **One plan per call:** each call picks straight decode, the hybrid
loop or flash attention, following a calibration sweep based on what
compiles. The kernels might change once the compiler evolves.

- **Several chips:** each chip runs the kernels over its own heads
inside a manual computation.
2026-11-04 18:52:05 +01:00
Guillaume Wenzek
db119b2d35
zml.Shape: improve parseTags (#1023)
avoid `stdx.BoundedArray` for tags, use `[]const Tag`, this allows to
simplify some logic and more explicitly separate comptime work from
runtime work.
It reduces binary sizes, preventing big functions to be duplicated
because of different tags.
2026-11-04 16:04:19 +01:00
Guillaume Wenzek
8992fbe434
zml.Shape.assertTags (#1022)
we often use `withTags` to document the current shape, but it could also
hide bug by overriding previous tags.
I introduce `assertTags` that should be favored when possible
2026-11-04 16:04:18 +01:00
Guillaume Wenzek
17ba49cce8
zml.ops.manualComputation: fix error message (#1021)
use `.format` instead of default Zig formatting
2026-11-04 16:04:18 +01:00
Guillaume Wenzek
350c6c1606
zml.Mesh: axis/hasAxis helpers (#1019) 2026-11-04 16:04:17 +01:00
Guillaume Wenzek
4707e58774
zml.Compiler: allow to propagate tensor store names (#1018)
The name can appear in the IR and thanks to #1003 it shows in the web
view of the model making all compute blocks pretty obvious.

The screenshot below without names would be quite confusing, 
but now we can see this is the QKV of first layer of a Llama model.
<img width="2211" height="1208" alt="image"
src="https://github.com/user-attachments/assets/1917edb0-a772-4a1d-9aa7-ad5aaf7b5260"
/>
2026-11-04 16:04:17 +01:00
erwanvivien
194175b36d
zml/kernels/flash_attn: add prefill flash_attention kernel and add Causal (#1000)
Prefill kernel is based on libflashattn.so from zml/flash-attention
2026-11-04 14:35:55 +01:00
Hugo Mano
6a1698f270
zml/ops: add manualComputation2 with explicit partitioning (#1013)
Accept a logical mesh and typed input/output partition specs at manual
computation boundaries.

Will replace `zml.ops.manualComputation` in a future PR.
2026-11-03 18:22:59 +01:00
Hugo Mano
4c355c61b7
zml/sharding: keep unconstrained graph results open (#1004)
Ordinary graph results should have an unspecified layout until
constrained or inferred by the partitioner.

Previously, we could report replication simply because a tensor had no
mesh, even when its stored partition specs were open. Other paths
preserved selected operations’ sharding metadata, making the frontend’s
interpretation depend on how a tensor was produced.

This change gives those results a consistent open state while requiring
concrete layouts at executable boundaries.

Shardy explicitly defines the intended semantics: [“no sharding
attribute on a tensor is equivalent to a fully open tensor
sharding.”](https://openxla.org/shardy/sharding_representation#open)

Open dimensions remain available for further partitioning; closed
dimensions fix the specified layout.
Replication is therefore a constraint, whereas an open layout lets the
partitioner choose either sharding or replication:

This distinction also applies to unused mesh axes. Marking an axis
explicitly replicated prevents propagation from using it, even when a
tensor dimension is open. For example, `[{?}, {}]` must leave unused
axes available; adding replicated={"data", "model"} removes those
choices.

Keeping intermediates open lets the partitioner use constraints from
producers and consumers, with operation-specific rules for
transformations such as dot and reshape.
2026-11-03 18:11:15 +01:00
Corentin Kerisit
439ba6c0a6
bazel: register runfiles initialization from C (#1005)
ZML initializes runfiles before `main` so its loaders can locate
packaged libraries and data.

When using `-fno-llvm -fincremental` and linking with the incremental
ELF linker, declaring the registration pointer in Zig produces incorrect
`.init_array` data.

See https://codeberg.org/ziglang/zig/issues/37453

Use a C constructor on Linux and macOS so the C compiler emits the
platform-specific startup registration, while Zig exports the existing
runfiles initialization function directly.
We can revert when https://codeberg.org/ziglang/zig/issues/37453 is
fixed.

With this fix, we bypass the wrong behavior in incremental ELF linker
and get closer to ZML compliance with zig incremental using their
incremental linker.

The registration is shared by the `use_cc_common_link=true` and `false`
paths. Keep the explicit linker retention so the constructor is included
even when its object is in a static archive.
2026-11-03 14:45:23 +01:00
Benjamin Rabier
86ab457d0b
dsl/cute: add SM100 primitives and CTA-group support (#1007)
Add helpers for TMA gather/prefetch, weight-stationary MMA,
shared-memory descriptors, packed FP32 arithmetic, async cluster stores,
and register limits.

Make CTA groups and barrier scopes explicit across TMA, block-scaled
MMA, TMEM, and synchronization, including two-CTA operations and
multicast completion. Consolidate named CTA barriers and update the MoE
and paged-attention callers.

Cover instruction operands, descriptor encoding, and MLIR round-trips in
builder tests.
2026-11-03 10:26:31 +01:00
174 changed files with 15967 additions and 26314 deletions

File diff suppressed because it is too large Load diff

Some files were not shown because too many files have changed in this diff Show more