0
Fork 0
mirror of https://github.com/zml/zml.git synced 2026-11-18 18:57:29 +00:00
Commit graph

1,078 commits

Author SHA1 Message Date
Raphael Nahoum
567434be79
zml/nn: Linear.quantizationScheme (#772) 2026-09-04 19:13:27 +02:00
Tristan Gautié
6b4b790a00
nn/gated_delta_net: bound GDN recurrence (#769)
Fixing an issue raised in https://github.com/zml/zml/pull/754, to bound
the GDN recurrence in the Qwen3.5 example model to the number of active
tokens.

We also make qwen3.5 and qwen3.5-moe slightly more consistent.
2026-09-04 12:45:57 +02:00
Guillaume Wenzek
59dfe4216b
zml.Buffer.freeDeviceBufferMemoryButKeepHostMetadataMemory (#760) 2026-09-04 10:24:26 +02:00
Tristan Gautié
0b41272a08
attention/flashattn: fix sliding window off-by-one error (#763)
We fix an off-by-one error in Flash-Attn: `window_size_left` is
inclusive of the current position, so it is minus 1 compared to the
actual `sliding_window` size.

Ref:
https://github.com/Dao-AILab/flash-attention#how-to-use-flashattention
2026-09-03 18:26:40 +02:00
Benjamin Rabier
54bf33a02f
zml/Compiler: Provide accurate device memory size to XLA (#764)
By default, XLA assumes 80% of the memory to be available during
compilation. It wasn't an issue as long as we had one executable per
layer, because most of the weights weren't parts of the inputs, but for
a full forward executable we exceed that 80% threshold in llmd with the
cache being sized to use almost everything up to the BFC limit. So XLA
ends up generating an executable the least possible temporary buffers
even though it has more than that.
2026-09-03 17:17:09 +02:00
Raphael Nahoum
1b1cfe89ab
zml/nn: unpack FP4 along an explicit axis (#767)
Make unpackFp4 accept the packed source axis explicitly instead of
requiring the hardcoded .kw tag.

Updates the Linear and MoE callers and adds regression coverage for
expanding and renaming the packed axis.
2026-09-03 17:15:45 +02:00
Damien Mehala
e97abbaad4
zml/ops: refactor manualComputation (#757)
Simplify `manualComputation` by merging `inputs` and `body_context` into a single `input` argument.
Internally, collect all and shard all tensors (including `?Tensor`) from `input` before invoking the body function.
2026-09-03 11:02:59 +02:00
Corentin Godeau
67ac9c7feb
zml/moe/moe: handle optional scales (#761)
While waiting for #757, this fixes triton moe for mxfp4
2026-09-02 16:37:56 +02:00
Olivier Boulant
de5850a99c
ci: prevent RunOn runner stealing (#759)
Give each concurrent RunsOn job a unique routing label using the
workflow run ID, run attempt, and job name. This prevents runners
provisioned for one matrix job from being assigned to another compatible
job.
- [Runner
stealing](https://runs-on.com/docs/maintenance/troubleshooting/#runner-stealing)

---

Replace the deprecated image-level `disk: large` setting with
`volume=80gb` on each RunsOn job. Storage belongs to the runner
configuration rather than the custom image definition.
- [RunsOn storage labels](https://runs-on.com/docs/runners/labels/#disk)
- [Custom runners and images](https://runs-on.com/docs/runners/custom/)
2026-09-02 13:51:29 +02:00
Raphael Nahoum
3178484b27
zml/io: record computed bindings in the store (#749)
A binding created through `maybeCreateBinding` is computed from its
sources by an executable and must never be streamed from the checkpoint.
`Loader.defaultCallback` inferred that from `sources.len != 1`, which a
single-source binding fails resulting to the raw bytes being loaded and
producing wrong results
2026-09-02 12:51:09 +02:00
nzinci
5d0686de02
zml/safetensors: close model discovery handles (#758)
Model discovery leaves its safetensors entrypoint and repository handles
open after building the tensor registry, including when parsing fails.

Close handles owned by `fromRepo` and `fromPath`, and close repositories
at the existing `fromRepo` call sites.
2026-09-02 09:10:18 +02:00
Damien Mehala
e1e983c802
zml/moe: support Flashinfer cutlass MoE for sm 10.3 (#740)
Update flashinfer cutlass moe to retrieve sm103 compiled .so. 
Unpack forwardMoe abi by removing the Nvfp4Scales type.

---------

Co-authored-by: Louis Lechevalier <louis@zml.ai>
Co-authored-by: Corentin Godeau <corentin@zml.ai>
2026-09-01 16:11:57 +02:00
Benjamin Rabier
6e5829a061
zml: Use zml.nn.Linear for forwardMoe and rename quantization structs (#725)
`forwardMoe` is effectively consuming a `gate_up` and `down` linear MLP,
it "just" only applies parts of it.
I've renamed:
```
Quant -> Quantization
QuantSchem -> Quantization.Scheme
TensorScale -> Quantization.GlobalScale
Direction { multiplier, divisor } -> Operation { multiply, divide }
```
and moved it outside of zml.nn.Linear

This makes it easier in LLMD to just load zml.nn.Linear
2026-09-01 14:17:48 +02:00
Steeve Morin
de134385c5 zml/vfs: handle invalid GCS json
we do not support "external_account", but for now at least do not crash when we
can't parse the JSON file.
2026-09-01 11:23:43 +02:00
nzinci
325cf2185e
zml/tensor: fix triangularSolve batched RHS contract (#753)
`Tensor.triangularSolve` currently rejects batched inputs and derives
the result shape from the coefficient matrix.

Allow tensors with rank 2 or greater and preserve the right-hand side's
shape in the result.

This was found while working on a separate Qwen3.5 GDN optimization.

Tests:

- `bazel test //zml:test`
- `bazel build //zml`
2026-09-01 11:15:08 +02:00
Guillaume Wenzek
6e171a5ef9
zml: remove implicit donation [bugfix] (#751)
in #738 I accidentally introduced implicit donation `fn foo(x: Tensor)
Tensor { return x; }` would not make a copy of the input tensor but
return the same pointers.

I'm not sure an implicit copies is better than an implicit aliasing, but
fixing since this is an un-planned breaking change.

Also fixing a compiler error for `.items(bool)` after #724.

Note: this is an argument for snapshot testing: the CI should check if a
given PR changes the emitted IR for a given model.
2026-08-31 17:01:49 +02:00
Guillaume Wenzek
a46e4cac06
zml: handle packed fp4 on CPU side (#724)
The XLA API assumes that all sub-bytes data is given to it unpacked.
But when you read safetensors files the data is packed.

My proposition is the following: when we have a zml.fp4 tensor/buffer we
say to pjrt it's a u8 buffer instead.
Since PJRT infers the input/output shape from stablehlo I had to
introduce some bitcast at the edge of the compilation:
- fp4 input are declared as u8, but I insert bitcast operand before
calling the zig code, so the model code only see the fp4 tensors.
- when the model code returns a fp4 output, I bitcast to u8 before
exiting the final compilation

Breaking changes: none
Currently fp4 can't cross the pjrt boundary, so all fp4 code already
manually lie about the data being u8, and it will keep working. In the
future this code will be able to be "upgraded" by explicitly tagging the
input as fp4 and remove the "unpacking" code from their model code.

Note: the tracking XLA issue seems pretty cold
https://github.com/openxla/xla/issues/16795
2026-08-31 13:21:52 +02:00
Guillaume Wenzek
fcb0d2a9c1
zml.Compiler: cleanup (#738)
* use multiArrayList in `emitMlirFunc` to store information about
input/output tensors.
* merge OutputInfo and InputInfo: The code is collecting same info for
input/output, some code was merge.
`emitMlirFunc` is now clearly split between the function specific
wrapper, and the part that works only with slices of tensor info.
* zml.Sharding: clarify error sources, added more error messages. Most
`zml.Sharding` function can fail because they resolve the sharding on
the fly. Changed some APIs to explicitly take a resolved sharding,
leaving only OOM errors to handle. I've put a TODO for you @hugomano
about GSPMD sharding that can still fails pretty late when generating
the attribute. I feel this should be checked when constructing the
Partitionning.
* add Tensor.Id instead of old `usize` this allows accidental confusion
or operation on type id.
* improve Compiler.Scope API: `pushBlock` now returns the `scope` this
remove the old pattern of `pushBlock(); const scope = currentScope();`
2026-08-31 13:19:52 +02:00
Damien Mehala
fdcdade631
zml/moe: fix manualComputation scales usage (#743) 2026-08-28 19:38:39 +02:00
Corentin Godeau
54c0f16d22
zml/attention/triton_attention: use specific block_m on GB300 (#742)
Investigation has been done with the help of a clanker. The problem
might manifest on other blackwell hardware but for the time being,
simply gate to GB300.

@elogir If you can add some details
2026-08-28 18:18:46 +02:00
Benjamin Rabier
450abb6131
zml/attention: optimize sparse MLA value accumulation (#737)
GLM 5.2 represents each MLA query/key head as 512 latent dimensions
followed by 64 RoPE dimensions. All 576 dimensions participate in the
attention score, but the value is only the 512-dimensional latent
prefix.

Previously, sparse MLA accumulated a 576-dimensional context and sliced
off the RoPE suffix afterward. Return the 512-dimensional value context
directly instead. This also reduces the padded value accumulator from
1024 to 512 elements.

Make the value width explicit so the same implementation also supports
models such as DeepSeek V4, where the RoPE dimensions are included in
the value.

Keys and values share the same latent cache. Remove the redundant value
pointer and strides, reuse the loaded key tile for value accumulation,
and remove paging arguments that are no longer needed after converting
top-k indices to physical cache positions.

Add a grouped partial-reduction path for the high split counts that
improve occupancy on MI300, and tune the shape-based launch selection
more aggressively for GB300.
2026-08-28 15:38:11 +02:00
Guillaume Wenzek
cd477b858a
zml: refactor module.zig into Compiler.zig (#736)
Migration instructions:

zml.module.CompilationContext -> zml.Compiler
zml.module.CompileError -> zml.Compiler.Error
zml.module.CompilationOptions -> zml.Compiler.Options
zml.module.compile -> zml.Platform.compiler or zml.Compiler.compile

Breaking changes: none, I've kept a small compatibility layer at the top
level, with upgrade instructions.

We can remove it in a second time.
2026-08-27 18:31:50 +02:00
Guillaume Wenzek
805cf8cd74
zml/moe: fixup triton.zig (#734)
fix after conflict between #729 and #730
2026-08-27 17:32:20 +02:00
Guillaume Wenzek
082c48b211
zml: refactor slice API (#729)
## Overview

* merge `dynamicSlice` and `slice` into `slices`
* merge `dynamicSlice1d` and `slice1d` into `slice` 
* drop `choose`, `choose1d`
* handle singleton for dynamic slices too
* typed API so that it's easier to specify arguments

## Migration guide:

```diff
// new .slice accepts all old .slice1d inputs 
- const x1 = x.slice1d(-1, .{ .start = 0, .end = half_dim });
+ const x1 = x.slice(-1, .{ .start = 0, .end = half_dim });

- k.dynamicSlice(.{ .layer = zml.Tensor.DynSlice{ .start = cache_index, .len = 1 } }).squeeze(.layer);
+ k.slice(.layer, .dynSingle(cache_index));

- k.dynamicSlice(.{
-    .layer = zml.Tensor.DynSlice{ .start = cache_index, .len = 1 }, 
- }).squeeze(.layer).choose1d(.k, kv_head);
+ k.slice(.{ .layer, .k }, &.{ .dynSingle(cache_index), .single(kv_head) });

- o.choose1d(ax, 5),
+ o.slice(ax, .single(5))
```

Now the `Tensor.Slice` struct is bigger and now as a Tensor in it, which
may break user code if your model was storing a `Slice` inside it,
because now `zml.Bufferize(Model)` will contain one more `?Buffer`. You
can either pass an extra `null` to the `zml.Exe` or refactor to remove
`Slice` out of `Model`. I haven't seen this happen so far. The idea is
that `Slice` will mostly be a stack only struct, only created to call
`slice`
2026-08-27 16:06:36 +02:00
Damien Mehala
8cb020ca43
zml/moe: introduce A16WMXFP4 kernel (#730)
Changes:
  - add `Options` struct to the MoE API.
  - `num_experts_per_tok` field is now mandatory.
  - updated qwen 3.5 moe to use the new api.
2026-08-27 13:58:21 +02:00
Damien Mehala
a05d7f920a
mlir/func: support up to 64 results (#731) 2026-08-27 11:13:09 +02:00
nzinci
163a387673
zml/attention: fix latent KV cache update (#727)
Latent `KvCache.update` currently passes one tuple to `scatterSlices`,
so the branch fails to compile when instantiated.

Pass the three arguments directly and add a latent-cache compile
regression test.

Tests:

- `bazel test --jobs=2 //zml:test`
- `bazel build --jobs=2 //zml`
2026-08-26 16:54:27 +02:00
Guillaume Wenzek
3fe735f491
zml: mlir known types (#722)
cache base MLIR type corresponding to zml.DataType
2026-08-26 16:54:04 +02:00
Guillaume Wenzek
98f20d0ec3
zml.Slice: improve formatting (#723) 2026-08-26 10:50:14 +02:00
Guillaume Wenzek
a452efe972
zig 0.17: replace ** by splat (#728)
update in prevision of Zig 0.17
2026-08-26 10:49:19 +02:00
Damien Mehala
c0dcc7cf7f
zml/nn: detect deepseek-like MXFP4 quantization (#726) 2026-08-26 10:34:47 +02:00
Guillaume Wenzek
1b6a3cb395
zml/testing: apply close fraction across strided slices (#721)
correctly accumulates error statistics across the full Buffer instead of checking each sub - slice independently.
2026-08-25 16:47:49 +02:00
Benjamin Rabier
baa6bb4354
zml/attention/mla: Add 3d kernel (#717)
On 8 GPUs, we ended up only using one SM per GPU without 3d attention.
2026-08-24 15:03:59 +00:00
Raphael Nahoum
007be52c54
zml/nn: load MXFP8 and MXFP4 checkpoints (#719) 2026-08-24 16:17:41 +02:00
Damien Mehala
c8cae615c5
zml/tensor: add constant remainder and substraction helpers (#716) 2026-08-24 13:54:54 +02:00
Damien Mehala
60bd1ee233
zml/attention: add sparse MLA (#714) 2026-08-24 13:40:08 +02:00
Damien Mehala
b132a65010
zml: support F8_E8M0 tensors (#715)
Adds safetensors parsing and MLIR type support for F8_E8M0 tensors
2026-08-24 12:58:05 +02:00
Raphael Nahoum
f223acccb7
zml/nn: accept an f32 per-channel FP8 scale (#711) 2026-08-20 13:13:31 +02:00
Louis Lechevalier
533afb8915
zml/moe: flashinfer nvfp4 x nvfp4 moe backend (#707)
This PR add the support of nvfp4 x nvfp4 moe backend on sm120 / sm100
(natively supported nvfp4 hardware). It :
- retrieves the released artifact cutlass moe flashinfer :
https://github.com/zml/flashinfer/releases/tag/cutlass-moe-v0.1.0
- loads the .so
- implements the nvfp4 x nvfp4 backend (added to existing triton and
mosaic tpu).
2026-08-20 11:51:32 +02:00
Benjamin Rabier
12211f3765
zml/exe: borrowed Exe in Runner (#708)
It's a bit of a mess to transfer ownership of the Exe to the runner as
that needs careful handling of defers. Fundamentally there is also no
particular reason why the Runner should own the Exe and it was not
consistent across zml.Exe and zml.TypedExe. Now it's always borrowed.

Renaming `zml.TypedExe` to `zml.FnExe` because it's shorter. I've
removed all the aliases `type LayerExe = zml.TypedExe(...)` it's useless
with ZLS, it's a lot easier to have `zml.FnExe(<function>)`
2026-08-20 10:56:32 +02:00
Raphael Nahoum
19f562560d
zml/nn: classify quantization schemes and serve them on Linear (#710)
`QuantScheme` classifies a checkpoint's scheme from the weight and scale
shapes rather than from key names, and `Linear.forward` serves it:

| scheme | stored as |
|---|---|
| NVFP4 | `f4e2m1`, or two values per byte in `u8` |
| FP8 per-channel | `f8e4m3fn` + `[N, 1]` bf16 |
| FP8 block-128 | `f8e4m3fn` + `[N/128, K/128]` bf16 |
2026-08-19 16:25:50 +02:00
Raphael Nahoum
5eff1ed501
zml/moe: NVFP4 and block-FP8 Metal custom calls (#709)
Emits `__metal$moe_gemm` / `$f4` / `$f8` so a MoE layer can run bf16,
NVFP4 or DeepSeek block-FP8 (`f8e4m3fn` weights + bf16 128×128 scales)

`forwardMoe` takes two new optional arguments, `w1_global_scale` and
`w2_global_scale`. The per-expert global (`weight_scale_2`) is applied
in-graph after the GEMM.
2026-08-19 15:43:42 +02:00
Benjamin Rabier
c58d81a7b3
zml/exe: TypedExe (#697)
in LLMD and //examples/llm, we compile multiple executable, re-use the
args & results. We do all of this manually, in addition we don't have
any type-safety on the arguments/outputs which is a bit too easy to
misuse IMHO.

So adding two abstractions:
- Runner: Packs the args, exe and results together to avoid calling
`args.set`, `exe.run` and `results.fill` with similar arguments. Today
it owns the Exe. It doensn't strictly needs it, but it's simpler and
works for us for now.
- TypedExe: Type-safe version of an Exe, expecting a single input &
ouptut to have keyed arguments to have fully unambiguous arguments &
results mapping.

I've updated //example/llm with this new pattern.

Also adding an explicit `zml.module.CompileError`.
2026-08-18 16:36:38 +02:00
Kevin Barre
bb860a050b
platforms/oneapi: bundle oneDNN and TBB with PJRT update PJRT_ONEAPI_RELEASE (#706)
platforms/oneapi: bundle oneDNN and TBB with PJRT
Add oneDNN 2026.0 and TBB 2023.1 to the hermetic oneAPI packages.
the pjrj plugin need it for the mistral models type with the support of
the FP8 scaled matmul with libdnn.

platforms/oneapi: update PJRT_ONEAPI_RELEASE and SHA256 for new artifact
version
2026-08-18 09:56:05 +02:00
Raphael Nahoum
d539f6bdb8
zml/io: wait for loadExecute call before deiniting input buffers (#705)
Sometimes there is a race where the input buffers get deinited at the
end of loadExecute before the executable call finishes, which can
corrupt data.

Waiting on the call closes that race, but it also prevents concurrency,
so we may need another way to keep the buffers alive without blocking.
2026-08-17 19:11:16 +02:00
Kevin Barre
49a12235bb
zml/pjrt: propagate ready-event failures and validate opaque pointers (#703)
Fix two PJRT error-handling cases that could allow execution to continue
after a backend failure.

- Use the same error conversion and destruction path for ready-event and
callback errors.
- Reject a successful `PJRT_Buffer_OpaqueDeviceMemoryDataPointer` result
when `device_memory_ptr` is null.
2026-08-17 15:16:33 +02:00
Louis Lechevalier
067602b241
zml/shape: Effective byte size on device (#699)
This PR introduces deviceByteSize which computes the byte size of a shape on device i.e. with packed elements when they are less than 8 bits.
2026-08-17 09:41:28 +02:00
David Zbarsky
a37f903c3f
[build] bump rules_swift to 3.6.1 (#640)
This prevent eager toolchain load that resulted in 10ish seconds of
`swiftc` calls
2026-08-03 03:56:42 +02:00
David Zbarsky
154b212caa
build: disable default Xcode discovery (#630) 2026-08-03 09:35:09 +09:00
Kevin Barre
039fc9516a
zml/cuda : update PJRT CUDA asset SHA256 checksums for amd64 and arm64 (#700) 2026-07-31 20:08:08 +02:00