Fixing an issue raised in https://github.com/zml/zml/pull/754, to bound
the GDN recurrence in the Qwen3.5 example model to the number of active
tokens.
We also make qwen3.5 and qwen3.5-moe slightly more consistent.
By default, XLA assumes 80% of the memory to be available during
compilation. It wasn't an issue as long as we had one executable per
layer, because most of the weights weren't parts of the inputs, but for
a full forward executable we exceed that 80% threshold in llmd with the
cache being sized to use almost everything up to the BFC limit. So XLA
ends up generating an executable the least possible temporary buffers
even though it has more than that.
Make unpackFp4 accept the packed source axis explicitly instead of
requiring the hardcoded .kw tag.
Updates the Linear and MoE callers and adds regression coverage for
expanding and renaming the packed axis.
Simplify `manualComputation` by merging `inputs` and `body_context` into a single `input` argument.
Internally, collect all and shard all tensors (including `?Tensor`) from `input` before invoking the body function.
Give each concurrent RunsOn job a unique routing label using the
workflow run ID, run attempt, and job name. This prevents runners
provisioned for one matrix job from being assigned to another compatible
job.
- [Runner
stealing](https://runs-on.com/docs/maintenance/troubleshooting/#runner-stealing)
---
Replace the deprecated image-level `disk: large` setting with
`volume=80gb` on each RunsOn job. Storage belongs to the runner
configuration rather than the custom image definition.
- [RunsOn storage labels](https://runs-on.com/docs/runners/labels/#disk)
- [Custom runners and images](https://runs-on.com/docs/runners/custom/)
A binding created through `maybeCreateBinding` is computed from its
sources by an executable and must never be streamed from the checkpoint.
`Loader.defaultCallback` inferred that from `sources.len != 1`, which a
single-source binding fails resulting to the raw bytes being loaded and
producing wrong results
Model discovery leaves its safetensors entrypoint and repository handles
open after building the tensor registry, including when parsing fails.
Close handles owned by `fromRepo` and `fromPath`, and close repositories
at the existing `fromRepo` call sites.
`forwardMoe` is effectively consuming a `gate_up` and `down` linear MLP,
it "just" only applies parts of it.
I've renamed:
```
Quant -> Quantization
QuantSchem -> Quantization.Scheme
TensorScale -> Quantization.GlobalScale
Direction { multiplier, divisor } -> Operation { multiply, divide }
```
and moved it outside of zml.nn.Linear
This makes it easier in LLMD to just load zml.nn.Linear
`Tensor.triangularSolve` currently rejects batched inputs and derives
the result shape from the coefficient matrix.
Allow tensors with rank 2 or greater and preserve the right-hand side's
shape in the result.
This was found while working on a separate Qwen3.5 GDN optimization.
Tests:
- `bazel test //zml:test`
- `bazel build //zml`
in #738 I accidentally introduced implicit donation `fn foo(x: Tensor)
Tensor { return x; }` would not make a copy of the input tensor but
return the same pointers.
I'm not sure an implicit copies is better than an implicit aliasing, but
fixing since this is an un-planned breaking change.
Also fixing a compiler error for `.items(bool)` after #724.
Note: this is an argument for snapshot testing: the CI should check if a
given PR changes the emitted IR for a given model.
The XLA API assumes that all sub-bytes data is given to it unpacked.
But when you read safetensors files the data is packed.
My proposition is the following: when we have a zml.fp4 tensor/buffer we
say to pjrt it's a u8 buffer instead.
Since PJRT infers the input/output shape from stablehlo I had to
introduce some bitcast at the edge of the compilation:
- fp4 input are declared as u8, but I insert bitcast operand before
calling the zig code, so the model code only see the fp4 tensors.
- when the model code returns a fp4 output, I bitcast to u8 before
exiting the final compilation
Breaking changes: none
Currently fp4 can't cross the pjrt boundary, so all fp4 code already
manually lie about the data being u8, and it will keep working. In the
future this code will be able to be "upgraded" by explicitly tagging the
input as fp4 and remove the "unpacking" code from their model code.
Note: the tracking XLA issue seems pretty cold
https://github.com/openxla/xla/issues/16795
* use multiArrayList in `emitMlirFunc` to store information about
input/output tensors.
* merge OutputInfo and InputInfo: The code is collecting same info for
input/output, some code was merge.
`emitMlirFunc` is now clearly split between the function specific
wrapper, and the part that works only with slices of tensor info.
* zml.Sharding: clarify error sources, added more error messages. Most
`zml.Sharding` function can fail because they resolve the sharding on
the fly. Changed some APIs to explicitly take a resolved sharding,
leaving only OOM errors to handle. I've put a TODO for you @hugomano
about GSPMD sharding that can still fails pretty late when generating
the attribute. I feel this should be checked when constructing the
Partitionning.
* add Tensor.Id instead of old `usize` this allows accidental confusion
or operation on type id.
* improve Compiler.Scope API: `pushBlock` now returns the `scope` this
remove the old pattern of `pushBlock(); const scope = currentScope();`
Investigation has been done with the help of a clanker. The problem
might manifest on other blackwell hardware but for the time being,
simply gate to GB300.
@elogir If you can add some details
GLM 5.2 represents each MLA query/key head as 512 latent dimensions
followed by 64 RoPE dimensions. All 576 dimensions participate in the
attention score, but the value is only the 512-dimensional latent
prefix.
Previously, sparse MLA accumulated a 576-dimensional context and sliced
off the RoPE suffix afterward. Return the 512-dimensional value context
directly instead. This also reduces the padded value accumulator from
1024 to 512 elements.
Make the value width explicit so the same implementation also supports
models such as DeepSeek V4, where the RoPE dimensions are included in
the value.
Keys and values share the same latent cache. Remove the redundant value
pointer and strides, reuse the loaded key tile for value accumulation,
and remove paging arguments that are no longer needed after converting
top-k indices to physical cache positions.
Add a grouped partial-reduction path for the high split counts that
improve occupancy on MI300, and tune the shape-based launch selection
more aggressively for GB300.
Migration instructions:
zml.module.CompilationContext -> zml.Compiler
zml.module.CompileError -> zml.Compiler.Error
zml.module.CompilationOptions -> zml.Compiler.Options
zml.module.compile -> zml.Platform.compiler or zml.Compiler.compile
Breaking changes: none, I've kept a small compatibility layer at the top
level, with upgrade instructions.
We can remove it in a second time.
## Overview
* merge `dynamicSlice` and `slice` into `slices`
* merge `dynamicSlice1d` and `slice1d` into `slice`
* drop `choose`, `choose1d`
* handle singleton for dynamic slices too
* typed API so that it's easier to specify arguments
## Migration guide:
```diff
// new .slice accepts all old .slice1d inputs
- const x1 = x.slice1d(-1, .{ .start = 0, .end = half_dim });
+ const x1 = x.slice(-1, .{ .start = 0, .end = half_dim });
- k.dynamicSlice(.{ .layer = zml.Tensor.DynSlice{ .start = cache_index, .len = 1 } }).squeeze(.layer);
+ k.slice(.layer, .dynSingle(cache_index));
- k.dynamicSlice(.{
- .layer = zml.Tensor.DynSlice{ .start = cache_index, .len = 1 },
- }).squeeze(.layer).choose1d(.k, kv_head);
+ k.slice(.{ .layer, .k }, &.{ .dynSingle(cache_index), .single(kv_head) });
- o.choose1d(ax, 5),
+ o.slice(ax, .single(5))
```
Now the `Tensor.Slice` struct is bigger and now as a Tensor in it, which
may break user code if your model was storing a `Slice` inside it,
because now `zml.Bufferize(Model)` will contain one more `?Buffer`. You
can either pass an extra `null` to the `zml.Exe` or refactor to remove
`Slice` out of `Model`. I haven't seen this happen so far. The idea is
that `Slice` will mostly be a stack only struct, only created to call
`slice`
Latent `KvCache.update` currently passes one tuple to `scatterSlices`,
so the branch fails to compile when instantiated.
Pass the three arguments directly and add a latent-cache compile
regression test.
Tests:
- `bazel test --jobs=2 //zml:test`
- `bazel build --jobs=2 //zml`
This PR add the support of nvfp4 x nvfp4 moe backend on sm120 / sm100
(natively supported nvfp4 hardware). It :
- retrieves the released artifact cutlass moe flashinfer :
https://github.com/zml/flashinfer/releases/tag/cutlass-moe-v0.1.0
- loads the .so
- implements the nvfp4 x nvfp4 backend (added to existing triton and
mosaic tpu).
It's a bit of a mess to transfer ownership of the Exe to the runner as
that needs careful handling of defers. Fundamentally there is also no
particular reason why the Runner should own the Exe and it was not
consistent across zml.Exe and zml.TypedExe. Now it's always borrowed.
Renaming `zml.TypedExe` to `zml.FnExe` because it's shorter. I've
removed all the aliases `type LayerExe = zml.TypedExe(...)` it's useless
with ZLS, it's a lot easier to have `zml.FnExe(<function>)`
`QuantScheme` classifies a checkpoint's scheme from the weight and scale
shapes rather than from key names, and `Linear.forward` serves it:
| scheme | stored as |
|---|---|
| NVFP4 | `f4e2m1`, or two values per byte in `u8` |
| FP8 per-channel | `f8e4m3fn` + `[N, 1]` bf16 |
| FP8 block-128 | `f8e4m3fn` + `[N/128, K/128]` bf16 |
Emits `__metal$moe_gemm` / `$f4` / `$f8` so a MoE layer can run bf16,
NVFP4 or DeepSeek block-FP8 (`f8e4m3fn` weights + bf16 128×128 scales)
`forwardMoe` takes two new optional arguments, `w1_global_scale` and
`w2_global_scale`. The per-expert global (`weight_scale_2`) is applied
in-graph after the GEMM.
in LLMD and //examples/llm, we compile multiple executable, re-use the
args & results. We do all of this manually, in addition we don't have
any type-safety on the arguments/outputs which is a bit too easy to
misuse IMHO.
So adding two abstractions:
- Runner: Packs the args, exe and results together to avoid calling
`args.set`, `exe.run` and `results.fill` with similar arguments. Today
it owns the Exe. It doensn't strictly needs it, but it's simpler and
works for us for now.
- TypedExe: Type-safe version of an Exe, expecting a single input &
ouptut to have keyed arguments to have fully unambiguous arguments &
results mapping.
I've updated //example/llm with this new pattern.
Also adding an explicit `zml.module.CompileError`.
platforms/oneapi: bundle oneDNN and TBB with PJRT
Add oneDNN 2026.0 and TBB 2023.1 to the hermetic oneAPI packages.
the pjrj plugin need it for the mistral models type with the support of
the FP8 scaled matmul with libdnn.
platforms/oneapi: update PJRT_ONEAPI_RELEASE and SHA256 for new artifact
version
Sometimes there is a race where the input buffers get deinited at the
end of loadExecute before the executable call finishes, which can
corrupt data.
Waiting on the call closes that race, but it also prevents concurrency,
so we may need another way to keep the buffers alive without blocking.
Fix two PJRT error-handling cases that could allow execution to continue
after a backend failure.
- Use the same error conversion and destruction path for ready-event and
callback errors.
- Reject a successful `PJRT_Buffer_OpaqueDeviceMemoryDataPointer` result
when `device_memory_ptr` is null.