On cpu, neuron and metal, `toMemory` returned self. Donating a result to
a device buffer and its `toMemory` copy to a host buffer then called
`reuseBuffer` twice on the same tensor id: the second donation replaced
the first, and compilation panicked because both outputs aliased the
host buffer. This broke llmd async scheduling on Metal and CPU.
Return the same value under a new tensor id instead.
Bump plugins to
[202611132003.375.1.7d765f57a407](2026111320.375.1.7d765f57a407)
oneAPI: weight-only FP8 and FP4 (NVFP4, MXFP4) scaled dots now run in a
Triton kernel that reads the
weights in their stored format.
Separate dispatch from kernel generation. Use one factory to select BF16
and block-128 FP8 kernels by format and schedule, and reuse the shared
ActivationKind.
Pass the requested activation to both BF16 TCL MoE kernels and compute
tanh GELU on the gate before multiplying by the up projection. This lets
Gemma 4 compile and run with its required activation.
Verified both kernel paths against a CPU reference and confirmed
coherent Gemma 4 generation on two and four RNGD cards.
Replace nonlocal expert IDs with an in-range placeholder and zero their
routing weights before dispatch. This prevents the activation-stationary
and blockwise kernels from using -1 in gathers and scatters.
Verified BF16 outputs against the CPU reference and coherent
Qwen3.5-35B-A3B generation on two and four RNGD cards.
Last PR for manualComputation v2 which promote as default and only one
API for manual computation. Custom calls now require to be wrapped in a
manual computation too.
- Select TCL for plain E4M3FN block-128 experts with matching BF16 or
F32 scales. Preserve rejection of unsupported quantization.
- Quantize activations per block and apply checkpoint scales during
native projections. Keep BF16 materialization and F32 down and
routing accumulation.
- Reuse local expert routing and AllReduce for expert parallelism.
Choose expert-stationary for small FP8 token counts and use the
existing activation-stationary and blockwise recipes for larger
counts.
- Share FP8 and BF16 dispatch preparation while retaining the token
padding and BF16 SiLU rounding workarounds.
Refactor `zml/safetensors.zig`
* Drop metadata parsing
* refactor code to be more "object oriented", code is owned by the
`TensorRegistry`
* implement a new format `model.safetensors.full_index.json` that is
similar to `index.json` but actually has data inside it:
```
{
"metadata": {"num_tensors":96085,"total_size":510286023000},
"weight_map": {
"aligner.w1.bias": {"filename":"model-00001-of-00048.safetensors","shape":[5120],"data_offsets":[27384,37624],"dtype":"BF16"},
"aligner.w1.weight": {"filename":"model-00001-of-00048.safetensors","shape":[5120,9216],"data_offsets":[37624,94409464],"dtype":"BF16"},
"aligner.w2.bias": {"filename":"model-00001-of-00048.safetensors","shape":[5120],"data_offsets":[94409464,94419704],"dtype":"BF16"},
"aligner.w2.weight": {"filename":"model-00001-of-00048.safetensors","shape":[5120,5120],"data_offsets":[94419704,146848504],"dtype":"BF16"},
"vision.blocks.0.attn.wo.bias": {"filename":"model-00001-of-00048.safetensors","shape":[1024],"data_offsets":[146848504,146850552],"dtype":"BF16"},
"vision.blocks.0.attn.wo.weight": {"filename":"model-00001-of-00048.safetensors","shape":[1024,1024],"data_offsets":[146850552,148947704],"dtype":"BF16"},
"vision.blocks.0.attn.wqkv.bias": {"filename":"model-00001-of-00048.safetensors","shape":[3072],"data_offsets":[148947704,148953848],"dtype":"BF16"},
...
```
I followed conventions of `model.saftensors.index.json`
Replaces the `zml$flash_attn` custom call with Metal DSL kernels over
contiguous K/V:
- `fa_vec` + `fa_vec_reduce` for bf16 decode, at most 256 threads per
threadgroup.
- `fa_tiled` for prefill and f16.
- Batched queries work (the old path panicked). Other dtypes and head
dims fall back to sdpa.
Bumps the plugins to
[202611101651.363.1.d46c7471c419](2026111016.363.1.d46c7471c419)
Fixes Metal decode on M1/M2: `fa_vec_paged` launched up to 512 threads
per threadgroup, over the 384 those GPUs accept.
- A threadgroup holds up to 4 query heads of one KV head times position
splits, at most 256 threads.
- The split count follows the PJRT core count and the number of rows.
- Head dims 32 and 64 now use the vector kernel instead of the tiled
one.
llmd decode on M5 Max vs master: no change at short context, +4–27% at
8K, +12–74% at 24K, bs16 unchanged.
Head-dim-64 models: 2–7.6× at long context.
Should fix https://github.com/zml/zml/issues/1037 but needs some testing
on a real M1/M2
This PR enable the MegaMoe kernel support in llmd by :
- Making the symmetric workspace an arg of CuteOps
- Adding cuda.grid_constant arg to the DSL builder, enabling GPU kernels
to receive TMA tensormap
- Adapting the existing mxfp4 kernel (non mega) to the new shared
swizzling `interleaved_8`
`DirectMemoryWriter.drain` only wrote the first slice of data and
ignored the splat count. The `std.Io.Writer` contract is to write every
slice in order, then repeat the last one splat times.
- A one-byte splat advanced one byte per drain call.
- After a partial write, `splatBytesAll` passes an empty first slice
followed by the pattern. The old drain returned 0 for that forever, so
the fill never finished.
`drain` now writes every slice and repeats the last one splat times.
* Stopped using `zml.ops.CustomCall` because it creates cycles
impossible to break.
* renamed some `cutlass_flashinfer` to `flashinfer_cutlass` for
consistency
I didn't replace `CustomCall` because flashinfer_cutlass is the last
user and will be removed soon.
`zml/kernels/moe.zig` is not imported by backend implementation anymore.
I introduced `options.zig` to contain the shared options. I also
introduced `weights.zig` to expose the unpacking helpers.
MOE API currently relies on input sharding metadata or
partitioning choices made inside backend kernels. Ordinary graph
operations can leave that metadata unspecified, making the resulting
sharding depend on the backend and preceding operations.
Move the manual computation boundary into the shared
`zml.kernels.moe.forwardMoe` API using `manualComputation2`.
Callers provide a logical mesh and typed input/output partitioning specs.
2026-11-09 15:26:03 +01:00
100 changed files with 7589 additions and 5720 deletions