0
Fork 0
mirror of https://github.com/zml/zml.git synced 2026-11-16 19:11:09 +00:00

Compare commits

..

1,042 commits

Author SHA1 Message Date
erwanvivien
1a26a051a5
zml/tensor: return a distinct tensor from toMemory on single-memory platforms (#1087)
On cpu, neuron and metal, `toMemory` returned self. Donating a result to
a device buffer and its `toMemory` copy to a host buffer then called
`reuseBuffer` twice on the same tensor id: the second donation replaced
the first, and compilation panicked because both outputs aliased the
host buffer. This broke llmd async scheduling on Metal and CPU.

Return the same value under a new tensor id instead.
2026-11-16 14:17:19 +01:00
Raphael Nahoum
8fcff8c462
workspace: bump ZML plugins (#1082)
Bump plugins to

[202611132003.375.1.7d765f57a407](2026111320.375.1.7d765f57a407)

oneAPI: weight-only FP8 and FP4 (NVFP4, MXFP4) scaled dots now run in a
Triton kernel that reads the
weights in their stored format.
2026-11-13 22:27:25 +01:00
Raphael Nahoum
f65072a740
platforms/furiosa: use all RNGD chips by default (#1083) 2026-11-13 22:09:27 +01:00
Benjamin Rabier
9c9161bac6
platforms/oneapi: update pjrt plugin and bundle libmkl_def (#1070)
Update PJRT OneAPI plugin with fixes for Triton
2026-11-13 17:59:02 +01:00
Kevin Barre
d2a69b9718
zml/moe: [refacto] consolidate Furiosa TCL kernel setup (#1066)
Separate dispatch from kernel generation. Use one factory to select BF16
and block-128 FP8 kernels by format and schedule, and reuse the shared
ActivationKind.
2026-11-13 17:34:09 +01:00
Kevin Barre
4c8f520251
zml/moe: support BF16 GEGLU-tanh on Furiosa (#1063)
Pass the requested activation to both BF16 TCL MoE kernels and compute
tanh GELU on the gate before multiplying by the up projection. This lets
Gemma 4 compile and run with its required activation.

Verified both kernel paths against a CPU reference and confirmed
coherent Gemma 4 generation on two and four RNGD cards.
2026-11-13 17:34:09 +01:00
Kevin Barre
3ecf1ff68b
zml/moe: mask nonlocal BF16 routes on Furiosa (#1062)
Replace nonlocal expert IDs with an in-range placeholder and zero their
routing weights before dispatch. This prevents the activation-stationary
and blockwise kernels from using -1 in gathers and scatters.

Verified BF16 outputs against the CPU reference and coherent
Qwen3.5-35B-A3B generation on two and four RNGD cards.
2026-11-13 17:34:08 +01:00
erwanvivien
4af281a0be
stdx/flags: cleanup flag_name using mem.replaceScalar (#1068)
Just porting 5253ea1362 from #1065
2026-11-13 15:38:46 +01:00
Damien Mehala
a0533da7b8
zml/kernel: introduce packages per DSL (#1061)
Split `kernel.zig` into one package per DSL
2026-11-13 14:13:10 +01:00
Hugo Mano
8c41b89801
zml/ops: manualComputation v2 (#1045)
Last PR for manualComputation v2 which promote as default and only one
API for manual computation. Custom calls now require to be wrapped in a
manual computation too.
2026-11-13 13:52:58 +01:00
Damien Mehala
2d45ef0804
zml/platform: fix zml$print leak (#1060) 2026-11-13 13:52:07 +01:00
Corentin Godeau
dc829cfb7d
[Modularization] zml/kernels/moe: replace options.zig with api.zig (#1059)
Also moved `LocalRouting` to `api.zig`
2026-11-13 10:50:15 +01:00
Kevin Barre
c40fa52d2d
zml/moe [Platform:tcl]: support block-128 FP8 SwiGLU experts (#1053)
- Select TCL for plain E4M3FN block-128 experts with matching BF16 or
  F32 scales. Preserve rejection of unsupported quantization.

- Quantize activations per block and apply checkpoint scales during
  native projections. Keep BF16 materialization and F32 down and
  routing accumulation.

- Reuse local expert routing and AllReduce for expert parallelism.
  Choose expert-stationary for small FP8 token counts and use the
  existing activation-stationary and blockwise recipes for larger
  counts.

- Share FP8 and BF16 dispatch preparation while retaining the token
  padding and BF16 SiLU rounding workarounds.
2026-11-12 13:47:48 +01:00
Guillaume Wenzek
8ebed54029
zml: refactor safetensors (#1048)
Refactor `zml/safetensors.zig`

* Drop metadata parsing
* refactor code to be more "object oriented", code is owned by the
`TensorRegistry`
* implement a new format `model.safetensors.full_index.json` that is
similar to `index.json` but actually has data inside it:

```
{
"metadata": {"num_tensors":96085,"total_size":510286023000},
"weight_map": {
  "aligner.w1.bias": {"filename":"model-00001-of-00048.safetensors","shape":[5120],"data_offsets":[27384,37624],"dtype":"BF16"},
  "aligner.w1.weight": {"filename":"model-00001-of-00048.safetensors","shape":[5120,9216],"data_offsets":[37624,94409464],"dtype":"BF16"},
  "aligner.w2.bias": {"filename":"model-00001-of-00048.safetensors","shape":[5120],"data_offsets":[94409464,94419704],"dtype":"BF16"},
  "aligner.w2.weight": {"filename":"model-00001-of-00048.safetensors","shape":[5120,5120],"data_offsets":[94419704,146848504],"dtype":"BF16"},
  "vision.blocks.0.attn.wo.bias": {"filename":"model-00001-of-00048.safetensors","shape":[1024],"data_offsets":[146848504,146850552],"dtype":"BF16"},
  "vision.blocks.0.attn.wo.weight": {"filename":"model-00001-of-00048.safetensors","shape":[1024,1024],"data_offsets":[146850552,148947704],"dtype":"BF16"},
  "vision.blocks.0.attn.wqkv.bias": {"filename":"model-00001-of-00048.safetensors","shape":[3072],"data_offsets":[148947704,148953848],"dtype":"BF16"},
 ...
 ```
 
 I followed conventions of `model.saftensors.index.json`
2026-11-12 13:12:25 +01:00
Raphael Nahoum
2b604ebfc6
zml/kernels/flash_attention/metal: port the kernels to the Metal DSL (#1054)
Replaces the `zml$flash_attn` custom call with Metal DSL kernels over
contiguous K/V:

- `fa_vec` + `fa_vec_reduce` for bf16 decode, at most 256 threads per
threadgroup.
- `fa_tiled` for prefill and f16.
- Batched queries work (the old path panicked). Other dtypes and head
dims fall back to sdpa.

Bumps the plugins to
[202611101651.363.1.d46c7471c419](2026111016.363.1.d46c7471c419)
2026-11-10 18:51:08 +01:00
Raphael Nahoum
9f8da93eaf
zml/kernels/paged_attention/metal: fit decode in 256 threads and split it over the GPU cores (#1046)
Fixes Metal decode on M1/M2: `fa_vec_paged` launched up to 512 threads
per threadgroup, over the 384 those GPUs accept.

- A threadgroup holds up to 4 query heads of one KV head times position
splits, at most 256 threads.
- The split count follows the PJRT core count and the number of rows.
- Head dims 32 and 64 now use the vector kernel instead of the tiled
one.

llmd decode on M5 Max vs master: no change at short context, +4–27% at
8K, +12–74% at 24K, bs16 unchanged.
Head-dim-64 models: 2–7.6× at long context.

Should fix https://github.com/zml/zml/issues/1037 but needs some testing
on a real M1/M2
2026-11-10 18:51:08 +01:00
Guillaume Wenzek
6ee4f84ea7
zml: Exe.memoryStats (#1052) 2026-11-10 14:24:41 +01:00
Louis Lechevalier
b61eede864
zml/kernels: Enable MegaMoe support in llmd (#1032)
This PR enable the MegaMoe kernel support in llmd by :
- Making the symmetric workspace an arg of CuteOps
- Adding cuda.grid_constant arg to the DSL builder, enabling GPU kernels
to receive TMA tensormap
- Adapting the existing mxfp4 kernel (non mega) to the new shared
swizzling `interleaved_8`
2026-11-10 14:23:27 +01:00
Raphael Nahoum
7b40b1b0f8
zml/io: honor slices and splat in DirectMemoryWriter.drain (#1050)
`DirectMemoryWriter.drain` only wrote the first slice of data and
ignored the splat count. The `std.Io.Writer` contract is to write every
slice in order, then repeat the last one splat times.

- A one-byte splat advanced one byte per drain call.
- After a partial write, `splatBytesAll` passes an empty first slice
followed by the pattern. The old drain returned 0 for that forever, so
the fill never finished.

`drain` now writes every slice and repeats the last one splat times.
2026-11-09 20:48:45 +01:00
Corentin Godeau
b23dbe334a
[Modularization] zml/kernels/moe: break cycle between flashinfer_cutlass and platform (#1042)
* Stopped using `zml.ops.CustomCall` because it creates cycles
impossible to break.
* renamed some `cutlass_flashinfer` to `flashinfer_cutlass` for
consistency

I didn't replace `CustomCall` because flashinfer_cutlass is the last
user and will be removed soon.
2026-11-09 19:24:38 +01:00
Corentin Godeau
2c57959202
[Modularization] zml/kernels/moe: break cycle (#1041)
`zml/kernels/moe.zig` is not imported by backend implementation anymore.

I introduced `options.zig` to contain the shared options. I also
introduced `weights.zig` to expose the unpacking helpers.
2026-11-09 19:24:38 +01:00
Corentin Godeau
b73de969eb
[Modularization] zml: break cycle in zml/kernels/paged_attention (#1039) 2026-11-09 19:24:37 +01:00
Corentin Godeau
7fbd15eee4
[Modularization] zml: break cycle in zml/attention (#1038)
Break cycle between `zml/attention/attention.zig` and
`zml/kernels/flash_attention/flash_attention.zig`
2026-11-09 19:24:37 +01:00
Hugo Mano
1042cba3ff
zml/kernels/moe: make mesh and partitioning explicit (#1028)
MOE API currently relies on input sharding metadata or
partitioning choices made inside backend kernels. Ordinary graph
operations can leave that metadata unspecified, making the resulting
sharding depend on the backend and preceding operations.

Move the manual computation boundary into the shared
`zml.kernels.moe.forwardMoe` API using `manualComputation2`. 
Callers provide a logical mesh and typed input/output partitioning specs.
2026-11-09 15:26:03 +01:00
Corentin Godeau
59a888f7e2
third_party/zls: allow all 0.17 zig versions (#1036) 2026-11-06 15:14:38 +01:00
Raphael Nahoum
5b079127bb
workspace: bump ZML plugins (#1034)
Bump plugins to
[202611060925.355.1.2ca211fa5fbd](2026110609.355.1.2ca211fa5fbd)
2026-11-06 10:56:05 +01:00
PierreLgol
488c21c899
workspace: add Helix editor configuration for Zig (#1029)
adding basic support for the helix editor, to automatically work
2026-11-06 10:29:31 +01:00
erwanvivien
837e45022a
zml/kernels/paged_attention: fix stablehlo backend slot_id increment (#1027)
If sequences were not 16-aligned we would have sequence leaks. Now we
increase slot_id by query_start_len and pad when needed
2026-11-06 09:57:09 +01:00
Corentin Godeau
630d96726b
zml/io: add consolidated version of safetensors to defaults (#1035)
Some models entrypoints are exposed as
`consolidated.safetensors.index.json` or `consolidated.safetensors`.
2026-11-05 19:34:17 +01:00
Raphael Nahoum
3285b7e0f4
zml: FP8 block activations and paged attention query padding on Furiosa and bump plugins (#1033)
- Quantize block FP8 activations to E4M3FN on Furiosa, so scaled dots
run natively. Other block sizes keep BF16 activations and XLA's
fallback.

- Pad paged attention query groups to a power of two with zero queries,
as TCC fails other group sizes. Marked as a workaround.

- Bump to
[202611051630.351.1.48d539621544](2026110516.351.1.48d539621544)
2026-11-05 17:38:31 +01:00
Corentin Godeau
5decef1fca
zml/kernels/paged_attention: enable cute by default when available (#1017) 2026-11-05 14:17:44 +01:00
Hugo Mano
2438b9c364
zml/kernels/paged_attention: make mesh and partitioning explicit (#1024)
Paged attention currently relies on input sharding metadata or
partitioning choices made inside backend kernels. Ordinary graph
operations can leave that metadata unspecified, making the resulting
sharding depend on the backend and preceding operations.

Move the manual computation boundary into the shared `pagedAttention`
API using `manualComputation2`. Callers provide a logical mesh and typed
input/output partitioning specs.
2026-11-05 13:19:58 +01:00
Hugo Mano
47f50667d8
zml/attention: declare manual partitioning in the shared API instead of kernels (#1016)
Move manual regions out of flash attention kernels and use
`manualComputation2` for every shared attention backend.

Also fully replicate the embedding executable outputs in Qwen and Llama.
Partial graph constraints leave their batch and sequence dimensions
open, which is invalid at executable boundaries.

Fix two further blockers exposed by the Qwen run that we forgot to move
recently: softplus comparisons must use the input dtype and ROCm DNN
runfiles must include the MIOpen kernel archives.
2026-11-05 13:00:00 +01:00
Guillaume Wenzek
4baec25924
kernels/moe: remove confusing bitcast (#1030)
the bitcast discard partitioning info
the API issue is tracked by https://github.com/zml/zml/issues/914
2026-11-05 11:53:31 +01:00
Louis Lechevalier
69a6b2bb22
workspace: bump ZML plugins (#1031) 2026-11-05 11:44:39 +01:00
Damien Mehala
8898051c7b
zml/kernels: fix Triton split-K (#1012)
The split-K/3D path writes partial results only for active queries, but
the reduction kernel processes inactive queries in the padded batch.
These queries have no valid results and cause invalid
computation/outputs.

This fix zeroes the ouput for inactive queries and return before before
accessing sequence metadata.

Resolves #1006
2026-11-04 20:48:56 +01:00
Raphael Nahoum
5764202e5f
zml/kernels/paged_attention/tcl: follow sequence lengths and support several chips (#1026)
- **Work follows sequence lengths:** decode's compute no longer runs
over the block table's whole width, and flash prefill loops only over
the chunks in use.

- **One plan per call:** each call picks straight decode, the hybrid
loop or flash attention, following a calibration sweep based on what
compiles. The kernels might change once the compiler evolves.

- **Several chips:** each chip runs the kernels over its own heads
inside a manual computation.
2026-11-04 18:52:05 +01:00
Guillaume Wenzek
db119b2d35
zml.Shape: improve parseTags (#1023)
avoid `stdx.BoundedArray` for tags, use `[]const Tag`, this allows to
simplify some logic and more explicitly separate comptime work from
runtime work.
It reduces binary sizes, preventing big functions to be duplicated
because of different tags.
2026-11-04 16:04:19 +01:00
Guillaume Wenzek
8992fbe434
zml.Shape.assertTags (#1022)
we often use `withTags` to document the current shape, but it could also
hide bug by overriding previous tags.
I introduce `assertTags` that should be favored when possible
2026-11-04 16:04:18 +01:00
Guillaume Wenzek
17ba49cce8
zml.ops.manualComputation: fix error message (#1021)
use `.format` instead of default Zig formatting
2026-11-04 16:04:18 +01:00
Guillaume Wenzek
350c6c1606
zml.Mesh: axis/hasAxis helpers (#1019) 2026-11-04 16:04:17 +01:00
Guillaume Wenzek
4707e58774
zml.Compiler: allow to propagate tensor store names (#1018)
The name can appear in the IR and thanks to #1003 it shows in the web
view of the model making all compute blocks pretty obvious.

The screenshot below without names would be quite confusing, 
but now we can see this is the QKV of first layer of a Llama model.
<img width="2211" height="1208" alt="image"
src="https://github.com/user-attachments/assets/1917edb0-a772-4a1d-9aa7-ad5aaf7b5260"
/>
2026-11-04 16:04:17 +01:00
erwanvivien
194175b36d
zml/kernels/flash_attn: add prefill flash_attention kernel and add Causal (#1000)
Prefill kernel is based on libflashattn.so from zml/flash-attention
2026-11-04 14:35:55 +01:00
Hugo Mano
6a1698f270
zml/ops: add manualComputation2 with explicit partitioning (#1013)
Accept a logical mesh and typed input/output partition specs at manual
computation boundaries.

Will replace `zml.ops.manualComputation` in a future PR.
2026-11-03 18:22:59 +01:00
Hugo Mano
4c355c61b7
zml/sharding: keep unconstrained graph results open (#1004)
Ordinary graph results should have an unspecified layout until
constrained or inferred by the partitioner.

Previously, we could report replication simply because a tensor had no
mesh, even when its stored partition specs were open. Other paths
preserved selected operations’ sharding metadata, making the frontend’s
interpretation depend on how a tensor was produced.

This change gives those results a consistent open state while requiring
concrete layouts at executable boundaries.

Shardy explicitly defines the intended semantics: [“no sharding
attribute on a tensor is equivalent to a fully open tensor
sharding.”](https://openxla.org/shardy/sharding_representation#open)

Open dimensions remain available for further partitioning; closed
dimensions fix the specified layout.
Replication is therefore a constraint, whereas an open layout lets the
partitioner choose either sharding or replication:

This distinction also applies to unused mesh axes. Marking an axis
explicitly replicated prevents propagation from using it, even when a
tensor dimension is open. For example, `[{?}, {}]` must leave unused
axes available; adding replicated={"data", "model"} removes those
choices.

Keeping intermediates open lets the partitioner use constraints from
producers and consumers, with operation-specific rules for
transformations such as dot and reshape.
2026-11-03 18:11:15 +01:00
Corentin Kerisit
439ba6c0a6
bazel: register runfiles initialization from C (#1005)
ZML initializes runfiles before `main` so its loaders can locate
packaged libraries and data.

When using `-fno-llvm -fincremental` and linking with the incremental
ELF linker, declaring the registration pointer in Zig produces incorrect
`.init_array` data.

See https://codeberg.org/ziglang/zig/issues/37453

Use a C constructor on Linux and macOS so the C compiler emits the
platform-specific startup registration, while Zig exports the existing
runfiles initialization function directly.
We can revert when https://codeberg.org/ziglang/zig/issues/37453 is
fixed.

With this fix, we bypass the wrong behavior in incremental ELF linker
and get closer to ZML compliance with zig incremental using their
incremental linker.

The registration is shared by the `use_cc_common_link=true` and `false`
paths. Keep the explicit linker retention so the constructor is included
even when its object is in a static archive.
2026-11-03 14:45:23 +01:00
Benjamin Rabier
86ab457d0b
dsl/cute: add SM100 primitives and CTA-group support (#1007)
Add helpers for TMA gather/prefetch, weight-stationary MMA,
shared-memory descriptors, packed FP32 arithmetic, async cluster stores,
and register limits.

Make CTA groups and barrier scopes explicit across TMA, block-scaled
MMA, TMEM, and synchronization, including two-CTA operations and
multicast completion. Consolidate named CTA barriers and update the MoE
and paged-attention callers.

Cover instruction operands, descriptor encoding, and MLIR round-trips in
builder tests.
2026-11-03 10:26:31 +01:00
Louis Lechevalier
2532589572
zml/moe: CuTe MoE nvfp4 on sm103 (#981)
This PR port the CuTe MoE for nvfp4 x nvfp4 from cutlass flashinfer into
zml. It uses the CuTe zig DSL. It is the first PR of a serie that aim to
replace the covering of flashinfer cutlass in ZML. The final goal is to
remove the flashinfer cutlass binding. It begins by removing the binding
of the bf16 kernel on sm90 (never used, we have triton).
2026-11-02 14:46:41 +01:00
Benjamin Rabier
d4b413f5f9
zml/tokenizer: support Mistral's numeric lookahead (#997)
Mistral Large 4's Tekken tokenizer uses positive lookahead to split
number runs into groups of three from the right. IREE's regex compiler
rejects this pattern, preventing the tokenizer from loading.

Patch IREE to recognize the exact isolated Split pattern, match complete
number runs with its DFA, and emit the original groups using Unicode
codepoint counts. Preserve progress when output fills during streaming
and finalization. Register the patch in ZML's IREE repository rule.

Treat RESOURCE_EXHAUSTED as normal encoder backpressure instead of
logging errors before retrying. Add regressions for numeric token IDs,
Unicode numbers, byte-at-a-time input, and output overflow.
2026-10-29 14:10:32 +01:00
Raphael Nahoum
ca39742f14
zml/platform: bump plugins and preallocate Metal memory again (#1001)
- Turns preallocation back on by default on Metal, as before #979: not
preallocating seems to cause memory issues we haven't looked into yet.
- Bumps the PJRT plugins to `202610290951.309.1.785550677e75`.
2026-10-29 12:52:21 +01:00
Raphael Nahoum
e4e031b2c5
zml/kernels/paged_attention/metal: port the kernels to the Metal DSL (#999)
Moves Metal paged attention from the `zml$paged_attn` custom call to
Metal DSL ports of the same kernels.

- `fa_vec_paged`, with a split-K reduce, handles causal bf16 decode.
`paged_attention_tiled` and `zero_padding_rows` handle everything else.
- The shared paged attention test now runs on Metal. Triton counts as
available only on CUDA, ROCm and oneAPI. StableHLO is the reference on
Metal for the causal case.
2026-10-29 12:52:20 +01:00
Raphael Nahoum
70f1bd4af4
dsl/metal: add the Metal DSL (#989)
Adds a Zig DSL for Metal kernels over the AIR dialect, and moves the
Metal paged attention kernels onto it.

- `mlir/dialects/air`: a `thread` address space. Each op has an
`llvmBuilder` that calls its `air.*` builtin with Apple's attributes,
and the dialect's translation interface builds the `!air.kernel`
metadata, so lowering to LLVM IR is MLIR's own translation. The verifier
rejects what Apple's compiler can't take: bool, scalable and n-D
vectors, buffers of non-Metal scalars, scalar f32 converts, and
`air.kernel` on a declaration.
- `dsl/metal`: an MSL-shaped builder. It emits `air` plus the
`arith`/`cf`/`vector`/`llvm` IR that clang produces for MSL, and covers
MSL's math, SIMD-group, barrier and `simdgroup_matrix` functions.
- `zml.kernel.metal` / `ops.metal`: wrap a kernel into a
`__gpu$xla.gpu.air` custom call. XLA lowers it, compiles it to a
metallib and launches it.
2026-10-29 12:52:20 +01:00
Damien Mehala
b725ff2481
zml/Compiler: add XLA flags to improve generated kernels for ROCm (#998)
Enable two flags for ROCm:
- multi output fusion: allows to search for fusion opportunity
generating different output.
- latency hiding scheduler: overlap async communication with
computation.
2026-10-29 11:16:03 +01:00
Damien Mehala
47e692c300
platforms/oneapi: bump to IGC 2.41.15 (#996)
Bump to resolve a compiler bug encountered on LFM3.

LFM3 broadcasts one sigmoid gate per attention head. This generated code
of sigmoid introduced a BF16 conversion and conditional select. The
compiler then lower it to a scalar computation which had a
register-aligment bug. The fix introduced in 2.41.11
<5d92530a07>
enforces alignment.
2026-10-29 11:15:39 +01:00
Raphael Nahoum
b37ba31500
mlir/dialects: build every remaining op and attribute through the dialect packages (#993)
The DSLs and the kernels no longer make ops or set dialect attributes by
name: each dialect package gets the builder or helper they were missing,
and every consumer goes through it. This now fix ZML core files as well
2026-10-27 17:53:31 +01:00
Raphael Nahoum
3736f3dc29
dsl: unify the kernel DSLs (#991)
Makes kernels written in the DSLs read alike, without changing what they
emit.

- Every DSL target is `//dsl/<name>:builder`, and `dsl/common` is
imported as common.
- Scopes without values end with `close()`.
- The scf DSLs (triton, mosaic, cute, fly) share
`openFor`/`openIf`/`openIfElse`/`openWhile` in `dsl/common`. cuda_tile
keeps its own scopes with the same shapes.
- Scopes take tuples or arrays, and result types can be dtypes (.f32),
Values, or MLIR types.
- Typed constants are `constant(dtype, value)`, alongside constIndex and
constantLike.
- Comparisons and bitcasts have the same names in every DSL: Fly gains
`lt`/`eq`/…, and every DSL spells it `value.bitCast(dtype)`.
- One block stack and one `common.verify`, which logs the module that
fails to verify.
- READMEs updated.
2026-10-27 17:53:31 +01:00
Damien Mehala
aa30110932
platforms/rocm: default rocprofiler log level to error (#992)
By default rocprofiler log level is `warning`, which can produce
unnecessary log output. Now, we override this log level to error, unless
`ROCPROFILER_LOG_LEVEL` environment variable is already set.
2026-10-27 13:01:40 +01:00
Kevin Barre
8f40b8b09f
zml/kernels: add TCL paged attention and MoE backends for Furiosa (#973)
zml: add TCL kernels and Furiosa attention and MoE backends

Introduce a typed TCL kernel DSL and MLIR bindings, and select TCL
automatically for Furiosa paged attention and BF16 SwiGLU MoE.

- Support native kernel calls, fusion, in-place outputs, device loops,
  compiler configuration, and chip-aware layouts.
- Implement standard paged attention for decode, prefill, and mixed
  batches, including causal/non-causal masks, sliding windows, custom
  scales, and per-head sinks.
- Use BF16 contractions with FP32 normalization and accumulation.
  Exclude unused KV entries before contractions and return zero for
  inactive or fully masked queries.
- Reuse prepared metadata for the optimized decode path and process
  longer contexts in page-aligned chunks.
- Add activation-stationary and blockwise fused-expert kernels.
- Fix StableHLO attention metadata for variable batch sizes and add
  host-reference, emission, and hardware regression coverage.
- Bump the Furiosa sandbox to 202610221403.283.1.d9f4f68759e1.

---------

Co-authored-by: Raphael Nahoum <raphael@zml.ai>
2026-10-26 15:47:48 +01:00
Raphael Nahoum
b71982fda7
mlir/dialects: add AIR dialect (#988)
Adds Apple's AIR dialect, for writing Metal kernels in MLIR.

- `mlir/dialects/air`: the AIR that Apple's Metal compiler emits for
MSL. Each op is named after its `air.*` intrinsic without the type
suffix (`fast_exp`, `max.u`, `simd_shuffle_xor`, `wg.barrier`,
`simdgroup_matrix_8x8_load`, ...), with fastmath flags on float ops.
Kernel arguments are described the way that compiler describes them:
builtins, buffers with access/noalias, and `air.arg_name`.
- `llvm`, `arith`: bind array and named struct types, linkage, loop
annotations, and integer overflow flags.
2026-10-26 11:41:06 +01:00
Raphael Nahoum
c74d5ff637
mlir/dialects: add TCL dialect and DSL (#986)
Support Furiosa's TCL dialect and add the builder as well to write
custom tcl kernels
2026-10-26 11:41:06 +01:00
Raphael Nahoum
3f60e626e4
mlir/dialects: split into one package per dialect and unify them (#982)
Splits the monolithic mlir/dialects library into one Bazel package per
dialect, then makes every package follow the same layout, C API and Zig
API. Each package owns its Zig bindings, its C API when upstream doesn't
provide one, and its tests. Dialect registration moves out of the core
mlir package, so a target only pulls in the dialects it actually uses.
The conventions are documented in `mlir/dialects/README.md`, with a
porting checklist and conformance checks.

Along the way:

- cute_ir is flattened, and its Cuda, CuteNVGPU and NVVM sub-dialects
become their own packages
- fly's ROCDL extension moves to fly_rocdl
- new packages for ROCDL and UB
- kernels/ is renamed to dsl/, since it holds the DSLs
- every package registers through its own registerDialect, and each DSL
through registerDialects, instead of dialect name strings
- the C and C++ follow the LLVM and MLIR C API conventions, formatted
with a new mlir/.clang-format
- BUILD files share one template
- accessors are named as nouns like in the rest of ZML
- each package tests itself in its own test.zig, including that every
bound operation is registered and that every wrapper method compiles

Fixes:

- memref atomic_rmw emitted the wrong kind, because the AtomicRMWKind
values didn't match the pinned LLVM
- the pinned Triton and Mosaic no longer have tt.cat and tpu.repeat, so
the DSL's cat and repeat now lower the way tl.cat and pltpu.repeat do
- the ROCDL bindings are completed: buffer address spaces, vector types,
waves_per_eu range

Breaking changes:

- register dialects through their package instead of
DialectHandle.fromString, and mlir.registerFuncExtensions becomes
func.registerExtensions
- accessors drop the get prefix, so getValue() becomes value()
- ttir.cat and mosaic_tpu.repeat are removed, and stablehlo pad and
mosaic_tpu sort take their options before the location
- zml.kernel.fly.rocdl becomes fly_rocdl
- Mesh.sdyMeshAttr takes a context and returns an attribute instead of
text
- func.func no longer prints public, its default visibility

---------

Co-authored-by: Steeve Morin <steeve@zml.ai>
2026-10-26 11:41:05 +01:00
Raphael Nahoum
79bcc6af03
workspace: bump plugins (#985)
Update the metal plugin to improve performance under load
2026-10-23 17:08:20 +02:00
Louis Lechevalier
2e7871a802
zml/platforms: bump ZML plugins (#984) 2026-10-23 15:30:55 +02:00
Tristan Gautié
e81b0ecdc9
llmd/tpu/ragged_paged_attn: fix for sequences with empty kv (#938)
Drafter scheduling in speculative decoding
(https://github.com/zml/monorepo/pull/685) can create empty slots with
empty kv content in full-attn layers of the drafter.

This fixes this issue by aligning the TPU logic with other platforms (eg
Triton) by avoiding to request DMA transfers of KV buffers for invalid
slots.
2026-10-22 18:07:07 +02:00
Damien Mehala
fcdef99e1f
zml/kernels: remove clamp to CuTe DSL (#983)
Seems like this is not properly lowered to CuTe DSL. Revert and
investigate later why.
2026-10-22 18:00:35 +02:00
Tristan Gautié
29b67dd10e
zml/manual_computation: translate output shapes to local before calling the callback (#980)
We fix a bug introduced in https://github.com/zml/zml/pull/925: the
output shapes passed to the local callback were left in their global
state and not translated any more to the local state.
2026-10-22 15:45:17 +02:00
Raphael Nahoum
5d85f0f521
platforms: bump plugins and stop preallocating by default on Metal (#979)
Metal does not need to pre-allocate memory
2026-10-22 14:25:28 +02:00
Damien Mehala
9658a71ec0
zml/kernels: relax MoE Fly restrictions (#978)
Remove hardcoded hidden and intermediate hidden dimensions check.
2026-10-22 13:00:00 +02:00
Corentin Godeau
93c0fd5ddb
zml/kernels/paged_attention/cute: add sm10x specific kernels (#974)
This PR adds a `sm10x` specific implementation of paged attention in
`cute`.

It replaces the gemma 4 specific introduced in #969
2026-10-22 10:48:09 +02:00
Corentin Godeau
be49a71bff
zml/kernels/paged_attention/cute: add sm120 specific kernels (#972)
This PR adds a `sm120` specific implementation of paged attention in
`cute`
2026-10-22 10:48:08 +02:00
Corentin Godeau
ddb520cf39
zml/kernels/paged_attention/cute: add SM100 specialization for gemma 4 (#969)
This PR adds a `sm10x` specific implementation for the head dim used in
Gemma 4
2026-10-22 10:48:08 +02:00
Damien Mehala
a1c9049f44
zml/kernels: relax CuTe MoE conditions (#976)
Changes:
  - introduce `clamp` function to CuTe builder
  - make swiglu limit field optional
  - add 16 and 32 reduction tile
2026-10-22 10:38:37 +02:00
Damien Mehala
b168417cce
zml/kernels: rework sparse GQA (#945)
Move sparse GQA into an standalone kernel rather than maintaining it as
a separate branch of the unified attention kernel. This separation
improves maintainability and keeps the unified attention kernel more
manageable, as previous performance optimizations had significantly
increased its complexity.

To reviewer: there's remaining work to merge the oneapi kernel into
unified attention and clean the kernels/kernels_sparse import.
2026-10-21 13:20:31 +02:00
Raphael Nahoum
24f8c2b986
examples/llm: sample only the last prompt token (#965)
The LM head normalized and projected every prompt position at prefill,
and the session read back the whole token buffer just to keep the last
token.

Every model now slices the hidden state at `active_length - 1` before
the norm, so prefill computes logits for a single position and returns a
single token. Decode writes the next token over the one it read, so it
allocates no buffer per step.

lfm2's `actual_seq_len` is renamed `active_length`, the name the other
models use.
2026-10-21 11:47:42 +02:00
Damien Mehala
21f5b1a55f
zml/kernels: vectorize KV load for triton paged attention (#971)
Make KV-cache strides compile-time constants so Triton can infer
alignment and simplify address calculations.

On Qwen3.5-35B-A3B, both 2D and 3D attention kernels now use 128-bit
vector loads for K/V, fetching eight bf16 values per instruction per
active thread instead of one.

Thanks @fior512 for your contribution 🙏

---------

Co-authored-by: fior512 <161968556+fior512@users.noreply.github.com>
2026-10-21 11:08:35 +02:00
Damien Mehala
0db5229cd4
zml/kernels: parallelize MoE expert block assignment (#961)
Assign blocks for multiple experts in parallel instead of processing
experts one at a time. Each lane writes the expert ids for its own block
range.

Reduces assignment cost from 7.6 us to 1.8 us per call.
2026-10-21 10:58:19 +02:00
Corentin Godeau
886a6b8a97
zml/kernels/paged_attention: add cute backend (#963)
This PR is the ground work that adds a `cute` backend for paged
attention.

It's not use by default yet because it's a very naive implementation
that is far from performant.

This PR also includes two targets that are useful when working on the
kernels.

The first one is `//zml/kernels/paged_attention/cute:correctness_test`.
As its name suggest, it's a correctness test that ensures the
computation done is correct and returns valid results. It's verified
against a CPU reference implementation (that is also checked against a
hardcoded truth). It relies on `Case` representing various real-life
scenarios.

The second one is `//zml/kernels/paged_attention/cute:benchmark`. It's
an executable that can be run and will output execution stats for
various `Case`. This is the target to use when working on the
performance of paged attention cute kernels.
2026-10-20 16:41:55 +02:00
Hugo Mano
6cd00a0d00
ci: add supported platforms with bumped and audited deps (#967) 2026-10-20 14:06:49 +02:00
Hugo Mano
b238e5e6e7
ci: update runs-on images and runners manifest (#968)
Needed configuration for an upcoming CI change. Also remove not used
anymore bzlcache over S3 since we use BuildBuddy.
2026-10-20 12:51:06 +02:00
Kevin Barre
e9685a3662
zml/bazel: simplify patchelf execution and fix RPATH removal (#916)
The `@patchelf` version 0.18.0 is more capable and can now construct
something like

```bash
patchelf \
  --add-needed somelib.so \
  --force-rpath \
  --set-rpath '/home/kevin/sdk/lib' \
  --rename-dynamic-symbols patched_tcc.symbols \
  --output patched_tcc/furiosa-tcc \
  somebinary
```

So now we do not need to spawn a shell and do all the cp, chmod, and the
successive patchelf command

Before 

```bash
remove_rpath = ["some/directory"]
```

`--remove-rpath` is a flag that `Removes the entire embedded search
path` and not an individual directory

will contruct something like `patchelf --remove-rpath some/directory
patched.so` but `--remove-rpath` do not take arguments

Switch to `remove_rpath = True` to ensure something like `patchelf
--remove-rpath --output patched.so original.so` the `--remove-rpath` is
now used just after the patchelf to ensure correct usage flag placement.
2026-10-20 11:49:41 +02:00
Damien Mehala
dd82a8a10b
third_party/zls: bump to 0.17.0 (#966) 2026-10-20 11:36:07 +02:00
Hugo Mano
1179f605f7
ci: remote build (#924)
Local runner growing disk usage will be handled in another PR.
2026-10-20 09:31:03 +02:00
Hugo Mano
a5dfa567d4
bazel: update to 9.3.0 (#964) 2026-10-20 09:30:12 +02:00
Arthur
231154ef14
zml/dtype: fix Value.as panicking for f4e2m1/f8e3m4/f8e4m3/f8e8m0 (#878)
`DataType.Value.as` converts a tagged-union `Value` to a native Zig type. The float branch of the inline switch lists which float dtypes it knows how to convert via `.toF32()`, but that list was never updated
when f4e2m1, f8e3m4, f8e4m3 (the plain variant, distinct from f8e4m3b11fnuz/fn/fnuz), and f8e8m0 were added to `DataType`. All four already implement `.toF32()` in zml/floats.zig, so calling `.as(f32)` on
a `Value` of any of these dtypes panics with "Unsupported conversion ...
-> f32" instead of returning the converted float.

The fix is a one-line addition of the four dtypes to the existing switch arm, the same pattern
as the other float dtypes already listed there.
2026-10-19 18:45:42 +02:00
Louis Lechevalier
94ead7094e
zml/platforms: bump ZML plugins and enable all reduce collective in cmd buffer (#959)
This enable Lamport all reduce collective if compute capabilities matched on CUDA and ROCm
2026-10-19 15:18:23 +02:00
Corentin Godeau
9c7448da3c
zml: re-structure moe kernels (#941)
Akin to #926, this PR introduces a restructuration of the moe kernels in
ZML.

No need to re-explaing the organizatiopn because it follows what has
been done in the mentioned PR.

Here is the detailed changes of this PR:
* Kernels reorganization. `zml/kernels/moe` is the place to put all the
moe implementation. There is one sub-folder per backend.
* Changed the expected shape and tags when calling the moe forward
function. It's now expecting a 2D tensor tagged `.{.t , .d}`.

Tests still passes without key modifications. The only small tweak that
was done is related to fp32 -> fp8 triton quantization on sm89. Due to
how triton lowers the convert, it was producing slightly incorrect
result. A small preparation was added to ensure the conversion is
identical on this platform.

**Future TODO**:
* Ideally, this conversion on sm89 should be gated so that we don't pay
the cost on other platforms. We don't have a way to detect the platform
in the triton `Builder` yet but it could probably be added
* Settle on a complete API that encodes all the scenario we could
encounter. One example of that is what is required for Deepseek V4.1
Flash, the moe gemms should be done in `mxfp8 x mxfp4` and we should
have a way to request that in a better way than `quantize_input: bool`.
2026-10-19 15:15:11 +02:00
Hugo Mano
c651a9de45
rbe: use GCR image registry instead of Docker Hub (#962)
GCR mirror is more reliable than fetching through the Docker
Hub
2026-10-19 12:45:42 +02:00
Damien Mehala
2a57062b29
zml/nn: softplus (#954) 2026-10-19 10:22:38 +02:00
Damien Mehala
2d516eeb78
zml/platforms: rollback to CUDA 13.3.0 (#955)
CUDA 13.4.2 ships PTAX 13.4.92, which miscompiles a valid atomic CAS in
some LLM models, causing crashses. The generated code uses a different
byte offset for the load and atomic update, so the two operations access
different memory locations.
2026-10-16 18:46:10 +02:00
Guillaume Wenzek
a7a4fcd3f8
zml/sharding: use logical meshes (#943)
## Motivation

This is the big sharding rewrite we need to achieve complex sharding.

Currently we always resolve a mesh logical axis to a mesh physical axis
(or several of them).
But on GPU we always have 1D topology, so we can't have 2D logical
meshes.
So this PR drops the logical -> physical axis bindings, and instead
store a logical -> devices binding.
This is more flexible, and has the nice side effect of simplifying
`Buffer.from`

On stablehlo side this changes how we generate the MLIR because we now
use logical meshes, and `manualComputation` uses logical axes.
(In shardy, GSPM is not affected AFAICT because it uses device ID
directly)

## Buffer.from

This also accelerates `Buffer.from` that I rewrote to be FAST.
Version | Fixed cost | Per device
-- | -- | --
origin/master | 109.6 ns | 35.39 ns
Rewrite | 35.8 ns | 4.38 ns
Improvement | 3.1× | 8.1×

## Creating mesh

I'm adding a new syntax to create LogicalMesh, to allow the more complex
meshes we want to do, and in particular, I now allow to force a specific
number of dimensions.
You can see it in action in `mnist.zig`. 

```
const dp_mp = try platform.registerMesh("DP-MP", try .init(
        .{ .data, .model },
        &.{ .low_bandwidth, .high_bandwidth },
        &.{ 2, -1 },
    ));
 
info: topology used: Mesh(name=DP-MP, partitions=4, replicas=1)
┌ Mappings (logical coordinates from physical coordinates):
├── data (low_bandwidth) -> (bus / 2)
└── model (high_bandwidth) -> (bus % 2)

┌ PhysicalMesh(platform=cpu shardable_axes={ .bus } num_devices=4)
└── [bus] 0..4 (tree)
    ├── Device(id=0, coords={ 0 })
    ├── Device(id=1, coords={ 1 })
    ├── Device(id=2, coords={ 2 })
    └── Device(id=3, coords={ 3 })

┌ Logical coordinates:
└── data=0
    └── model=0,1 -> devices=0,1
└── data=1
    └── model=0,1 -> devices=2,3
```

The previous syntax is still available, and ZML will align logical axes
to physical axes like we do before this PR.
This part as unit tests, and I checked it work as I expect on CUDA but I
would need to double check the logics still works on TPU/Neuron, because
there is a few platform specific branches in that code.

## Test plan

As part of the benchmark Codex generated a test suite with hard-coded
offsets, allowing to check that Buffer.from didn't change behavior
through the commits, checking we correctly compute each individual
shard.

TODO check that LLMD perf isn't degraded.
2026-10-16 15:27:30 +02:00
Damien Mehala
8329631da1
platforms/cuda: fix redist json checksum (#949) 2026-10-16 15:22:13 +02:00
Benjamin Rabier
c36b3f8f1e
platforms: Bump plugins for reduced Thunk Prepare/Initialize overhead (#946)
For DeepSeek 4.1-Flash on 4 GB300
```
|                                    |       current |     new |
|------------------------------------|--------------:|--------:|
| Start of XlaModule → cuGraphLaunch | 1,008 µs      | 132 µs  |
| ...of which on CPU                 | 896 µs        | 70 µs   |
| Prepare + Initialize               | ~790 µs       | ~2 µs   |
```
2026-10-16 13:43:03 +02:00
Guillaume Wenzek
7bb457ebeb
zml: refactor manualComputation (#925)
here I'm promoting a pattern I would like to extend to more of ZML:

function that takes `anytype` argument should do as little work as
possible and called a fully typed function.

I kept `manualComputation` signature to not break existing code,
I moved the code to `zml/ops/manual_computation.zig` to better
encapsulate the required helper functions.
(and `zml/ops.zig` is pretty big atm)

Similarly `Sharding.Partitioning.parse` now calls `const logical =
comptime Logical.parse(partitioning);` then `logical.applyUseAll(tags,
mesh, default_value);` which is fully typed.

I'm also documenting `manualComputation` better
2026-10-16 12:30:05 +02:00
Hugo Mano
bc9d6c2c8d
workspace: bump ZML plugins and CUDA/ROCm runtimes (#942)
Update XLA plugins to release 202610151602.227.1.e215eb02db73
for CPU, CUDA, ROCm, Metal, and Furiosa.

Updated CUDA to 13.4.3 and ROCm with TheRock 10.1.

Now we use CUDA compatibility libraries and NCCL 2.32.3 through NVIDIA's
APT repositories. Starting with CUDA 13.4's redistribution manifest
omits the generic cuda_compat package
(https://docs.nvidia.com/deploy/cuda-compatibility/why-cuda-compatibility.html).
So we resolve it through NVIDIA's APT repositories instead.
These repositories expose flat package indexes at separate x86_64
and SBSA URLs, so we extend rules_distroless to support flat sources.

For NCCL we leverage the same APT source workflow, replacing
separate cryptic wheel URLs.
2026-10-16 10:31:34 +02:00
Raphael Nahoum
e385d9c272
workspace: bump plugins to fix Metal GDN overflow (#940)
Metal GDN used to overflow on state indexes since the rework.
2026-10-15 14:45:20 +02:00
Raphael Nahoum
cddc9cf607
zml/io: load Furiosa weights through pinned host memory (#928)
Staging blocks are uninitialized `.host_pinned` buffers, and tensors are
sent with the direct writer. The plugin backs `.host_pinned` with SDK
pinned (dma-buf) memory, which the SDK recognizes by address and
transfers without pinning pages on every request.

Pinned memory comes from `/dev/dma_heap/system`, so the loading user
needs access to it.
2026-10-14 19:32:13 +02:00
Hugo Mano
016376f2f4
workspace: remove flashattn (#939) 2026-10-14 18:29:02 +02:00
Benjamin Rabier
fe25f16b46
zml/buffer: replace emptyShell by PjrtBufferMetadata.init (#935) 2026-10-14 18:09:21 +02:00
Hugo Mano
82e800dc5c
workspace: bump ZML plugins (#937)
This introduce the new tiling path support.
2026-10-14 15:18:04 +02:00
fior512
3b0f556a74
zml/nn: reducing sync in GDN decode (#932)
Fast-path inside the forward of
`GatedDeltaNet` in order to reduce syncs in decode path.

Master vs this commit:
- tok/s : 167 to 204
- cuStreamSynchronize call/tok : 37.24 to 1.24
- GPU idle/tok : 3504 us to 2240 us
2026-10-14 13:31:42 +02:00
Tristan Gautié
f83f5ece0b
zml/kernels/attention: Resolve TPU and NKI attention meshes by logical axis (#933)
We align TPU and NKI attention meshes expectation on the CuTe GPU
attention, which simply resolves the mesh from the logical axis.

This avoids expecting a hard-coded `model` mesh, which breaks in some
TPU workflows where the mesh is called `tp_mesh`.
2026-10-14 13:12:54 +02:00
Kevin Barre
a5c40b7614
zml/platform/furiosa: rename environment variables from to FURIOSA_* (#921) 2026-10-13 19:54:27 +02:00
Guillaume Wenzek
00a0af4a55
zig 0.17 (second attempt) (#923)
Bump for zig 0.17

Deps:
- uucode bumped to
1fb73433bb
- zigimg bumped to
734fa919fb
- libvaxis bumped to
14b7684272
- zio bumped to:
417757935e
- rules_zig upgraded to [version
0.17](https://github.com/hermeticbuild/rules_zig/releases/tag/v0.17.0)
- zls is using a commit from master, but they don't have yet a tagged
release so Codex did a bit of packaging. I confirmed it correctly
autocompletes on my machine
2026-10-13 17:11:38 +02:00
Corentin Godeau
8e95da3f2d
zml/kernels/paged_attention: add kernel documentation (#930) 2026-10-13 13:34:14 +02:00
Corentin Godeau
a0c358db12
zml/kernels/paged_attention: removed latent kvcache (#929)
It's not used anymore
2026-10-13 13:34:13 +02:00
Corentin Godeau
bd0886c3e5
zml: re-structure attention kernels (#926)
This PR introduces a restructuration of the kernels in ZML. This is done
to simplify and speedup bringup on new platforms.

From now on each "kernel" will have its own directory under
`zml/kernels`. "Kernel" is in quote because we need to think about it
being more of an "operation in a graph" than an actual kernel. For
example, `moe` will eventually be part of this `kernels` directory
despite being an operation potentially launching multiple actual kernels
on some hardware.

Also it's probable that in the future, this `kernels` directory will be
put outside of zml or, at least, properly separated so that we don't
have circular imports in the codebase. Currently, we have
`@import("../../../zml.zig")` in the kernel code and this is a recipe
for disaster.

On top of the reorganization, it also adds a level of indirection for
attention. Where previously `attention` was a generic term, we now have
`attention` and `flash_attention` because `flash_attention` is an
implementation of `attention`.

Here is a detailed summary of the changes introduced by this PR:
* Kernels reorganization. `flash_attention` and `paged_attention` are
now separated. Each folder has a subfolder for each backend we support.
`flash_attention` is also now a `BackendType` of `zml.attention` with
its own implementations.
* Removed `cuda_fa2` and `cuda_fa3` backend because they were obsolete.
They have been replaced by a `cute` backend that has been generated
automatically following the operation documentation, showing that this
approach of scoping kernels per DSL actually scales.
* Removed `attnd` support for now
2026-10-12 20:31:17 +02:00
Corentin Godeau
3a201a67fa
workspace: fix typo in nvim config (#927)
`mod` was not defined resulting in bazel file formatting being broken.
2026-10-12 17:55:27 +02:00
fior512
ecea83bda7
examples/llm: Qwen 3.5 Dense and MoE memory leak (#919)
Fix a memory leak in Qwen 3.5 Dense and MoE examples: 
`layer_index` was never freed.

Before (74562d5) / After
Dense 0.8B VRAM : 11079 MB vs 5461 MB
Dense 0.8B tok/s : 169 vs 198

tok/s increases because it reduces `cuMemCreate` calls from 26 to 2
per tokens
2026-10-12 11:08:52 +02:00
Hugo Mano
3500e88cf8
zml/moe: fit triton kernel launch config with LDS limit in mind (#920)
Temporary until we refactor zml.kernels
2026-10-09 17:49:32 +02:00
Corentin Godeau
c0992073fd
workspace: bump other plugins (#922) 2026-10-09 17:36:28 +02:00
Louis Lechevalier
8173a5b7f6
platforms/cuda: Bump zml CUDA plugin (#918) 2026-10-09 17:19:04 +02:00
Kevin Barre
74562d518d
zml/platform: Add furiosa platform (#871)
Co-authored-by: Raphael Nahoum <raphael@zml.ai>
2026-10-09 12:21:16 +02:00
Corentin Godeau
f82a474971
zml/moe: add backend compliance tests (#886)
This PR adds tests to the MoE backend. The comparison is done using a
StableHLO backend that has been tested against an oracle with known
value.

It's adding quite some time to the test so I don't know if we want to
keep that many tests. However, this is the only way we have to ensure
compliance across all backends and configuration.

Each backend is tested with the whole matrix it supports.

A nice side-effect of this PR is that we have a working implementation
of MoE that will work on all platforms. It's going to be slower but
allows for a quick platform bringup.
2026-10-08 16:40:59 +02:00
Benjamin Rabier
3972d4a011
zml/buffer: add Buffer.HostAccessible (#899)
`Buffer` placed in `.host_pinned` memory can be read by executables,
but the host can't address it. `Buffer.HostAccessible` gives the host
typed views over its allocations, to write inputs and read outputs in
place between executions, without staging copies. Like `Buffer`, it is
a value: executions read its `buffer` field or donate it through
`&buffer`, and `await` then picks up the handles they replaced.

- `init` takes a `Buffer.Donation`. On CPU, CUDA and ROCm, the devices
  holding a shard of an `.undonatable` buffer share one allocation: the
  first allocates it and the others read it through
  `createViewOfDeviceBuffer`, which now takes the memory the view
  belongs to; an external reference keeps the allocation alive.
  Elsewhere each device owns an allocation, and writes reach all of
  them. Each device owns its allocation of a `.donatable` buffer, so
  that no device reads another's output.
- `View(T)` addresses a row-major range of the array: `slice`, `get`,
  `set`, `fill`, `fillIota`, `copyFrom`, `copyTo`, `copyFromView`,
  `indexOfScalar`, `indexOfAny`, and `blocks`/`reverseBlocks`, which
  visit runs contiguous both in memory and in global order. Reads use
  the first replica of a region, and writes reach every replica.
  `constSlice` and `edit` expose a range as one slice: in place when
  every shard holds the whole array densely, through a scratch slice
  otherwise.

Addressing follows PJRT's layouts: row-major, strided, and tiled as
XLA's LayoutUtil::LinearIndexForNestedTiling expands them, including
TPU's `T(8,128)(2,1)` packing of bf16 and 8-bit arrays. The test
reference is a direct port of that function, checked against both
figures of https://openxla.org/xla/tiled_layout. Buffers whose shards
all hold the whole array densely, the case on every platform but TPU,
index elements directly.

The buffer is built for llmd's decode loop, which touches it thousands
of times per step:
- Reading a host pointer from PJRT takes and releases a hold on the
  buffer and allocates an external reference, about 0.9 us. Pointers
  are read at creation, and by `await` only when execution replaced the
  handles; `view` panics before that `await` instead of writing to a
  released allocation.
- The layout metadata and allocations take several KB. They are
  prepared once in a heap-allocated mapping, which views (24 B) and
  blocks borrow rather than copy.
On Qwen3.8-27B in llmd (TP=2 on GB300, batch 256, 8k context), this
takes host time between graph completion and the next Execute from
2.07 ms, when views copied the layout and donated buffers read their
pointers in every view, to 0.27-0.30 ms per step (bpftrace probes).

`initForTests` builds a buffer over host memory, for the tiled layouts
and shardings that no available platform produces. `Buffer.from` and
`Buffer.uninitialized` share the new `emptyShell` with
`HostAccessible.init` to compute the PJRT shape, placement and layout.
2026-10-08 14:52:10 +02:00
Tristan Gautié
734ec46230
workspace: bump ZML plugins to 202610081055.168.1.56e99704ada4 (#915) 2026-10-08 14:22:40 +02:00
Benjamin Rabier
728eedfbb3
zml/Compiler: Add COLLECTIVES_KERNEL to command buffers (#912)
We don't have a single command buffer in Deepseek since
`xla_gpu_experimental_use_collective_kernels=COLLECTIVE_KERNEL_ALL_REDUCE`
was added
2026-10-08 13:40:37 +02:00
Corentin Godeau
08391af750
zml/Compiler: remove conditionals from command buffer (#913)
Due to recent changes in XLA (introduced in
3cf18a01c7), the way child are
counted to decide if a conditional is expressed as a graph conditional
or kept outside has been changed.

With the new corrected way, the body of the conditional used in the
paged attention StableHLO backend was recorded as a child graph. The
problem is that when we are sharded, there are collectives inside this
child graph and XLA adds event wait/record around the actual collective
kernel. This is explicitly forbidden because
https://docs.nvidia.com/cuda/cuda-programming-guide/04-special-topics/cuda-graphs.html#conditional-node-body-graph-requirements
states:

> The graph can only contain kernel nodes, empty nodes, memcpy nodes,
memset nodes, child graph nodes, and conditional nodes.

The temporary solution is to remove `CONDITIONAL` from the
`xla_gpu_enable_command_buffer` XLA flag. In practice this reverts to
the old behavior we had when the counting was done differently.

Long term, I don't have a solution yet and. There is an environment
variable that solves it
[NCCL_GRAPH_MIXING_SUPPORT](https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/env.html#nccl-graph-mixing-support)
but I don't know if it's safe to disable it.

Maybe we could also check if the events introduced by NCCL are actually
required.
2026-10-08 12:33:34 +02:00
Raphael Nahoum
b774b7ad16
mlir/dialects: bind missing attributes and types through the C API (#905)
Some dialects (i.e. `gpu`) were missing the C bindings, this is now
done.
2026-10-08 10:39:26 +02:00
Guillaume Wenzek
edd3e0b4e1
zml/moe: avoid unecessary bitcast (#910)
let the partitioning information propagate down to manual computation
2026-10-07 22:13:30 +02:00
Olivier Boulant
daab01572f
platforms: switch LLVM libunwind packages from 23 to 22 (#907)
`apt.llvm.org` removed some links, breaking downloads from our lockfiles
and resulting in 404.

Switch CPU/CUDA to `llvm-toolchain-bookworm-22` and ROCm to
`llvm-toolchain-jammy-22`, and regenerate the lockfiles.

All test passed on:
- macOS
- 9985wx-5090x4
- gb300-2* 
- mi300

* See reverted commit
[f7eb1cb](https://github.com/zml/zml/pull/907/commits/f7eb1cbe3b715da465bd4e9cba8a2b071499b045)
@hugomano

---

It is a follow up from https://github.com/zml/zml/pull/875
2026-10-07 19:13:33 +02:00
Hugo Mano
b148f0b919
workspace: bump XLA + LLVM + IRs (#903) 2026-10-07 16:33:44 +02:00
Guillaume Wenzek
303ff3e6a4
Revert "zig 0.17 (#693)" (#909)
This reverts commit 5a18757368.

Our internal repos has a few zig patches, I need to also upgrade them
before we ugprade ZML
2026-10-07 11:34:51 +02:00
Benjamin Rabier
6c279b469e
zml/Compiler: add extra_device_memory_bytes (#843)
Adding `extra_device_memory_bytes` to the compilation options to have
XLA compute the accurate amount of memory left for the temporaries with
our engram hack.
2026-10-07 11:18:10 +02:00
Guillaume Wenzek
5a18757368
zig 0.17 (#693)
Bump for zig 0.17

Deps:
- uucode bumped to
1fb73433bb
- zio bumped to:
417757935e
- rules_zig upgraded to [version
0.17](https://github.com/hermeticbuild/rules_zig/releases/tag/v0.17.0)

There is one Codex generated patch:
- zigimg: minor fixes to allow zig 0.17 to compile it, until they
officially support
2026-10-07 11:08:06 +02:00
Tristan Gautié
948f21c26c
zml/attention: introduce new AttentionMask entry (#904)
Follow-up to https://github.com/zml/zml/pull/888:

We introduce a new `.standard` entry to `AttentionMask` enum, that
accepts both flags `causal` and `sliding_window`. This makes the
interface more flexible, to allow cases with both non-causal _and_
sliding_window options.
2026-10-06 18:07:42 +02:00
Guillaume Wenzek
857c59d331
fix: remove debug log (#906)
of course
2026-10-06 17:29:50 +02:00
Guillaume Wenzek
b2f0e03043
zml/Tensor: explicit sharding in Tensor code (#868)
In Buffer code we already have explicit sharding, but not for tensor.

I'm adding a `zml.Sharding` to `.withPartitioning` to make it explicit
which mesh we are working with.
To simplify the migration, `.withPartitioning` in Tensor code can use
`.model` syntax to lookup a sharding by name.
This replace the previous logic of `selectSharding(shape)` which would
look up a sharding that matches the axes of the given partitioning which
could yield surprising behavior.

This also modifies how sharding are stored in zml.Shape.
Before it was storing a list of axis names + a tag (2 * 8 * 64bits), now
I'm storing a zml.Sharding (64 bits) and a compact PartitionArray (32
bits) that references the logical axis of the sharding by offset instead
of by name.

There is an impact on the `TensorRegistry` because it also needs to be
aware of the possible shardings, instead of delegating the matching
partition spec -> sharding to loading. This means error will be detected
sooner.

`store.createTensor` accepts `@enumLiteral()` to lookup the relevant
sharding.

This also simplifies `zml.Buffer` because the full sharding is now
stored in the shape, so it doesn't need to be passed independently. It
also means that it will propagate from the tensor who are graph inputs,
to the buffers that are eg inputs.

`Placement` logic also got a bit simpler.
2026-10-06 15:37:22 +02:00
Steeve Morin
3831492aa0 third_party/zio: bump zio=v0.19.0 2026-10-05 18:11:42 +02:00
Kevin Barre
3b70cc23a0
zml/workspace: bump llvm to 0.8.24 (#901)
Bump the hermetic LLVM Bazel dependency from 0.8.9 to 0.8.24
2026-10-05 17:51:01 +02:00
Damien Mehala
108c2d429b
zml/attention: support sparse attention (#892) 2026-10-05 17:23:18 +02:00
Damien Mehala
3b8b05b9d9
zml/attention: unify masking options under AttentionMask (#888)
Replace `is_causal` and `sliding_window` fields in `AttentionOptions`
with a tagged union. This prepares the API for a subsequent commit that
will introduce sparse attention mask. A tagged union makes these
mutually exclusive behaviour explicit.
2026-10-05 17:23:16 +02:00
Benjamin Rabier
177f81f21b
stdx: Add cpu.zig with cpu utilities (#902) 2026-10-05 16:48:54 +02:00
Corentin Godeau
11aceb7430
zml/moe: refactor MoE (#884)
This PR removes a lot of slop in the MoE part of ZML, as well as some
global improvements.

Among the improvements:
* Unify activation functions and parameters across backends
* Unify activation computation across backends
* Replace error bubbling with explicit asserts (because we have no
meanignful way to handle them)
* Add a StableHLO MoE backend
2026-10-05 16:02:38 +02:00
Corentin Godeau
98dc929608
zml/moe: cleanup entrypoint (#865)
Move logic out of the entry point while keeping everything the same
2026-10-05 16:02:37 +02:00
Damien Mehala
daa0c0dca8
zml/quantization: MXFP8 and MXFP4 (#842)
Introduce MXFP8 and MXFP4 quantization functions.

Changes:
   - add `QuantizedInput.reuseBuffer`
2026-10-05 13:52:44 +02:00
Benjamin Rabier
427499e81e
zml/exe: Exe.Runner destroys donated input buffer handles (#896)
Record donated input indices during compilation and destroy their
PJRT_Buffer handles once execution is enqueued for `zml.Exe.Runner`.

I'm not doing this automatically in `call()` because it's a bit
invasive. Not all the code uses the pattern of overwriting an input
buffer like `run(.{cache}, .{&cache})`, but most do. So I'm cleaning it
up on easy path and leave `call` as is for finer control.
2026-10-05 13:52:13 +02:00
Benjamin Rabier
af4e079f34
zml/exe: callOpts/zml.testing.autoCall replaced by eval + remove run|callOpts (#897)
Removing callOpts & runOpts to simplify the API and changing `io` to be
required for API clarity. I'm replacing `zml.testing.autoCall` with
`Exe.eval` which solves most needs in a simpler way IMHO
2026-10-05 13:51:36 +02:00
Kevin Barre
daac9c8833
zml/model: [refactor] Change llama to a whole forward pass (#881) 2026-10-05 13:21:19 +02:00
Benjamin Rabier
699add8fd2
zml/io: fix error handling when a direct shard writer fails to init (#894)
DirectShardWriter.init treated retrieveBuffer errors as unreachable and
leaked its transfer manager and buffer when a later step failed.
DirectMemoryWriter.init counted a shard writer as initialized even when
its init failed, so its errdefer deinitialized an undefined writer.
2026-10-05 11:16:39 +02:00
Benjamin Rabier
8c2e4c6dc9
pjrt,zml/platform: expose devices' NUMA node (#891)
Allows placing all the threads/memory on the right numa nodes. It's
faster for a device to access memory on the same numa node and overall
the it's better to have all threads on the same numa node if all devices
are on the same one.
2026-10-05 11:15:57 +02:00
Benjamin Rabier
773fb8f2f3
platforms/oneapi: update latest pjrt artifact (#883) 2026-10-02 22:30:34 +02:00
Louis Lechevalier
75aa9878e1
mlir/dialects: keep pointerType general in LLVM dialect (#889)
NVPTXAddressSpace is NVPTX specific. In LLVM, addressSpace is defined
for each target (nvptx, amdgpu)
The pointerType is general in llvm dialect by accepting an anytype.
2026-10-02 15:45:47 +02:00
Louis Lechevalier
9e6793c604
mlir/dialects: rename NVPTX specifics in LLVM dialect (#887) 2026-10-02 13:33:21 +02:00
Louis Lechevalier
e2a882077d
zml/attention: CuTe flashmla kernel for sm100, sparse MLA related backend (#882)
- Implement the flashmla kernel for sm100 using the cute dsl in zig.
- Extend the cute dsl with specialized sm100 (tcgen05) ops.
- Introduces a new cute backend which calls the cute kernel.
- Uses a new cache for MLA (latent or quantized).
2026-10-02 10:13:50 +02:00
Olivier Boulant
8cbd3bd38c
platform/metal: support toolchain extraction on Linux x86_64 (#880)
Even if `exec_compatible_with = ["@platforms//os:macos"],` is removed
from `platforms/metal/BUILD.bazel`, fetching the pinned Metal toolchain
on Linux currently fails:
```
ERROR: /home/oboulant/.cache/bazel/_bazel_oboulant/4ea653214b23f6728bdb9f5b40de2f50/external/zml+/bazel/http_aea_archive.bzl:20:13: An error occurred during the fetch of repository 'zml++http_aea_archive+metal_toolchain':
   Traceback (most recent call last):
        File "/home/oboulant/.cache/bazel/_bazel_oboulant/4ea653214b23f6728bdb9f5b40de2f50/external/zml+/bazel/http_aea_archive.bzl", line 43, column 19, in _http_aea_archive_impl
                _require_macos(rctx)
        File "/home/oboulant/.cache/bazel/_bazel_oboulant/4ea653214b23f6728bdb9f5b40de2f50/external/zml+/bazel/http_aea_archive.bzl", line 20, column 13, in _require_macos
                fail("http_aea_archive is macOS-only; current repository OS is %r" % rctx.os.name)
Error in fail: http_aea_archive is macOS-only; current repository OS is "linux"
```

Add checksum-pinned `ipsw` and `7-Zip` binaries for Linux x86_64.
`bazel/http_aea_archive.bzl` now selects the right binaries according to
running host. It preserves the existing macOS extraction path.
Linux ARM64 remains uncovered.

Validation:

```bash
bazel  build \
  # selects compiler binaries from the extracted external repository \
  # filegroup defined in platforms/metal/metal_toolchain.BUILD.bazel \
  @metal_toolchain//:metal_compile_tools \
  # selects the compiler’s runtime files \
  # filegroup also defined in platforms/metal/metal_toolchain.BUILD.bazel \
  @metal_toolchain//:metal_compile_runtime \
  # copies compiler binaries and runtime files into the layout consumed by ZML \
  //platforms/metal:metal_toolchain \
  # selects macOS ARM64 as the build target \
  --platforms=//platforms:macos_arm64 \
  --lockfile_mode=off
```

---------

Co-authored-by: Corentin Kerisit <corentin.kerisit@gmail.com>
2026-10-01 18:41:45 +02:00
Olivier Boulant
676f26a4e6
platforms/cuda: fix libunwind downloads blocking Linux ARM64 build (#875)
The error it fixes:
```bash
 bazel build //llmd:upload_sentry_symbols_linux_amd64 //llmd:upload_sentry_symbols_linux_arm64 --config=remote
...
ERROR: no such package '@@zml++cuda_packages+llvm-libunwind1_linux_arm64//': java.io.IOException: Error downloading [https://apt.llvm.org/bookworm/pool/main/l/llvm-toolchain-23/llvm-libunwind1_23.1.2~++20260916013112+21a77e7bb6bf-1~exp1~20260916133123.74_arm64.deb] to /Users/oboulant/Library/Caches/bazel/_bazel_oboulant/50730f1e2ab11dae9d7f564452062ab5/external/zml++cuda_packages+llvm-libunwind1_linux_arm64/.downloaded.llvm-libunwind1_23.1.2~++20260916013112+21a77e7bb6bf-1~exp1~20260916133123.74_arm64.deb: GET returned 404 Not Found
...
```

The `platforms/cuda/packages.lock.json` was updated with `bazel run
@apt_cuda//:lock`.
2026-09-30 22:28:09 +02:00
Benjamin Rabier
4d6c61f595
platforms/oneapi: update pjrt plugin (#877) 2026-09-30 19:24:34 +02:00
Raphael Nahoum
a34428e05c
workspace: bump plugins (#876) 2026-09-30 18:24:24 +02:00
Corentin Godeau
f37e845672
zml/safetensors: support hf snapshots (#872)
This PR adds support for models downloaded via `hf download` without the
`--local-dir` flag.

This is done by keeping the given path instead of getting it from the
`std.Io.File`. The problem with `File.realpath` is that in the case of
models downloaded from HF, the files points to blobs whose name are hex
strings, thus losing the required file extension. Unfortunately, we rely
on the file extension to detect the kind of repository the path
represents.

I also added `//examples/io:playground` to the `//:completion` target so
that we have completion when working on it.
2026-09-29 18:23:08 +02:00
Damien Mehala
35d5bf9282
zml/tokenizer: remove homemade tokenizer (#870) 2026-09-29 14:37:42 +02:00
Damien Mehala
8392254013
zml/attention: support sink (#864)
Required for Mimo 2.6
2026-09-29 13:57:37 +02:00
Benjamin Rabier
94278ee4ed
zml/quantization: Improve FP8 quantization (#859)
The changed code needs to round up the scale to the next power of two.
The math is:
```
(bits + 0x7fffff) & 0xff800000
```

Since 0x7fffff is \(2^{23}-1\):
- When \(m=0\), adding it fills the fraction bits without carrying into
the exponent.
- When \(m>0\), the addition carries exactly once into the exponent.
- The mask then clears all 23 fraction bits.
2026-09-29 12:45:01 +02:00
Benjamin Rabier
8745ef8b30
Compiler: disable cudnn fusion (#867)
Too many problems for now in XLA:
- https://github.com/zml/zml/pull/863
- a bf16 dot failed

We have fixes incoming for those, but unclear whether it's enough
2026-09-29 12:43:12 +02:00
Damien Mehala
337ce099fc
zml/tokenizer: add function to retrieve the token corresponding using its ID (#851) 2026-09-29 10:29:37 +02:00
Louis Lechevalier
74c590b0b0
zml/moe: Refactoring MoE selection and packing contracts (#862)
Refactoring of the MoE selection and add repacking contract required by
the model inits.
This solve a design problem in the model init :
- Before, backend selection required the resulting MoE after repacking
(dtype), and repacking required to know the selected backend.
- Now, backend selection is based on raw weights loaded from HF
(quantization scheme and dtype). Then it induces the packing.
Add useful structs for packing that are specific to MoE
(`ExpertsLayout`).
2026-09-28 18:53:26 +02:00
Louis Lechevalier
90618b8a7d
zml/ops: Up tensors size to trigger the gemm fusion dot lowering in XLA (#863) 2026-09-28 18:36:44 +02:00
Raphael Nahoum
c6530f08d3
workspace: bump plugins (#857) 2026-09-25 13:52:26 +02:00
Steeve Morin
2397718ca3 zml: enable xla_gpu_cudnn_gemm_fusion_level=2 on cuda 2026-09-25 07:15:52 +00:00
Steeve Morin
94e79b1ea7 third_party/zio: add dynamic page size support
this enables GB300 (notably) to work
2026-09-24 23:03:14 +00:00
Steeve Morin
9b7d8fb58c bazel: rework zig_options to be config_settting compatible
zig_option can now take a `flag` dependency to expose existing flags or
`bool`, `int or `string` for config_setting flags
2026-09-24 22:21:44 +02:00
Benjamin Rabier
cabcdd8f59
platforms/oneapi: Use latest pjrt artifact (#855) 2026-09-24 20:13:04 +02:00
Damien Mehala
90f914ad07
platforms/cuda: use info level instead of warn (#827) 2026-09-24 17:57:54 +02:00
Raphael Nahoum
778a5ab3c4
zml/compiler: enable experimental all-reduce kernels on CUDA and bump plugins (#852)
Enable XLA's experimental all-reduce kernels on CUDA (custom one-shot /
two-shot kernels instead of NCCL)
2026-09-24 17:56:20 +02:00
Louis Lechevalier
e2da790b1c
zml/moe: CuTe MoE mxfp4 backend (#853)
Introduces the CuTe mxfp4 moe backend. The kernels use the cute dsl and
are optimized (and so at the moment constraint to its shapes).
Introduces the cute launch host func and so the associated cuda dialect
in zml.
2026-09-24 13:12:08 +02:00
Guillaume Wenzek
50e68877a6
ci: use bazel --experimental_disk_cache_gc_max_size (#824) 2026-09-24 10:31:19 +02:00
Kevin Barre
58cda1283b
buffer: ensure proper deinitialization of host buffer events (#854) 2026-09-23 23:17:48 +02:00
Raphael Nahoum
1e045ec017
mlir/dialects/cute_ir: carry the DSL compiler's CuTe dialects (#849)
The CuTe dialects no longer build on a git checkout with a private
overlay.

- Every operation, type, attribute and enum of the compiler's cute and
cute_nvgpu dialects is typed, with its documentation, operand
constraints, traits and result inference. Nothing is kept as an opaque
payload, and the compiler's IR round-trips through it unchanged.
- The definitions are generated following `REGENERATING.md`, from the
release's Python bindings and by probing the compiler.
- The C API and the Zig builders follow: atom types take their
parameters instead of a text payload, constrained integers are one type,
and pointers take an address space enum.
2026-09-23 17:54:37 +02:00
Damien Mehala
11d49d8bc5
zml/tokenizer: add normalizer support (#847)
Support normalization from HF JSON and typed normalizer sequences.

Changes:
   - Patch IREE memory ownership and chunk-boundary bugs
   - Add regression tests for IREE bugs.
2026-09-23 16:57:04 +02:00
Guillaume Wenzek
e1356ee4ed
zml: Compiler.pushLoc (#846)
Add explicit debug info in ZML.
Model author can now add `pushLocation` to mark specific sections of the
IR

---------

Co-authored-by: Guillaume Wenzek <gwenzek@users.noreply.github.com>
2026-09-23 00:17:41 +02:00
Hugo Mano
ca9a342377
platforms: update plugins and tune ROCm graph execution (#848)
Update CPU, CUDA, Metal, and ROCm plugin references to release
202609221707.75.1.0cf3c5a71e94.

Default ROCm to one HIP graph stream and one hardware queue per device
to reduce synchronization overhead and throughput variability. Disable
rocprofiler queue interposition by default, while preserving explicit
environment overrides.

Restrict collective command-buffer capture to all-reduce and all-gather.
Other collective types execute outside HIP graphs.
2026-09-22 21:05:06 +02:00
Damien Mehala
71396f389a
zml/moe: fix cutlass expert local routing (#845)
Prevent non local experts to fallback on expert 0.
2026-09-22 16:31:08 +02:00
Corentin Godeau
397aa43e0b
pjrt: properly interpret error when an event is ready (#844)
A ready event can still contain an error that we need to report.
2026-09-21 19:42:46 +02:00
Steeve Morin
d0cf9562a2 zml: enable nccl user buffers by default
With SKIP_TEMP, captured NCCL collectives can retain stale physical
mappings if VMM reclaims their backing under memory pressure, causing
silent corruption.

Enable user buffers to assign color 1 and receive the reclaim exemption:
https://github.com/openxla/xla/pull/46029
2026-09-21 19:26:09 +02:00
Steeve Morin
63d8204be2
zml: add Platform.io_impl to enable threaded or zio at runtime (#841)
This enables runtime selection of the std.Io runtime. This also fixes
the compilation hangs by using zio.blockInPlace for CPU bound ops.

This may require some more `zio.blockInPlace`, so don't merge until
we're sure.
2026-09-21 18:59:29 +02:00
Steeve Morin
fc0eb7d878
platforms: bump plugins to 202609181631.57.1.f601577e1ab1 (#837)
Notably, adds LTO
2026-09-18 20:00:56 +02:00
Corentin Godeau
c5d476d5e8
zml/testing: fix environment init (#836) 2026-09-18 18:44:39 +02:00
Corentin Godeau
8bae3485f5
zml/Compiler: enable optimizations flags (#834)
* `xla_gpu_experimental_use_collective_kernels=CONCURRENT` allows
parallel execution
* `xla_gpu_experimental_use_collective_kernels=SKIP_TEMP` avoids graph
updates
* `xla_gpu_experimental_use_collective_kernels` disables custom
collectives while we are looking for a fix
* `xla_gpu_enable_command_buffer` adds collectives to the graphs
2026-09-18 17:22:07 +02:00
Corentin Godeau
a6c4464dba
zml/platforms: use vmm by default on CUDA and ROCm (#833)
So that we benefit from `SKIP_TEMP`
2026-09-18 17:22:07 +02:00
Corentin Godeau
00d5fcfb00
kernels/triton/builder: add explicit cast to avoid compilation error (#831) 2026-09-18 16:34:20 +02:00
Olivier Boulant
1f0c465f9c
bazel: fix runfiles initializer retention on macOS (#829) 2026-09-18 16:27:00 +02:00
Hugo Mano
86e55c0c91
zml/mem: use UninitializedBufferAllocator for ROCm (#832)
It's faster than using the default pageable memory allocator with the
latest version of the plugin,
2026-09-18 16:22:21 +02:00
Benjamin Rabier
243dcda29b
zml/attention/triton: expose tokenToSequence (#818) 2026-09-18 10:33:00 +02:00
Corentin Godeau
ebb9813157
zml/platform: disable async dispatch on cuda (#828)
This has a negative impact on single graph forward
2026-09-18 10:17:10 +02:00
Steeve Morin
b31dca5b12
bazel: make lto work again with bazel runfiles init (#822)
in LTO, the linker would garbage collect that exported symbol since it
wouldn't be used inside the final binary

forcing it undefined via a link flag prevents it from being removed.
2026-09-17 23:52:43 +02:00
Hugo Mano
5b48183654
zml/quantization: recognize FNUZ block-32 FP8 (#826)
Allow block-32 scales to accompany E4M3FNUZ weights after load-time
normalization on CDNA3. The tensor layout and quantization scheme stay
unchanged, so existing Linear paths can use native FP8 conversion
without materializing software-decoded FN weights.
2026-09-17 12:48:01 +02:00
Hugo Mano
cf7c93c81b
zml/io: use direct memory writer on ROCm (#825) 2026-09-17 12:47:28 +02:00
Guillaume Wenzek
f09060e5bf
//tools/xprof: bump to xprof 2.32.2 (#823)
I found a non-nightly version that seems to work well for us.
Previous nightly versions we were using is not hosted anymore on
Pypi.org
2026-09-17 10:02:57 +02:00
Steeve Morin
d60a11e2a2
zml: add support for zio (#821)
Add a [zio](https://github.com/lalinsky/zio) dependency.

We require ZML to be aware of which `std.Io` implementation it's running
because we use `threadlocal` to store the current `zml.Compiler`
instance when compiling a graph (to make the DSL pleasant to use).

This is our only `threadlocal` usage but it is one too much. So we must
make it compatible with `zio.TaskLocal`. Not a big deal, but this
requires opt-in from the user.

And so `zio` support can be opt-in enabled by setting the build flag
(default is `std`):
```
--@zml//zml:io_impl=zio
```

Sadly there is no `std.Io` response to `threadlocal` (or task local).
Hopefully this will be fixed in the future.

Note that this creates problem if using `std.Io.Evented`.
2026-09-16 22:27:34 +02:00
Steeve Morin
6de80971d2 platforms/oneapi: add missing header in oneapi target 2026-09-16 21:53:11 +02:00
Steeve Morin
df3a43d29f
bazel: add zig_options rule (#820)
Introduce `zig_option` and `zig_options` rules to generate the bazel
equivalent of option modules plugged on bazel flags.

On `string` flags, if the flag has a `allowed_values`, the `string` is
converted into a `enum` in zig code.

Also, port platforms to this and remove the ugly `defines` hack to
comptime test for platforms enablement.

In the process, port `arocc` to the `zig_options`.
2026-09-16 21:50:22 +02:00
Benjamin Rabier
51235e7eb6
zml: block32 FP8 with E8M0 scales, and an explicit Linear output dtype (#811)
Two things, the second one falling out of the first.

**`fp8_block32`**: E4M3FN weights with one scale per 32x32 tile, the
layout
DeepSeek V4.1 ships. `quantizeBlockFp8` takes a block size and a scale
dtype;
the E8M0 path floors amax at 1e-4 and rounds the exponent up, so the
activation
side is MXFP8: one E8M0 scale per row per 32 contracted values. That
pairing is
what XLA's fp8_block_gemv arm needs to hand both scales to
`tt.dot_scaled`
(`Fp8BlockGemvScaleInMma` requires E8M0 on *both* sides); F32 activation
scales
would quietly fall to the per-block multiply. The Triton MoE path gets
the same
scheme at `block_size_k = 32`.

`Linear.forward` also stops rounding the activation to BF16 before
quantizing,
so amax is computed on the input as given.

**An explicit output dtype** on `scaledDot` and `Linear.forward`.
Inference
cannot answer what an FP8 x FP8 product should return: the accumulation
is F32
either way, so BF16 or F32 back is the caller's choice. `forwardScaled`
now
hands the caller's type to `scaledDot` rather than rounding through BF16
and
converting after. Narrow outputs are rejected.

Separately, `Linear.forward` converts the weight and bias to the
activation
dtype. `dotGeneral` asserts both operands share one, which broke the
moment a
model raised its activation to F32 against a BF16 weight; the wrappers
this API
replaced converted for exactly that reason.

**Breaking:** `Linear.forward` takes a second argument and `scaledDot` a
fifth.

---------

Co-authored-by: Raphael Nahoum <raphael@zml.ai>
2026-09-16 18:13:08 +02:00
Corentin Godeau
78a89679fa
zml/buffer: introduce simple createView (#817)
`Buffer.createView` can be used to create a view from a host pinned
buffer and use it as if it was one the device. This avoids XLA
limitations of doing operations directly on host pinned buffers.
2026-09-16 17:57:41 +02:00
Corentin Godeau
ae9ce333ed
zml/io: introduce support of memory in the loader (#815)
This only works on OneAPI and Cuda for now.

For other platform we would need a writer that allocates host pinned
uninitialized buffers for the shards and write directly to them.
2026-09-16 17:57:40 +02:00
Corentin Godeau
7adc862cbd
pjrt: rename ambiguous field of CreateViewOfDeviceBufferArgs (#814) 2026-09-16 17:57:40 +02:00
Raphael Nahoum
ace231388d
platforms/cpu: fix sandbox lib path after plugin layout change (#819)
The CPU plugin archives now ship the shared library under lib/, and
copy_to_directory outputs to sandbox/ instead of sandbox/lib. Look up
the sandbox directory in runfiles and join lib/ when building the
library path, matching metal, rocm and cuda.
2026-09-16 17:48:02 +02:00
Hugo Mano
4adb31b4d6
platforms: bump plugins for CPU, ROCm, CUDA and Metal (#812)
Bump to build `202609161226.43.1.b992d049cec2`
2026-09-16 14:53:44 +02:00
Raphael Nahoum
94257d3747
kernels/cute: Zig CuTe DSL and XLA custom call and bump xprof (#810)
Adds a Zig front end for NVIDIA's CuTe DSL, shaped after `cutlass.cute`,
plus the `__gpu$xla.gpu.cute` custom call that compiles and launches it
on CUDA.

- `mlir/dialects/cute_ir`: bindings for the `cute` and `cute_nvgpu`
dialects
- `zml.ops.cute`: emits the custom call with the textual module, `grid`
and `block`.
- `kernels/cute` + `zml.kernel.cute.Kernel`: builder, values, tensors
with coordinate `get`/`set`, thread/block indices, static smem, sync,
`scf`. See `kernels/cute/README.md`.
- bump `xprof-nightly`

---------

Co-authored-by: Steeve Morin <steeve@zml.ai>
2026-09-16 14:02:46 +02:00
Corentin Godeau
91c8b31502
platforms: bump plugins for CPU, ROCm, CUDA and Metal (#809)
Bump to `202609151311.37.1.f1b6432d55a0`
2026-09-15 16:07:39 +02:00
Corentin Godeau
550f2bbee0
zml/Compiler: allocate inputs/outputs for func.func (#806)
When we have a lot of inputs/outputs, it can overflow the defaults we
put in the `func.func` helper.
2026-09-14 18:19:03 +02:00
Hugo Mano
fdb79caa05
kernels/fly: MOE + MLA (#805)
Add native FlyDSL kernels for MXFP4 MoE and BF16 sparse MLA on `gfx942`,
with SSA builder operations for matrix loops
2026-09-14 15:42:56 +02:00
Raphael Nahoum
b78c0f03d7
kernels/fly: FlyDSL Zig DSL (#799)
Adds a Zig front end for the FlyDSL dialect, so AMD GPU kernels can be
written in Zig alongside the existing Mosaic, Triton and tile ones, and
dispatched the same way.

Types and layouts are built through the dialect's own C entry points
rather than by printing and reparsing assembly. That holds in both
directions: host-side layout literals become attributes directly, and
anything the dialect computes is read back through typed accessors.

Launch geometry is derived from the device rather than hard-coded, so
the same kernels run on both AMD architecture families and the tests
stay architecture agnostic. Everything builds and tests on any host, no
GPU needed.

Also carries one unrelated cleanup, dropping a needless null terminator
from the Triton and tile kernel strings.
2026-09-14 09:57:36 +02:00
Corentin Godeau
4788ccc284
zml/moe: add triton_mxfp4 backend (#801)
This is a kernel that is quite specific to Deepseek V4 Flash

Ideally, we would want to unify the triton kernels.
2026-09-11 18:28:51 +02:00
Alhussein Jamil
ecb824e859
bin/zml-smi: keep NVML polling on one thread so --top does not freeze (#766)
## Summary
- `zml-smi --top` was stalling under load because each GPU and the
process list hit NVML on their own thread. Those calls serialize on the
driver RM lock, so one slow ioctl froze the whole TUI.
- All NVIDIA queries now run on a single poller.
`nvmlDeviceGetPcieThroughput` (it blocks while sampling) is only taken
on the first tick and every fourth tick after that; link speed/width
stay on the hot path.
- Process-query sizes above 4096 are ignored, and a failed per-process
util sample no longer skips the rest of that GPU.

---------

Co-authored-by: alhussein jamil <alhussein@zml.ai>
2026-09-11 16:29:29 +02:00
Benjamin Rabier
f75f5bd27e
zml/Compiler: Allow providing a Progress.Node (#798) 2026-09-11 13:07:45 +02:00
Benjamin Rabier
7204aaacbf
zml/moe: Merge MXFP4 & MXFP8 paths (#795) 2026-09-11 12:56:35 +02:00
Benjamin Rabier
46bc6df684
zml/moe: proper FP8 scaling and bias in triton experts (#793)
- The current Fp8 scaling wasn't properly finished. We didn't use the
scales at all.
- Adding a `quantize_input` option that GLM5.3-Flash uses
- `applyActivation` wasn't properly using the threshold for relu
- The bias was silently ignored for the MoE.
2026-09-11 12:55:59 +02:00
Corentin Godeau
432609ed22
zml/triton: wire TMA and global scratch memory size in Triton kernels (#797)
XLA supports TMA and scratch memory in the triton custom call but we
never wired it.

It is now solved.
2026-09-11 11:21:20 +02:00
Benjamin Rabier
28ea0571e6
zml/moe/triton: remove makeFusedMoeConfig and use clearer names (#792)
`makeFusedMoeConfig` was just adding extra complexity and we used `a`,
`b` and `c` everywhere. I think it's fine in the triton kernel itself,
but outside of it's just confusing IMHO.
I removed a bunch of useless parameters also that just end up returning
an `Unsupported` error.
2026-09-10 19:19:34 +02:00
Hugo Mano
0ebb0c21a9 platforms: bump plugins for CPU, ROCm, CUDA and Metal (#788) 2026-09-10 14:34:40 +00:00
Benjamin Rabier
662bc8ea5d
zml/moe: Simplify triton API and config loader (#790) 2026-09-10 09:50:57 +02:00
Benjamin Rabier
f6af7f3e52
zml/moe: remove unused metadata (#789) 2026-09-09 19:07:33 +02:00
Damien Mehala
0f7b3bb4b0
zml/moe: enforce padded_num_experts as a power of two (#786)
Fix an issue when `padded_num_experts` computed doesn't land as a power of two. Notably for DeepSeek-v4 Pro with its 384 experts, divided by 32 warp is not of power of two, leading to a crash in the triton which expects a power of two number of experts.
2026-09-09 15:17:32 +02:00
Benjamin Rabier
41fce4f2fc
zml/nn: Add zml.nn.forwardQuantized (#783) 2026-09-09 09:32:43 +02:00
Benjamin Rabier
06074d81f6
zml: support block-128 FP8 quantized input and f32 scale & add rocm computeCapability (#782)
Imitating `platform.cuda.computeCapability` as fp8 support is native for
e4m3fnuz on gfx942 but e4m3fn for everything else that has hardware
support AFAIK.
2026-09-09 09:32:26 +02:00
Guillaume Wenzek
2e5150eb0f
stdx/json: {Union,TaggedUnion}Helpers (#770) 2026-09-09 09:01:10 +02:00
Benjamin Rabier
afb3559f7f
zml/moe: apply activation thresholds (#781) 2026-09-08 15:56:58 +02:00
Raphael Nahoum
77e41879d3
zml/kernel: add a CUDA Tile IR (cuda_tile) Zig DSL (#779)
A third kernel language beside Triton and Mosaic TPU. It emits NVIDIA's
cuda_tile dialect and reaches the GPU through XLA's
`__gpu$xla.gpu.cuda_tile` custom call.

- `third_party/cuda_tile`: cuda-tile v13.3.3
- `mlir/dialects/cuda_tile`: every type and enum through the dialect's C
API, one constructor per op (all 100, in Ops.td order), and a small C
shim for the TileView interface the C API does not expose.
- `kernels/cuda_tile`: views, tokens, for/if/loop, reduce/scan (single
and variadic), atomics, MMA, every elementwise op.
- `zml.kernel.cuda_tile.Kernel` and `ops.cudaTile`: the custom call,
with a grid and no num_warps/num_stages (those are optimization_hints in
the IR).
2026-09-08 14:57:40 +02:00
Raphael Nahoum
9c92ff6203
zml/platform: one ComputeCapability for every CUDA gate (#777)
Every CUDA gate used to parse the compute capability string itself. They
now share `platform.cuda.computeCapability`, which returns null off CUDA
and offers `.is` / `.atLeast` / `.sm`.

Also folds the NVFP4 scale clamp into its reduce to remove one launch
per projection, and is equivalent.
2026-09-08 14:42:32 +02:00
Raphael Nahoum
520f8d63c1
zml/Compiler: set the fork's flag defaults per target (#775)
The fork keeps upstream's flag defaults, so zml turns on what each
target actually wants:
- the Tile IR scaled-dot backend, subchannel dequantize fusion and
Triton multi-output fusion on CUDA
- Metal AIR fast math off, since it lets the AIR compiler assume no NaNs
and is_finite(NaN) comes back true.
2026-09-08 14:39:08 +02:00
Raphael Nahoum
12b299186a
platforms/cuda: bump the plugin and cuDNN to 9.24.0 (#774) 2026-09-08 14:34:30 +02:00
Benjamin Rabier
7f5c678a58
kernels: Add support for e4m3fnuz dtype (#780) 2026-09-08 13:52:52 +02:00
Raphael Nahoum
1fd10b6528
platforms/metal: update plugin (#773)
improve gated delta net and rebase on latest XLA
2026-09-07 17:02:57 +02:00
Tristan Gautié
053577d82d
zml/buffer: switch uninitialized buffer log to debug level (#771) 2026-09-07 09:37:39 +02:00
Raphael Nahoum
567434be79
zml/nn: Linear.quantizationScheme (#772) 2026-09-04 19:13:27 +02:00
Tristan Gautié
6b4b790a00
nn/gated_delta_net: bound GDN recurrence (#769)
Fixing an issue raised in https://github.com/zml/zml/pull/754, to bound
the GDN recurrence in the Qwen3.5 example model to the number of active
tokens.

We also make qwen3.5 and qwen3.5-moe slightly more consistent.
2026-09-04 12:45:57 +02:00
Guillaume Wenzek
59dfe4216b
zml.Buffer.freeDeviceBufferMemoryButKeepHostMetadataMemory (#760) 2026-09-04 10:24:26 +02:00
Tristan Gautié
0b41272a08
attention/flashattn: fix sliding window off-by-one error (#763)
We fix an off-by-one error in Flash-Attn: `window_size_left` is
inclusive of the current position, so it is minus 1 compared to the
actual `sliding_window` size.

Ref:
https://github.com/Dao-AILab/flash-attention#how-to-use-flashattention
2026-09-03 18:26:40 +02:00
Benjamin Rabier
54bf33a02f
zml/Compiler: Provide accurate device memory size to XLA (#764)
By default, XLA assumes 80% of the memory to be available during
compilation. It wasn't an issue as long as we had one executable per
layer, because most of the weights weren't parts of the inputs, but for
a full forward executable we exceed that 80% threshold in llmd with the
cache being sized to use almost everything up to the BFC limit. So XLA
ends up generating an executable the least possible temporary buffers
even though it has more than that.
2026-09-03 17:17:09 +02:00
Raphael Nahoum
1b1cfe89ab
zml/nn: unpack FP4 along an explicit axis (#767)
Make unpackFp4 accept the packed source axis explicitly instead of
requiring the hardcoded .kw tag.

Updates the Linear and MoE callers and adds regression coverage for
expanding and renaming the packed axis.
2026-09-03 17:15:45 +02:00
Damien Mehala
e97abbaad4
zml/ops: refactor manualComputation (#757)
Simplify `manualComputation` by merging `inputs` and `body_context` into a single `input` argument.
Internally, collect all and shard all tensors (including `?Tensor`) from `input` before invoking the body function.
2026-09-03 11:02:59 +02:00
Corentin Godeau
67ac9c7feb
zml/moe/moe: handle optional scales (#761)
While waiting for #757, this fixes triton moe for mxfp4
2026-09-02 16:37:56 +02:00
Olivier Boulant
de5850a99c
ci: prevent RunOn runner stealing (#759)
Give each concurrent RunsOn job a unique routing label using the
workflow run ID, run attempt, and job name. This prevents runners
provisioned for one matrix job from being assigned to another compatible
job.
- [Runner
stealing](https://runs-on.com/docs/maintenance/troubleshooting/#runner-stealing)

---

Replace the deprecated image-level `disk: large` setting with
`volume=80gb` on each RunsOn job. Storage belongs to the runner
configuration rather than the custom image definition.
- [RunsOn storage labels](https://runs-on.com/docs/runners/labels/#disk)
- [Custom runners and images](https://runs-on.com/docs/runners/custom/)
2026-09-02 13:51:29 +02:00
Raphael Nahoum
3178484b27
zml/io: record computed bindings in the store (#749)
A binding created through `maybeCreateBinding` is computed from its
sources by an executable and must never be streamed from the checkpoint.
`Loader.defaultCallback` inferred that from `sources.len != 1`, which a
single-source binding fails resulting to the raw bytes being loaded and
producing wrong results
2026-09-02 12:51:09 +02:00
nzinci
5d0686de02
zml/safetensors: close model discovery handles (#758)
Model discovery leaves its safetensors entrypoint and repository handles
open after building the tensor registry, including when parsing fails.

Close handles owned by `fromRepo` and `fromPath`, and close repositories
at the existing `fromRepo` call sites.
2026-09-02 09:10:18 +02:00
Damien Mehala
e1e983c802
zml/moe: support Flashinfer cutlass MoE for sm 10.3 (#740)
Update flashinfer cutlass moe to retrieve sm103 compiled .so. 
Unpack forwardMoe abi by removing the Nvfp4Scales type.

---------

Co-authored-by: Louis Lechevalier <louis@zml.ai>
Co-authored-by: Corentin Godeau <corentin@zml.ai>
2026-09-01 16:11:57 +02:00
Benjamin Rabier
6e5829a061
zml: Use zml.nn.Linear for forwardMoe and rename quantization structs (#725)
`forwardMoe` is effectively consuming a `gate_up` and `down` linear MLP,
it "just" only applies parts of it.
I've renamed:
```
Quant -> Quantization
QuantSchem -> Quantization.Scheme
TensorScale -> Quantization.GlobalScale
Direction { multiplier, divisor } -> Operation { multiply, divide }
```
and moved it outside of zml.nn.Linear

This makes it easier in LLMD to just load zml.nn.Linear
2026-09-01 14:17:48 +02:00
Steeve Morin
de134385c5 zml/vfs: handle invalid GCS json
we do not support "external_account", but for now at least do not crash when we
can't parse the JSON file.
2026-09-01 11:23:43 +02:00
nzinci
325cf2185e
zml/tensor: fix triangularSolve batched RHS contract (#753)
`Tensor.triangularSolve` currently rejects batched inputs and derives
the result shape from the coefficient matrix.

Allow tensors with rank 2 or greater and preserve the right-hand side's
shape in the result.

This was found while working on a separate Qwen3.5 GDN optimization.

Tests:

- `bazel test //zml:test`
- `bazel build //zml`
2026-09-01 11:15:08 +02:00
Guillaume Wenzek
6e171a5ef9
zml: remove implicit donation [bugfix] (#751)
in #738 I accidentally introduced implicit donation `fn foo(x: Tensor)
Tensor { return x; }` would not make a copy of the input tensor but
return the same pointers.

I'm not sure an implicit copies is better than an implicit aliasing, but
fixing since this is an un-planned breaking change.

Also fixing a compiler error for `.items(bool)` after #724.

Note: this is an argument for snapshot testing: the CI should check if a
given PR changes the emitted IR for a given model.
2026-08-31 17:01:49 +02:00
Guillaume Wenzek
a46e4cac06
zml: handle packed fp4 on CPU side (#724)
The XLA API assumes that all sub-bytes data is given to it unpacked.
But when you read safetensors files the data is packed.

My proposition is the following: when we have a zml.fp4 tensor/buffer we
say to pjrt it's a u8 buffer instead.
Since PJRT infers the input/output shape from stablehlo I had to
introduce some bitcast at the edge of the compilation:
- fp4 input are declared as u8, but I insert bitcast operand before
calling the zig code, so the model code only see the fp4 tensors.
- when the model code returns a fp4 output, I bitcast to u8 before
exiting the final compilation

Breaking changes: none
Currently fp4 can't cross the pjrt boundary, so all fp4 code already
manually lie about the data being u8, and it will keep working. In the
future this code will be able to be "upgraded" by explicitly tagging the
input as fp4 and remove the "unpacking" code from their model code.

Note: the tracking XLA issue seems pretty cold
https://github.com/openxla/xla/issues/16795
2026-08-31 13:21:52 +02:00
Guillaume Wenzek
fcb0d2a9c1
zml.Compiler: cleanup (#738)
* use multiArrayList in `emitMlirFunc` to store information about
input/output tensors.
* merge OutputInfo and InputInfo: The code is collecting same info for
input/output, some code was merge.
`emitMlirFunc` is now clearly split between the function specific
wrapper, and the part that works only with slices of tensor info.
* zml.Sharding: clarify error sources, added more error messages. Most
`zml.Sharding` function can fail because they resolve the sharding on
the fly. Changed some APIs to explicitly take a resolved sharding,
leaving only OOM errors to handle. I've put a TODO for you @hugomano
about GSPMD sharding that can still fails pretty late when generating
the attribute. I feel this should be checked when constructing the
Partitionning.
* add Tensor.Id instead of old `usize` this allows accidental confusion
or operation on type id.
* improve Compiler.Scope API: `pushBlock` now returns the `scope` this
remove the old pattern of `pushBlock(); const scope = currentScope();`
2026-08-31 13:19:52 +02:00
Damien Mehala
fdcdade631
zml/moe: fix manualComputation scales usage (#743) 2026-08-28 19:38:39 +02:00
Corentin Godeau
54c0f16d22
zml/attention/triton_attention: use specific block_m on GB300 (#742)
Investigation has been done with the help of a clanker. The problem
might manifest on other blackwell hardware but for the time being,
simply gate to GB300.

@elogir If you can add some details
2026-08-28 18:18:46 +02:00
Benjamin Rabier
450abb6131
zml/attention: optimize sparse MLA value accumulation (#737)
GLM 5.2 represents each MLA query/key head as 512 latent dimensions
followed by 64 RoPE dimensions. All 576 dimensions participate in the
attention score, but the value is only the 512-dimensional latent
prefix.

Previously, sparse MLA accumulated a 576-dimensional context and sliced
off the RoPE suffix afterward. Return the 512-dimensional value context
directly instead. This also reduces the padded value accumulator from
1024 to 512 elements.

Make the value width explicit so the same implementation also supports
models such as DeepSeek V4, where the RoPE dimensions are included in
the value.

Keys and values share the same latent cache. Remove the redundant value
pointer and strides, reuse the loaded key tile for value accumulation,
and remove paging arguments that are no longer needed after converting
top-k indices to physical cache positions.

Add a grouped partial-reduction path for the high split counts that
improve occupancy on MI300, and tune the shape-based launch selection
more aggressively for GB300.
2026-08-28 15:38:11 +02:00
Guillaume Wenzek
cd477b858a
zml: refactor module.zig into Compiler.zig (#736)
Migration instructions:

zml.module.CompilationContext -> zml.Compiler
zml.module.CompileError -> zml.Compiler.Error
zml.module.CompilationOptions -> zml.Compiler.Options
zml.module.compile -> zml.Platform.compiler or zml.Compiler.compile

Breaking changes: none, I've kept a small compatibility layer at the top
level, with upgrade instructions.

We can remove it in a second time.
2026-08-27 18:31:50 +02:00
Guillaume Wenzek
805cf8cd74
zml/moe: fixup triton.zig (#734)
fix after conflict between #729 and #730
2026-08-27 17:32:20 +02:00
Guillaume Wenzek
082c48b211
zml: refactor slice API (#729)
## Overview

* merge `dynamicSlice` and `slice` into `slices`
* merge `dynamicSlice1d` and `slice1d` into `slice` 
* drop `choose`, `choose1d`
* handle singleton for dynamic slices too
* typed API so that it's easier to specify arguments

## Migration guide:

```diff
// new .slice accepts all old .slice1d inputs 
- const x1 = x.slice1d(-1, .{ .start = 0, .end = half_dim });
+ const x1 = x.slice(-1, .{ .start = 0, .end = half_dim });

- k.dynamicSlice(.{ .layer = zml.Tensor.DynSlice{ .start = cache_index, .len = 1 } }).squeeze(.layer);
+ k.slice(.layer, .dynSingle(cache_index));

- k.dynamicSlice(.{
-    .layer = zml.Tensor.DynSlice{ .start = cache_index, .len = 1 }, 
- }).squeeze(.layer).choose1d(.k, kv_head);
+ k.slice(.{ .layer, .k }, &.{ .dynSingle(cache_index), .single(kv_head) });

- o.choose1d(ax, 5),
+ o.slice(ax, .single(5))
```

Now the `Tensor.Slice` struct is bigger and now as a Tensor in it, which
may break user code if your model was storing a `Slice` inside it,
because now `zml.Bufferize(Model)` will contain one more `?Buffer`. You
can either pass an extra `null` to the `zml.Exe` or refactor to remove
`Slice` out of `Model`. I haven't seen this happen so far. The idea is
that `Slice` will mostly be a stack only struct, only created to call
`slice`
2026-08-27 16:06:36 +02:00
Damien Mehala
8cb020ca43
zml/moe: introduce A16WMXFP4 kernel (#730)
Changes:
  - add `Options` struct to the MoE API.
  - `num_experts_per_tok` field is now mandatory.
  - updated qwen 3.5 moe to use the new api.
2026-08-27 13:58:21 +02:00
Damien Mehala
a05d7f920a
mlir/func: support up to 64 results (#731) 2026-08-27 11:13:09 +02:00
nzinci
163a387673
zml/attention: fix latent KV cache update (#727)
Latent `KvCache.update` currently passes one tuple to `scatterSlices`,
so the branch fails to compile when instantiated.

Pass the three arguments directly and add a latent-cache compile
regression test.

Tests:

- `bazel test --jobs=2 //zml:test`
- `bazel build --jobs=2 //zml`
2026-08-26 16:54:27 +02:00
Guillaume Wenzek
3fe735f491
zml: mlir known types (#722)
cache base MLIR type corresponding to zml.DataType
2026-08-26 16:54:04 +02:00
Guillaume Wenzek
98f20d0ec3
zml.Slice: improve formatting (#723) 2026-08-26 10:50:14 +02:00
Guillaume Wenzek
a452efe972
zig 0.17: replace ** by splat (#728)
update in prevision of Zig 0.17
2026-08-26 10:49:19 +02:00
Damien Mehala
c0dcc7cf7f
zml/nn: detect deepseek-like MXFP4 quantization (#726) 2026-08-26 10:34:47 +02:00
Guillaume Wenzek
1b6a3cb395
zml/testing: apply close fraction across strided slices (#721)
correctly accumulates error statistics across the full Buffer instead of checking each sub - slice independently.
2026-08-25 16:47:49 +02:00
Benjamin Rabier
baa6bb4354
zml/attention/mla: Add 3d kernel (#717)
On 8 GPUs, we ended up only using one SM per GPU without 3d attention.
2026-08-24 15:03:59 +00:00
Raphael Nahoum
007be52c54
zml/nn: load MXFP8 and MXFP4 checkpoints (#719) 2026-08-24 16:17:41 +02:00
Damien Mehala
c8cae615c5
zml/tensor: add constant remainder and substraction helpers (#716) 2026-08-24 13:54:54 +02:00
Damien Mehala
60bd1ee233
zml/attention: add sparse MLA (#714) 2026-08-24 13:40:08 +02:00
Damien Mehala
b132a65010
zml: support F8_E8M0 tensors (#715)
Adds safetensors parsing and MLIR type support for F8_E8M0 tensors
2026-08-24 12:58:05 +02:00
Raphael Nahoum
f223acccb7
zml/nn: accept an f32 per-channel FP8 scale (#711) 2026-08-20 13:13:31 +02:00
Louis Lechevalier
533afb8915
zml/moe: flashinfer nvfp4 x nvfp4 moe backend (#707)
This PR add the support of nvfp4 x nvfp4 moe backend on sm120 / sm100
(natively supported nvfp4 hardware). It :
- retrieves the released artifact cutlass moe flashinfer :
https://github.com/zml/flashinfer/releases/tag/cutlass-moe-v0.1.0
- loads the .so
- implements the nvfp4 x nvfp4 backend (added to existing triton and
mosaic tpu).
2026-08-20 11:51:32 +02:00
Benjamin Rabier
12211f3765
zml/exe: borrowed Exe in Runner (#708)
It's a bit of a mess to transfer ownership of the Exe to the runner as
that needs careful handling of defers. Fundamentally there is also no
particular reason why the Runner should own the Exe and it was not
consistent across zml.Exe and zml.TypedExe. Now it's always borrowed.

Renaming `zml.TypedExe` to `zml.FnExe` because it's shorter. I've
removed all the aliases `type LayerExe = zml.TypedExe(...)` it's useless
with ZLS, it's a lot easier to have `zml.FnExe(<function>)`
2026-08-20 10:56:32 +02:00
Raphael Nahoum
19f562560d
zml/nn: classify quantization schemes and serve them on Linear (#710)
`QuantScheme` classifies a checkpoint's scheme from the weight and scale
shapes rather than from key names, and `Linear.forward` serves it:

| scheme | stored as |
|---|---|
| NVFP4 | `f4e2m1`, or two values per byte in `u8` |
| FP8 per-channel | `f8e4m3fn` + `[N, 1]` bf16 |
| FP8 block-128 | `f8e4m3fn` + `[N/128, K/128]` bf16 |
2026-08-19 16:25:50 +02:00
Raphael Nahoum
5eff1ed501
zml/moe: NVFP4 and block-FP8 Metal custom calls (#709)
Emits `__metal$moe_gemm` / `$f4` / `$f8` so a MoE layer can run bf16,
NVFP4 or DeepSeek block-FP8 (`f8e4m3fn` weights + bf16 128×128 scales)

`forwardMoe` takes two new optional arguments, `w1_global_scale` and
`w2_global_scale`. The per-expert global (`weight_scale_2`) is applied
in-graph after the GEMM.
2026-08-19 15:43:42 +02:00
Benjamin Rabier
c58d81a7b3
zml/exe: TypedExe (#697)
in LLMD and //examples/llm, we compile multiple executable, re-use the
args & results. We do all of this manually, in addition we don't have
any type-safety on the arguments/outputs which is a bit too easy to
misuse IMHO.

So adding two abstractions:
- Runner: Packs the args, exe and results together to avoid calling
`args.set`, `exe.run` and `results.fill` with similar arguments. Today
it owns the Exe. It doensn't strictly needs it, but it's simpler and
works for us for now.
- TypedExe: Type-safe version of an Exe, expecting a single input &
ouptut to have keyed arguments to have fully unambiguous arguments &
results mapping.

I've updated //example/llm with this new pattern.

Also adding an explicit `zml.module.CompileError`.
2026-08-18 16:36:38 +02:00
Kevin Barre
bb860a050b
platforms/oneapi: bundle oneDNN and TBB with PJRT update PJRT_ONEAPI_RELEASE (#706)
platforms/oneapi: bundle oneDNN and TBB with PJRT
Add oneDNN 2026.0 and TBB 2023.1 to the hermetic oneAPI packages.
the pjrj plugin need it for the mistral models type with the support of
the FP8 scaled matmul with libdnn.

platforms/oneapi: update PJRT_ONEAPI_RELEASE and SHA256 for new artifact
version
2026-08-18 09:56:05 +02:00
Raphael Nahoum
d539f6bdb8
zml/io: wait for loadExecute call before deiniting input buffers (#705)
Sometimes there is a race where the input buffers get deinited at the
end of loadExecute before the executable call finishes, which can
corrupt data.

Waiting on the call closes that race, but it also prevents concurrency,
so we may need another way to keep the buffers alive without blocking.
2026-08-17 19:11:16 +02:00
Kevin Barre
49a12235bb
zml/pjrt: propagate ready-event failures and validate opaque pointers (#703)
Fix two PJRT error-handling cases that could allow execution to continue
after a backend failure.

- Use the same error conversion and destruction path for ready-event and
callback errors.
- Reject a successful `PJRT_Buffer_OpaqueDeviceMemoryDataPointer` result
when `device_memory_ptr` is null.
2026-08-17 15:16:33 +02:00
Louis Lechevalier
067602b241
zml/shape: Effective byte size on device (#699)
This PR introduces deviceByteSize which computes the byte size of a shape on device i.e. with packed elements when they are less than 8 bits.
2026-08-17 09:41:28 +02:00
David Zbarsky
a37f903c3f
[build] bump rules_swift to 3.6.1 (#640)
This prevent eager toolchain load that resulted in 10ish seconds of
`swiftc` calls
2026-08-03 03:56:42 +02:00
David Zbarsky
154b212caa
build: disable default Xcode discovery (#630) 2026-08-03 09:35:09 +09:00
Kevin Barre
039fc9516a
zml/cuda : update PJRT CUDA asset SHA256 checksums for amd64 and arm64 (#700) 2026-07-31 20:08:08 +02:00
Raphael Nahoum
830c8facff
zml/nn: treat Linear global scales as dequant multipliers (#698)
Make `Linear.global_scale` / `input_global_scale` match checkpoint
conventions
2026-07-31 19:22:03 +02:00
Guillaume Wenzek
4976dfbeb3
bazel: use github.com instead of codeberg.org as reference (#695) 2026-07-31 14:59:58 +02:00
Olivier Boulant
57eabe72de
kernels/mosaic_tpu: add log1p builder operation (#691)
Add `Builder.log1p` to Mosaic TPU kernel DSL

The MLIR math dialect already exposes `math.log1p`. 
This change makes it available through the Mosaic TPU builder alongside
the already existing operations.

--> This avoids kernels constructind `math.log1p` operations manually
and makes the operation reusable for future DSL kernel translations.

A companion DSL-harness changes replaces existing manual `math.log1p`
construction with `Builder.log1p`.
2026-07-31 14:52:11 +02:00
Raphael Nahoum
14a0a8f507
zml: start of quantization support (#690)
Add MLIR composites, `scaledDot`, and a Linear path for NVFP4 weights
(unpack, optional activation quant on Blackwell, global scales). Skip
fused multi-source tensors in the generic loader so pack/`loadExecute`
can fill them (may be used later for fused QKV etc).
2026-07-31 11:02:07 +02:00
Guillaume Wenzek
648c5fd2ee
zml/attention: stablehlo paged attention (#688)
Implement paged_attention in pure stablehlo.

introduces `zml.ops.if` and `zml.ops.if2`
The first one is similar to `zml.ops.while` and requires a full struct
with two functions. it's a bit heavy to use, but it follows the
stablehlo API, and if one of the branch has side effects it will
correctly evaluate the side effect only in that branch.

`zml.ops.if2` is a simpler version that doesn't require to create
intermediary struct, but as a consequence, if you have side effect they
will always evaluate, and it relies on the compiler to correctly avoid
duplicated compute by moving relevant compute inside the branches.

(open to feedback  about if/if2)
2026-07-31 10:57:02 +02:00
Guillaume Wenzek
2b9320fe72
//zml/io/vfs: promote to //vfs (#680)
Simplify vfs architecture: merge `//zml/io` and `//zml/io/vfs`. They move to top level `@zml//vfs`

The PR also include two bug fix:
- S3: when no content length we were trying to read exactly 1M, while we
should read up to 1M
- GCP: restore the commented code for parsing bucket info

## Test plan:

```
bazel run --config=debug //examples/io:playground -- ls gs://gcp-public-data-landsat/
bazel run --config=debug //examples/io:playground -- tree s3://noaa-goes19/ABI-Flood-Day-Shapefiles/2025/08
```
2026-07-30 14:21:36 +02:00
Guillaume Wenzek
09a7a9130a
zml/attention: test paged attention (#681)
Add unit test for paged attention, enforcing all our backends produce
the same outputs.

To simplify memory management of many small buffer provided in a struct
(like paged attention metadata buffers), I introduce
`zml.Buffer.deinitAll` backed by `zml.meta.visitFlatStruct`. Contrary to
og `zml.meta.visit` the `flatStruct` variant emit a compile error if the
input has a slice/ptr inside it. The idea is that since it enforces the
struct to be a trivial, it means the struct won't require complex
`deinit` and `zml.Buffer.deinitAll` is in fact everything you need to
`deinit`. This avoid previous footgun of `zml.io.unloadAll`.

This led to more improvement with `zml.Tensor.toMemoryAll` and
`zml.Tensor.onMemoryAll` that also leverage the new
`zml.meta.visitFlatStruct`
2026-07-30 13:21:00 +02:00
Guillaume Wenzek
bdbfeb19ec
zml.ops: refactor while loop API (#686)
removes `anytype` from `zml.ops.while`, helping with type checking and
simplifying a bit the logic.
A `While` struct needs 
* a State struct definition that captures all the loop state that will
change on each iteration
* a `cond(While, State) Tensor` function returning a scalar boolean
tensor
* a `body(While, State) State` function that preserves the state shapes.

example:

```
const While = struct {
    // While struct can contain tensor and non-tensor fields.
    limit: i64,
    step: Tensor,

    pub const State = struct {
        i: zml.Tensor,
        sum: zml.Tensor,
    };

    pub fn cond(self: @This(), state: State) Tensor {
        return state.i.cmp(.LT, Tensor.scalar(self.limit, .i64));
    }

    pub fn body(self: @This(), state: State) State {
        return .{
            .i = state.i.add(self.step),
            .sum = state.sum.add(self.step),
        };
    }

    fn forward(i: Tensor, sum: Tensor) State {
        return zml.ops.@"while"(
            @This(),
            .{ .limit = 10, .step = .scalar(1, .i64) },
            .{ .i = i, .sum = sum },
        );
    }
};
```

The code is a bit more verbose for simple example, but for GatedDeltaNet
it's equivalent, and I have also used it for paged attention.
2026-07-30 11:29:42 +02:00
Benjamin Rabier
fdb6d5cd23
platforms/rocm: Update package.lock.json (#687) 2026-07-29 18:01:59 +02:00
Guillaume Wenzek
1affa657ae
zml/Slice: correctly handle strides (#685)
While working on paged attention I realize that
`zml.testing.expectClose` wasn't properly handling strided Slice.
To fix that I'm adding a `ContiguousItemsIterator` that loops over the
contiguous part of the slice.
Only used in test now, so no perf considerations.
2026-07-29 17:34:53 +02:00
Guillaume Wenzek
bf7e277f06
zml: simplify memory allocation during compilation (#682) 2026-07-28 18:34:52 +02:00
Louis Lechevalier
f4282a47ae
platforms/cuda: Update libpjrt cuda (#684)
Update of libpjrt cuda after rebase and patches update
2026-07-28 15:57:04 +02:00
Olivier Boulant
a0d0af2a8d
flashattn: harmonize zml.Tensor.zeroes() calls (#659)
Following https://github.com/zml/zml/pull/653
2026-07-23 11:17:28 +02:00
Benjamin Rabier
c6415035db
tools/xprof: Fix xprof after update (#664) 2026-07-23 10:54:22 +02:00
Olivier Boulant
d1d313cf67
mosaic_tpu: support custom buffered-ref transfer strategies (#676)
## Context

Translate
https://github.com/vllm-project/tpu-inference/tree/main/tpu_inference/kernels/gdn/v3
kernels.

## Summary

- add `BufferedRefStrategy` callbacks for overriding `copyIn`, `waitIn`,
`copyOut`, and `waitOut` on individual `PipeSpec`s
- add the `input_output` buffer role, with separate receive/send
semaphores and four loop-carried slot counters: copy-in, wait-in,
copy-out, and wait-out (input-only refs use `copy_in_slot` and
`wait_in_slot` ; output-only refs use `copy_out_slot` and
`wait_out_slot`, `input_output` refs use all four)
- provide custom transfer callbacks with the references, buffer slots,
and semaphores needed to emit their DMA operations (`sourceRef`,
`windowRef`, `copyInSlot`, `waitInSlot`, `receiveSemaphoreAt`,
`copyOutSlot`, `waitOutSlot`, `sendSemaphoreAt`)
- fall back to the existing transfer implementation for callbacks that
are not overridden

## Validation

- all 7 `fused_conv1d_gdn_batched` sweeps produce matching Python and
Zig MLIR, covering tile size, state count, head counts, Q/K-to-V head
ratio, distinct head dimensions, and convolution kernel size
- all 19 existing `decoding_gdn-bt_8` sweeps continue to produce
matching Python and Zig MLIR
2026-07-22 15:59:11 +02:00
Benjamin Rabier
252113095c
zml/testing: Do not pre-allocate memory for the zml.testing.env() (#679)
Pre-allocating 85% of memory for simple tests is overkill, we'll just
let the BFC grow as needed. That should hopefully limit the problem of
having concurrent CI runs.
2026-07-22 15:30:46 +02:00
Hugo Mano
3e589c9772
platforms/rocm: bump to rocm-jaxlib-v0.10.2 with TheRock 7.14 (#671)
Linked PR: https://github.com/zml/pjrt-artifacts/pull/96
2026-07-21 16:18:28 +02:00
Corentin Godeau
031b2a56a8
zml/io: add some feedback to loadExecute (#670)
It's a subotimal solution as we print a generic message but it makes the
progress behave as intended.
2026-07-21 09:13:57 +02:00
Louis Lechevalier
62a4d3c651
zml/platform: set async dispatch to true when platform is cuda (#673)
This PR introduce a new option gpu_async_dispatch that set to true by
default the legacy pjrt API option: `use_tfrt_gpu_client`. This option
in xla creates an other thread per device to dispatch the thunk and
command buffers asynchronously on the gpu. It strongly reduces the cpu
overhead when the model is sharded across several devices.
2026-07-20 20:57:36 +02:00
Louis Lechevalier
a35fac997e
platforms/cuda: Bump CUDA to 13.3.0 (#672)
Bump CUDA to 13.3.0 and CUDNN to 9.22.
2026-07-20 20:57:19 +02:00
Corentin Godeau
d57d851b63
zml: ground work to support Poolside's Laguna models (#669)
This PR contains two main changes:
* A patch for the tokenizer to support the tokenizer config file
* A temporary workaround to speed-up loading of the model when executing
arbitrary executable at load time
2026-07-20 13:31:54 +02:00
Damien Mehala
9ecd60f4d4
nvim: merge configurations table instead of overwriting (#611)
Update the local neovim configuration to override only the fields that
needs to change (such as `cmd` for LSP) instead of replacing the user's
entire configuration.

The main difference is `vim.lsp.config("zls", { ... })` which merges the
provided table into the existing configuration, while
`vim.lsp.config["zls"] = { ... }` overwrite the LSP configuration for
ZLS.

I also fixed the same issue for conform.

Signed-off-by: dmehala <damien@zml.ai>
2026-07-19 11:12:00 +02:00
Corentin Godeau
f4ffbbc017
zml/io: use correct shape when loading tensor (#665)
Small mistake happened while cleaning #586, I factorized code that
shouldn't have been factorized blindly.

This PR fixes a bug where tensor loaded using `Loader.load` were not
correctly sharded.
2026-07-17 22:27:55 +02:00
Benjamin Rabier
9c38bad5be
platforms/oneapi: update pjrt artifact with collectives fixes (#666) 2026-07-17 22:23:57 +02:00
Corentin Godeau
577d6452bf
zml/io: rework zml.io.load (#586)
The old `zml.io.load` was plagued with problems. This PR is an attempt
at improving the state of things.

One major problem of `zml.io.load` was that it was not very composable.
It was either using it to load all the model or doing everything by
hand.

Another problem was that it didn't offer a way to apply transformations
at load time to the tensors stored in a safetensors file. For example,
it happens that for MoE models, instead of the experts weights to be
tightly packed in a nice 3D tensor, the weight file contain one tensor
for each expert. This is nice until your implementation required a 3D
tensor as input. Using the old `zml.io.load` API this resulted in very
convoluted way to do concatenation at load time. With this new API, we
offer convenience so that it can be handled nicely and as fast as
possible.

The main change is the introduction of the `Loader` struct. It's a
struct that you instantiate once and reuse whenever you want to load
tensors. It relies on `Binding`, which are a way to store which tensor
are required and how they need to be combined (if it's a complex binding
involving more than one source tensor).

One advantage of this `Loader` is that it's composable. Loading a struct
using `Loader.auto(Struct)` or loading each individual field of `Struct`
has essentially the same behavior. This solved the major problem of the
old solution.

For simplicity I kept the old model loading code almost identical, as
this new solution is a drop-in replacement.

# Update

After some discussions with @gwenzek I've changed my mind. Instead of
having a declarative API, we'll provide helper to do transformation
easily. The main changes that this PR now bring are:
* Introduce `Loader` struct
* `Loader` exposes `load` which is the composable replacement of
`zml.io.load`
* `Loader` exposes `loadExecute` that allows you to execute an
executable with the buffers loaded from the sources associated with a
tensor. This allows for example to concatenate tensors.

We'll continue to work on it, this is just a preliminary work that
allows us to do transformation at load time in LLMD.
2026-07-17 18:41:18 +02:00
Guillaume Wenzek
d6489fbb84
zml/attention: implement batching in cu_fa2 backend (#641) 2026-07-17 14:10:41 +02:00
Guillaume Wenzek
787d8cafa3
zml: handle passthroug in zml.nn.rope (#662)
alternative to #660

---------

Co-authored-by: Corentin Godeau <corentin@zml.ai>
2026-07-17 12:21:19 +02:00
Benjamin Rabier
f29ed52ea8
platforms/oneapi: 4 GPUs fixes (#661)
Co-authored-by: Kevin Barre <kevin@zml.ai>
2026-07-17 11:18:53 +02:00
Guillaume Wenzek
1c65836a44
pjrt: better error message on dlopen failure (#657)
I hit an issue with the plugin refusing to load on macos.
There was no information.
After adding this I saw that the the file was apparently not existing in
the bazel sandbox,
so I could reduce the bug to a Bazel issue.
2026-07-16 17:44:50 +02:00
Guillaume Wenzek
2e05c383e7
zml: cleanup attention code (#653) 2026-07-16 10:11:08 +02:00
Corentin Godeau
2e1a07e7f1
stdx/json: fix potential invalid free (#631)
When we parsed from a std.json.Value we weren't correctly duping the
strings, potentially leading to invalid free.
2026-07-15 16:43:55 +02:00
Kevin Barre
24165146b9
platform/intel: bump oneapi=2026.1 and rules_distroless=0.8.0 (#644)
# Bump intel oneapi and rules_distroless

The Debian package set is now also exposed separately through the
rules_distroless APT extension:

Now use `rules_distroless` to install the oneAPI packages in a
distroless container image.

So the platforms/oneapi/packages.lock.json file is now generated from
the platforms/oneapi/packages.yaml file, and the packages.yaml file is
now the source of truth for the oneAPI packages.

The MODULE.bazel contain rules_distroless patch
`third_party/rules_distroless/accept-nospace-version-constraints.patch`
because the rules_distroless APT extension does not support version
constraints with spaces, e.g. `>= 2023.2.0` is not supported, but
`>=2023.2.0` is supported.

A PR has been submitted to rules_distroless to fix this issue, but until
it is merged and released, the patch is needed in zml
https://github.com/bazel-contrib/rules_distroless/pull/229.

`_ROOT_MODULE_DIRECT_DEPS` does not need anymore to list the individual
oneAPI and Ubuntu package repositories.

`_ROOT_PACKAGES` is now in `_UBUNTU_PACKAGES` because all the package
share the same `_UBUNTU_STRIP_PREFIX`

## Tests

```
bazel run @apt_oneapi//:lock
```

```
ONEAPI_DEVICE_SELECTOR=level_zero:0 \
    bazel run  \
    --config=release  \
    --@zml//platforms:cpu=false  \
    --@zml//platforms:oneapi=true   \
    //examples/llm --  \
    --model=/var/models/meta-llama/Llama-3.1-8B-Instruct  \
    --topk=4  \
    --prompt="Tell me a story about evariste galois"
```

__-> decode 26.248s · 34.9tok/s__

```
ONEAPI_DEVICE_SELECTOR=level_zero:0,1 \
    bazel run  \
    --config=release  \
    --@zml//platforms:cpu=false  \
    --@zml//platforms:oneapi=true   \
    //examples/llm --  \
    --model=/var/models/meta-llama/Llama-3.1-8B-Instruct  \
    --topk=4  \
    --prompt="Tell me a story about evariste galois"
```

__-> decode 13.017s · 56.6tok/s__
2026-07-15 16:21:56 +02:00
Corentin Godeau
16ea2e2718
zml/tensor: remove concatenate and stack limits (#656)
We can now rely on the compilation context to allocate when needed.
2026-07-15 14:27:50 +02:00
Guillaume Wenzek
dbf3834b29
zml: improve Buffer sharding related error messages (#655)
It's easy to break `Buffer.from(x_tensor.shape(), data, sharding)` by
creating `x_tensor` in the model code with a specific sharding and you
pass a different one to `from`.

I think this should be a crash (like when you mix-up allocators in
init/deinit) and it should have a dedicated error message to guide the
developers.
2026-07-15 14:17:22 +02:00
Guillaume Wenzek
ece52d46bf
zml: allow PJRT compilation to fail (#654)
We used to panic when PJRT compilation failed.

I now think it's better to return an error, since we introduced the WIP
Metal plugin that can fail on legitimate programs.
This will make it possible to run the ZML test suite on the Metal plugin
and seeing all failing tests.

```
bazel run --@zml//platforms:metal=true --@zml//platforms:cpu=false //zml:test

33/138 nn.decltest.sampleTokens...FAIL (Internal)
...
34/138 nn.decltest.sampleTokensDynamic...FAIL (Internal)
...
102/138 tensor.Tensor.decltest.convert...FAIL (Internal)
...
115/138 tensor.Tensor.decltest.argsort...FAIL (Internal)
```
2026-07-15 13:53:54 +02:00
Steeve Morin
715047e025
workspace: add LLMD issue template 2026-07-13 14:52:35 +02:00
Kevin Barre
ed2bd190e8
bin/zml-smi: reuse shared shiny logo art (#647)
Render the zml-smi text logo fallback from the shared ZML logo data,
including the shiny per-glyph colors. Add a lightweight //zml:logo
target so the TUI can depend on the logo art without pulling in //zml.

Verified with:
- bazel test //bin/zml-smi/tui:test
- bazel build //bin/zml-smi:zml-smi
2026-07-10 18:35:37 +02:00
Pierre Chapuis
6970e7923e
bin/zml-smi: fix memory & compute over-reporting for oneapi (#645)
When a DRM fd is duplicated or inherited through fork(), the same oneAPI
DRM client can appear multiple times under /proc/<pid>/fdinfo. Summing
each fdinfo entry double-counts GPU memory and engine time.

Parse drm-client-id and skip entries whose (device_idx, client_id) pair
has already been seen during the collection pass. Include the device in
the key because drm-client-id may be scoped per DRM device.

Add a focused test covering duplicate clients on the same device and
distinct clients across devices or client IDs.

---------

Co-authored-by: Kevin Barre <kevin@zml.ai>
2026-07-10 16:52:05 +02:00
Benjamin Rabier
87a211d3ea
zml/vfs/hf: Increased chunk size for HF (#646)
With only 16MiB chunks, I'm hitting the authenticated user rate limits
when loading Qwen3.6-27B sharded over two devices.
2026-07-10 13:21:30 +02:00
Ben Younes
a1b595043b
zml/tensor: fix IRFFT output shape (#642)
## Summary

Fix IRFFT shape inference so it uses the requested FFT output length and
validates that the complex input length matches the corresponding RFFT
output length.

Fixes #411.

## Why

RFFT changes the innermost real dimension `n` into `n / 2 + 1` complex
bins. IRFFT receives those complex bins plus the requested real FFT
length, so its output shape should restore that requested length instead
of deriving a smaller value from the complex input dimension.

## Test verification (RED -> GREEN)

Command used for all runs:

```bash
npx -y @bazel/bazelisk@1.25.0 test //zml:test --test_output=errors
```

RED, with the new regression test before the fix:

```text
102/139 tensor.Tensor.test.Tensor.fft IRFFT restores RFFT last dimension...thread 1701155 panic: exact division produced remainder
INFO: Build completed, 1 test FAILED, 5355 total actions
//zml:test                                                               FAILED in 24.9s
Executed 1 out of 1 test: 1 fails locally.
```

GREEN, with the fix applied:

```text
INFO: Build completed successfully, 6 total actions
//zml:test                                                               PASSED in 8.1s
Executed 1 out of 1 test: 1 test passes.
```

Revert RED, with the production fix reverted while keeping the test:

```text
102/139 tensor.Tensor.test.Tensor.fft IRFFT restores RFFT last dimension...thread 1711292 panic: fft(.{ .kind = .IRFFT, .length = { 10 } }) expects tensor last dimensions to match given lengths, got 1 and 1
INFO: Build completed, 1 test FAILED, 6 total actions
//zml:test                                                               FAILED in 22.9s
Executed 1 out of 1 test: 1 fails locally.
```

## Additional checks

```text
zig fmt --check zml/tensor.zig
fmt_status=0
```
2026-07-10 13:13:29 +02:00
Benjamin Rabier
0b4db7f108
zml/io/vfs: Retry on retryable errors (#639) 2026-07-10 11:55:38 +02:00
Adrien
dcad988918
zml/vfs: treat scheme:// paths as absolute in the router (#635)
In the VFS router, a scheme-qualified path like `hf://owner/model/file`
is meant to be absolute: it should resolve from the scheme's root. But
`lookupDir` only treated it as absolute when the base directory was
`cwd`. When such a path is opened *relative to an already-open directory* of
the same backend, the router kept the directory's path and glued the new
path after it, duplicating it. So `hf://owner/model` +
`hf://owner/model/file` became `owner/model/owner/model/file`, and the
lookup failed with `FileNotFound`.

Fix: on `lookupDir`, any `scheme://` sub-path resolves from the scheme's root
regardless of the base directory, the same way an absolute POSIX path
already does. About four lines.
2026-07-09 16:40:20 +02:00
Benjamin Rabier
19a15ca966
examples/llm: shiny ZML logo (#636)
Co-authored-by: Tristan Gautie <tristan@zml.ai>
2026-07-09 14:02:57 +02:00
Hugo Mano
5cacbe606e
platforms/intel: full hermetic sandboxing (#638)
Needed sandboxing for SDK less and distroless OCI image

---------

Co-authored-by: Kevin Barre <kevin@zml.ai>
2026-07-08 12:02:28 +02:00
Kevin Barre
8a89d73fb8
build/tools xprof: refresh xprof-nightly lock (#637)
Regenerate the xprof uv lockfile to move xprof-nightly off the removed
2.23.2a20260505 build and onto 2.23.3a20260707.

Fixes Bazel fetch failures for //tools/xprof:xprof.
2026-07-08 09:33:36 +02:00
Benjamin Rabier
6f4e8b8365
io/vfs: file handle race (#634) 2026-07-07 17:42:36 +02:00
Kevin Barre
75b820acad
zml/attention: pad block_m for oneAPI decode (#633)
Use the standard minimum/power-of-two block_m sizing for paged attention
instead of the raw GQA group size. This keeps oneAPI decode tile sizing
valid for Triton IR generation.
2026-07-06 17:12:45 +02:00
David Zbarsky
3b5530833c
workspace: replace deprecated new_git_repository (#629)
## Summary

- replace the legacy `new_git_repository` alias with `git_repository`
for `arocc`, `libvaxis`, `linenoise`, and `translate-c`

## Why

`git_repository` is the canonical Bazel repository rule.
`new_git_repository` doesn't exist at bazel HEAD
2026-07-04 21:40:33 +02:00
Kevin Barre
73cbeca7cf
platform/intel: update intel pjrt and limit the intel device to 2 (#628) 2026-07-03 17:02:23 +02:00
elogir
f9183137a1
examples/llm: fix single-chip TPU sharding crash (#627)
A 1-chip TPU mesh collapses to rank 1 (only `link_x`), so the
unconditional `link_y` fold tripped InvalidPhysicalAxis. Only fold
`link_y` when the physical mesh actually has it.
2026-07-03 14:32:41 +02:00
Hugo Mano
833f42029d
platforms: bump cpu + cuda plugins and align xla (#626)
Related PR: https://github.com/zml/monorepo/pull/304
2026-07-03 14:22:23 +02:00
elogir
2057d68556
platforms/metal: bump plugin & remove metadata from struct (#620)
- Bump Metal plugin version
- Drop the attention metadata fields from the llama and lfm2 Sessions. Prefill and decode now each build their own metadata inline and free it.
2026-07-03 11:23:43 +02:00
Corentin Godeau
7349826476
zml/nn: improve sampleTokensDynamic() (#625)
This PR improves the `zml.nn.sampleTokensDynamic()` function. Most
notably:
* Convert parameters to input data type
* Support scalar/batched parameters
* Correctly use the temperature parameter
2026-07-02 17:02:51 +02:00
Corentin Godeau
51db372455
third_party/iree: bump iree tokenizer to 4d4e97d00f099a21f38eeff26f82a6d9e3643a11 (#624) 2026-07-02 15:46:02 +02:00
Olivier Boulant
28e5d9ed7f
pjrt: fix misleading pjrt plugin load success log (#622)
Move the PJRT plugin “Loaded” log so it is only emitted after the
dynamic library `.so` is opened and the PJRT API is initialized
successfully.

Previously, `Api.loadFrom` used a `defer` for the success log. Failed
plugin load attempts could still print `Loaded: ...`. This was
misleading during platform auto-detection, for example when probing TPU
before falling back to CUDA. Example:

```
2026-07-01 13:12:17   ███████╗███╗   ███╗██╗             ██╗    ██╗     ██╗     ███╗   ███╗██████╗
2026-07-01 13:12:17   ╚══███╔╝████╗ ████║██║            ██╔╝    ██║     ██║     ████╗ ████║██╔══██╗
2026-07-01 13:12:17     ███╔╝ ██╔████╔██║██║           ██╔╝     ██║     ██║     ██╔████╔██║██║  ██║
2026-07-01 13:12:17    ███╔╝  ██║╚██╔╝██║██║  .ai     ██╔╝      ██║     ██║     ██║╚██╔╝██║██║  ██║
2026-07-01 13:12:17   ███████╗██║ ╚═╝ ██║███████╗    ██╔╝       ███████╗███████╗██║ ╚═╝ ██║██████╔╝
2026-07-01 13:12:17   ╚══════╝╚═╝     ╚═╝╚══════╝    ╚═╝        ╚══════╝╚══════╝╚═╝     ╚═╝╚═════╝
2026-07-01 13:12:17                                                          Copyright (c) 2026 ZML
2026-07-01 13:12:17 warning(zml/io/vfs/s3): S3 initialized without full credentials (AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY). Access will be unauthenticated. Using default endpoint: https://s3.amazonaws.com (set AWS_ENDPOINT_URL or AWS_ENDPOINT_URL_S3). Using default region: us-east-1 (set AWS_REGION or AWS_DEFAULT_REGION).
2026-07-01 13:12:17 info(pjrt): Loading: libpjrt_tpu.so...
2026-07-01 13:12:17 info(llmd): Loading tokenizer
2026-07-01 13:12:17 error(pjrt): Unable to dlopen plugin: /llmd/llmd.runfiles/zml++tpu_packages+libpjrt_tpu/sandbox/lib/libpjrt_tpu.so
2026-07-01 13:12:17 info(pjrt): Loaded: libpjrt_tpu.so           #################### HERE ####################
2026-07-01 13:12:17 info(llmd): Resolving model repo
2026-07-01 13:12:17 info(llmd): Loading model config
2026-07-01 13:12:17 info(llmd): Loaded model config [67.405ms]
2026-07-01 13:12:18 info(pjrt): Loading: libpjrt_cuda.so...
2026-07-01 13:12:18 warning(zml/platforms/cuda): Detected NVIDIA GPU that requires CUDA compatibility libraries.
2026-07-01 13:12:18 info(zml/platforms/cuda): Loaded CUDA compatibility libraries.
2026-07-01 13:12:18 info(pjrt): Loaded: libpjrt_cuda.so
2026-07-01 13:12:18 info(llmd): Loaded tokenizer [334.168ms]
2026-07-01 13:12:20 info(llmd): Devices:
2026-07-01 13:12:20 info(llmd):         - NVIDIA RTX PRO 6000 Blackwell Server Edition (cuda:0)
2026-07-01 13:12:20 info: Using 31711 pages for the KvCache
2026-07-01 13:12:20 info: sizes: { 256 }
2026-07-01 13:12:27 info: Compiled all models [7.336s]
2026-07-01 13:12:50 info: Listening on 0.0.0.0:8000
2026-07-01 13:12:50 info(llama): Loaded weights [14.96GiB, 22.778s, 672.41MiB/s]
```

Moreover make `fromDynLib` return errors for missing `GetPjrtApi` and
failed `PJRT_Plugin_Initialize` instead of panicking or using `catch
unreachable`. So that `Api.loadFrom()` can catch it.

After the change, the output is:
```
2026-07-01 14:50:50   ███████╗███╗   ███╗██╗             ██╗    ██╗     ██╗     ███╗   ███╗██████╗
2026-07-01 14:50:50   ╚══███╔╝████╗ ████║██║            ██╔╝    ██║     ██║     ████╗ ████║██╔══██╗
2026-07-01 14:50:50     ███╔╝ ██╔████╔██║██║           ██╔╝     ██║     ██║     ██╔████╔██║██║  ██║
2026-07-01 14:50:50    ███╔╝  ██║╚██╔╝██║██║  .ai     ██╔╝      ██║     ██║     ██║╚██╔╝██║██║  ██║
2026-07-01 14:50:50   ███████╗██║ ╚═╝ ██║███████╗    ██╔╝       ███████╗███████╗██║ ╚═╝ ██║██████╔╝
2026-07-01 14:50:50   ╚══════╝╚═╝     ╚═╝╚══════╝    ╚═╝        ╚══════╝╚══════╝╚═╝     ╚═╝╚═════╝
2026-07-01 14:50:50                                                          Copyright (c) 2026 ZML
2026-07-01 14:50:51 warning(zml/io/vfs/s3): S3 initialized without full credentials (AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY). Access will be unauthenticated. Using default endpoint: https://s3.amazonaws.com (set AWS_ENDPOINT_URL or AWS_ENDPOINT_URL_S3). Using default region: us-east-1 (set AWS_REGION or AWS_DEFAULT_REGION).
2026-07-01 14:50:51 info(llmd): Resolving model repo
2026-07-01 14:50:51 info(llmd): Loading model config
2026-07-01 14:50:51 info(llmd): Loaded model config [101.024ms]
2026-07-01 14:50:51 info(llmd): Loading tokenizer
2026-07-01 14:50:51 info(pjrt): Loading: libpjrt_tpu.so...
2026-07-01 14:50:51 error(pjrt): Unable to dlopen plugin: /llmd/llmd.runfiles/zml++tpu_packages+libpjrt_tpu/sandbox/lib/libpjrt_tpu.so
2026-07-01 14:50:52 info(llmd): Loaded tokenizer [349.499ms]
2026-07-01 14:50:53 info(pjrt): Loading: libpjrt_cuda.so...
2026-07-01 14:50:53 warning(zml/platforms/cuda): Detected NVIDIA GPU that requires CUDA compatibility libraries.
2026-07-01 14:50:53 info(zml/platforms/cuda): Loaded CUDA compatibility libraries.
2026-07-01 14:51:11 info(pjrt): Loaded: libpjrt_cuda.so
2026-07-01 14:51:14 info: Using 31711 pages for the KvCache
2026-07-01 14:51:14 info: sizes: { 256 }
2026-07-01 14:51:14 info(llmd): Devices:
2026-07-01 14:51:14 info(llmd):         - NVIDIA RTX PRO 6000 Blackwell Server Edition (cuda:0)
2026-07-01 14:51:22 info: Compiled all models [8.395s]
2026-07-01 14:51:46 info: Listening on 0.0.0.0:8000
2026-07-01 14:51:46 info(llama): Loaded weights [14.96GiB, 23.629s, 648.20MiB/s]
```
2026-07-02 14:01:18 +02:00
Corentin Godeau
001dbbbde8
zml/tokenizer: Have matching tokenizer output (#623)
The IREE tokenizer was not producing the same output as the HF
tokenizer, potentially degrading the performance of the models. I put a
clanker to the task (because the tokenizer is already full clanked).
Outputs are now matching without any performance drop.

I also fixed an edge case issue on the `Writer` of the tokenizer.
2026-07-02 12:26:27 +02:00
Corentin Godeau
e8eb3ca689
zml/moe/triton: reduce log level (#621) 2026-07-01 18:10:32 +02:00
Guillaume Wenzek
e04ccb9ecb
zml: introduce zml.Compiler (#618)
Move the type out of `zml.compile` signature so that we can call
`io.async` more easily.

Sample code:

```
var layer_compile_future = io.async(zml.Compiler(TransformerLayer.forward).compile, .{
    allocator,
    io,
    platform,
    layer_compile_opts,
    .{
        model.model.layers[0],
        hidden,
        device_state.kv_cache.layers[0],
    },
});
```
2026-07-01 16:21:04 +02:00
Guillaume Wenzek
d6e9b47391
zml/examples/llm: simplify dispatch code (#619)
`@unionInit` ftw
2026-07-01 16:20:48 +02:00
Louis Lechevalier
0e12e4c998
platforms/cuda: update pjrt plugin (#617)
related: https://github.com/zml/pjrt-artifacts/pull/91/changes
From 130 to 200 tok/s for Gemma 4 26B A4B on GH200
2026-06-30 17:31:14 +02:00
Olivier Boulant
cb12e2ea71
zml/io: update defaults for zml.io.load and read parallelism 2026-06-30 15:58:00 +02:00
Reese Chong
89b0908ca5
zml/attention/flashattn: make fa2 window_size_left configurable (#595)
Add a sliding_window field to the FA2 parameters and use it to set the window_size_left argument in zml.attention.attention().
2026-06-29 17:37:13 +02:00
Reed Meyerson
be684a6dce
docs/tutorials: fix 'write_first_model' example code (#612)
Fix in the `write_first_model.md` tutorial: Update main() code section in to match full example
2026-06-29 17:32:32 +02:00
Damien Mehala
d1e4ef9e29
zml/tensor: fix segmentation fault when using split Tensor (#614)
Previously, split tensors where allocated using an arena that would be
free at the end of the function scope. This commit uses the arena
context instead.

Signed-off-by: dmehala <damien@zml.ai>
2026-06-29 14:54:07 +02:00
Corentin Kerisit
4c2393e5cf
bazel: scope Zig cache sandbox mounts to Linux (#615)
## Summary

- Limit Zig cache sandbox mount flags to Linux, where linux-sandbox and
Bazel's Docker sandbox consume host path mounts.
- Remove the macOS mount entry because sandbox-exec uses a symlinked
execroot and ignores bind-mount settings.
- Mark `/tmp` and `/var/tmp` writable alongside the Linux mount pairs.
2026-06-29 14:43:42 +02:00
elogir
6251953668
platforms/metal: add moe support (#613) 2026-06-29 10:43:40 +02:00
Corentin Kerisit
076417363a
workspace: add missing loads (#606)
nits
2026-06-27 00:51:55 +02:00
Corentin Kerisit
4792fba3c1
bazel: improve remote build resilience (#610)
Enable lost-input rewinding and local fallback for the remote
downloader.
2026-06-27 00:26:50 +02:00
Olivier Boulant
1dbe39d3e8
zml/io/vfs: add parallel read to GCS and S3 (#604)
# Summary

This PR extends the support to chunked parallel read introduced
initially for HF in the VFS, to GCS and S3.

It:
- Adds a shared `zml/io/vfs/parallel_read.zig` helper for chunked
parallel HTTP range reads
- Add configurable read pool options for S3/GCS initialization
- Register the `gs://` VFS backend in `examples/llm`.


# Tests

## S3

We used [MINIO](https://hub.docker.com/r/minio/minio) docker image from
the `5090` GPU machine. From there, we mirror the
`meta-llama/Llama-3.1-8B-Instruct` model through the S3 server.

Then from the `4090` GPU machine, we run 

```bash
bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=s3://zml-test/Llama-3.1-8B-Instruct --prompt="hello, how are you ?"
```

It works both on 1 and 2 GPU. 

## GCS

In the same spirit, we put the `meta-llama/Llama-3.1-8B-Instruct` in a
GCS bucket we own. Then, on the `4090`, we run

```bash
bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct --prompt="hello, how are you ?"
```

It works both on 1 and 2 GPU. 

### Performance

As a reference, we run:
```bash
gcloud storage cp gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00001-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00002-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00003-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00004-of-00004.safetensors /tmp/gcs-gcloud-llama-shards/
```

It gives:
```
Completed files 4/4 | 15.0GiB/15.0GiB | 942.7MiB/s 
Average throughput: 918.8MiB/s
```

Using the VFS in various setups:

* `dma_chunks = 32`, `dma_chunk_size = 128`, `parallelism = 16`,
`chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64`

```bash
info (llama): Loaded weights [14.96GiB, 21.302s, 719.02MiB/s]
```

* `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`,
`chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64`

```bash
info (llama): Loaded weights [14.96GiB, 18.227s, 840.32MiB/s]
```

* `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`,
`chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64`

```bash
info (llama): Loaded weights [14.96GiB, 17.376s, 881.43MiB/s]
```

* `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`,
`chunk_size = 16 MiB`, `num_workers = 32`, `queue_capacity = 64`

```bash
info (llama): Loaded weights [14.96GiB, 17.28s, 886.36MiB/s]
```

* `dma_chunks = 32`, `dma_chunk_size = 512`, `parallelism = 16`,
`chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64`

```bash
info (llama): Loaded weights [14.96GiB, 18.658s, 820.90MiB/s]
```
2026-06-26 19:24:31 +02:00
Corentin Kerisit
3099106f17
platforms: set linux amd64 legacy cpu (#607)
part of highway XLA target selects which header to include based on
legacy --cpu flags but the rest not...
So even if we targeted linux amd64, the top part of the highway would be
used correctly, but the extension specific part would fallback to --cpu
darwin_arm64 and would not include hh_avx2.h resulting in compilation
errors.

The normal solution here is to use platform_mappings but since we pass
`--force_pic=true` in platforms, this takes precedence and the mappings
are ignored.

The temporary solution while https://github.com/openxla/xla/pull/44823
is merged, is to pass `--cpu` in all platforms we use for cross
compilation.
2026-06-26 15:35:27 +02:00
Louis Lechevalier
c2cbcb82f1
zml/attention: Optimize heuristic and parameters choices of triton attention (#605)
Limits the number of segments for the 3d kernel. Effect : reduce the
cost of the reduce kernel which follows.
2026-06-26 14:44:48 +02:00
elogir
a4993854da
platforms/metal: sandbox Metal toolchain (#598)
Fetch Apple's encrypted Metal toolchain and only exposes the minimal
compile tools + runtime needed
2026-06-26 14:37:56 +02:00
Corentin Kerisit
aa90f92832
workspace: bump to bazel 9.1.1 (#601)
A few missing loads
Remove all flipped flags or deperacted flags from .bazelrc

Important:
- Keep autoload rules_cc and rules_python because XLA still has
transitive deps that are missing loads, PR in progress.
2026-06-25 19:37:51 +02:00
Kevin Barre
33ced8fa07
zml/backend: add oneAPI as Triton backend selector for MoE (#603) 2026-06-25 15:57:35 +02:00
Corentin Godeau
c023e26f2c
zml/moe: enable moe for any compute capability (#600) 2026-06-25 14:32:44 +02:00
Corentin Kerisit
f9ec63fd4e
workspace: Update to XLA @ 5e83525729fd21c4ad8d55073a3d047360d13f9c (#599)
Updates to latest XLA which added support for bazel 8.x workspace and
fixed a bunch of things that made bazel 9 impossible.

3 problematic changes with this bump:
- PJRT headers were again incompatible with translate-c so a patch is
made and PR has been done in XLA
- PJRT introduced a proper type fo PJRT_Error so I used the InnerMixin
pattern for those.
- //tsl/platform/default package force evaluates Label("@cuda_nvcc")
even for CPU builds and we don't have this repo. I commented for now
until I find a solution on XLA side.

Bumped a few deps necessary for this and additional llvm deps that are
now evaluated as part of the llvm-project graph on XLA side (llvm_zlib
and llvm_zstd).

Next will be to investigate bazel 9 for real.
2026-06-25 14:17:46 +02:00
Olivier Boulant
d19949e255
zml/io/vfs(hf): introduce parallel read (#594)
---------

Co-authored-by: Hugo Mano <hugo@zml.ai>
2026-06-24 17:45:39 +02:00
elogir
1d71ee7bd3
zml/safetensors: limit reads to 1GB (#597)
On macOS, `pread` returns INT_MAX when the requested size is larger than 2GB
2026-06-24 16:50:01 +02:00
elogir
33a3b37900
platforms/metal: add Metal support (#593)
Co-authored-by: Steeve Morin <steeve@zml.ai>
2026-06-24 09:54:11 +02:00
Louis Lechevalier
d942ce014a
platforms/tpu: bump plugin from v0.0.40 to v0.0.42 (#596) 2026-06-23 17:51:37 +02:00
elogir
f586cbfc3d
third_party/iree: match all added tokens (#591)
IREE's tokenizer only registered added tokens for pre-segmentation
matching if they were flagged special or normalized. Tokens that are
neither (e.g. <think> was dropped)

The fix is to match all added tokens
2026-06-23 09:16:21 +02:00
Kevin Barre
a1acbd36d5
zml/onapi: set defaults for oneAPI PjRT create options (#592)
same as https://github.com/zml/zml/pull/579
And update the pjrt plugin with the collective fix
2026-06-23 05:41:40 +02:00
elogir
5131467252
zml/attention: tune triton kernel for oneAPI performance (#588)
Speeds up paged-attention decode on the Intel/oneAPI backend
2026-06-22 19:06:48 +02:00
Louis Lechevalier
7c3b60202d
zml/sharding : introduce shardableDim api (#583)
This PR introduces shardableDim helper in Sharding to handle the case
where : (dimension % number of sharding axes) != 0

Useful to stutter and shard correctly kv heads dim for example.

---------

Co-authored-by: Hugo Mano <hugo@zml.ai>
2026-06-22 18:52:35 +02:00
Tristan Gautié
6ca94f34bd
attention: support is_causal option in Triton and Mosaic paged-attn kernels (#587)
We add suport for `is_causal` option in Triton and Mosaic paged-attention
kernels, by adapting the mask-generation logic
2026-06-22 15:25:02 +02:00
Corentin Godeau
984b60b75e
zml: various QoL improvements (#585)
These are QoL improvements to problems I stumbled upon while working on
#586
2026-06-20 17:58:18 +02:00
Kevin Barre
4cd7139477
zml/platform: add hermetic oneAPI 2026 runtime deps (#582)
Fetch oneAPI runtime libraries from locked DEB packages and wire the oneAPI sandbox packaging to use those repositories.
2026-06-19 18:30:57 +02:00
Kevin Barre
2cc6a541b3
zml/build: bump bazel to 8.7.0 (#584) 2026-06-19 15:15:41 +02:00
Hugo Mano
56a6a4db64
zml/moe: enable triton moe kernels on ROCm (#580) 2026-06-19 11:38:58 +02:00
Hugo Mano
ec776c53e0
zml/platform: set defaults for ROCm PjRT create options (#579)
ROCm was using only 0.75 of available memory. Align with CUDA, this
permit us to have Gemma 4 26B A4B on 2xRTX 9700 for example.
2026-06-18 16:17:12 +02:00
Hugo Mano
700b57466b
zml/io: expose DMA chunk windows to readers from DirectMemoryWriter (#576)
DirectMemoryWriter used to expose the next shard-sized slice as its writable
window. With HF/VFS this often produced tiny reads, even when adjacent tensor
bytes could be streamed through a larger direct-load window.

This PR changes direct loading to use an alias-window over the active primary
shard DMA buffer. The public `std.Io.Writer` buffer still points at real DMA
memory, but its visible length can coalesce across dispatch spans until a DMA
chunk, shard buffer, or tensor boundary is reached. On commit, the first active
primary segment stays zero-copy, while crossed spans and mirrors are scattered
before any buffer is submitted or rotated.

`DirectShardWriter` keeps ownership of DMA pool rotation, flip-flop events, and
PJRT transfer state. `DirectMemoryWriter` only walks the precomputed dispatch
spans with a small cursor.
2026-06-18 15:35:09 +02:00
Reese Chong
85e71e405c
zml/moe: fix - move allReduce into manual computation block (#577)
Move the MoE allReduce inside the manual computation block.

Previously, zml.moe.forwardMoe placed allReduce outside the manual
computation block which caused XLA SPMD partitioning to throw the 
following error with certain models.

"Cross-partition allreduce must be in (partial) manual partitioning mode."
2026-06-16 17:58:35 +02:00
Hugo Mano
1fead1c014
platforms/neuron: introduce NKI stack with SDPA kernel (#528)
**Summary**

This PR update to Neuron v2.30, introduces the Neuron NKI stack into ZML and wires it into the
LLM attention path.

The main goal is to make Neuron attention run through NKI kernels.
It adds the compiler/runtime plumbing needed to embed NKI kernels as
Neuron custom-native calls, then uses that path for SDPA-style
prefill/decode attention. Paged Attention will come later.

On inf2.8xlarge it runs :
- 25 tok/s for Llama 3.1 8B BF16
- 63.5 tok/s for LFM2.5-1.2B-Instruct BF16
- 19 tok/s for Qwen3.5-9B BF16
2026-06-16 10:57:00 +02:00
Louis Lechevalier
be29637c07
zml/moe: Mosaic TPU MoE backend. (#571)
This PR add the TPU backend for MoE. It uses the GMM Expert parallelism
kernel to perform the grouped gemms on gate_up and down weights (these
weights must have a precise shape to run optimally).

This backend is currently optimized for small batches. A specialized
mosaic tpu kernel will be introduced to handle bigger ones.

This implementation come with few fixes concerning the TPU sharding and
the Qwen 3.5 moe model which uses the backend :
- TPU sharding : PJRT reports TPU device coordinates in a max rank form.
So it breaks folding. To avoid this we compute the Rank by not counting
where coords =0.
- Linear attention was reading padding token in the Qwen 3.5 model. So
we introduce masking of them.

n.b. : The kv head repetition to satisfy the sharding will be handled a
more generic way in a next PR

---------

Co-authored-by: Corentin Godeau <corentin@zml.ai>
2026-06-16 10:06:52 +02:00
Benjamin Rabier
1278805b87
zml/pjrtx: alignCast the host buffer for host offloading (#575) 2026-06-12 15:03:34 +02:00
Benjamin Rabier
5d4e4db01c
zml/profiling/tracer: Add simpler infaillible span (#573)
Renamed `scope` to `span` to match OpenTelemetry convention I'm more
familiar with. Traceme doesn't seem to use `Scope` explicitly but talks
more about a scope object AFAIK.

Now `span` fails at compile time. if it cannot compute the appropriate
buffer size for the name. In that case the caller should use
`formatSpanName` which allocates. This makes `span` infaillible at
runtime.
2026-06-09 16:10:34 +02:00
Corentin Godeau
c5bdcac632
zml/sharding: fix max_rank for TPU (#572)
Topology coords on TPU have rank 4 and not 3
2026-06-04 19:04:34 +02:00
Corentin Godeau
39ffd27f6f
zml/module: expose autotune cache dir through an env var (#570)
This opt-in flag allows XLA to save the autotuning resulting from
previous runs in a folder, greatly speeding up the compilation.

It's not meant to be used at all times, so it's undocumented and opt-in.
2026-06-04 15:32:17 +02:00
Louis Lechevalier
4b0c7ed1c0
zml/moe : Mosaic GMM EP TPU kernel. DSL update. (#569)
This PR introduce the new GMM Expert parallelism kernel (basis grouped
gemm for moe on TPU). It induces some updates into the DSL mosaic TPU,
such as Builders helpers or Pipelines and schedulers :
- Buffer counts (for scheduling overlapping) > 2 allowed (until 7)
- source_reshape_shape reshaped at DMA slice/use sites to match Pallas
IR placement
- More grid dimensionality for pipeline (up to 4)
2026-06-03 15:58:16 +02:00
Steeve Morin
f9a282162a zml: fix multiple enabled platforms
more than one platform can be enabled at the same time
2026-06-01 15:53:48 +00:00
elogir
8c03af9987
stdx/flags: Update typeInfo tag to Zig 0.16 .@"union". (#565) 2026-06-01 17:51:42 +02:00
Guillaume Wenzek
807a9116d4
zml: make Buffer.from faster (#556)
`Buffer.from` is in the hot path everytime we want to send smth to the
GPU.
Sometimes we can overlap it with GPU compute but not always, and
sometimes it's quite complicated to do so.

I've rewrote Sharding.Placement to be device agnostic instead of device
specific.
This allow to reuse most of the logic for multiple devices.

According to LLVM MCA (with script included here), we got from:

```
Iterations:        100
Instructions:      324200
Total Cycles:      175191
Total uOps:        369700
```

to 

```
    Iterations:        100
    Instructions:      162900
    Total Cycles:      85585
    Total uOps:        182600
```

So about twice as fast for this part of the code. I would need some help
to confirm the numbers on TPU.


(Once the PR is approved, I'll remove the mca script as well as the
build.zig)

---------

Co-authored-by: Hugo Mano <hugo@zml.ai>
2026-06-01 10:51:32 +02:00
Kevin Barre
02c1a90c08
bin/smi: add oneapi smi monitoring (#557)
Add a oneAPI backend for `zml-smi` that detects Intel DRM render devices
and reports GPU telemetry from sysfs, hwmon, and fdinfo.

The new backend collects utilization, memory usage, power, temperature,
clocks, PCIe link data, and per-process usage for Intel GPUs. It is
wired into device detection, Bazel builds, Prometheus metrics, and the
TUI, including Intel GPU imagery and GPU detail/process table support.

Also clean up related TUI handling for GPU clock/PCIe fields and process
table lifetime management.
2026-05-29 18:14:38 +02:00
loupicaaa
2530747949
zml/attention: Fix the triton attention kernel by always masking the padding tokens (#563)
Fix the triton attention kernel by avoiding the case where `TILE_SIZE ==
BLOCK_SIZE`. Now we enforce masking this way : `const tile_mask: Value =
seq_offset.lt(max_seq_prefix_len);`.

Co-authored-by: loupicaaa <louis@zml.ai>
2026-05-29 17:42:35 +02:00
Hugo Mano
cd941143dc
examples/llm: remove useless sharding constraints (#560)
Perf iso to https://github.com/zml/zml/pull/555
2026-05-28 20:47:02 +02:00
Hugo Mano
950c80dee8
examples/llm: split Llama and Qwen 3.5 with LFM cleanup (#555)
This PR splits Llama in `examples/llm` into separate `embed_tokens`,
`layer`, and `lm_head` kernels for both prefill and decode, and brings
LFM2 onto the same composed execution path. And done same work for Qwen
3.5 .

It removes the old LFM2 `--single` mode, reworks Llama KV-cache access
to use explicit layer indices, and reuses more decode-time state through
a persistent runner and prebuilt token index buffers.

It also tightens sharding/partitioning in LFM2 for multi-device runs,
adds clearer per-phase compile logging/program names, removes the unused
`--single` CLI flag.
2026-05-28 10:10:29 +02:00
Corentin Godeau
3db711aa6f
zml/ops: fix typo in type name (#553)
Names of types should be `PascalCase`
2026-05-27 17:52:33 +02:00
Corentin Godeau
6d596964eb
workspace: bump llvm to 0.8.3 (#554) 2026-05-27 15:00:47 +02:00
loupicaaa
b6e2b026bc
zml/moe: Add sharding for triton MoE (#552)
This PR introduces the sharding as expert parallelism for MoE :

- Introduce a .experts logical sharding (required to express an other
semantic than TP (.model))
- Wrap the whole MoE impl into a manualComputation
- Implement a manual stableHlo all reduce since shardy can not infer
the correct communication after a manualComputation (blackbox)
- Partition qwen 3.5 moe example with TP for the rest of the model (non
MoE)

---------

Co-authored-by: loupicaaa <louis@zml.ai>
2026-05-27 10:13:38 +02:00
Benjamin Rabier
a06265264f
zml/Sharding: Rename Placement.Shard to Placement (#549)
Keeping `devices_in_canonical_order: []Device` in the `PhysicalMesh` and
inside `Sharding.Data` I've changed `physical: PhysicalMesh` to
`physical: *const PhysicalMesh` to have the platform own it now that
PhysicalMesh has a `deinit()`.
2026-05-26 09:29:49 +02:00
Benjamin Rabier
289c29f365
workspace: update AGENTS.md (#548)
More concise AGENTS.md and add a command to help the agent find the
sources of the Zig library it regularly struggles with it
2026-05-25 17:14:28 +02:00
Kevin Barre
73fab92393
zml/platforms: add experimental oneapi (intel) (#550)
Add experimental Intel GPU backend

Add oneAPI as a ZML platform target with Bazel flags, module extension
repositories, and PJRT runtime packaging for Linux Intel GPUs.

- package libpjrt_oneapi with oneAPI runtime libraries, Level Zero
loader,
  libsycl, and a zmlxoneapi dlopen shim for versioned shared libraries
- register the oneAPI platform loader and detect available render
devices
  before loading the packaged PJRT plugin from runfiles
- thread the oneAPI target through platform selection, sharding,
placement,
  buffers, execution, memory, IO, tensor behavior, and XLA compile flags
- document oneAPI prerequisites and example LLM usage
- update Bazel module lockfile entries for the new runtime dependencies

---------

Co-authored-by: Steeve Morin <steeve@zml.ai>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Steeve Morin <steeve@9960x-b70x2.tailb57cfe.ts.net>
2026-05-25 15:52:10 +02:00
elogir
cebc4cebf8
kernels: extend Mosaic TPU + Triton kernel DSL (#545)
This PR changes the following:

- Name-based `output_operand_aliases` for MosaicTPU and Triton DSLs
- New `kernels/mosaic_tpu/pipeline.zig`, a Zig port of `pltpu.emit_pipeline`
- Some fixes to MosaicTPU
2026-05-25 09:41:08 +02:00
Corentin Godeau
36300dee2e
zml/attention: small changes required for llmd (#551)
* Made the `MAX_NUM_SPLITS` public
* Inject max num pages from args in `Options.fromBackend()` instead of
computing it
2026-05-22 10:58:27 +02:00
Benjamin Rabier
5d372aa4c0
zml/Sharding: Reduce Placement size by allocating shards on the stack. (#547)
`Placement` is 27KiB big, it led to stack allocation failure with
gigantic struct sizes (~400KiB) in llmd. So changing the shards to be
allocated on the heap with functions to avoid allocations when we don't
need it (`format`, etc.).
2026-05-20 16:34:47 +02:00
Corentin Kerisit
77711b949d
workspace: support zig-only local execution (#546)
This PR bumps rules_zig back to upstream/main where we now have all the
necessary
features to enable zig local actions while the reste on remote.

It also adds the necessary configurable to actually enable this.

Additionally, since rules_zig now ships ZLS support, we remove zls
entirely from zml.

### How the Bazel resolution works

Bazel resolves execution platforms and toolchains during analysis for
each
target that requires toolchains. If a rule defines exe, resolution
happens separately for the default target execution group and for each
`exec_group`;
actions assigned to an exec group use that group’s selected execution
platform
and toolchains.

Execution platforms come from an ordered set. In this repository’s
remote
configuration, the relevant order is:

1. `//:rbe_platform`, supplied by
`--extra_execution_platforms=@zml//:rbe_platform`
2. `//bazel/zig_toolchain_config:local_execution_platform`, registered
by this PR
3. the host platform, included automatically by Bazel

Earlier platforms are selected when they can satisfy the requested
toolchains.

For each target or exec group, Bazel takes the required toolchain types,
the
target platform, the ordered execution platforms, and the ordered
registered
toolchains. It filters execution platforms by execution constraints,
then tries
to find a matching toolchain for every required toolchain type. A
toolchain
matches when:

- its `exec_compatible_with` constraints are present on the candidate
execution
platform
- its `target_compatible_with` constraints are present on the target
platform
- its `target_settings` match the current configuration

Any execution platform missing a required toolchain is rejected. The
first
remaining execution platform wins, and the first matching toolchain for
each type
is selected.

### How this makes Zig local-only

This PR defines a custom execution constraint:

- `//bazel/zig_toolchain_config:local_zig_execution`

It then defines a host-derived execution platform:

- `//bazel/zig_toolchain_config:local_execution_platform`

That platform inherits the host OS/CPU constraints from
`@platforms//host`, adds
`local_zig_execution`, and sets:

```starlark
exec_properties = {
    "no-remote-exec": "1",
}
```

The zig.toolchain(...) module extension call now passes:

extra_exec_compatible_with =
["//bazel/zig_toolchain_config:local_zig_execution"]

rules_zig appends that extra execution constraint to every generated Zig
SDK
toolchain. As a result, when rules_zig resolves
@rules_zig//zig:toolchain_type
for its zig execution group:

- the remote execution platform is considered first, but it does not
have
local_zig_execution, so it cannot match the Zig SDK toolchain
- the custom local execution platform has both the host OS/CPU
constraints and
local_zig_execution, so it matches
- the selected platform carries no-remote-exec, so Zig actions execute
locally
even in remote builds

Other toolchains are not constrained by local_zig_execution, so they can
continue
to resolve to the remote execution platform first. This keeps the build
remote by
default while carving out Zig actions as local-only.

---------

Co-authored-by: Steeve Morin <steeve@zml.ai>
2026-05-20 16:13:59 +02:00
Reese Chong
3f2666dba6
zml/testing: resolve type of forward in testLayer() (#540)
Change made in response to the following error:

```
zml/testing.zig:299:43: error: root source file struct 'meta' has no member named 'FnArgs'
    const ArgsT = stdx.meta.Tail(stdx.meta.FnArgs(forward));
                                 ~~~~~~~~~^~~~~~~
stdx/meta.zig:1:1: note: struct declared here
```
2026-05-19 09:19:56 +02:00
Reese Chong
986532bb33
tensor/zml: fix SwiGLU (#539)
Update SwiGLU implementation to match the SwiGLU paper.

- Formula: `(xW + b) * sigmoid(β(xW + b)), outer product with (xV + c)`
- Add test for SwiGLU
2026-05-18 16:52:05 +02:00
Corentin Godeau
61ed004556
zml/attention: big rework of paged attention (#536)
List of changes:
* Removed `Context`
* Simplified metadata to not contain scratch buffers. They are created
uninitialized on the fly instead (thanks to #535)
* Added helper for memory movement (host_pinned -> device) (thanks to
#534)
* FA2/FA3 don't read the `max_seqlen_q` from host anymore. This was
error prone. Instead rely on a fixed value for now. (It's error prone
because it's baked in the command buffer and can result in wrong value
if the value changes but the command buffer is not recorded again).
2026-05-18 15:40:04 +02:00
Corentin Godeau
57424e6c41
zml/buffer: added fromBytesOpts (#544)
Since there is `fromSliceOpts`, there should be `fromBytesOpts`
2026-05-18 15:31:00 +02:00
Corentin Kerisit
55474ea31f
zls: resolve translate-c toolchain in zls rules (#538)
This one slipped from #537 

It worked because if there is no translate-c toolchain, zig uses zig
translate-c.
But we want to respect user choice here too.
2026-05-16 09:58:34 +02:00
Corentin Kerisit
886d9000be
workspace: use latest version of upstreamable rules_zig (#537)
The upstreamable branch now contains only 2 minor commits. Everything
else inccluding test-obj, translatec and bootstrap have been upstreamed.
2026-05-15 21:01:50 +02:00
loupicaaa
dd0ab6c334
zml/moe : Moe backend triton zig, qwen3_5_moe model (#529)
### Overview
Context: Modern models utilize a MLP called Mixture of Experts (MoE). In
a MoE, the MLP is organized into "experts," where specific sub-tensors
are specialized to process particular linguistic patterns or elements.
During inference, the hidden states of each token are routed to the most
relevant experts, ensuring that matrix multiplications (matmul) are
performed only on the active parameters.

Problem: Due to the sparse and dynamic nature of MoE where token
assignment is determined at runtime via routing, treating an MoE layer
as a standard matmul results in significant performance degradation.
Standard kernels are not optimized for the irregular memory access
patterns in sparse routing.

Approach: MoE is implemented as a primitive in the ZML operation set.
The generic MoE Backend is designed to support various specialized
kernels tailored to specific hardware and weight precisions. The primary
implementation currently features a Triton MoE backend optimized for
NVIDIA Hopper architecture using BF16 and FP8 weights. This
implementation is utilized in the Qwen 3.5 MoE model.

### Features
**Generic MoE Backend (zml/moe/moe.zig)**
- Integrate specialized MoE backends (e.g., Triton).
- Select the optimal backend based on hardware and precision.
- Implements essential utilities and structures for registration,
parameter loading, and metadata management.
- The forwardMoe() function acts as the entry point to invoke the
appropriate backend.

**Triton MoE Backend (zml/moe/triton.zig)**
- Generates TTIR (Triton Intermediate Representation) at compile time.
- Leverages Triton's custom calls exposed via XLA.
Execution Flow:
1. Select the optimal configuration for the fused MoE kernel.
2. Determine token distribution: use naive assignment if the
distribution is sparse; otherwise, sort and align tokens for efficiency.
3. Quantization: quantize activations if using FP8 weights.
4. Fused Kernel: execute the fused MoE kernel for the gate and
up-projection layers.
5. Post-Processing: split the output and apply the activation function.
6. Quantization : re-quantize activations for the next stage if required
by the FP8 path.
7. Fused Kernel: execute the fused MoE kernel for the down-projection
layer.
8. Aggregation: sum the Top-K results for each token.

Qwen 3.5 MoE
- Fully utilizes the Triton MoE backend.
- BF16 weights implementation (also tested with FP8 but not implemented
here)

---------

Co-authored-by: loupicaaa <louis@zml.ai>
Co-authored-by: Raphael NAHOUM <raphael@zml.ai>
2026-05-15 11:06:02 +02:00
Guillaume Wenzek
37ef0a87c0
mlir: fluent api (#533) 2026-05-15 10:53:54 +02:00
Steeve Morin
c8cc476042 third_party/zls: adapt to rules_zig changes
notably when using with_cfg targets
2026-05-13 11:35:34 +00:00
Corentin Godeau
3b718a5a89
zml/tensor: add Tensor.uninitialized() (#535)
This PR exposes the XLA-specific custom calls that allow to create
uninitialized tensors (thus not paying the cost of a kernel to
essentially do a memset).

The `stablehlo` part of it was already done, so I added a wrapper in
`tensor.zig`
2026-05-13 10:40:20 +02:00
Corentin Godeau
8418c97624
zml/tensor: add Tensor.onMemory() (#534)
It is now required with newer plugin to annotate the executable
arguments with their location.

Like `Tensor.reuseBuffer()`, you can now use `Tensor.onMemory()` to
specify on which memory a tensor is located.
2026-05-13 10:39:07 +02:00
Benjamin Rabier
ca8599aea0
zml/attention: attnd (#450)
Simple reference implementation of attnd client as a attention backend
relying on POSIX. I've tried various things with `sendmmsg`/`recvmmsg`
and `io_uring` out of curiosity but they weren't worth the extra code
here.

The integration as a attention backend is a bit clunky, we're still
allocating a KV cache and there are a few panics here and there because
the current initialization mechanism of attention backend doesn't work
for attnd, but that's something for later.
2026-05-12 17:27:37 +02:00
elogir
c917acb572
kernels/triton: add LLVM dialect, pipelining hints and fixes (#525)
This PR introduces the following:

- Wire up the LLVM dialect so the Triton builder can emit
`llvm.intr.assume` so we can match
[`triton.language.assume`](https://triton-lang.org/main/python-api/generated/triton.language.assume.html#triton.language.assume)
- Thread `tt.num_stages` through `scf.for` for pipelining hints
- Add an inline-return mode to `ReturnIfScope`. Not necessary but better
for IR matching with Python reference
- Build DSL kernels IR in a new MLIR context

Also fixes i1 extension to use `extui` instead of `extsi`.
2026-05-12 14:54:42 +02:00
Hugo Mano
eae31a51b8
zml/sharding: fix segfault in tensorShardingAttr for gspmd (#532) 2026-05-12 13:34:08 +02:00
Corentin Kerisit
e67f60fe25
workspace: remove custom registry (#531)
Unused and noisy
2026-05-12 10:32:51 +02:00
Corentin Kerisit
31887689a9
workspace: add several bazel 9 required changed (#530)
Just a few stuff to get us closer to Bazel 9
XLA is the main blocker and i'll tackle this as soon as other
dependencies merge my PRs.
2026-05-12 10:27:18 +02:00
elogir
144f3eb0af
platforms/neuron: bump toolchain and stabilize support (#476)
- bump Neuron dependencies in `platforms/neuron/requirements.lock.txt`
- `libneuronxla` `2.2.15515.0+50c26cbd` -> `2.2.16408.0+50c26cbd`
- `neuronx-cc` `2.23.6484.0+3b612583` -> `2.24.8799.0+6f62ff7c`
- make `neuronx-cc` invocation more robust by fixing path resolution,
releasing the Python GIL during compilation, checking the compiler
process exit status, and propagating NEFF wrapping/read
failures
- resolve PJRT devices and memories by PJRT handles instead of assuming
stable numeric ids during platform initialization, including Neuron
cases where memory ids are reused across devices
- avoid unsupported Neuron synchronization/event paths by always
requesting execution events on Neuron, deinitializing them without
awaiting, and removing the explicit await from the sharding
example
- avoid unsupported Neuron runtime paths by:
- falling back to host-backed buffer initialization for
`Buffer.uninitialized`
- using passthrough DMA allocation on Neuron
- skipping `Tensor.print()` custom calls on Neuron
- add NRT-backed Neuron physical mesh topology construction, including
chip/core placement, ring/torus/point-to-point topology inference, and
Neuron local-hardware-id based shard placement
- let the sharding example pick the default partitioner from the current
target
2026-05-12 09:58:32 +02:00
Benjamin Rabier
532c22955d
zml/ops: Merge the two customCall and add compute_on_host (#511)
Add `compute_on_host` to force host offloading and merging the
`customCallInternal` and `typeCustomCall` function into one to avoid
duplication. I've slightly simplified the signature of some of the
functions I wrote initially to avoid passing the types explicitly.
2026-05-11 18:15:42 +02:00
Hugo Mano
10536817a4
zml/sharding: fix leak and use mlir.Attribute instead of concrete types (#527)
Co-authored-by: Guillaume Wenzek <wzk@zml.ai>
2026-05-11 11:16:56 +02:00
Reese Chong
86aeed775e
docs: fix excerpts don't match final code (#524)
Two fixes to the `write_first_model.md` tutorial:

- `allocator` on line 221 is deinitialized, however, `io` is not. `io`
should be deinitialized as well.

- `sharding` on line 216, is an undeclared identifier. It should be
replaced with `.replicated`, which is used in the full code example.
2026-05-11 10:37:52 +02:00
Hugo Mano
13cee57bf2
zml/profiling: unify profiling flow and tracing (#443)
This PR modernizes ZML profiling in four areas:

- replaces the old C++ `xspace_to_perfetto` tool with a native Zig
exporter built on upb
- integrates trace export directly into `zml/profiler.zig`
- introduces a scoped tracing API in `zml.tracer` and wires it through
examples
- adds platform-specific annotation support and Bazel-native profiler
wrappers for CUDA and ROCm

The result is a simpler profiling flow, better large-trace behavior, and
a more consistent experience across Linux CUDA, Linux ROCm, and macOS.
2026-05-11 09:52:11 +02:00
Guillaume Wenzek
5d94f9b83c
mlir: add shardy dialect (#526)
Add bindings to shardy dialect,
remove the string generation code from zml.Sharding, and replace it with
mlir calls.
2026-05-11 09:37:05 +02:00
Guillaume Wenzek
0979289ffe
zml/sharding: pass sharding by pointer and simplify replicated usage (#514)
This PR simplifies the common replicated case while making sharding
intent explicit and use sharding by pointer ( sharding struct is quite
big: ~600 bytes).

Using pointers for shardings also avoids unnecessary sharding copies and
makes it easier to share stable sharding instances across compilation,
loading, and runtime paths.

- `Platform` owns the canonical replicated sharding
- `Buffer` APIs now take explicit sharding specs:
    - `.replicated`
    - `.{ .sharded = sharding_ptr }`
- `createTensor` / `maybeCreateTensor `now require explicit partitioning
instead of accepting null
- compilation/loading now use pointer-based sharding lists

This makes simple cases like mnist and benchmark much cleaner: callers
no longer need to manually construct and thread a replicated sharding
just to use the default case, but they still state placement explicitly
at the call site.

---------

Co-authored-by: Hugo Mano <hugo@zml.ai>
2026-05-06 12:46:15 +02:00
Guillaume Wenzek
c38179513d
bump xprof-nightly (#523)
the previous nightly we depend on is not on pypi anymore.
I wanted to switch to stable release but 2.23 isn't public yet:
https://pypi.org/project/xprof/#history
2026-05-06 12:26:09 +02:00
Hugo Mano
92b8e34c7b
zml/profiler: post-process saved xplane profiles and clean up profiler FFI (#522)
This change fixes profile timestamp handling so saved xplane.pb data is
post-processed like JAX/XLA’s `ProfilerSession`, and also cleans up the
Zig/C++ FFI used by the profile export bridge.

Before this, we were writing raw profiler output from the PJRT extension
directly to `profiling.xplane.pb`, which bypassed XLA’s single-host
post-processing step done by JAX that we can find here:
efac222ebf/tsl/profiler/lib/profiler_session.cc (L100)

That made ZML behave differently from JAX, including wrong timeline
origins in downstream viewers like XProf.
To fix that, we now run
`tsl::profiler::PostProcessSingleHostXSpace(...)` before writing the
protobuf, and we treat that post-processed xplane.pb as the source for
subsequent Perfetto conversion.
We also register start/stop timestamps in `zml/profiler.zig` so they
match XLA’s expectations for session timestamps with `real` clock.

While touching this path, I tightened the FFI boundary as well. 
Error text returned from the xspace export bridge is now allocated with
the provided Zig allocator instead of malloc, C receives a pointer to
the `zig_allocator wrapper` to avoid lifetime issues from passing
allocator state by value, and error propagation in
`tools/xspace_to_perfetto_lib.cc `now uses `zig_slice` outputs directly
instead of building std::string error objects just to copy them back
out. This removes mixed allocation ownership between C++ and Zig and
fixes the allocator lifetime bug that could crash across the FFI
boundary.
2026-05-05 14:52:15 +02:00
Steeve Morin
57d191d092 zml/attention: make triton the default paged attention provider on cuda 2026-05-04 13:04:11 +00:00
Steeve Morin
b230be2f3f platforms: bump PJRT=manual-2026-04-30T17-13-00Z
notably, this bumps fixes TTIR gen problems now that we switched away
from python based TTIR
2026-04-30 17:34:49 +00:00
Steeve Morin
5e4d1b1d05 zml: bump allocation to 90% of the GPU memory
up from 85%

since we now usually compile _before_ loading models (at least on llmd)
2026-04-30 17:34:06 +00:00
Steeve Morin
e2ab3022de platforms: for aarch64 cpu flag when selecting the linux_arm64 platform
this fixes highwayhash outdated build files that select on this

sadly we can't update highwayhash just yet
2026-04-30 17:32:56 +00:00
elogir
8e4c333007
zml: replace Python Triton/Mosaic IR generation with native Zig DSLs (#517)
Drops the Python build-time dependency for Triton and Mosaic TPU
kernels. ZML now emits TTIR and Mosaic TPU IR directly from Zig, against
the same MLIR context as the rest of the compiler. So no more JAX,
Triton at build time and a lighter sandbox.

Split into two commits:

- **`c72f44884` — Add Triton and Mosaic Zig DSL.** Additive: IR builders
(`kernels/triton/`, `kernels/mosaic_tpu/`), upstream dialect bindings
(`ttir`, `mosaic_tpu`, plus `arith`/`scf`/`math`/`memref`/`vector`/
`affine`/`cf`), and the high-level
`zml.kernel.{triton,mosaic_tpu}.Kernel(Config, spec)` declarative form.
Comes with author guides in each `README.md`.

- **`cb61e529f` — Remove Python and replace with Zig kernels.** Migrates
the two production kernels (`unified_attention` for Triton,
`ragged_paged` for Mosaic TPU) and deletes everything Python related in
the sandbox and what was generating the IR.

---------

Co-authored-by: Steeve Morin <steeve@zml.ai>
2026-04-30 16:13:48 +02:00
Olivier Boulant
1c3b505b9e
swig(tokenizer): remove unsupported SWIG optimal hint (#501)
This PR removes `optimal="1"` from the `[const std::string&]` out
typemap in `zml/tokenizer/sentencepiece/sentencepiece.i`.

SWIG cannot apply the requested optimal code path for the `%extend
std::string { data() }` wrapper and emitted `Warning 474: ` during
SwigC. Removing the unsupported hint preserves the generated behavior
while eliminating the warning.

**Reproduce**
```
$ bazel build //examples/llm
[...]
zml/tokenizer/sentencepiece/sentencepiece.i:41: Warning 474: Method std::string::data() const usage of the optimal attribute ignored
zml/tokenizer/sentencepiece/sentencepiece.i:25: Warning 474: in the out typemap as the following cannot be used to generate optimal code: cppresult = (std::string *) &std_string_data__SWIG((std::string const *)arg1);
[...]
```

**Validation**

* Rebuilt `//examples/llm:llm` and a clean
* Confirmed `Warning 474: ` no longer appears
* Confirmed build still succeeds
* Warning 524 remains and is not addressed in this PR
2026-04-29 18:26:58 +02:00
Olivier Boulant
df4bb4c3dd
bazel: suppress SWIG c-target warning due to intended behavior (#504)
This suppresses SWIG Warning 524 in `bazel/swig.bzl`. 

The custom SWIG rule intentionally parses C++ interfaces while
generating a C ABI wrapper for Zig consumption (in
[sentencepiece.zig](zml/tokenizer/sentencepiece/sentencepiece.zig)), so
this warning is expected noise from an intentional -c backend selection
rather than a build problem.

**Reproduce**
```
$ bazel build //examples/llm
[...]
SWIG:1: Warning 524: Experimental target language. Target language C specified by -c is an experimental language. See the 'Target Languages' section in the Introduction chapter of the SWIG documentation.
[...]
```

**Validation**
* Built `//examples/llm:llm` after a clean
* Confirmed Warning 524 no longer appears
* Build still succeeds without changing generated binding behavior
2026-04-29 18:24:43 +02:00
Steeve Morin
b0b7e02692
zml: remove hftokenizers (#518)
This is the second take of #463, now that #482 is merged.
2026-04-29 18:15:26 +02:00
Guillaume Wenzek
039a638aeb
examples/llm: cleanups (#512)
I'm doing a pass on the example code we have.

* llama/model.zig, qwen/model.zig: change layer_index from being a
tensor to being an integer.
It's clearer this way what is happening. Note that lfm doesn't have the
issue because the layer index is passed everywhere.

* tokenizePrompt: the code was doing unnecessary copies of data, I added
some helper to tokenizer.encoder to streamline the code by removing the
extra copies and allocations.

* `.stdout().writer() -> .stdout().writerStreaming()` this one is weird
but apparently using stdout by position isn't always allowed, in
particular when running the exe from within SublimeText.

* runDecode -> show case bake
2026-04-29 10:38:19 +02:00
Kevin Barre
0fd551e4d5
zml/zls: Add compatibility for aarch64-linux platform in zls_toolchain (#508) 2026-04-28 09:32:40 +02:00
elogir
5300905228
platforms/rocm: update amdgpuids URL and bump llvm version (#510)
This PR fixes two issues:

- Update `amdgpu.ids` URL to Gitlab, now that the old link is dead
- Bump LLVM version to `0.7.3`
2026-04-27 19:59:31 +02:00
Kevin Barre
29ad498772
zml/tpu: support padded KV-cache head dims in ragged paged attention (#507)
fix(mosaic_tpu): support padded KV-cache head dims in ragged paged
attention

Pass both logical and cache head dimensions through the Mosaic TPU paged
attention path so KV pages can remain 128-byte aligned without padding Q
or trimming the final output.

- add backend helpers and tests for Mosaic TPU kernel head-dim alignment
- keep query/output tensors and softmax scaling on the logical head dim
- allow cache head dims greater than or equal to query head dims in TPU
IR and kernel validation
- slice padded K/V vectors back to the logical head dim before attention
math
- fix the tensor pad call signature used by tensor padding
2026-04-27 10:56:57 +02:00
Guillaume Wenzek
9211a1770e
Remove BoundedArray.init (#506)
When looking at the new Sharding code I was triggered by the high number
of "try".

I then realized most of the try where traced back to `axes = try
.init(0)` which can never fail.
So I removed BoundedArray.init (aka Andrew mode)

It's generally unsafe because it makes `slice()` return undefined.;
I replaced`try init(0)` by `.empty`
I replaced `try init(n)` by `.fromSlice` or with `.{ .buffer =
undefined, .len = n }` to explicitly show the unsafeness

This leads to widespread changes because it means that most of the
sharding operations can't fail anymore.
2026-04-27 10:31:52 +02:00
Kevin Barre
6ee021faf9
zml/tpu: vendor ragged paged attention kernel for strided KV loads (#505)
Improve TPU ragged paged attention strided KV loading

- add the TPU ragged paged attention kernel to the repo and package it
with gen_ir
- switch gen_ir to import the vendored kernel instead of JAX's default
one
- update tpu_attention tensor axis tags to match the kernel's expected
shapes
2026-04-24 13:05:24 +02:00
Kevin Barre
bfb792342b
zml/io: do not fail .auto when application default can't be loaded (#502)
Co-authored-by: Copilot <copilot@github.com>
2026-04-23 17:43:37 +02:00
Benjamin Rabier
e79311ccf4
platforms/cpu: remove sha256: prefix from the pjrt artifact's sh256 (#499) 2026-04-23 11:17:40 +02:00
Kevin Barre
94d1b1582f
zml/tpu: implement pagedAttentionDense for fused K V cache (#489)
refactor: introduce KVCache union and wire dense/split layout through
pagedAttention

Adds a `KVCache` union type to `paged_attention.zig` with two variants:
- `.split { k, v }`: separate K and V page tensors (used by cuda_fa2,
cuda_fa3, triton)
- `.dense`: a single pre-fused K/V page tensor (used by mosaic_tpu)

Changes:

- `pagedAttention()` in `paged_attention.zig` now accepts a `kv_cache:
KVCache`
parameter alongside `k`/`v` (unused); dispatches `.split` to CUDA/Triton
  backends and the full `KVCache` union to `mosaic_tpu`.
- `mosaic_tpu.pagedAttention()` in `tpu_attention.zig` now takes
  `layout: paged_attention.KVCache` instead of separate k/v tensors, and
  passes `layout.dense` to the existing `prepareInputs()` helper.
- `KvCache.Layer` in `llmd/models.zig` is now a tagged union of `.dense`
(fused tensor) and `.split { k, v }`. `attentionLayout()` converts it to
`attention.KVCache`. The `.dense` update path in `Layer.update()` fuses
  K and V by inserting a `.kv` axis, concatenating, and merging `.hkv`
  before scattering into the page cache.
2026-04-22 16:48:30 +02:00
Hugo Mano
f80f925409
examples/llm: add profiling option (#497)
This allow use to easily trace what going on
2026-04-22 16:23:36 +02:00
Imran Alam
83ebe37e93
bin/zml-smi: Fix zml logo rendering in KDE Konsole (#494)
Specifying both `rows` and `cols` makes the zml logo in `zml-smi` render
correctly in KDE Konsole. Verified that the fix didn't cause any
regression in a terminal emulator where logo rendering was already
functional (`Ghostty`).

`BUILD.bazel` is also updated to run the `zml-smi`  tests.

**Before**
<img width="348" height="238" alt="image"
src="https://github.com/user-attachments/assets/7abc604d-14b9-4c0a-a561-9acde1d5359e"
/>
**With Fix**
<img width="348" height="162" alt="image"
src="https://github.com/user-attachments/assets/d93a7253-c1fc-4ff9-891b-2e0c6ec46a65"
/>

**Test Discovery Fix**

```
master $ bazel query 'kind(zig_test, //bin/zml-smi/...)'
INFO: Empty results

master $ git checkout -
Switched to branch 'fix/zml-smi-logo-size'

fix/zml-smi-logo-size $ bazel query 'kind(zig_test, //bin/zml-smi/...)'
//bin/zml-smi/tui:test
```
2026-04-22 12:39:14 +02:00
Corentin Kerisit
241aaf7d3b
platforms/cuda: add arm64 support (#493)
Co-authored-by: Steeve Morin <steeve@zml.ai>
2026-04-22 12:37:48 +02:00
Benjamin Rabier
fa3811fcd0
tokenizer/iree: Fix integer underflow (#492) 2026-04-21 15:09:21 +02:00
Benjamin Rabier
154a2cb641
workspace/xla: Use rolledback version of pjrt (#491)
Rolling back XLA to versions that work with triton 3.6
2026-04-21 15:06:39 +02:00
Hugo Mano
b25349b384
ci: bump ci actions (Node.js 20 actions are deprecated) (#490)
Use newer versions of several GitHub Actions. Also bump used Zig
version.
`mlugg/setup-zig` is still using Node.js v20 so warning still here but
only for this action. Check
https://codeberg.org/mlugg/setup-zig/pulls/55 in a near future in order
to update the action.
2026-04-21 14:27:00 +02:00
Benjamin Rabier
49c805f7e3
zml/attention/triton: thread-safe generateTtir (#488)
generateTtir isn't thread-safe as we keep a single process alive. Using
different processes didn't seem to be better from a quick try. So just
locking it with a mutex.
2026-04-20 19:28:49 +02:00
Benjamin Rabier
f258280ab8
platforms/cuda+rocm: update pjrt artifact (#487)
Update with the XLA patches for improved parallel compilation.

`//examples/llm` goes from 27s to 8s for llama3.1 8B compilation
2026-04-20 18:24:01 +02:00
Kevin Barre
c7f26eeeac
zml/attention: standardization of the mosaic ragged page attention structure (#486)
Aligns Mosaic TPU ragged paged-attention input/parameter naming with the
terminology used by other backends (e.g., block table + sequence
lengths)

Changes:
- Rename Mosaic TPU ragged attention inputs/parameters from
kv_lens/page_indices/cu_seqlens_q to
seq_lens/block_table/query_start_len.
2026-04-20 18:14:09 +02:00
Benjamin Rabier
c1a902b131
workspace/xla: update to nightly-2026-04-20 (#485) 2026-04-20 17:16:45 +02:00
Benjamin Rabier
df86cdfd96
zml/tokenizer/iree: Improve IREE tokenizer with upstream fix (#482)
I've reworked the Tokenizer API to actually work with all tokenizers and
not just IREE and moved the `Writer` part into the IREE tokenizer. I'm
now avoiding intermediate copies as much as possible now.

I'm also more defensive now to avoid the same infinite loop and put more
effort into providing appropriate output buffers to IREE. It fails with
empty output buffers and may return a `ResourceExhausted` if it's too
small. Also added a bunch of tests.

IREE tokenizer itself had a bug which I fixed here
https://github.com/iree-org/iree/pull/24150. In the meantime, I'm
pointing bazel to our fork.
2026-04-20 11:15:37 +02:00
Steeve Morin
3e3f2f7297 zml/tensor: for argxmax, mirror more closely emitted stablehlo for pattern matching
For instance, the tenstorrent plugin won't match it
2026-04-20 10:18:45 +02:00
Steeve Morin
da99fe93b5 platforms/rocm: bump ROCm=7.2.2 2026-04-17 16:48:08 +00:00
Kevin Barre
77bdfadc1a platforms/tpu: add ragged paged attention (#481) 2026-04-17 16:47:25 +02:00
Corentin Kerisit
45f65c7b97
workspace: expose translate-c so that downstream can use it (#484) 2026-04-17 14:54:14 +02:00
Corentin Kerisit
bd66cfa1f3
Update ZLS to 0.16.0 and rules_zig new providers (#483)
Bump rules_zig to latest zig_toolchain changes (for bootstrap)
Bump ZLS to 0.16.0
Amend ZLS completion rules and runners to match rules_zig new providers
2026-04-17 13:42:37 +02:00
Hugo Mano
8b12d2ef1e
bazel: make repo-wide cquery work across host and linux target analysis (#465)
This PR fixes several Bazel analysis regressions so both of these
commands work reliably:

- `bazel cquery //...`
- `bazel cquery --platforms=@zml//platforms:linux_amd64 //...`

After recent changes and the rebase, repo-wide cquery was failing for
three separate reasons:

- ZLS toolchains were over-constrained by target platform: updated ZLS
toolchain registration to constrain only the exec platform.
This avoids cross-target analysis failures when host-side completion
tooling is analyzed under a foreign target platform.
- OCI packaging targets were analyzed on host platforms where their base
images do not apply: marked Linux OCI image targets as Linux x86_64
compatible only. Also switched the Debian base image pull from Docker
Hub to the GCR mirror to avoid CI fetch failures when resolving
packaging images manifests due to rate limiting
- Triton Python deps were fetched for the host Python platform instead
of the Linux wheel platform the lockfile was generated for: restored
Linux x86_64 wheel resolution for `triton_py_deps` in `MODULE.bazel`.
This prevents macOS host analysis from attempting to fetch a
`triton==3.6.0` wheel for an unsupported host platform.


Also added a step in ci to prevent regression
2026-04-17 11:48:09 +02:00
Corentin Godeau
0c67102033
zml/module: provide an Io to the CompilationContext (#457)
This might come in handy in the future if we need to do some IO in the
compilation pipeline
2026-04-17 11:41:09 +02:00
Corentin Godeau
f23a8a14d3
zml/ops: improve metadata conversion (#474)
In #470 I introduced support for optional float in custom call metadata
but it was incomplete.

This PR cleans the code a bit and introduce support for all basic
optional types.
2026-04-17 11:35:58 +02:00
Benjamin Rabier
1e79469136
examples/llm: fix prompt tokenization (#480)
Tested with IREE and HF on all three models.
2026-04-16 11:09:47 +02:00
Corentin Kerisit
169f7a021b
workspace: bump llvm=0.7.1 (#479)
This fixes #467
2026-04-16 10:20:08 +02:00
Steeve Morin
185e75f884 workspace: bump zig=0.16.0
also bump translate-c and remove a genrule in arocc
2026-04-15 21:42:47 +02:00
Steeve Morin
b912b08c40 zml/attention/triton: sandbox zlib1g when running the triton codegen
else python fails in a container
2026-04-15 21:32:51 +02:00
Benjamin Rabier
91f58097cc
zml/tokenizer: use correct API for decoding with HF (#478)
The current API only worked with IREE. Reworked it to work with both HF
and IREE
2026-04-14 17:13:26 +02:00
Steeve Morin
2f11092deb workspace: correctly set zig mode on host tools
this was a typo
2026-04-14 13:10:33 +02:00
Steeve Morin
46862fa85a workspace: put overrides next to the dep
it's clearer, also, do a buildifier pass
2026-04-14 13:09:50 +02:00
Steeve Morin
e58c6af965 workspace: pin hermeticbuild/hermetic-llvm to git
this is until the release is done, this fixes duplicate symbols
between clang and zig compiler_rt
2026-04-14 13:07:54 +02:00
Corentin Godeau
35e8ae9082
zml/ops: fix comptime for optional float value in custom call metadata (#472) 2026-04-14 10:07:03 +02:00
Corentin Godeau
6ad3e26790
zml/attention: add option to set the softmax scaling (#470)
I added a `scale` field in the `AttentionOptions` struct that is passed
through the attention backend.

I had to add support for optional in the custom call metadata. For now
it only supports optional floats, we'll add support for other types
later.
2026-04-14 09:48:32 +02:00
Corentin Godeau
e47b7ea390
zml/nn: add proportional scaling option to RopeOpts and related tests (#471)
Co-authored-by: Kevin Barre <kevin@zml.ai>
2026-04-14 09:44:05 +02:00
Benjamin Rabier
d343b84fd8
zml/tokenizer: Re-introduce HF tokenizer (#469) 2026-04-14 09:42:02 +02:00
Benjamin Rabier
16fb132dec
zml/attention/triton: rename .num_pages_per_seq to .p (#466)
Using the same axis name as FlashAttention for consistency.
2026-04-13 15:16:18 +02:00
KB
bd113697d4
zml/attention: add Linux compatibility only for Triton dependencies (#464)
Co-authored-by: Kevin Barre <kevin@zml.ai>
2026-04-13 14:56:55 +02:00
Steeve Morin
8ec67ee5c3
zml: remove HuggingFace tokenizer (#463)
Following #416, the HuggingFace tokenizer is redundant.

This also removes the Rust code from the repo, as well as rules_rust and
rules_rs.
2026-04-13 12:42:44 +02:00
KB
8edf02d0c3
zml/chore: bump Zig to 0.16.0-dev.3133 and adapt codebase (#462)
Update Zig compiler to 0.16.0-dev.3133+5ec8e45f3 and adjust code
for breaking standard library changes:

- Replace ArrayListUnmanaged/ArrayList init with .empty
- Switch to unmanaged AutoArrayHashMap/StringArrayHashMap
- Use debug.GeneralPurposeAllocator (renamed upstream)
- Replace std.Io.File.ReadStreamingError with error.EndOfStream
- Refactor meta.visit to use structured visitor pattern
- Fix alignment allocation in mem.bufferizeInner

Also update third-party dependencies: flashattn 0.0.6-rc8, arocc,
uucode, libvaxis, and zigimg patches.

Co-authored-by: Kevin Barre <kevin@zml.ai>
2026-04-13 11:56:32 +02:00
Corentin Godeau
ac9bc32d89
zml/attention: add Triton Unified Attention backend (#417)
This PR introduces a Triton paged attention backend, unlocking ROCm
support.

It relies on a Python sandbox to generate the TTIR.

The major change is that the KVCache now needs to be per-layer for paged
attention. The `layer_index` is now superfluous and will probably be
removed in the future.
2026-04-13 11:39:08 +02:00
elogir
1362a2cc45
bin/zml-smi: fix release script (#460)
- bucket name in rclone config
- version name will be tag
2026-04-13 11:09:43 +02:00
Benjamin Rabier
177fddabeb
zml/attention & zml/ops: mark fa2 as sharding aware and fix implementation (#461)
Sharding aware case wasn't properly handled and I marked fa2 as sharding
unaware even though it is.
2026-04-11 10:18:44 +02:00
elogir
572ba04a0c
bin/zml-smi: fix mapfile (#458)
Replace mapfile with manual IFS read
2026-04-10 17:55:27 +02:00
Steeve Morin
e71c523605 workspace: swap strip linkopt to bazel flag
because --strip-all doesn't exist on macos
2026-04-10 17:30:09 +02:00
elogir
8701a54882
bin/zml-smi: add prometheus endpoint, json and macos host infos (#455)
This PR introduces:

- Prometheus metrics endpoint through `--prometheus` and optionally
`--prometheus-port`
- JSON output mode with `--json`
- macOS arm64 host information
- An install and release script
2026-04-10 17:26:09 +02:00
Steeve Morin
ef66f310ec workspace: bump rules_zig
to segregate exec platform only options
2026-04-09 10:23:11 +02:00
Steeve Morin
adee932efa
zml/io/vfs: add GCS VFS (#453)
This pull request adds support for Google Cloud Storage (GCS) to the
virtual file system (VFS) module, alongside some build configuration
updates. The main changes introduce the new `GCS` backend, make it
available in the VFS API, and register it in the example application.

Authentication is handled like other GCS libraries via the
`GOOGLE_APPLICATION_CREDENTIALS` env or application default account
files.

**IMPORTANT NOTE**: Only Authorized User and Metadata Server auth are
implemented. Service Account auth isn't implemented yet.
2026-04-07 19:05:43 +02:00
elogir
57546912f1
bin/zml-smi: Add CSV export and driver version reporting (#452)
- Restructured `bin/zml-smi/`
- Added CSV export
- Added driver version reporting
- Minor fixes
2026-04-07 16:44:52 +02:00
Benjamin Rabier
448084908a
zml/tokenizer: add IREE tokenizer (#416)
Adding the [IREE
tokenizer](https://github.com/iree-org/iree/blob/main/runtime/src/iree/tokenizer/README.md).Being
made from the ground up with streaming (de-)tokenization in mind, I've
extended the APIs to expose a `std.Io.Writer` for the tokenization.

I've quickly benchmarked IREE's tokenizer vs HuggingFace's one with a
request I retrieved from a coding agent (opencode) in the middle of a
session. The request body was 360KB encoded in 95K tokens. I also did
another test with a similarly sized request with the same results as
shown hereafter.

I've tested two cases:

- oneshot: the template is rendered all at once and only then we encode
the whole string.
- stream: the template rendering and encoding are pipelined to avoid the
intermediate allocation.

There is a slight difference in the generated tokens between HuggingFace
and IREE. Some tokens like `introdu` were encoded as `i`, `n`, `t`, etc.
Decoding them returns the same text. It's very close in terms of total
token count:

- hf: 95779
- iree/stream: 95705
- iree/oneshot: 95587

| Model | IREE avg (oneshot / stream) ms | HuggingFace avg (ms) | Uplift
vs HF (oneshot / stream) |
|---|---:|---:|---|
| liquidai/LFM2.5-1.2B-Instruct | 8.5 / 7.3 | 40.5 | 4.8x / 5.5x |
| Qwen/Qwen3.5-9B | 6.0 / 8.2 | 59.4 | 9.9x / 7.2x |
| mistralai/Ministral-3-3B-Reasoning-2512 | 7.4 / 6.9 | 59.2 | 8x / 8.6x
|


It's unclear why the Qwen's chat template is somehow slower in the
stream case, maybe because there are more writes. Adding non-empty
buffer in the interface helps a bit, but tends to slow down mistral
case. So well

---------

Co-authored-by: Corentin Kerisit <corentin.kerisit@gmail.com>
2026-04-07 16:42:55 +02:00
Benjamin Rabier
70c5d1c588
zml/ops: Stronger typing of custom calls and more code re-use beyond flashattn (#454)
Today, creating a custom call is a bit tedious and error prone. The
primary goal of this PR is to only accept structs as
input/output/attributes and rely more strongly on the names everywhere
when creating as custom call. I've migrated the `fa2` custom to provide
an example. Can migrate the rest.
- Custom call relies on the position of the input/outputs/attributes to
be in the correct order to work. The `output_operand_aliases` also maps
the outputs to the input field position.
- If the custom call is "sharding aware", as in it knows it works on a
single partition on multi devices setup, one must use
`zml.ops.manualComputation` which is a bit tedious to set up.
- `flashattn.zig` has a bunch of logic to register said custom call but
also to map inputs into something more usable which can be shared.
- both `zml.ops.customCall` and `zml.ops.manualComputation` tend to
accept to many different types IMHO which makes it a bit hard to follow.
But well, to be fair not sure how much simpler my code is. I do
constrain myself to only accepting simple structs for the generic types.

I've re-used the existing `manualComputationInternal` and
`customCallInteral` for this new API without touching them. The only
exception is the `output_operand_alises` where I needed a bit more
flexibility.
2026-04-07 15:17:33 +02:00
Steeve Morin
1a9e66c1c3 bazel/stamp: generate .c file and replace genrule by run_binary+tool
This makes it remote build compatible and cross platform compatible.
2026-04-03 15:53:54 +02:00
Steeve Morin
96414d29c4 workspace: add stamping and helpers 2026-04-03 12:55:13 +00:00
Steeve Morin
e69508f364 zml/io/vfs/s3: don't treat 404 as Unexpected
It's completely normal to stat a missing file.
2026-04-03 12:06:18 +00:00
Tristan Gautié
333ec93c47
Fix slice lifetime issue (#451)
We fix a small slice lifetime issue in the prompt template of the Qwen3.5 model in LLM CLI
2026-04-03 13:11:45 +02:00
Steeve Morin
c6931ba2be bazel/patchelf: use --force-rpath in rpath ops
This allows setting DT_RPATH instead of DT_RUNPATH. DT_RPATH takes
precendence to LD_LIBRARY_PATH.

Sometimes LD_LIBRARY_PATH is useful, sometimes it's a pain. Using this
enables us at least to precede it.
2026-04-03 09:18:40 +00:00
Benjamin Rabier
c3c0a69689
ci: update actions (#447)
- Update setup-bazel & checkout to latest
- Add a bunch of `tags = ["manual"]` for docker image related targets
2026-04-01 10:48:10 +02:00
Benjamin Rabier
c817fe94f8
Revert "flashattn: update flash-attention to v2026.3.23" (#446)
This reverts commit 7f71fa58c2.
2026-04-01 10:13:12 +02:00
Hugo Mano
65e9b4487e
examples/llm: replicate Qwen 3.5 KV heads when TP exceeds KV head count (#444)
Handle Qwen3.5 full-attention K/V heads like vLLM when
num_key_value_heads < tp: replicate K/V heads across the model mesh
instead of sharding the KV head axis.

**Motivation**

Qwen3.5 9B has 4 KV heads. On 8-way TP, sharding the KV head axis over
`.model` is invalid because the dimension is not divisible by the
tensor-parallel degree.

This is a placement issue, not a model-shape issue. The model still has
4 logical KV heads, so the right fix is to replicate those heads across
ranks rather than materialize fake extra heads.

This also matches cost modeling better: replication represents
duplicated placement of the same logical data, while repeating would
incorrectly make the tensor look larger than the model actually defines.

**Changes**

- Add a policy helper to detect when KV heads should be replicated
instead of model-sharded.
- Apply that policy to projected K/V tensors in Qwen3.5 self-attention.
- Apply the same policy to KV cache shape partitioning.
- Thread model partition count through the Qwen3.5 inference compilation
path so cache placement follows the same rule.

**Why replicate instead of repeat**

Replicating:

- preserves the logical model shape
- matches grouped-query attention semantics
- keeps cost modeling honest by representing placement, not model
expansion

Repeating:

- would change the graph from 4 KV heads to 8 KV heads
- would increase KV cache size and bandwidth for no new information
- would distort cost modeling by turning a placement choice into fake
tensor growth

**Result**

Qwen3.5 now supports TP setups such as 8 GPUs where KV head count is
smaller than the tensor-parallel degree.
2026-03-31 18:07:02 +02:00
elogir
67b4fbdbb8
bin: Introduce zml-smi (#440)
This PR introduces **zml-smi**, a tool to monitor metrics of any
platform supported by ZML, featuring but not limited to:

- real time `--top` mode
- per process tracking
- responsive layout

---------

Co-authored-by: Steeve Morin <steeve@zml.ai>
2026-03-31 12:47:07 +02:00
Steeve Morin
b675cfbbe3 workspace: bump @llvm=0.7.0
For macOS SDK fix
2026-03-31 10:28:15 +02:00
elogir
4ecdd92cb0
platforms/rocm: merge rocm and mesa amdgpu.ids (#441)
This PR introduces a script to merge `amdgpu.ids` from two sources: Mesa
and ROCm.

Both provide GPU ID-to-name mappings, but each may contain entries the
other lacks. For example, Mesa includes `AMD Radeon 8060S Graphics`
while ROCm does not, and ROCm lists newer GPUs absent from Mesa.

The script merges the two sources together, with ROCm entries taking
priority in case of conflicts.
2026-03-27 14:45:35 +01:00
Hugo Mano
6e7cb90728
zml/platforms/rocm: update to ROCm 7.2.1 with full hermeticity (#439)
Fix ROCm runtime hermeticity in the packaged LLM sandbox by patching transitive ROCm libraries to use zmlxrocm_dlopen and by adding missing libdw/liblzma/libbz2 runtime deps for distroless execution. Also update the ROCm packaging to 7.2.1 and refresh the pinned libpjrt_rocm
2026-03-26 17:52:48 +01:00
Benjamin Rabier
9fa7768808
examples/llm: better single prompt UX + --topk (#438)
- Same UX as the chat
- Add `--topk` argument
2026-03-26 15:58:03 +01:00
Corentin Kerisit
127b195a52
workspace: minor README edits (#432) 2026-03-26 15:47:47 +01:00
Steeve Morin
3828a571bc platforms/neuron: minor neuron fixes 2026-03-25 18:05:23 +00:00
Benjamin Rabier
268ef6e058
examples/llm: fix --help & relative/home paths (#437) 2026-03-25 15:14:22 +01:00
Benjamin Rabier
bba9230f0b
ci: Increase CPU core count on runners (#434)
```
4cpus: 
    total: 2m56s
    ci-cpu: 2m14
    ci-neuron: 2m07s
    ci-tpu: 1m58s
8cpus:
    total: 2m26s
    ci-cpu: 1m37s
    ci-neuron: 1m47s
    ci-tpu: 1m36s
16cpus (mi8-flex):
    total: 2m07s
    ci-cpu: 1m24s
    ci-neuron: 1m26s
    ci-tpu: 1m19s
16cpus (c8i-flex):
    total: 1m58s
    ci-cpu: 1m17s
    ci-neuron: 1m15s
    ci-tpu: 1m17s
32cpus:
    total: 1m48s
    ci-cpu: 1m05s
    ci-neuron: 1m06s
    ci-tpu: 1m02s
```

RunsOn tends to prefer m8i instance (memory-optimzed) rather than c8i
ones for some reason, so forced the family with `c8+c7`, maybe we could
just use `c8`. But not sure if it's worth being that selective. RunsOn
selects the best there is.
2026-03-25 14:01:03 +01:00
Hugo Mano
22db5548ec
zml/examples/llm: add sharding to Qwen 3.5 (#435)
Using tensor parallelism strategy. From 86 toks/s to 119 tok/s for Qwen
3.5 on 2x5090
2026-03-25 13:55:06 +01:00
Benjamin Rabier
0d0d5020cf
example/llm: Further simplifications (#433) 2026-03-25 10:16:26 +01:00
Corentin Kerisit
f17291e7a4
workspace: update readme for zml/v2 (#431) 2026-03-24 18:18:33 +01:00
Benjamin Rabier
8de2c42fe6
examples/llm: Simplifications (#430) 2026-03-24 18:16:25 +01:00
Corentin Godeau
2eb8423e8a
Update doc for V2 (#428) 2026-03-24 17:00:10 +01:00
Benjamin Rabier
81299c67c8
examples/llm: Common model example (#425) 2026-03-24 16:55:29 +01:00
Steeve Morin
140af6716e platforms/cuda: bump PJRT plugin 2026-03-24 15:21:47 +00:00
Steeve Morin
942d6a1f0a Revert "Set xla flags (#429)"
This reverts commit 47c0bdd94d.
2026-03-24 15:11:03 +00:00
Tristan Gautié
47c0bdd94d
Set xla flags (#429)
Fix XLA flags to avoid autotune crash on Nvidia 4090:
- Enable CublasLT on Cuda 
- Disable autotune and triton gemm
2026-03-24 15:57:14 +01:00
Hugo Mano
463ca4a762
zml: add zml.ops.while and zml.nn.GatedDeltaNet (#427) 2026-03-24 14:28:34 +01:00
Corentin Kerisit
7f71fa58c2 flashattn: update flash-attention to v2026.3.23 2026-03-23 19:03:01 +01:00
Hugo Mano
a68f0f9c85
workspace/zls: add non regression test (#426) 2026-03-23 18:33:02 +01:00
Hugo Mano
2f175540bf
ci: rework pipeline + add remote with buildbuddy
Co-authored-by: Benjamin Rabier <benjamin@zml.ai>
2026-03-23 16:07:40 +01:00
Benjamin Rabier
3b76703430
workspace: fixes for build and test //... green
Fixes:
- Added `target_compatible_with = ["@platforms//os:linux"],` for neuron
& rocm
- Updated the zig in neuron
- Updated neuron to use Python 3.12, it's at best a mess to handle two
different python version with `build //...`
- renamed `third_party/flashattn/BUILD.bazel` to
`third_party/flashattn/flashattention.BUILD.bazel` as it's not part of
this module.
- Used HTTPS & fallback URL for swig, sometimes we would end up with an
incorrect archive (wrong sha256)
- Using `--@rules_python//python/config_settings:bootstrap_impl=script`
for hermetic Python build, it relied on the system Python otherwise.
- Used a separate pip extension for neuron to avoid the extra pip args
to leak to other installations.
- Fix `zml/testing` `fn autoCall` function

---------

Co-authored-by: Hugo Mano <hugo@zml.ai>
2026-03-23 14:55:48 +01:00
Hugo Mano
a96b8bd84f platforms/cuda: fix CUDNN redist sha 2026-03-23 13:23:22 +00:00
Corentin Godeau
90fccf5f5a
Proper sharding for paged .cuda_fa3 (#423) 2026-03-23 13:07:42 +01:00
Steeve Morin
5e2bf7fc16 workspace: bump buildifier prebuilt 2026-03-21 17:33:46 +01:00
Steeve Morin
b065d976d3 third_party/zls: fix completion on single dep targets
take the opportunity to bump ZLS to a version compatible with our
current nighly.
2026-03-21 17:33:46 +01:00
Steeve Morin
78546350af stdx: remove meta.Signature
has proven unstable on Zig nightlies and mostly needed since std.Io
anyway.
2026-03-21 17:33:45 +01:00
Steeve Morin
c8e8db75e6 third_party/translate-c: bump translate-c and arocc 2026-03-21 17:33:45 +01:00
Steeve Morin
58b2a4b552 examples/llama: fix bazel_lib and tar imports 2026-03-21 16:21:57 +00:00
Steeve Morin
13c7c4bb33 platforms/rocm: bump PJRT plugin, switch to hipBLASLt instead of triton
faster compile times!

but also +10tok/s on Llama 3.1 8B (125tok/s -> 136tok/s)
2026-03-21 15:30:23 +00:00
Steeve Morin
cb5443e5da platforms/cuda: bump PJRT plugin 2026-03-21 12:15:14 +00:00
Steeve Morin
06f13f9a22 platforms: uncompress http_archives using http_bsdtar_archive
Roughly a 2x speedup according to @cerisier. So take it for what it is.
2026-03-20 16:22:26 +00:00
Benjamin Rabier
a011fb0e52
Green CI (#419)
Co-authored-by: Corentin Godeau <corentin.godeau@zml.ai>
2026-03-20 10:31:57 +01:00
Corentin Godeau
b56e86a8e5 zml/attention/flashattn: enable sharding for Fa3 backend 2026-03-19 18:10:07 +01:00
Kevin Barre
5ce4825cee fix: Correct divCeil usage in fromBackend function for proper type handling 2026-03-19 16:15:40 +00:00
Corentin Godeau
a0ef362b65 zml/attention/paged_attention: replace custom divCeil with std.math.divCeil 2026-03-18 14:59:33 +01:00
Steeve Morin
ac971b4bc6 workspace: fix upb rule loading 2026-03-18 13:03:04 +01:00
Steeve Morin
7d373133ff workspace: bump protobuf and prefer prebuilt protoc 2026-03-18 11:49:49 +01:00
Steeve Morin
a5f7e155e7 workspace: add remote build configuration 2026-03-18 11:49:49 +01:00
Benjamin Rabier
ba44bf47e7
bazel: Fix rules_zig for tests (#415) 2026-03-18 08:19:07 +01:00
Hugo Mano
619fea8b18
zml: introduce profiler (#413)
Move profiling behind a zml-owned primitive instead of exposing PJRT
details in user code. Add the XLA/TSL Bazel deps needed to convert
collected XSpace protobufs into Perfetto traces with XLA/TLS APIs
instead of having a custom fallible converter.
2026-03-17 18:14:24 +01:00
Benjamin Rabier
9422836884
third_party/zls: Remove absent attributes from latest rules_zig (#414) 2026-03-17 18:06:31 +01:00
Benjamin Rabier
ba8134f4a2
stdx/json: consistent serialization of Union, TaggedUnion and NeverNull (#412) 2026-03-17 13:58:31 +01:00
Tristan Gautie
880aeaf101 Use http archive for translate-c repo 2026-03-17 11:32:33 +01:00
Steeve Morin
9385ffbcfe workspace: update rules_zig translate-c handling 2026-03-17 11:10:14 +01:00
Hugo Mano
9fef9b85ef zml/sharding: remove partitionning for shard shape 2026-03-12 11:01:21 +00:00
Olivier Boulant
1b50eb526c
chore(zls): remove transfer-benchmark from zls completion (#410)
This allows the ZLS server to work.

Before this fix, `bazel run //:completion.print_build_config` would fail
into
```bash
ERROR: no such package 'examples/transfer-benchmark': BUILD file not found in any of the following directories. Add a BUILD file to a directory to mark it as a package.
 - examples/transfer-benchmark
 ```
2026-03-11 16:49:58 +01:00
Corentin Godeau
70e4c9953d
Bump zig to 0.16.0-dev.2722+f16eb18ce (#409)
Co-authored-by: Hugo Mano <hugo@zml.ai>
2026-03-10 15:58:48 +01:00
Hugo Mano
7b048b7e51 zml/io/hf: fix request using identity header for tree api call 2026-03-10 12:51:08 +00:00
Hugo Mano
a6058eb354 third_party: stb is a non module dep 2026-03-10 12:49:33 +00:00
Hugo Mano
abde0a04dd
zml: sharding (#407)
Co-authored-by: Benjamin Rabier <benjamin@zml.ai>
2026-03-10 11:17:57 +01:00
KB
9df955adad
feat: add stb library integration with BUILD and repo files (#408)
Co-authored-by: Kevin Barre <kevin@zml.ai>
2026-03-10 11:09:07 +01:00
Benjamin Rabier
c7ca488f3b
third_party/zls: faillible canonicalization of modules (#406)
Files provided by bazel may not exist when zls starts, so
canonicalization would fail and break zls. Now we fall back to the
provided path instead. Furthermore, to make it a bit more robust, if any
part of the post process/canonicalization fails, we return the provided
input string instead of failing.

I'm now also using the same struct definition as ZLS instead of
manipulating `std.json.Value`.
2026-03-10 09:34:22 +01:00
Hugo Mano
26c832dc38 workspace/bazelrc: switch config native from rules_rust to rules_rs 2026-03-09 11:22:45 +00:00
Steeve Morin
fd53bfcf31 platforms: set glibc version for linux platform 2026-03-07 10:53:59 +00:00
Steeve Morin
e17da6892f platforms/cuda: probe cuda-compat via an external binary
Detect CUDA Compat by spawning a probe binary that will try
to initialize it. If it succeeds, the main binary can use it
too.
2026-03-07 10:53:14 +00:00
Steeve Morin
54c2153389 platforms/cuda: enable cuda-compat only on supported GPUs
It turns out it's only supported on enterprise GPUs.

The trick is to set a soname on the compat/libcuda.so.1 and preload
it so that later resolution uses this one.

See
https://docs.nvidia.com/deploy/cuda-compatibility/forward-compatibility.html
2026-03-06 23:05:52 +00:00
Steeve Morin
2780770da7 workspace: re-enable rules_rs 2026-03-06 23:03:14 +00:00
Steeve Morin
0fda597649 platforms/cuda: add WSL support 2026-03-06 21:00:30 +00:00
Steeve Morin
8bd1676e0f Revert "bazel: Revert cuda_compat & rules_rs (#392)"
This reverts commit c53d2275b4.
2026-03-06 20:01:46 +00:00
Hugo Mano
9f1f46f686 mlir: introduce func extension registration via C++ bridge 2026-03-05 11:24:55 +00:00
Hugo Mano
0535a5b056 zml/module: make MLIR init one-time and reuse global dialect registry 2026-03-05 10:52:11 +00:00
Benjamin Rabier
8995f3a8dc
bazel: update llvm toolchain to 0.6.1 (#401) 2026-03-05 09:23:35 +01:00
Benjamin Rabier
cadf6f5759
zml/attention: Use divCeil form max_num_page and expose it (#400) 2026-03-04 10:47:32 +01:00
Benjamin Rabier
1df41a8ee1
zls: neovim fix canonicalizing paths (#399)
- Fix incorrect build config path by using runfiles.
- Canonicalize paths like neovim does for ZLS (doesn't work otherwise)
2026-03-04 08:05:37 +01:00
Benjamin Rabier
e258ecb147 Add devenv files
It defines my nix environment for this project, and it's a lot easier to
manage by committing it.
2026-03-03 17:41:40 +01:00
Hugo Mano
6be028f873 runtimes: update tpu plugin and fix mem 2026-02-25 12:42:26 +00:00
Benjamin Rabier
569bd1dfd7
examples/transfer-benchmark: Use VFS & full model for transfer-benchmark (#390)
Co-authored-by: Benjamin Rabier <benjamin@zml.ai>
2026-02-24 09:52:40 +01:00
Benjamin Rabier
8f85fc1c06
bazel: fix zls completion for zig_test (#389)
Co-authored-by: Benjamin Rabier <benjamin@zml.ai>
2026-02-24 09:52:19 +01:00
Benjamin Rabier
de284c05e0
bazel: Use target_platforms for neuron instead of pip args to force x86 (#391)
Co-authored-by: Benjamin Rabier <benjamin@zml.ai>
2026-02-23 14:11:38 +01:00
Benjamin Rabier
c53d2275b4
bazel: Revert cuda_compat & rules_rs (#392)
Co-authored-by: Benjamin Rabier <benjamin@zml.ai>
2026-02-23 14:10:37 +01:00
Corentin Kerisit
45b21db589 Replace rules_rust by rules_rs 2026-02-22 23:36:30 +01:00
Steeve Morin
6636093cb4 platforms/cuda: prevent cuda-compat leaking to host 2026-02-20 10:19:39 +00:00
Steeve Morin
a1506d3ed4 platforms/cuda: add cuda_compat functionality 2026-02-19 15:39:39 +00:00
Hugo Mano
32a8d69b06 zml/io/vfs/hf: check performRead http status correctly (aligned with s3
and http)
2026-02-19 10:36:49 +00:00
Hugo Mano
a9f2687237
zml/io/vfs: fixes for Zig 0.16.0-dev.2611+f996d2866 + new S3 VFS (#388) 2026-02-19 10:48:15 +01:00
Tristan Gautié
a175578eba
examples/llama: improve defer pattern (#386)
We improve the defer pattern in the Llama example to:
- avoid double deinit of `prefill_exe` and `decode_exe` by cleaning at
future-definition time,
- using defer instead of errdefer for future-allocated resources cleanup
in the main function scope,
- unload buffers from the arena allocator.
2026-02-19 09:28:36 +01:00
Benjamin Rabier
48df9c85e9
examples/benchmark: Add transfer-benchmark & fix benchmark (#387)
Co-authored-by: Benjamin Rabier <benjamin@zml.ai>
2026-02-19 09:27:53 +01:00
Benjamin Rabier
82c9c25836
examples/lfm: Add LiquidAI LFM2.5 (#376) 2026-02-19 09:27:10 +01:00
Steeve Morin
44f28b0335 third_party/zls: allow using outside of ZML repo 2026-02-16 20:52:23 +00:00
Corentin Godeau
aa0b4924d6 zml/nn: remove useless convert 2026-02-16 20:52:23 +00:00
Corentin Godeau
73d734cc27 zml/nn: added default value for rope_theta + linear scaling 2026-02-16 20:52:23 +00:00
Corentin Godeau
41f793fd39 zml/attention: added options for paged_attention 2026-02-16 20:52:23 +00:00
Corentin Godeau
5234081d59 zml/testing: formatting fix 2026-02-16 20:52:23 +00:00
Corentin Godeau
2bd356e0f9 zml/nn: added common rope_theta parameter to rope variants 2026-02-16 20:52:23 +00:00
Corentin Godeau
87d3eb83ed zml/attention: update to latest zml/v2 changes 2026-02-16 20:52:23 +00:00
Corentin Godeau
0f694a8903 zml: added some convenience to paged_attention 2026-02-16 20:52:23 +00:00
Corentin Godeau
bbe3af7c67 zml: added ops.scaledDot()
It implements a scaled dot product
2026-02-16 20:52:23 +00:00
Corentin Godeau
13a5ad77cc zml/testing: Made testlayer take a runtime name 2026-02-16 20:52:23 +00:00
Corentin Godeau
1302b0c166 zml/nn: Improved Yarn logic 2026-02-16 20:52:23 +00:00
Corentin Godeau
e11c687929 zml/attention: Re-introduced allocationSize 2026-02-16 20:52:23 +00:00
Corentin Godeau
e4f831cc46 zml/io: added TensorStore.hasKey() to check key existence
It's a bit hacky since it does an array traversal to see if there is a
match.
2026-02-16 20:52:23 +00:00
Corentin Godeau
f81763a11a zml/nn: Use higher precision for rmsNorm 2026-02-16 20:52:23 +00:00
Corentin Godeau
44c249de07 zml/nn: removed commented code 2026-02-16 20:52:23 +00:00
Corentin Godeau
1200262da0 zml/attention: fix wrong causal option 2026-02-16 20:52:23 +00:00
Corentin Godeau
7b3474ab89 third_party/flashattn: bump libflashattn.so 2026-02-16 20:52:23 +00:00
Corentin Godeau
9e29f44642 small fixes 2026-02-16 20:52:23 +00:00
Corentin Godeau
64223d935b flashattn lib override 2026-02-16 20:52:23 +00:00
Corentin Godeau
6b351b9b40 zml: added simple tracer support back 2026-02-16 20:52:23 +00:00
Corentin Godeau
80692454e7 Added paged attention FA3 2026-02-16 20:52:23 +00:00
Corentin Godeau
1e2ace0ced wip: argument baking 2026-02-16 20:52:23 +00:00
Corentin Godeau
ea69b89eee wip it works 2026-02-16 20:52:23 +00:00
Corentin Godeau
ed27b2722d zml: added some paged attention bindings 2026-02-16 20:52:23 +00:00
Corentin Godeau
692825cd72 Better testLayer (#383) 2026-02-16 20:52:23 +00:00
Corentin Godeau
79cec4f9a4 V2: Tensor.print() (#380)
Added `Tensor.print()` back
2026-02-16 20:52:23 +00:00
Corentin Godeau
2bafd65458 Various improvements (#379)
This pull request includes several improvements and bug fixes across the
codebase, focusing on memory handling, correctness in tensor testing,
and API consistency. The most significant changes are grouped below:

* Fixed a bug in `zml/testing.zig`'s `testlayer` by updating the
callback signature to take a mutable context and increment the index
correctly, ensuring tensors are correctly initialized.
* Improved memory management for expected output slices in `testlayer`
by switching to `zml.Slice.alloc` for allocation, using `readSliceAll`,
and freeing only the data instead of the whole slice, removing the
potential alignment problems.
* Enhanced test logging to include the tolerance value in success
messages for better diagnostics.

* Increased the evaluation branch quota in `MapRestrict` to support more
complex compile-time type mapping, preventing quota exceeded errors.
* Updated the `fromCStruct` function in `pjrt/pjrt.zig` to accept a
`meta.Struct`.

* Improved handling of output operand aliases in
`mlir/dialects/stablehlo/stablehlo.zig::custom_call` by passing
`output_tuple_indices` for multi-result functions, increasing
correctness in custom call attribute construction.
2026-02-16 20:37:34 +00:00
Steeve Morin
ee9155b7bb examples/llama: remove bufferized leak 2026-02-16 20:00:03 +00:00
Steeve Morin
f64ad1ef79 workspace: remove apple_support 2026-02-16 19:59:38 +00:00
Steeve Morin
8fd3fec244 platforms/cuda: fix cuda 13 loader names 2026-02-16 19:58:58 +00:00
Steeve Morin
18896ce10c platforms/rocm: bump to ROCm 7.2.0 2026-02-16 18:31:30 +00:00
Steeve Morin
1323d02f4b platforms/cuda: bump to CUDA 13.1 2026-02-16 14:37:37 +00:00
Steeve Morin
b72124af50 examples/llama: disable VFS (until reenabling it later) 2026-02-16 14:20:48 +00:00
Steeve Morin
f781bfdde3 workspace: make rules_zig work on linux again 2026-02-16 14:20:24 +00:00
Steeve Morin
51a36fa8d8 third_party/zls: update hashes 2026-02-16 14:13:14 +00:00
Steeve Morin
aaab6f7a86 zml: load safetensors sequencially 2026-02-16 14:12:40 +00:00
Steeve Morin
d511605d5b platforms: port to zig 0.16.0-dev.2611+f996d2866 2026-02-16 13:45:51 +00:00
Steeve Morin
dcbf5f2e27 workspace: update to 0.16.0-dev.2611+f996d2866, add remote exec
also:
- use toolchains_llvm_bootstrapped for macOS builds
2026-02-16 12:05:01 +01:00
Steeve Morin
2d65af067f examples/benchmark: platform is not a const pointer 2026-02-16 12:05:01 +01:00
Steeve Morin
639850347e pjrt: use dlopen on macos too
to keep symbols private
2026-02-16 12:05:01 +01:00
Steeve Morin
24354d34b3 pjrt: add from DynLib 2026-02-16 12:05:01 +01:00
Corentin Godeau
d7655bc029 zml/tensor: fix assert formatting 2026-02-13 11:59:30 +01:00
Benjamin Rabier
7bf0f66024
Add linenoise as bazel third party dependency (#384)
Co-authored-by: Benjamin Rabier <benjamin@zml.ai>
2026-02-12 18:31:09 +01:00
Corentin Godeau
d5cfd726ac
Better testLayer (#383) 2026-02-12 11:59:09 +01:00
Corentin Godeau
7d497907ef zml/testing: updated testLayer to new API
Also renamed to testLayer
2026-02-11 14:14:37 +01:00
Corentin Godeau
2f9f01e9b9 zml/tensor: Added checks on Tensor.dot() 2026-02-10 17:29:14 +00:00
Corentin Godeau
2218ec0f3f
V2: Tensor.print() (#380)
Added `Tensor.print()` back
2026-02-10 10:11:48 +01:00
Corentin Godeau
ef60536186 zml/slice: fix formatting
Also added a test
2026-02-09 18:03:03 +00:00
Corentin Godeau
24ca08692d zml: fix tests 2026-02-09 18:02:48 +00:00
Corentin Godeau
1fd90f1e45
Various improvements (#379)
This pull request includes several improvements and bug fixes across the
codebase, focusing on memory handling, correctness in tensor testing,
and API consistency. The most significant changes are grouped below:

### Tensor Testing and Memory Management Improvements

* Fixed a bug in `zml/testing.zig`'s `testlayer` by updating the
callback signature to take a mutable context and increment the index
correctly, ensuring tensors are correctly initialized.
* Improved memory management for expected output slices in `testlayer`
by switching to `zml.Slice.alloc` for allocation, using `readSliceAll`,
and freeing only the data instead of the whole slice, removing the
potential alignment problems.
* Enhanced test logging to include the tolerance value in success
messages for better diagnostics.

### API and Code Quality Enhancements

* Increased the evaluation branch quota in `MapRestrict` to support more
complex compile-time type mapping, preventing quota exceeded errors.
* Updated the `fromCStruct` function in `pjrt/pjrt.zig` to accept a
`meta.Struct`.

### StableHLO Output Operand Alias Handling

* Improved handling of output operand aliases in
`mlir/dialects/stablehlo/stablehlo.zig::custom_call` by passing
`output_tuple_indices` for multi-result functions, increasing
correctness in custom call attribute construction.
2026-02-09 12:01:22 +01:00
Benjamin Rabier
eb28352018
zml/io: memory leak in load (#378) 2026-02-04 18:03:24 +01:00
Benjamin Rabier
5a1d5041d9
zml/platform: small memory leak fix (#377)
devices and memories are not allocated within the arena when the rest of
the platform is.
2026-02-04 16:57:12 +01:00
Steeve Morin
0f460fb636 zml: expose Memory, pass Platform as *const, add zml.io.load 2026-02-03 10:59:39 +00:00
Steeve Morin
f76f02984e zml: add io.BufferedDeviceWriter, io.DirectDeviceWriter and
io.ProgressWriter
2026-01-28 18:13:17 +00:00
Steeve Morin
7832fd26ac stdx: rework stdx.Io.LimitedGroup to use atomics and conditions 2026-01-28 18:12:15 +00:00
Steeve Morin
7f936ba7b3 zml: create and pass Platform via const pointer
Platform is destined to hold more things, since it's effectively an
object
that is instanciated once per program. Make it a pointer and pass it via
const pointers around.
2026-01-28 18:11:35 +00:00
Steeve Morin
a8154fb5de zml: upgrade meta.TupleRange to Zig 0.16 2026-01-28 18:07:06 +00:00
Steeve Morin
24137cdc11 pjrt: sugarize Event.onReady
Dont't expose callconv stuff, just pass a comptimed function
2026-01-27 23:20:17 +00:00
Steeve Morin
a89c1a16b2 zml: introduce mem subpackage
This implement various host pinned allocators as well as
static and dynamic buffer pools
2026-01-27 22:53:04 +00:00
Steeve Morin
b48c122756 workspace: import cc_* rules from rules_cc
This becomes mandatory starting bazel 9

Also fixed in patchelf=0.18.0.bcr.1
2026-01-27 22:49:06 +00:00
Corentin Godeau
d56e5ce673 zml/testing: Re-introduced testLayer convenience
This function helps when porting models from torch.
2026-01-21 17:11:09 +01:00
Corentin Godeau
f89f4d1cac zml/io: quickfix for loadBuffersFromId
This fix should be properly addressed in the future.
2026-01-21 17:10:44 +01:00
Corentin Godeau
e04091b5af zml: Removed ConstSlice
constness is now handled at runtime via a union in `zml.Slice`
2026-01-21 11:00:19 +01:00
Corentin Godeau
16f8770602 zml: fix test_runner 2026-01-21 11:00:02 +01:00
Steeve Morin
0b25fbc6f5 platforms/cuda/flashattn: proper bazel dep chain
this won't depend on flashattn is cuda is disabled
2026-01-20 19:05:06 +00:00
Steeve Morin
ed709599f6 cuda/flashattn: package bindings in cuda platform 2026-01-20 18:55:21 +00:00
Steeve Morin
d26386d206 platforms: fix cuda, rocm and neuron sandboxes 2026-01-19 22:16:47 +01:00
Steeve Morin
0aff9cf4dc examples/llama: plug std.Progress to the compilation pipeline 2026-01-19 18:42:05 +01:00
Steeve Morin
1b644887b7 bazel,platforms: centralize runfiles loading
runfiles are process-wide in space and time.
so it makes sense to keep them around

because they are bound to the process, we use the smp_allocator and
except the memory to be kept around during the lifetime of the process.
2026-01-19 18:14:06 +01:00
Steeve Morin
21a7c5fec8 platforms: pass allocator to load 2026-01-19 17:41:57 +01:00
Steeve Morin
9e2c09066b runtimes: rename to platforms 2026-01-19 17:41:38 +01:00
Steeve Morin
ba57994cac runtimes: remove layers targets
it is now handled in the runtimes themselves
2026-01-19 16:22:34 +01:00
Hugo Mano
dc2e876b21 Fix llama stack corruption and remove Buffer.fromIoBufferize 2026-01-19 14:27:23 +00:00
Steeve Morin
32ca342f2c examples/llama: remove S3 vfs init 2026-01-16 21:32:39 +01:00
Corentin Godeau
27cf49cbe4 zml: added missing @alignCast
It was removed for a test and wasn't added back
2026-01-16 18:15:07 +01:00
Corentin Godeau
7bbce6c3d0
Attention Backend on ZML V2 (#373) 2026-01-16 10:59:12 +01:00
Corentin Godeau
2ea598a90f pjrt: missing fixes 2026-01-15 18:24:34 +00:00
Corentin Godeau
de41199c68 examples/llama: fix buffer size allocation 2026-01-15 18:24:34 +00:00
Hugo Mano
015ed2245f zml/io/vfs: add s3 vfs 2026-01-15 16:12:43 +00:00
Hugo Mano
3c96903568 Refactor llama and VFS to new CLI and IO APIs 2026-01-15 15:19:10 +01:00
Steeve Morin
7a798bb057 zml/io/vfs: use indices to address backends, switch to
StringArrayHashMapUnmanaged
2026-01-14 19:19:09 +01:00
Steeve Morin
6ee968e033 WIP 2026-01-14 18:08:56 +01:00
Hugo Mano
42633d133d use model name as progress 2026-01-14 18:04:56 +01:00
Hugo Mano
25a5ee10f6 rework zml.io.* 2026-01-14 18:04:55 +01:00
Steeve Morin
3112f1138a stdx: add utf32 helpers 2026-01-14 18:00:22 +01:00
Steeve Morin
b03b6299fc pjrt: remove get prefixes when not matching with PJRT 2026-01-11 22:23:25 +01:00
Steeve Morin
b0f35d4815 pjrt: remove dep to stdx 2026-01-11 22:13:57 +01:00
Steeve Morin
dc4f6e2f00 zml: remove zml.init and deinit 2026-01-11 21:24:57 +01:00
Steeve Morin
faee73c5ee zml: explicit TF_CPP_LOG_LEVEL 2026-01-11 17:19:30 +01:00
Steeve Morin
989f5aef25 workspace: bump bazel=8.5.0 2026-01-11 17:14:37 +01:00
Steeve Morin
431dcd7a1c pjrt: guard against out of bounds access
when calling into an API, make sure that the method is inside the
struct_size, else we would read into undefined memory
2026-01-11 16:07:33 +01:00
Steeve Morin
a05fa3299e pjrt: rework extensions support through tagged union 2026-01-11 16:07:33 +01:00
Steeve Morin
8c8f233f6a third_party/xla: bump to head 2026-01-11 16:07:33 +01:00
Steeve Morin
c1c97d8dec pjrt: more cosmetics 2026-01-11 16:07:33 +01:00
Steeve Morin
89aa189d8d pjrt: add addressableMemory from pjrtx
it should be part of the binding
2026-01-11 16:07:33 +01:00
Steeve Morin
470359f39a pjrt: memorystats structs in structs 2026-01-11 16:07:33 +01:00
Steeve Morin
9eae5baaf4 pjrt: meta 2026-01-11 16:07:33 +01:00
Steeve Morin
55488f676d PJRT: meta 2026-01-11 16:07:33 +01:00
Steeve Morin
132c7cf390 pjrt:WIP 2026-01-11 16:07:33 +01:00
Steeve Morin
75903732c1 pjrt: do not deinit Event after await
PJRT_Buffer_ReadyEvent semantics say it allocates a new object
everytime, of which the caller is the owner.
2026-01-11 16:07:33 +01:00
Steeve Morin
ab62c536e2 runtimes/neuron: wip 2026-01-11 16:07:33 +01:00
Steeve Morin
008390f305 runtimes/neuron: bump neuron dependencies 2026-01-11 16:07:33 +01:00
Steeve Morin
a39fece38c WIP BUFFEREDWRITER 2026-01-09 10:45:00 +01:00
Steeve Morin
fe2d33eca1 zml: force unwrap on compute_capability attribute on rocm
this should absolutely fail if this doesn't exist
2026-01-08 09:31:44 +00:00
Steeve Morin
2b0a8e653c zml: on cuda, re-enable triton gemm by default
NVIDIA doesn't recommend disabling it anymore
2026-01-08 09:13:24 +00:00
Steeve Morin
33775378b9 zml: disable command buffer on rocm
This avoids weird crashes, AMD recommended disabling it for now.
2026-01-08 09:13:24 +00:00
Steeve Morin
e20e224833 zml: on rocm, only enable triton gemm when using CDNA 2026-01-08 09:13:24 +00:00
Steeve Morin
d42e4f99ad pjrt: add DeviceDescription.attributes
and attribute helper for getting only one
2026-01-08 09:13:24 +00:00
Steeve Morin
1f73e85005 pjrt: rework NamedValue
it's simpler
2026-01-08 09:13:24 +00:00
Steeve Morin
b3614f6f0c third_party: bump swig=4.4.1 2026-01-06 17:32:19 +00:00
Steeve Morin
fd44f8e63e workspace: bump hf tokenizers and rules_rust 2026-01-06 16:36:35 +00:00
Steeve Morin
bb01106ace runtimes/rocm: bump PJRT plugin 2026-01-06 15:35:28 +00:00
Steeve Morin
f3a2992365 zml: disable XLA logs by default, unless overriden 2026-01-06 12:30:12 +00:00
Steeve Morin
2ba6d73af2 zml: expose stdx from inside zml 2026-01-06 10:52:28 +00:00
Steeve Morin
84e8b28131 examples: remove outdated examples 2026-01-06 10:51:18 +00:00
Steeve Morin
31c9357a37 workspace: remove clap from MODULE 2026-01-06 10:49:45 +00:00
Steeve Morin
53a4b6905d examples: remove outdated examples (will reintroduce later) 2026-01-06 10:45:28 +00:00
Steeve Morin
93d92f4757 third_party: remove clap 2026-01-06 10:44:02 +00:00
Steeve Morin
73950032f5 zml: remove pjrtx proxy, only keep helpers 2026-01-06 10:41:40 +00:00
Steeve Morin
c2780ca7c5 workspace: zml2 -> zml 2026-01-06 10:08:17 +00:00
Steeve Morin
3a94593467 workspace: fix and bump ZLS 2026-01-06 09:05:56 +00:00
Steeve Morin
e7ee7175dd third_party/zls: update to 0.16.0-dev.82+cbb62284 2026-01-06 08:54:16 +00:00
Steeve Morin
a1ce0a387b runtimes: remove usage of std.fs, add std.Io 2026-01-05 23:39:33 +00:00
Corentin Kerisit
0f1d49daf4 Fix ZLS 2026-01-05 16:02:06 +00:00
Steeve Morin
bdce67fb49 zml/io/vfs: hf2 working again 2026-01-05 11:46:36 +00:00
Steeve Morin
48eddd98b6 examples/mnist: cosmetic changes 2026-01-05 09:09:43 +00:00
Steeve Morin
541a0b2f41 examples/llama: remove clap, take prompt from stdin 2026-01-05 09:09:34 +00:00
Steeve Morin
749a22c881 examples/benchmark: update 2026-01-05 09:09:17 +00:00
Steeve Morin
df09615256 zig bump 2026-01-05 09:09:04 +00:00
Steeve Morin
3934392360 runtimes/cuda: disable XLA logging 2026-01-05 09:08:43 +00:00
Steeve Morin
03e7144692 zml: cosmetic logging changes 2026-01-05 09:08:27 +00:00
Steeve Morin
1b0242b2f1 zml: bump Zig 2026-01-05 09:07:50 +00:00
Steeve Morin
bfcdf27527 stdx: add time.hzFloat 2026-01-05 09:07:18 +00:00
Steeve Morin
8e9a28fc2c stdx: mirror new std layout 2026-01-04 19:37:42 +00:00
Steeve Morin
799b69e2b7 examples/llama: properly cancel tokenizer loading 2026-01-03 11:05:16 +00:00
Steeve Morin
7eaf650d0e zml: Platform.Device and iterator 2026-01-03 11:04:45 +00:00
Steeve Morin
a854a29cb3 examples/llama: cosmetic changes 2026-01-03 10:44:50 +00:00
Steeve Morin
1ec373368b examples/llama: remove stdx.time.Timer usage 2026-01-03 10:43:26 +00:00
Steeve Morin
7cdce4ffae zml/platform: new compile prototype 2026-01-03 10:36:44 +00:00
Steeve Morin
39876f6936 zml/tokenizers: add fromBytes 2026-01-03 09:16:16 +00:00
Steeve Morin
89afb43e64 pjrt,zml: trim pjrtx
with the aim to remove it
2026-01-02 22:14:00 +00:00
Steeve Morin
f9828e445f examples/llama: disable VFS 2026-01-02 20:37:56 +00:00
Steeve Morin
e7b6651c49 runtimes/cuda: update for Zig 0.16 2026-01-02 20:37:42 +00:00
Steeve Morin
b7a9bd68cc third_party/clap: switch to fork for Zig 0.16 compat 2026-01-02 20:37:14 +00:00
Steeve Morin
39fb5e590b WIP 2026-01-02 16:02:30 +01:00
Steeve Morin
f3245c5968 MNIST running again 2026-01-01 17:56:29 +00:00
Steeve Morin
db3ded4f80 WIP 2026-01-01 08:49:18 +00:00
Corentin Godeau
5ca3dad667 Change parameter order for Buffer.toSlice
std.Io should be as close to the beginning as possible for consistency
2025-12-22 15:57:56 +01:00
Corentin Godeau
11ca632931 Introduced variant of Exe.call
By default, we don't wait for execution completion when calling
`Exe.call`. If you want to await the result, you can use `Exe.callOpts`.
2025-12-22 15:18:16 +01:00
Corentin Godeau
c0522ca6aa Events can be null when executing a LoadedExecutable 2025-12-22 15:18:16 +01:00
Corentin Godeau
1c7c520d8e Use best platform for tests by default 2025-12-22 15:18:16 +01:00
Corentin Godeau
da596e30f3 Remove Tensor.matmul as it's not used anymore 2025-12-22 15:18:16 +01:00
Corentin Kerisit
573fabad95 Upgrade TPU to latest nightly 2025-12-22 12:27:41 +01:00
Corentin Godeau
e3ad311536 Ported simple_layer example to new API 2025-12-22 12:13:58 +01:00
Corentin Godeau
2d0aa3ceae Removed autofix comments 2025-12-22 12:13:58 +01:00
Corentin Godeau
2739c925a6 Fixed mnist example with new VFS.File changes 2025-12-22 12:13:58 +01:00
Corentin Godeau
058bd48abe Implemented formatNumber on Slice and ConstSlice 2025-12-22 11:53:03 +01:00
Hugo Mano
5d2dc7c8c1
io: fixes (#369)
Co-authored-by: Corentin Godeau <corentin.godeau@zml.ai>
2025-12-22 11:29:55 +01:00
Corentin Godeau
09df6d757d Make Llama more async 2025-12-19 15:18:36 +01:00
Hugo Mano
14d8f2359b
io: HF VFS v2 + alignment with HTTP VFS + fix for File VFS (#368) 2025-12-19 14:00:26 +01:00
Corentin Godeau
e17f07e1fc Use correct default parameter for TopK 2025-12-19 12:48:00 +01:00
Corentin Godeau
3988755c73 Fix VFS.File init in llama 2025-12-19 11:16:47 +01:00
Corentin Godeau
ec52dc22f1 Small fix in File VFS
DIRECT flag is not supported on macOS
2025-12-19 11:16:10 +01:00
Corentin Godeau
f46e649a29 Small fixe on LLama 2025-12-19 11:11:30 +01:00
Corentin Godeau
3581770d26 Fix tests 2025-12-19 11:09:34 +01:00
Hugo Mano
190141926f
io(vfs/file): implement direct io (#367) 2025-12-18 13:35:06 +01:00
Corentin Godeau
0776802b41 wip llama working 2025-12-17 18:32:52 +01:00
Corentin Godeau
ad93a58030 wip loading 2025-12-17 16:36:37 +01:00
Hugo Mano
860a52e8f0
io: fixes (#366) 2025-12-17 16:34:59 +01:00
Corentin Kerisit
cc0349fd70
Update ROCm runtime and sandbox to 7.1.1 (#364)
Co-authored-by: Corentin Godeau <corentin.godeau@zml.ai>
Co-authored-by: Hugo Mano <hugo@zml.ai>
2025-12-17 13:32:55 +01:00
Corentin Kerisit
0463e503c4 Await transfer event to avoid segfault 2025-12-17 12:01:23 +00:00
Corentin Godeau
95401d52aa wip llama 2025-12-16 17:58:33 +01:00
Corentin Godeau
850c8f3152 Re-introduced zml/tokenizer 2025-12-16 17:58:32 +01:00
Hugo Mano
be6d93990f
io: perf on linux (#365) 2025-12-16 17:19:14 +01:00
Corentin Godeau
56f16dadc7 Add output Memory Kind support 2025-12-16 12:07:23 +01:00
Corentin Godeau
f680e17178 First clean version of Mnist 2025-12-15 18:55:43 +01:00
Corentin Godeau
6df392bbe2 First port of Mnist to new API
Everything is done by hand so that's not the final version, but at least
it works!
2025-12-15 16:10:50 +01:00
Corentin Godeau
ed1e690578 Small fix in stdx 2025-12-15 16:10:37 +01:00
Hugo Mano
39911d6de7
io: hf + zml: safentensors (#363) 2025-12-15 13:39:48 +01:00
Corentin Godeau
03b0b5726d Ported benchmark examples to v2 2025-12-12 15:25:01 +01:00
Corentin Godeau
4294a19dd8 Expose compile and compileModel through Platform 2025-12-12 14:55:08 +01:00
Corentin Godeau
11a9ef2d58 Introduced Buffer.fromSlice 2025-12-12 14:27:16 +01:00
Corentin Godeau
1c51e65f90 Changed order of parameters for Buffer.from/fromBytes
std.Io should be put at the beginning (similar to std.mem.Allocator)
2025-12-12 14:25:59 +01:00
Corentin Godeau
1e9fe65c33 Some work on Platform
* Added a way to automatically select the best available platform
* Added a way to print available platforms
2025-12-12 14:00:21 +01:00
Corentin Godeau
b0f7f01a38 Simplified Tensor.init signature 2025-12-11 20:31:19 +01:00
Corentin Godeau
ed4bc7183d Use CompilationContext arena for chunkAllowTrailing 2025-12-11 19:06:45 +01:00
Corentin Godeau
9d5e54fddc Re-introduced ops.customCall + test 2025-12-11 19:03:07 +01:00
Corentin Godeau
92d42bde0a Added platform and an arena to CompilationContext 2025-12-11 19:02:49 +01:00
Corentin Godeau
13d2753c3b Re-enabled scatterConfig test 2025-12-11 18:26:44 +01:00
Hugo Mano
34dd16167e
io/file (#362) 2025-12-11 18:23:50 +01:00
Corentin Godeau
dc00398d1c Restored ops.triton 2025-12-11 18:23:11 +01:00
Corentin Godeau
51ec792d87 Migrate to @EnumLiteral() 2025-12-11 18:11:58 +01:00
Corentin Godeau
920fcbe9b2 Cleanup root main.zig 2025-12-11 18:11:58 +01:00
Corentin Godeau
cac4f38d03 Remove async package 2025-12-11 18:11:58 +01:00
Hugo Mano
ee18a2745b
zml/io (#361) 2025-12-11 17:41:19 +01:00
Corentin Godeau
f7aa2f3166 Some minor fixes
* Use std.testing.io to get test env
* Disable libnvptx when compiling
2025-12-11 16:05:17 +00:00
Corentin Godeau
447fc8c59b Made a simple main to test on other platforms 2025-12-11 16:02:28 +01:00
Corentin Godeau
33a5371dba Plug std.Io in runtimes 2025-12-11 16:02:20 +01:00
Corentin Godeau
742a8f1202 Introduced slice.zig with Slice and ConstSlice
This allows to easily manipulate shaped slice on the host.
2025-12-11 15:12:23 +01:00
Corentin Godeau
1574931db2 Removed useless refAllDecls + reordered imports 2025-12-11 13:44:51 +01:00
Corentin Godeau
1e9797f46b Re-introduced nn.zig + tests 2025-12-11 13:09:23 +01:00
Corentin Godeau
858b5bdd2d Fix partitioning for toDiagonal 2025-12-11 10:52:37 +01:00
Corentin Godeau
b4b8c2739b Various fixes in tensor.zig + Rng tests 2025-12-10 18:02:28 +01:00
Corentin Godeau
b0555085b3 Get new shape.zig from hugomano/explo-partitioning 2025-12-10 15:08:03 +01:00
Corentin Godeau
34b9283a6e Re-enabled dynamicUpdateSlice tests 2025-12-09 19:49:31 +01:00
Corentin Godeau
e0cba48ed7 Re-enabled scatterSlices tests 2025-12-09 19:34:15 +01:00
Corentin Godeau
1cfc13cd82 Re-enabled maxPool tests 2025-12-09 19:03:38 +01:00
Corentin Godeau
b4fddca0d6 Re-enabled unused tensor test 2025-12-09 18:49:50 +01:00
Corentin Godeau
eb1fad99e8 Re-enabled cartesianProductStacked tests 2025-12-09 18:47:37 +01:00
Corentin Godeau
0e4c4d9864 Re-enabled cartesianProduct tests 2025-12-09 18:45:37 +01:00
Corentin Godeau
d0da1c502f Re-enabled triangular tests 2025-12-09 18:39:27 +01:00
Corentin Godeau
f10a38c129 Re-enabled toDiagonal tests 2025-12-09 18:35:12 +01:00
Corentin Godeau
5db4a7ed29 Re-enabled dynamicSlice tests 2025-12-09 18:30:01 +01:00
Corentin Godeau
b0110a2b25 Re-enableds chunkAllowTrailing tests
I had to make choice about allocation to avoid memory leaks
2025-12-09 18:19:13 +01:00
Corentin Godeau
f59d5c5b19 Re-enabled chunkExact tests 2025-12-09 18:08:37 +01:00
Corentin Godeau
145a802491 Re-enabled argsort tests 2025-12-09 18:03:14 +01:00
Corentin Godeau
a56fe39534 Re-enabled argMax tests 2025-12-09 17:36:21 +01:00
Corentin Godeau
fe91568a2c Re-enabled gatherSlices tests 2025-12-09 17:12:24 +01:00
Corentin Godeau
09e3ddf55d Re-enabled gather tests 2025-12-09 16:56:29 +01:00
Corentin Godeau
0c90a0ff2d Re-enabled scalar tests 2025-12-09 16:33:59 +01:00
Corentin Godeau
ca49f06610 Re-enabled stutter1d tests 2025-12-09 16:32:24 +01:00
Corentin Godeau
4a71b381d6 Re-enabled repeat1d tests 2025-12-09 16:30:10 +01:00
Corentin Godeau
3772d657f0 Re-enabled slice tests 2025-12-09 16:10:10 +01:00
Corentin Godeau
9339b8d470 Re-enabled cumsum tests 2025-12-09 15:59:40 +01:00
Corentin Godeau
5b82edbc63 Added getValue in zml.Buffer 2025-12-09 15:59:28 +01:00
Corentin Godeau
0f05158a37 Re-enabled leakyReLU tests 2025-12-09 15:38:30 +01:00
Corentin Godeau
e50db03c21 Partially re-enabled Tensor.dot tests 2025-12-09 15:31:23 +01:00
Corentin Godeau
25df30598e Re-enabled convert tests 2025-12-09 15:13:58 +01:00
Corentin Godeau
6de7409a81 Reenabled all tests using refAllDecls 2025-12-09 15:00:33 +01:00
Corentin Godeau
edc451fcf3 Various fixes in stablehlo bindings 2025-12-09 14:59:47 +01:00
Corentin Godeau
715b0f745a Commented Writer in stdx.BoundedArray
It was causing compile issue in tests
2025-12-09 14:59:14 +01:00
Corentin Godeau
0916e7d7de Various PJRT fixes in bindings 2025-12-09 14:58:56 +01:00
Corentin Godeau
32d995ae10 Fixed DataType.minValue() and .maxValue() 2025-12-09 14:58:30 +01:00
Corentin Godeau
6391597f82 Cleanup pass on floats.zig 2025-12-09 14:58:05 +01:00
Corentin Godeau
054642f5b2 Restored basic test
* Only one Tensor test has been restored for now, Tensor.fmod
* Re-introduced helpers for testing
2025-12-08 17:57:53 +01:00
Corentin Godeau
86b5bb84df Various small fixes 2025-12-08 17:46:17 +01:00
Corentin Godeau
45c94dd5b5 Small fix in stablehlo.zig 2025-12-08 17:45:18 +01:00
Corentin Godeau
9cd56326e3 Basic support of zml.Buffer 2025-12-08 17:45:09 +01:00
Corentin Godeau
44fa0bf125 Added std.Io to compile functions 2025-12-05 18:55:50 +01:00
Corentin Godeau
2178b50c93 Moved MAX_RANK from shape.zig to constants.zig
This will be the single source of truth for constants throughout the
codebase.
2025-12-05 18:55:10 +01:00
Corentin Godeau
8e0e987968 Made it working on Zig 0.16 2025-12-05 16:42:28 +01:00
Corentin Godeau
fa8f75b9a1 Move some operations from tensor.zig to ops.zig 2025-12-05 16:42:28 +01:00
Corentin Godeau
d1fae650a4 Basic compilation support 2025-12-05 16:42:28 +01:00
Corentin Godeau
216c3bcab5 Tackled some TODOs on operations 2025-12-05 16:42:28 +01:00
Corentin Godeau
897c4dc87d Ported almost all previous Tensor operation to new mlir api
* Tests are disabled for now and marked with TODOs
* Missing operations are also marked with TODOs
2025-12-05 16:42:28 +01:00
Corentin Godeau
c182dda457 mlir and dialects modules rework
* Complete rework of mlir that uses pointers instead of values for basic
  types.
* Removed unused dialects.
2025-12-05 16:42:28 +01:00
Corentin Godeau
eba929b8cc Initial rework of Context and Platform
* Removed `zml.Context` for `zml.init()` and `zml.deinit()`
* `zml.Platform` is responsible for loading the PJRT api when
  initialized. This replace the previous behavior of trying to load all
  enabled runtimes.
2025-12-05 16:42:28 +01:00
Corentin Kerisit
299aaac7a7 cc_infos should be exposed as direct since translate_c use direct_public_headers 2025-12-05 16:08:15 +01:00
Corentin Kerisit
ddb05f5cde Remove cat 2025-12-02 16:26:23 +01:00
Corentin Kerisit
55b2073542 fix workspace_printer for all versions of zig 2025-12-02 16:23:25 +01:00
Steeve Morin
133abd1c7f WIP 2025-12-02 15:21:57 +01:00
Steeve Morin
de6b1a570b WIP 2025-12-02 15:21:25 +01:00
Steeve Morin
36760aec62 WIPREADER 2025-12-02 14:31:52 +01:00
Steeve Morin
bff3193026 WIP 2025-12-01 19:30:16 +01:00
Steeve Morin
3fe06706e9 WORKING !!! 2025-12-01 19:17:17 +01:00
Steeve Morin
255a2b6518 WIP 2025-11-27 15:48:19 +01:00
Steeve Morin
20747c4dcb WIP 2025-11-26 17:47:50 +01:00
Steeve Morin
42e9f801e6 WIP 2025-11-17 13:51:35 -08:00
Hugo Mano
e5bc971c75
xla: bump to 9a77a88 (#357) 2025-11-12 11:55:48 +01:00
Corentin Kerisit
06842784d3
workspace: use apple_support cc toolchains vs rules_cc (#355)
This is the recommended toolchain and is where all further improvements
is done.
2025-10-28 17:08:23 +01:00
Corentin Kerisit
e2c27b790b
workspace: default to LLVM backend for all zig compile actions (#356)
Use the new zigopt global setting from rules_zig to set -fllvm to all
zig copmile actions.

The Zig self-hosted backend is more fragile that we thought before. It
fails in numerous cases, in debug, but also in release sometimes. USe
LLVM backend until it is more stable.
2025-10-28 17:06:48 +01:00
Guillaume Wenzek
e58a5359f9
zml/aio: fix pickle loader for Zig 0.15.2 (#354)
zig 0.15.2 changed the meaning of `reader.takeDelimiterExclusive` to not
consume the separator. I rewrote the code to use
`takeDelimiterInclusive` since we want to consume the delimiter.
2025-10-22 17:43:44 +02:00
Guillaume Wenzek
13aa52b1d7
tools/hf: add hf_transfer implicit dependency (#344)
Fix #339
2025-10-20 12:28:45 +02:00
Corentin Kerisit
7d825caf2d
zml/aio/torch/pickle: disable test until implement is fixed (#352)
0.15.2 made this test fail with:

112/117 aio.torch.pickle.test.parse protocol 0...error(zml/aio): Unknow
pickle operator @enumFromInt(10), note we are only supporting pickle
protocol up to version 5.

cc @gwenzek
2025-10-17 21:19:08 +02:00
Corentin Kerisit
773253dfaf
zml: use .host_pinned memory for ROCm (#353)
It was failing with:

```
error(zml): Platform rocm doesn't have memory host_unpinned
```
2025-10-17 21:18:11 +02:00
Corentin Kerisit
a5b925f46b
examples/callback: fix compilation (#351)
Left over from when CI was not working most probably.
2025-10-17 19:06:17 +02:00
Corentin Kerisit
8ceacce067
runtimes/neuron: fix $ORIGIN now that zigopts can use Make Variables (#350)
zigopts is not subject to Make Variables expansion and so $ should be
escaped.
2025-10-17 19:06:05 +02:00
Corentin Kerisit
711ed46f5b
workspace: update zig version to 0.15.2 (#349) 2025-10-17 19:05:47 +02:00
Corentin Kerisit
f1dce257fe
all: provide -fllvm to all targets that requires it (#348)
Until zig x86 backend is fixed.
2025-10-17 19:05:29 +02:00
Corentin Kerisit
892766c652
zls: add missing cc toolchain (#347) 2025-10-17 19:04:39 +02:00
Corentin Kerisit
2613c1137c
workspace: use rules_zig archive_override pre 0.12.2 (#345)
Fixes shared_library and PJRT loading.
Fixes os/log.h non inclusion.
2025-10-17 19:04:17 +02:00
Guillaume Wenzek
2ff320253f
ci: handle no /tmp/zig-cache (#342) 2025-10-17 14:55:49 +02:00
Corentin Kerisit
1a263cf4e2
chore: remove unused target (#346) 2025-10-17 14:51:46 +02:00
Corentin Kerisit
319cf48b2c
pjrt/cpu: fix compilation without cpu (#343)
Zig modules that want to access the "c" module needs to have a non empty
CcCompilationContext for now.
So basically depending on an empty cc_library is not enough, it needs to
contain something (defines or headers or includes or wathever)
2025-10-17 14:51:29 +02:00
Corentin Kerisit
bc05231f30
workspace: use libxev from the BCR (#341)
This marks the last PR for getting out of our custom registry.

Module is now at https://registry.bazel.build/modules/libxev and @steeve
is a maintainer as well.
2025-10-16 11:48:09 +02:00
Guillaume Wenzek
876ca22ff2
add patches to http_deb_archive (#338)
add patches to http_deb_archive
add documentation for command_buffer_compatible
pinToCore
Optional head dim for Llama 3.1
2025-10-15 15:06:58 +02:00
Corentin Kerisit
36d583dd58
workspace: use upstreamed rules_zig instead of our fork (#340)
This PR uses the new release of rules_zig where we upstreamed all the
work from the zml rules_zig fork.

Not much changes from the POV of the public API except for `copts`
becoming `zigopts`.

Also, the global `c` module is only available to Zig modules that depend
on a non empty `cc_library`.
This is behavior that can probably be modified in `rules_zig` later on.

Finally, ZLS support had to be completely rethought to comply with the
change in shape of ZigModuleInfo providers and subsequent rules.

This PR is blocked until `rules_zig` exposes `ZigModuleInfo` for
`zig_binaries`.
2025-10-13 14:34:33 +02:00
Jean-Baptiste Dalido
53ad583089
workspace: add CI for cuda and rocm platforms, split jobs (#327) 2025-09-26 17:47:15 +02:00
0xC0
b04d483ecd
runtimes/cpu: fix darwin arm64 PJRT tarball checksum (#335)
Update sha256 to match v13.0.0 pjrt-cpu_darwin-arm64.tar.gz; unblocks
Bazel fetch on macOS arm64.
2025-09-26 11:25:00 +02:00
Guillaume Wenzek
25bd4d08ee
zml: cleanup some todos (#331) 2025-09-25 14:05:26 +02:00
Guillaume Wenzek
e9b42f0ca8
examples: gpt oss (#329)
This implement support for OpenAI "gpt-oss" models. 

It's mostly a take over of #292 branch, but MoE code is kept inside
examples folder cause it's still a bit warm.
In particular the routing logic can break apart when there are undefined
memory in the inputs, where it wasn't an issue for normal MLP.

I only tested with the 20B version.
2025-09-25 14:03:37 +02:00
Guillaume Wenzek
0312d6f678
zml: gather (#330)
Revamp the `gather` api to match `scatter`

(prerequesite of #329 )

## Gather new api

Now the syntax is to give a named list of indices. Each name must
correspond to one axis in the gathered tensor.
This match `scatter`, and avoid the user to have to concatenate indices
in the same order than the input axes,
which is error prone cause any screw up will compile but index at the
wrong offsets.

For 1 axis it doesn't change much: 

```
// before
self.weight.withTags(.{ .voc, .d }).gather(.{.voc}, .{ idx }, .{});
// after
self.weight.withTags(.{ .voc, .d }).gather(.{ .voc = idx }, .{});
```

But for two it helps a lot:

```
// before
self.weight.withTags(.{ .voc, .d }).gather(.{.voc, .d}, .{ .concatenate(&.{idx_voc, idx_d}, .last)}, .{});
// after
self.weight.withTags(.{ .voc, .d }).gather(.{ .voc = idx_voc, .d = idx_d }, .{});
```

Also adds a `gather_` that offers the same apis but with axis position
instead of names.
This one is more suitable inside library functions when dealing with
unknown names, making it complicated to create the "indices" struct.
I think we could generalize this pattern of offering "non-tagged" api
with an underscore suffix, simplifying the job of LSP.

## topk new API

I realized that most of the code calling topk was renaming the output
axis.
The idea is that logprob.topk(.voc) doesn't return you a tensor with voc
entry but only top_voc entries.
For the values it doesn't make much a difference but for the indices
it's embarrassing and it break usages of gather/scatter.

Now the user of `topK` can either use the unnamed api and pass a number,
of if using the name api must give a new name for the indexed axis.

## Yarn

Support Yarn rope embeddings. It's equivalent to the "llama3" embedding
except they do the interpolation in time space instead of freq space.
2025-09-25 12:29:46 +02:00
Guillaume Wenzek
21cd3ed532
mlir,pjrt,zml: expose all missing dtype (#328)
Adding the following missing data types:

- u2
- i1
- i2
- f4e2m1fn
- f8e3m4
- f8e4m3
- f8e8m0fnu

(ground truth is PJRT)

I've also realized the code in from/to Float32 wasn't correct and was
truncating to zero to often.
Hence the rewrite and bug fix.
2025-09-22 09:11:05 +02:00
Guillaume Wenzek
1480f77a55
zml: add the platform tag inside the Buffer (#326)
CPU buffers behave differently from Cuda/Rocm/TPU buffers,
and in particular they are addressable from main memory.
This was annoying and prevented us to add some safety checks that
weren't valid for CPU.

I'm adding a tag to buffers so that we know which target they refer to.

Also there is a bug in PJRT unitialized which doesn't respected the
memory. I've opened https://github.com/openxla/xla/pull/31292
but in the meantime I've a workaround in ZML that uses the regular
.fromHostBuffer.
(I've also a patched version of PJRT plugins in
https://github.com/zml/pjrt-artifacts/pull/41 but not merged yet)
2025-09-18 15:35:25 +02:00
Guillaume Wenzek
dda7f6c3f0
zml: remove old writer interface APIs (#324)
We still had a few old reader/writer left over.
I think we are good now
2025-09-18 11:36:23 +02:00
Guillaume Wenzek
df77b33d8e
async: remove asynk, asyncc, awaitt, await_ (#325) 2025-09-18 10:05:54 +02:00
Emmanuel Ferdman
03865b2f3d
Update bazel.sh source (#323)
## PR Summary
This small PR fixes the `bazel.sh` source.

Signed-off-by: Emmanuel Ferdman <emmanuelferdman@gmail.com>
2025-09-16 11:39:51 +02:00
Guillaume Wenzek
988f2711c5
zml: loadById (#322)
We already assign a unique tensor id to each tensor object created by a
BufferStore.
But we weren't using the id to load find the corresponding buffer in the
store, instead relying on the name.

This is allow to decouple the shape of the model in the struct vs the
shape of the model inside the safetensor file.
I also moved the id logic out of Tensor and into BufferStore.

I also updated llama/main.zig to use the new logic. 

Bonus:
* fix to `zml.call` tag hashing leading to different MLIR for every
recompile.
* expose  Cpu device count to control the number of CPU "devices"
2025-09-11 10:28:16 +02:00
Guillaume Wenzek
3f95bdd900
zml: callback api (#319)
Resuming work from @hugomano #244 
This is still a bit experimental.

The core API is here, but there are some caveats.

ZML should help with copying buffers to host memory (which is often
needed) but I'm not sure the current implementation using PJRT is good:
- does it allow graph capture
- what's the overhead ?
We should probably use directly cuMemcpy here instead of going through
PJRT.
Current implem should work with all plugins though.

This fix `tensor.print()`

I've also made a slightly more interesting examples/callback that
manually calls a cuda kernel. (there are other way to achieve this but
this shows you can do whatever you want with `zml.callback`

Note: 
the feature was originally named `custom_call` but it was conflicting
with existing bindings for xla builtin custom calls,
that are fully handled by the backend.
So I renamed it to `callback`

---------

Co-authored-by: Hugo Mano <hugo@zml.ai>
2025-09-04 16:57:34 +02:00
Hugo Mano
10d3da2449
xla: bump to b3fbfee (#321)
- Temporarily disable libnvptxcompiler usage since it's not available in
the PjRT CUDA plugin v13.0
- Add nvshmem in the sandbox, required by the PjRT CUDA plugin
2025-09-03 14:24:04 +02:00
Hugo Mano
5d6c04ec69
ci: revamp commit workflow (#320)
- Deduplicate workflows (introduced by
57fd9cda27)
- Fix linking issue on Linux since upb was introduced
- Fix duplicated `_start` symbol for some tests targets
- Temporarily use `-c opt` instead of `--config=debug` to bypass static
archive bug
2025-09-03 14:23:22 +02:00
Guillaume Wenzek
8994a37792
bump runtimes/* code for 0.15 (#318)
Fix #316, close #304 

- Bump platform specific Zig code that doesn't get built in CI.
- Restore pytorch loader, using `std.fs.File` instead of `async.File`,
allowing to reuse `std.zip`
- Update `zig fmt` in CI to `0.15.1` and ran it on the codebase
(third_party excluded)
- Remove `stdx.io` which was pointing to remove
`std.io.BufferededReader`.

The Neuron target is still broken though:
`bazel build //runtimes/neuron:libneuronxla`

As well as the Cuda plugin in `--config=debug` mode:
 `bazel build --config=debug //runtimes/cuda:zmlxcuda`
2025-09-02 15:16:09 +02:00
Steeve Morin
57fd9cda27
workspace: zig 0.15 (#316)
Much of the work was done by @gwenzek

Sadly we drop the torch loader in the process, to be added back later !

---------

Co-authored-by: Guillaume Wenzek <wzk@zml.ai>
2025-08-27 14:11:22 +02:00
Steeve Morin
56ac3e27dd
zml: remove gguf and torch loaders (#317)
Torch loader might be reintroduced later, if the need arises.
2025-08-27 09:43:39 +02:00
Steeve Morin
5f3a2e74c3
workspace: buildifier pass, remove rules_uv, rework tools/hf deps (#315)
Title says it all
2025-08-22 16:37:36 +02:00
Steeve Morin
fc384eb43e
runtimes/neuron: fix neuron runtime (#314)
This PR fixes the neuron runtime with the following:

Proxy the PJRT Api method to enforce the client struct sizes since the
neuron PJRT plugin doesn't use `>=` but `==` to assert them, breaking
PJRT compatibility guarantees.
Fixes https://github.com/aws-neuron/aws-neuron-sdk/issues/1095

Reimplement `libneuronxla` in Zig to control neuronx-cc sandboxing and
invocation.

Implement a python bootstrapper in Zig to create a full blown
`neuronx-cc` executable, avoiding the infamous chicken and egg problem
of python executables boostrapping when sandboxed (due to fixed path
shebangs).

---------

Co-authored-by: Corentin Kerisit <corentin.kerisit@gmail.com>
2025-08-22 16:28:23 +02:00
Steeve Morin
0c796302e2
runtimes/cuda: implement zmlxcuda in Zig (#313) 2025-08-21 18:34:09 +02:00
Steeve Morin
b0978b4b2d
runtimes/rocm: implement zmlxrocm in Zig (#312)
Also, sandbox `amdgpu.ids` and restore safetensors json parsing.
2025-08-21 18:15:30 +02:00
Steeve Morin
24575c2232
workspace: switch from zig_cc_binary to zig_binary (#311)
Following the rules_zig bump that adds the `linkmode` setting, we can
use the C linker directly.

We can now remove the `zig_cc_binary` hack.
2025-08-21 13:12:15 +02:00
Steeve Morin
cd8a65334f
zml: remove nemo, sentencepiece loaders, remove zig-yaml (#310)
Progress towards #298
2025-08-20 00:15:55 +02:00
Steeve Morin
de4a0e0beb
third_party/xla: de-modularize xla (#309)
This PR intends to remove XLA as a bzlmod and transfer it as a non
bzlmod dep.

This is because this module will never be upstreamed as is, so keep it
private.

Also, we fetch llvm-raw and stablehlo from it, which is fine.

While there, dummify the various local_config XLA symbols to please the
imports, as we don't use those parts in ZML itself.

Closes #305
2025-08-19 23:35:01 +02:00
Steeve Morin
aa87c024c1 workspace: re-add .vscode folder
removed by mistake in #308
2025-08-19 19:44:20 +02:00
Steeve Morin
3b81c5630e
zml: remove examples workspace (#308)
Closes #302
2025-08-19 19:42:50 +02:00
Steeve Morin
2044cd2de5
zml: remove zig-protobuf, switch to upb (#307)
Closes #301
2025-08-19 16:14:54 +02:00
Steeve Morin
8279e43038
pjrt,zml: remove profiler (#306)
Closes #303

Progress towards #301
2025-08-19 16:04:43 +02:00
Guillaume Wenzek
9f2ccf8a54
package HF cli so users don't have to create Python venv (#293) 2025-08-01 16:42:29 +02:00
Corentin Godeau
5263a170d6
examples/llama: switch to --hf-model-path flag (#270)
Instead of giving config, model weights and tokenizer paths, rely on
`huggingface-cli` download.
2025-08-01 09:46:07 +02:00
Corentin Kerisit
52d36e1c7e
runtimes/neuron: sandbox neuron dependencies (#290)
The difference with other sandboxing PRs is that we also define a
trampoline PJRT so file which will prove itself useful to circumvent
PJRT version mismatch because of
https://github.com/aws-neuron/aws-neuron-sdk/issues/1095

Additionally, we artificially create a empty repository so that we can
sandbox the distroless deps inside the neuron module extension.

---------

Co-authored-by: EC2 Default User <ec2-user@ip-172-31-64-93.ec2.internal>
2025-07-25 11:35:04 +02:00
Hugo Mano
4cf4ee3c14
workspace: xla 20250718.0-6319f0d with ROCm 6.4.1 (#289)
Note: CUDA plugin is not bumped due to an error with libnvptxcompiler. A
specific update will be done in another commit.
2025-07-22 23:50:38 +02:00
Guillaume Wenzek
ba8ff439ba
protobuf, libxev: remove usingnamespace (#280)
remove usingnamespace from libxev and zig-protobuf
    
relevant PRs:
* libxev:
7f803181b1
 * zig-protobuf: https://github.com/zml/zig-protobuf/pull/1/files

This means we can now build ZML with `-fincremental` (in debug mode only
and linux x86-64)

---------

Signed-off-by: Guillaume Wenzek <wzk@zml.ai>
2025-07-21 18:03:01 +02:00
Guillaume Wenzek
981b6becc3
remove usingnamespace from asyncio (#205)
Small PR to remove our last usingnamespace. 

Also usingnamespace will be out in 0.15.0:
https://github.com/ziglang/zig/pull/24362
2025-07-21 16:21:05 +02:00
Jean-Baptiste Dalido
a1752e8de9
ci: leverage disk cache and leverage runs-on snapshot feature (#288) 2025-07-18 16:30:54 +02:00
Steeve Morin
e13acae80b
runtimes/rocm: fetch libdrm from amdgpu repo, add amdgpu.ids layer (#287)
Properly fetch libdrm related artifacts from the amdgpu driver repo
instead of ubuntu.

While there, add `amdgpu.ids` file to the ROCm layers.
2025-07-17 22:16:27 +02:00
Steeve Morin
81fa2ec44c
workspace: disable glibc constraint, bump rules_rust, re-enable framepointers (#286)
Since the rework of runtimes sandboxing, explicitely compiling against a certain glibc version isn't needed anymore.
2025-07-17 18:20:07 +02:00
Corentin Kerisit
32216e8c54
runtimes/rocm: use inprocess lld instead of sandboxing it (#284)
XLA rencetly added the --xla_gpu_use_inprocess_lld bool flag which lets
use ditch the rocm-llvm//:lld from the sandbox :)

See
d01947de6f
2025-07-17 16:39:00 +02:00
Corentin Kerisit
4362fe323e
workspace: bump rules_zig to 20250714.0-b14a4f1 (#283)
This dummy fixes the presence of the static-lib in the runfiles of the
final target in case of kind != exe.
2025-07-16 18:01:09 +02:00
Hugo Mano
8066d2abdd
ci: use large disk on runs-on (#282)
With the latests developments the default disk space (40GB) is too small
for our new needs so we use large disks (80GB).
2025-07-16 15:21:37 +02:00
Corentin Kerisit
4c14652fb7
runtimes/rocm: sandbox ROCm dependencies (#276)
ROCm counterpart of #275 

The most important thing with this change is that ROCm dependencies
(along with the PJRT plugin) must be loaded in the main thread.

This is because some ROCm dependencies (amdhip64) use Thread Local
Storage in the destructor of static C++ instances.
Because all dependencies are unloaded at exit of the main program,
destructors are run on whatever thread exited (in general the main
thread), and any access to the TLS will except values from a different
threads....
2025-07-16 14:41:07 +02:00
Corentin Kerisit
d3dda29db5
runtimes/tpu: sandbox TPU PJRT Plugin (#278)
It has no dependencies so fairly straighforward
2025-07-16 14:13:22 +02:00
Hugo Mano
fc118e058f
deps/hftokenizers: bump dependency (#281) 2025-07-16 13:44:19 +02:00
Corentin Kerisit
12293da6af
runtimes/cpu: sandbox CPU PJRT Plugin (#279)
Same as TPU, simplified because there are no other NEEDED dependencies.
2025-07-16 13:08:55 +02:00
Corentin Kerisit
8274f9c2c0
runtimes/cuda: sandbox CUDA dependencies (#275)
This PR introduces CUDA sandboxing.

This essentially means:
- Removing all CUDA related dependencies off the leaf binary.
- Sandboxing the whole CUDA dependency graph.
- Mark dlopen direct dependencies of the PJRT shared library as NEEDED.
- Set the RPATH to the sandbox for all libraries that have sandbox
dependencies.
- Load the PJRT plugin from the sandbox.

And in the case of CUDA specifically for now:
- Enable usage of weak CUDA symbols without linking against CUDA
directly.
2025-07-16 11:33:48 +02:00
Corentin Kerisit
8ca8d069f9
pjrt: warn instead of panic if FFI Extension is not found (#277)
This is the case for TPU where FFI Extension doesn't not exist.
2025-07-15 14:55:15 +02:00
Corentin Kerisit
57abcb5aab
workspace: bump toolchains_llvm_bootstrapped to 0.2.4 (#274)
This update is important because it orders correctly the CRT which
contains .init and .fini family sections.
2025-07-15 14:54:57 +02:00
Guillaume Wenzek
ee1163a113
Bump rules_zig, add zig_srcs targets, cleanup build (#267)
This is the non-controversial part of #262 

It contains:
* the new `zig_srcs` helper,
* bug fix related to sources in rules_zig,
* cleanups of zml/BUILD.bazel
* test fixes in async/coro.zig (those tests weren't run in CI but now
are)
* disabled nemo and yaml models loader (don't compile anymore since
0.14.0 bump)
2025-07-15 14:18:15 +02:00
Corentin Kerisit
00ebef1a32
async/coro: make coroutines unwindable (#273)
The first stack of each new coroutines were holding not zeroed and hold
random memory data, which get popped in rbp during first coroutine stack
swap.

Which made unwinding random or even SIGSEGV when _Unwind_Backtrace was
used (during C++ exception for instance).

Zeroing the first part of the stack makes unwinding stop gracefully when
it encounters $rbp=0x00.
2025-07-12 20:48:37 +02:00
Corentin Kerisit
8d0e6e6348
bazel: add rpath manipulation to patchelf (#272) 2025-07-12 20:02:46 +02:00
Hugo Mano
b0168d1c63
workspace: bump xla to version 20250710.0-22ea002 (#271)
Also:
- Bump XLA deps : `com_github_grpc_grpc` and `com_google_protobuf`
- Inject `rules_ml_toolchain`
- Fix `zig_proto_library` rule
2025-07-11 17:45:57 +02:00
Corentin Godeau
67fc7d4b97
runtimes/cuda: downgrade cuda and cudnn (#269)
This commit reverts part of https://github.com/zml/zml/pull/238/files
This is required because XLA has a strong dependencies on CUDA 12.8 and
upgrading to 12.9 is impossible due to
https://github.com/NVIDIA/cccl/issues/4967
2025-06-23 13:11:56 +02:00
Corentin Godeau
0330e34788
pjrt: Bind PJRT_Client_CreateUninitializedBuffer (#266) 2025-06-19 09:11:53 +02:00
Corentin Godeau
3c2d319d20
zml/pjrt: added binding for PJRT_Device_MemoryStats (#264) 2025-06-18 18:32:15 +02:00
Corentin Godeau
a6469229d2
async: use stronger memory ordering (#268)
This was probably causing a segfault on some specific ordering of
things.
2025-06-18 12:55:36 +02:00
Victor Moin
8a6b4a9bdd
Patch aio.zig loadBuffersWithPrefix arguments (#265)
Corrects the type annotation for init_args parameter to use the same 
conditional type as `loadBuffers` function. 

@Corendos
2025-06-16 18:33:34 +02:00
Victor Moin
89344e256b
Add model prefix support (#226)
## Overview

This PR adds the ability to specify a prefix when loading a model from
safetensors file. This is useful when working with models like
[ModernBERT](https://huggingface.co/answerdotai/ModernBERT-large/tree/main?show_file_info=model.safetensors)
; the changes enable me to use `ModernBertModel` to generate embeddings
instead of the full `ModernBertForMaskedLM` model.

## Test

Changes have been tested with my text embeddings server project :
https://github.com/vctrmn/zig-text-embeddings-inference
2025-06-16 17:41:29 +02:00
Corentin Godeau
989d85be2a
workspace: bump xla to version 20250612.0-6e48cbb (#263) 2025-06-16 17:37:57 +02:00
Hugo Mano
b4716dba03
ci: revamp commit workflow (#246)
- Use runs-on s3 cache
- Use mlugg/setup-zig@v2 with ZML mirror instead of deprecated and unreachable goto-bus-stop/setup-zig@v2
- Setup Bazel with bazel-contrib/setup-bazel@0.15.0, leverage disk cache and bazelrc flags 
- Leverage Bazel `manual` tag to filter out unneeded targets for `//...`

Co-authored-by: Jean-Baptiste Dalido <jb@zml.ai>
2025-06-11 13:30:34 +02:00
Guillaume Wenzek
e15d3a9b90
remove usingnamespace from mlir (#204) 2025-06-10 15:51:30 +02:00
Guillaume Wenzek
e4705943a6
remove dead imports (#261) 2025-06-10 15:36:38 +02:00
Corentin Godeau
fb0a8e4071
zml/ops: wire the has_side_effect field (#260) 2025-06-10 15:11:45 +02:00
Guillaume Wenzek
657d4010a3
rename pjrt BufferType to match Zig and ZML conventions (#259) 2025-06-10 14:59:08 +02:00
Victor Moin
a38afbbe51
Fix ModernBERT compilation error by removing print calls (#258)
Fixes compilation error in ModernBERT examples on my intel-based macbook
pro.

### Problem
ModernBERT failed to compile with sharding error:
```logs
info(modernbert): ✅	Loaded weights in 397ms
info(modernbert): 	Compiling ModernBERT model...
info(zml/exe): Compiling modernbert.ModernBertForMaskedLM.forward with { Shape({b=1,s=64,u32}) }
info(zml/module): Wrote MLIR to /private/tmp/zml/modernbert/cpu/modernbert.ModernBertForMaskedLM.forward_192023e1d2d0c331/module.mlir
2025-06-10 12:42:13.905711: I xla/service/dump.cc:562] HloModule dump enabled with path prefix: , suffix: before_optimizations
2025-06-10 12:42:13.981006: I xla/hlo/utils/hlo_sharding_util.cc:3079] There is no registered layout_canonicalization_callback.
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1749552133.985201  627264 status_macros.cc:57] INTERNAL: RET_CHECK failure (xla/service/spmd/spmd_partitioner.cc:5555) hlo->has_sharding() Side-effect HLO must have sharding: %custom-call.0 = f32[1,64,64]{2,1,0} custom-call(%broadcast.2177), custom_call_target="zmlHostBufferCallback", operand_layout_constraints={f32[1,64,64]{2,1,0}}, custom_call_has_side_effect=true, output_to_operand_aliasing={{}: (0, {})}, api_version=API_VERSION_TYPED_FFI, metadata={source_file="ops.zig" source_line=1378}, backend_config={callback = 4553681824 : ui64, pjrt_api = 4795162728 : ui64, pjrt_client = 105553131915920 : ui64, user_context = 0 : ui64}
*** Begin stack trace ***
	GetPjrtApi
	GetPjrtApi
	GetPjrtApi
	GetPjrtApi
	GetPjrtApi
	GetPjrtApi
	GetPjrtApi
	GetPjrtApi
	GetPjrtApi
	GetPjrtApi
	GetPjrtApi
	GetPjrtApi
	GetPjrtApi
	GetPjrtApi
	GetPjrtApi
	GetPjrtApi
	async.callBlocking__anon_34323.TaskT.run
*** End stack trace ***

error(pjrt): [PJRT_Client_Compile] RET_CHECK failure (xla/service/spmd/spmd_partitioner.cc:5555) hlo->has_sharding() Side-effect HLO must have sharding: %custom-call.0 = f32[1,64,64]{2,1,0} custom-call(%broadcast.2177), custom_call_target="zmlHostBufferCallback", operand_layout_constraints={f32[1,64,64]{2,1,0}}, custom_call_has_side_effect=true, output_to_operand_aliasing={{}: (0, {})}, api_version=API_VERSION_TYPED_FFI, metadata={source_file="ops.zig" source_line=1378}, backend_config={callback = 4553681824 : ui64, pjrt_api = 4795162728 : ui64, pjrt_client = 105553131915920 : ui64, user_context = 0 : ui64}
error(zml/module): pjrt-cpu failed to compile: error.Internal
error(zml/module): mlir can be found at /private/tmp/zml/modernbert/cpu/modernbert.ModernBertForMaskedLM.forward_192023e1d2d0c331/module.mlir
error: Internal
```

### Solution
Removed `.print()` calls from `globalAttnMask()` and `localAttnMask()`
functions in `examples/modernbert/modernbert.zig`

```logs
info(modernbert): ✅	Loaded weights in 416ms
info(modernbert): 	Compiling ModernBERT model...
info(zml/exe): Compiling modernbert.ModernBertForMaskedLM.forward with { Shape({b=1,s=64,u32}) }
info(zml/module): Wrote MLIR to /private/tmp/zml/modernbert/cpu/modernbert.ModernBertForMaskedLM.forward_10a8f1a59c873a8f/module.mlir
2025-06-10 12:53:54.150319: I xla/service/dump.cc:562] HloModule dump enabled with path prefix: , suffix: before_optimizations
2025-06-10 12:53:54.227584: I xla/hlo/utils/hlo_sharding_util.cc:3079] There is no registered layout_canonicalization_callback.
info(zml/module): Compilation took 2.548s
info(modernbert): ✅	Loaded weights and compiled model in 2968ms
info(modernbert): 	Input text: [MASK] is the capital of France.
info(modernbert): ⏱️	Model inference in  347ms
info(modernbert): ✅	Top 5 predictions:
info(modernbert): 	  • score: 0.9755  word: 'Paris' token: 36062
info(modernbert): 	  • score: 0.0067  word: 'Nice' token: 29235
info(modernbert): 	  • score: 0.0063  word: 'It' token: 1147
info(modernbert): 	  • score: 0.0010  word: 'Or' token: 3980
info(modernbert): 	  • score: 0.0010  word: 'This' token: 1552
```
2025-06-10 13:49:18 +02:00
Corentin Godeau
b49b64df1b
zml/ops: added customCall (#253) 2025-06-09 15:11:08 +02:00
Corentin Godeau
0f6eb5076c
zml/meta: fix map tests (#257) 2025-06-09 13:33:17 +02:00
Corentin Godeau
b0cfb64a0f
pjrt: added Executable.getCompiledMemoryStats (#250) 2025-06-06 19:03:09 +02:00
Corentin Godeau
959936ffa1
zml/meta.zig: support unions in mapping helpers (#251) 2025-06-06 18:20:39 +02:00
Corentin Godeau
8bddaffcb8
mlir/dialect/stablehlo: set defaults to operand_layouts and result_layouts (#252) 2025-06-06 18:20:25 +02:00
Corentin Godeau
f323979cb2
zml/buffer: various improvements (#248)
* `Buffer.from` is blocking by default
* `Buffer.from` takes options to control the loading This includes
awaiting for the buffer or to which memory the buffer is copied
2025-06-06 18:12:49 +02:00
Corentin Godeau
7ac7a7f210
examples/llama: set model.norm.eps correctly (#256) 2025-06-06 18:12:25 +02:00
Corentin Godeau
88c627a309
Preliminary work for Custom Call (#247) 2025-06-06 11:24:21 +02:00
Corentin Godeau
c75e5d1aa5
modernbert: set embeddings layernorm eps default value (#255)
I'm not sure about the default value. @vctrmn maybe you know better 😄
2025-06-06 11:20:15 +02:00
Corentin Godeau
41e222f39f
workspace: updated default nvim configuration (#254)
This requires neovim >0.11.0
2025-06-06 11:20:03 +02:00
Steeve Morin
2502c44c0f
workspace: switch to toolchains_llvm_bootstrapped (#245)
Notably, this will enable using custom linkers or enable LTO.

---------

Co-authored-by: Corentin Kerisit <corentin.kerisit@gmail.com>
2025-06-05 21:45:15 +02:00
Guillaume Wenzek
56c71d97c0
zml: remove compile time fields from zml.Bufferized (#237)
this remove the need to pass `undefined` to `exe.call` when the
corresponding `.forward` function expects integer/float arguments that
have been inlined in the executable.

This was not needed and could just create confusion.
Now you may end up having to pass void to `exe.call` if the executable
doesn't have any runtime argument,
but it's still better than passing `undefined`.

The only place that was using the non-buffer parts of a `zml.Bufferized`
object was the infamous `zml.testing.compileAndCall`.
So I had to introduce a `zml.testing.BufferizedWithArgs` which is only
used for this function.
I put it under the `zml.testing` namespace to make it clear it's only
for this testing utility.
Note that because `compileAndCall` is strongly typed, you generally
don't need to explicitly mention `BufferizedWithArgs`
and the can just write the arguments in place and get help from ZLS.
2025-06-05 18:50:20 +02:00
Hugo Mano
bde41243bc
workspace: bump xla to version 20250527.0-cb67f2f (#238)
third_party/modules/xla: refactor XLA module extensions
ci: disable failing cache temporarily

XLA diff:
71c67e2a4f...cb67f2f7ce

---------

Co-authored-by: Steeve Morin <steeve@zml.ai>
2025-06-02 21:44:39 +02:00
Steeve Morin
23c7335592
workspace: bump to zig=0.14.1 (#242)
In the process, add a case where an empty tuple syntax (`&.{}`) was
detected as `*const @TypeOf(.{})`
2025-05-30 16:51:55 +02:00
Steeve Morin
aedcec638c
runtimes: use cloudflare debian snapshots (#243)
They are more reliable when pinning dependencies.

Replaces and Closes #111
2025-05-29 23:43:43 +02:00
Steeve Morin
42cf435f48
runtimes/neuron: bump runtime (#241)
In the process, expose the `nrt.h` header to zig.
2025-05-29 23:40:51 +02:00
Steeve Morin
6df48e39de
workspace: make zls runner target explicit (#240)
Easier for future reference
2025-05-29 23:04:20 +02:00
Steeve Morin
0f747f48b2
workspace: add Zed settings (#239)
This PR adds Zed configuration.
2025-05-29 16:46:44 +02:00
Steeve Morin
741c8be571
runtimes/cuda: add cuda.h to C namespace when using the CUDA runtimes (#236)
This allows for custom calls to potentially use CUDA functions.
2025-05-21 11:34:04 +02:00
Guillaume Wenzek
f2f37e6d8d
zml: add some buffer / hostbuffer helpers (#232)
* buffer/hostbuffer utils
* more precise f32 -> bf16 conversion
* visit should not read irrelevant fields
* HostBuffer.data -> HostBuffer.bytes()
* zml.aio loadBuffers: type inference
* store expected input shapes in the executable
* zml.meta.visit support std.BoundedArray
* stdx.json.TaggedUnion
* protection against undefined float hyperparams
* avoid meta.TagPayload which can be quite slow
* improve logging
2025-05-20 15:08:35 +02:00
Steeve Morin
35a5af6b88
third_party/sentencepiece: add missing protobuf_lite dependency (#235)
While we're here, bump it.
2025-05-20 11:53:52 +02:00
Guillaume Wenzek
09a3cb2d11
stdx.fmt: format slices (#231)
this is an improved version from what was implemented in "prettyPrinter"
but I had to dig a bit into the fmt internal to achieve it.
2025-05-19 22:39:56 +02:00
Steeve Morin
a566e396d2
workspace: add debug/release configurations and various stdx (#234)
This PR bumps rules_zig to add command line to the zig compilation in
order to enable debug and release configs from the bazel command line
(and bazelrc).

Most notably, this is useful for passing `-mcpu` flags to Zig
(especially for SIMD support like AVX512).

While we're there, add `stdx.time.Duration.hz()` and
`stdx.math.roundeven()`.
2025-05-19 21:26:28 +02:00
Guillaume Wenzek
bf79263919
Fix llama3 rope scaling (#230) 2025-05-09 14:32:15 +02:00
Steeve Morin
7ccfeaf524
workspace: various fixes and bumps (#229)
This PR add various fixes and bumps in the workspace, notably to prevent
dual builds for LLVM tools.
2025-04-22 16:12:26 +02:00
Corentin Godeau
cf038ccfa5
pjrt: bind PJRT_Device_AddressableMemories (#228) 2025-04-22 11:29:48 +02:00
Corentin Godeau
251e7d27db
zml/tensor: fix returned value in Tensor.toMemory (#227)
The `_output_memory_kind` in the returned result was not set correctly
2025-04-22 11:28:36 +02:00
Guillaume Wenzek
5aa1a22d9f
async should only use sync logging (#225)
currently using async.LogFn inside async runtime itself panics.
So I'm avoiding that by forcing async.zig logging to go through
std.log.defaultLog which is synchronous.

you can reproduce the crash by changing log level to .debug
2025-04-16 09:35:18 +02:00
Guillaume Wenzek
7db76fdaeb
pjrt: add ffi bindings for custom calls (#223)
---------

Co-authored-by: Hugo Mano <hugo@zml.ai>
Co-authored-by: Corentin Kerisit <corentin.kerisit@gmail.com>
2025-04-08 16:28:39 +02:00
Corentin Godeau
6bcb2194d1
zml/ops: added wiring for operand output alias in zml.ops.triton (#222) 2025-04-08 10:52:00 +02:00
Corentin Godeau
c348da9446
async: fix due to xev bump (#221) 2025-04-08 10:51:51 +02:00
Corentin Godeau
347c46f86c
zml/aio: handle bool in struct in populateStruct (#220) 2025-04-08 10:51:42 +02:00
Corentin Godeau
1bd161d46c
async: TCP connect fixes (#219)
Co-authored-by: Jean-Baptiste Dalido <jb@zml.ai>
2025-04-08 10:51:33 +02:00
Guillaume Wenzek
e9706222bf
add mini-DSL to create mlir common attributes and types (#208)
Leverages Zig 0.14 to simplify creation of mlir.Type and mlir.Attribute.
2025-03-31 17:31:53 +02:00
Steeve Morin
1bdde64755
workspace: fix vscode-zig update (#217)
The extension checks that the ZLS path is absolute, else it will run
which to find it in the PATH.

Add a heading / so that the path looks absolute. This works because we
use ${workspaceFolder}.
2025-03-28 13:41:01 +01:00
Hugo Mano
8ec91a0edf
zml: support nvtx tracing (#215) 2025-03-27 15:26:36 +01:00
Hugo Mano
d0f5f0c7be
async: detect xev only if the API is dynamic (fix macOS) (#216)
ref: 94ed6af7b2/src/dynamic.zig (L44)
2025-03-27 15:26:28 +01:00
Victor Moin
3fd5c49c32
workspace: fix ZLS macOS x86_64 incorrect checksum (#214) 2025-03-27 15:13:28 +01:00
Hugo Mano
6be509e5b6
async: detect xev most preferable backend to use (#212)
Since change allow use to leverage io_uring instead of xev if the
backend is supported.
2025-03-25 16:41:58 +01:00
Steeve Morin
e0336dc9b3
workspace: fetch llvm, stablehlo and triton from xla, bump XLA (#206)
This enables making sure versions are the same across the board.

Also, import `triton` repo for later use.

---------

Co-authored-by: Hugo Mano <hugo@zml.ai>
2025-03-19 10:00:27 +01:00
Hugo Mano
0376e5bb26
stablehlo: fix custom call optional attributes forwarding (#207)
Operand and result layouts are optionals
ref: 7639178ebd/xla/mlir_hlo/mhlo/IR/hlo_ops.td (L2568-L2569)
2025-03-18 17:58:49 +01:00
Victor Moin
123082de35
runtimes: add macOS x86_64 (#175)
This PR showcases an initial implementation to run ZML on MacOS Intel
(x86_64). While Mac Intel is *deprecated*, this work allowed me to
contribute to [modernbert example](https://github.com/zml/zml/pull/149).
It relies on a custom
[pjrt-cpu_darwin-amd64.tar.gz](https://github.com/vctrmn/pjrt-artifacts/releases/download/v3.0.0-rc/pjrt-cpu_darwin-amd64.tar.gz),
which is not officially supported by the ZML team.

This PR __is not intended for merging__ but serves as a reference for
users who want to experiment with ZML on older Mac.

https://github.com/zml/zml/issues/105
2025-03-18 16:52:18 +01:00
Guillaume Wenzek
2458981020
zml: remove usingnamespace from floats.zig (#203)
I tried to remove usingnamespace from ZML and it's deps.

Even though `-fincremental` works now, it doesn't provide improved
compilation speed.
I guess because our compilation time are dominated by the linking of all
our C++ deps, and the linking isn't incremental ATM.

I split this in 3 pull requests,
I find the code a bit uglier everywhere, not sure if it's worth it.

* floats.zig this PR
* mlir.zig #204
* asyncio.zig #205
2025-03-18 16:19:02 +01:00
Hugo Mano
1c87992ca3
mlir: rework stablehlo custom call (#201)
Also add Triton example

---------

Co-authored-by: Steeve Morin <steeve@zml.ai>
2025-03-17 16:03:07 +01:00
Steeve Morin
586a14da9f
mlir: rework DenseElementsAttribute to properly slice its inputs (#202)
In the process, make `.as()` not return an optional.
2025-03-17 14:04:09 +01:00
Corentin Godeau
97b99e7533
mlir: add MLIR index type (#198) 2025-03-14 23:27:36 +01:00
Steeve Morin
a93c7631e5
workspace,third_party/rules_zig: minor fixes for zls+zig 0.14.0 (#200)
ZLS now requires `--version` to be the first param, account for that.

Also, added missing keys to the BuildConfig object used for code
completion.
2025-03-14 13:42:10 +01:00
Guillaume Wenzek
a85b53fba0
zig 0.14.0 (#189)
Co-authored-by: Hugo Mano <hugo@zml.ai>
2025-03-14 09:49:23 +01:00
Steeve Morin
5bb1d46640
workspace: rework runtimes sandboxing, bump PJRT plugins, cuda=12.8 and ROCm=6.3.4 (#199)
This PR significantly reworks the runtimes sandboxing to leverage
rules_distroless apt resolution, bump pjrt-artifacts=7.0.0 which inclues
Triton ROCm fixes.

Finally, bump CUDA=12.8 cuDNN=9.8, ROCm=6.3.4
2025-03-13 22:17:41 +01:00
jaco-bro
7e62b6551c
workspace: update rules_cc=0.1.1 and bazel=8.1.1 (#196)
This PR updates the `rules_cc` dependency from `0.1.0` to `0.1.1` and
the Bazel version from `8.0.1` to `8.1.1`.

### Changes:
1. Updated `rules_cc` version in `zml/MODULE.bazel` from 0.1.0 to 0.1.1:
    ```bazel
    bazel_dep(name = "rules_cc", version = "0.1.1")
    ```
2. Updated the Bazel version in `zml/examples/.bazelversion` from
`8.0.1` to `8.1.1`.

### Context:
1. Initially, running `bazel run -c opt //mnist` failed due to a Bazel
version mismatch (required `8.0.1` but the currently brew-installable
version is `8.1.1`). This was resolved by updating `.bazelversion`.
2. After fixing the Bazel version, the build failed again this time due
to a yanked version of `rules_cc` (`0.1.0`).

> ERROR: Error computing the main repository mapping: Yanked version
detected in your resolved dependency graph: rules_cc@0.1.0, for the
reason: rules_cc 0.1.0 is yanked due to incompatible change (prematurely
removing cc_proto_library from defs.bzl), please upgrade to 0.1.1.
> Yanked versions may contain serious vulnerabilities and should not be
used. To fix this, use a bazel_dep on a newer version of this module. To
continue using this version, allow it using the --allow_yanked_versions
flag or the BZLMOD_ALLOW_YANKED_VERSIONS env variable.

3. Updating `rules_cc` to `0.1.1` resolved this issue.

### Testing:
1. Verified the update by running:
    ```zsh
    git clone https://github.com/jaco-bro/zml
    cd zml/examples
    bazel run -c opt //mnist
    ```
2. The build and execution succeeded without errors.

    ```
    info(mnist): ✅ RECOGNIZED DIGIT:
               +-------------+
               |     ###     |
               |    #   #    |
               |   #     #   |
               |   #     #   |
               |   #     #   |
               |    #   #    |
               |     ###     |
               +-------------+
    ```
2025-03-03 16:46:50 +01:00
Steeve Morin
cb1010948b
workspace: bump to libxev=20250222.0-07bcffa (#197)
This introduces `Dynamic`, which will switch between IO Uring and Epoll
in linux depending on availablity.
2025-03-03 16:46:34 +01:00
Guillaume Wenzek
a672bb8247
examples: add modernBERT (#192)
Modern bert implementation from Victor: @vctrmn

Tested with https://huggingface.co/answerdotai/ModernBERT-base
I started from his PR #149 and cleaned up a bit the code to make it more
what I consider "idiomatic ZML" even if it's still a bit in flux since
we are so young.

---------

Co-authored-by: vctrmn <victormoin.vm@gmail.com>
Co-authored-by: Rene Schallner <rene@zml.ai>
2025-02-27 18:07:42 +01:00
Hugo Mano
d712ab7534
workspace: fix libxev version (#195) 2025-02-21 12:48:00 +01:00
Corentin Kerisit
44b74dea40
workspace: fix mistakes in modules bumps (#191)
Fix missing = in stbalehlo integrity.
Changes back - by . in xla module name and folders.
Correctly depend on `xla@20250204.1-6789523`
2025-02-18 15:54:56 +01:00
Corentin Kerisit
9227fc9ad4
stablehlo: bump to head and use new dialect capi (#190)
This drastically reduce the number of build steps (from 3589 to 2553 steps)
2025-02-17 18:43:07 +01:00
Corentin Godeau
744e0d2843
zml/ops.zig: Added zml.ops.case operation (#188)
This can be used to select which branch will be run at runtime.

It wraps the `stablehlo.case` operation.
2025-02-13 20:47:53 +01:00
Hugo Mano
2fd9516773
workspace: query, build and test (#187)
Ensure that our Bazel setup for the framework and examples work through
queries, builds and tests.
Also fix broken code and deps in the codebase.

As of today only Neuron is broken (cf:
https://github.com/zml/zml/pull/111)
2025-02-12 10:40:20 +01:00
Guillaume Wenzek
1c2580b3c2
zml: FnCache should handle non-tensor values for less confusing behavior (#186)
Currently if you used the FnCache (through zml.call) on a function with
non tensor fields you would get a struct with undefined memory.
This wasn't nice. So now we copy the value of the function returned by
the first zml.call, and use this as base for future calls, reusing
non-tensor fields, and adding correct mlir.values to the previous
tensors.
This is easy because now zml.call only supports fixed size structs (no
slice/pointer)

---------

Co-authored-by: Corentin Godeau <corentin@zml.ai>
2025-02-11 16:49:05 +01:00
Guillaume Wenzek
53a43432e0
pjrt: profiler support std writer API (#185)
expose a more low-level function to customize where to write profile
reports
2025-02-11 16:47:28 +01:00
Hugo Mano
ceec8a8c9a
workspace: bump xla to version 20250204.0-6789523 (#184)
Also improves Bazel 8 compatibility

---------

Co-authored-by: Corentin Kerisit <corentin.kerisit@gmail.com>
2025-02-07 15:36:15 +01:00
Guillaume Wenzek
5de2adcec3
Fix llama generations (#181)
Current implementation of llama is discarding the first outputted token,
and feeding the last prompt token a second time in the model.

Due to how self attention works this wasn't too disturbing for attention
but it was technically wrong.
Also it was doing a bit extra compute for nothing.

---------

Co-authored-by: Corentin Godeau <corentin@zml.ai>
2025-02-06 14:10:02 +01:00
Steeve Morin
86dc444964
workspace: MODULE.bazel cleanup (#183)
Title says it all !
2025-02-05 17:34:06 +01:00
Steeve Morin
cbd9e50a79
workspace: bump libxev to 20252401.0-31eed4e (#182)
Includes:
- https://github.com/mitchellh/libxev/pull/128
- https://github.com/mitchellh/libxev/pull/141
2025-02-05 13:40:03 +01:00
Steeve Morin
17723f5f88
pjrt,zml: add memory bindings (#180)
This preliminary PR binds PJRT memory endpoints and adds them to
`zml.Buffer`.

A follow up PR will properly integrate it inside `zml.Buffer`
2025-02-04 11:11:02 +01:00
Rene Schallner
a9057737f3
llama: clean up llama3 prompt template encoding. (#172)
There was an unnecessary re-encoding of the newline string, and we added
a newline at the end.
2025-02-03 12:10:19 +01:00
Guillaume Wenzek
f1d0c64db3
Restore tinyllama models (#173)
This is a follow up on #162 

* restore homemade tokenizer, integrate it with the new zml/tokenizer
library. This will only be used for working with tinyllama, I could also
make it work to support GGUF if needed (couple lines of code).
* this allows us to use the original tinyllama files from karpathy repo
which generates actual sentences. This is important when working on
modeling because subtle bugs can degrade the output quality, which is
not detectable if the output is crap to begin with.
* fix casing of zml/tokenizer api: fromFile, tokenToId. Let's stick to
the Zig style guide. Function written in Rust follows rust convention,
but the Zig bindings should follow Zig convention.
* make tokenizer/main.zig a proper tokenizer test tool reusing
llama/test_tokenizer.zig

~~use native_binary instead of cc_binary~~
Reverted because apparently native_binary break the runfiles and
therefore the cuda sandbox
2025-02-03 12:06:33 +01:00
Corentin Kerisit
71fa3764d1
bazel: use prebuilt protoc binaries (#178)
This removes ~1300 build steps by depending on prebuilt binaries.

Note that this PR is blocked by
https://github.com/aspect-build/toolchains_protoc/pull/44 since
`rules_proto` depends on `protobuf@29.1` and
https://github.com/aspect-build/toolchains_protoc doesn't expose
`protoc@29.1` and we can't pin module versions unless adding an entry in
the local BCR which I think is overkill for this.
2025-01-31 23:12:04 +01:00
Steeve Morin
fd4310f418
bazel: fix random huggingface redownloads by using sha256 (#179)
If a file is in Git LFS, use the oid as the sha256 for the download.
2025-01-31 20:41:55 +01:00
Rene Schallner
edf5d9a93d
llama: fix dims & dtypes -> fix #170 (#171)
This PR fixes issue #170 , making Llama run again on CUDA and RoCM.

- Latest Llama changes had introduced an unnecessary batching dimension:
fixed
- Ops returning indices return i32 yet we now use u32 as token indices
-> dtype conversion requred: fixed

Tested on CUDA.
2025-01-27 17:06:03 +01:00
Steeve Morin
d76e5b9cd4
examples: fix logFn usages (#167)
Ooops! Forgot to update the examples.
2025-01-24 21:07:36 +01:00
Steeve Morin
f0c2be0280
async: async mutex on log and fixes on ResetCondition (#166)
Stop using a real mutex for the `logFn` and use an `Mutex` and allow for
a fallback logger if outside a coroutine.
2025-01-24 20:58:14 +01:00
Steeve Morin
8394632b19
runtimes: properly set sonames (#165)
Sometimes we require prebuilt PJRT plugins from outside (neuron,
TPU...), and their `SONAME` are wrong.

Let's ensure all the plugins have their soname proper.
2025-01-24 20:44:41 +01:00
Steeve Morin
a9e17d35fd
runtimes/cuda: downgrade the PJRT plugin back to 0.4.38 (#164)
There is a performance regression on XLA master and 0.5.0.

Get back to 0.4.38 for now.
2025-01-24 19:42:22 +01:00
Rene Schallner
f41e023e22
zml/tokenizer: use HuggingFace Tokenizers and SentencePiece & Llama improvements (#162)
# HF tokenizers & Llama improvements

1. Added HuggingFace tokenizer and SentencePiece bindings, removed old
tokenizers
2. Improved Llama example

## 1. zml/tokenizer: add HuggingFace tokenizers binding, remove old
tokenizers

`zml.Tokenizer` now uses the new HuggingFace tokenizer bindings when
loading a `tokenizer.json`.

To move fast, we dropped the `tinyllama`, `gguf`, and `sentencepiece`
tokenizers.

## 2. examples/llama

Changes to the llama example:

- config.json support -> less cmdline flags
- switched to [zig-clap](https://github.com/Hejsil/zig-clap) for cmdline
args parsing
- using HuggingFace tokenizer
- re-worked and simplified the code
    - single token prediction
- removed esoteric models from BUILD.bazel
- default container now uses Llama-3.2-1B-Instruct

---------

Co-authored-by: Steeve Morin <steeve@zml.ai>
Co-authored-by: Corentin Godeau <corentin.godeau@zml.ai>
Co-authored-by: Corentin Kerisit <corentin.kerisit@gmail.com>
2025-01-24 18:25:57 +01:00
Corentin Kerisit
52c5d4fe02
deps: Upgrade rules_distroless to 0.4.2 (#161)
The reason for the upgrade is because the //:lock target of the apt
repository checks for bazel marker files to write the lockfile at the
right relative path from the root module.

In 0.3.8, only WORKSPACE and WORKSPACE.bazel files were checked as
marker files, and since we are full bzlmod with a single MODULE.bazel
marker file, the run target would fail like so:

```bash
> bazel run @neuron_bookworm//:lock
Writing lockfile to runtimes/neuron/packages.lock.json
Error: neither WORKSPACE nor WORKSPACE.bazel file was found
```

0.4.2 brings that support.

# Tests

I only tested `cd examples; bazel run //mnist:image` to validate that it
still works. Let me know what else is needed.
2025-01-24 13:58:23 +01:00
Guillaume Wenzek
68d0b5b713
zml: misc oneliner fixes (#160)
* tinyllama tokenizer: rewrite "<0x0A>" token to "\n"
* HostBuffer.isContiguous: don't get triggerd by 1 dim axes
* HostBuffer.slice1d: improve error messages
* module.zig: just print where the .mlir file is instead of cluttering
the screen. most of the time the full IR is not very useful anyway
* setFlags: correctly handle comptime int/float
* tokenizer.zig: don't panic when detokenizing high number just use
`<oob>` instead.
* Buffer.constant -> speedup creation up to 2.5G/s on cuda.
2025-01-24 10:20:19 +01:00
Jean-Baptiste Dalido
3682e16da9
workspace: upgrade to Bazel 8 (#143)
This PR covers the migration to Bazel 8.

---------

Co-authored-by: Hugo Mano <hugo@zml.ai>
2025-01-23 18:36:57 +01:00
Hugo Mano
acce597ae9
workspace: bump xla to version 20250122.0-cc075be (#159)
* Bump XLA to 20250122.0-cc075be
* Use nvptx compiler and nvlink with nvjitlink support
* Warn user if cuda path detected in LD_LIBRARY_PATH
* Revert CUDA sandbox fix added here: 6d67fcf and fixed by openxla/xla@a04b262
2025-01-23 16:37:39 +01:00
Guillaume Wenzek
b517170674
Llama3.2: fix tokenizer (#157)
Fix #134 , superseed #133

Basically the llama3 tokenizer.json has the following config: 

```
  "model": {
    "type": "BPE",
    "dropout": null,
    "unk_token": null,
    "continuing_subword_prefix": null,
    "end_of_word_suffix": null,
    "fuse_unk": false,
    "byte_fallback": false,
    ...
}
```

this it technically invalid because it says the model has no byte
fallback and no unknown token either.
This means that unknown emoji can't be converted to bytes nor to the
special unknown token.
This ended up in tokenizer.zig using the value of MAX_INT for the UNK
token then panicking for out of bound.

So I've added some logic to recognize that in the case of GPT2 encoded
vocab (like llama3) that actually contains individual bytes, we should
use the "byte_fallback" algorithm even if disabled.

I've also added a Llama3 "well known" normalizer because HF tokenizers
doesn't add a dummy prefix for llama3 tokenizer.

Technically this is still not a correct implementation of llama3
tokenizer because we don't do the regex based splitting they do. But
even for quite long snippets I tested it now produces exact same tokens.
2025-01-21 13:41:57 +01:00
Steeve Morin
9cc0f96a7a
third_party: bump libxev epoll patch (#156)
This should fix crash and hangs in the epoll and kqueue implementations.
2025-01-20 14:48:29 +01:00
Steeve Morin
bc7fec1261
runtimes/rocm: fix sandboxing and zmlxrocm hooks (#155)
Only the PJRT plugin needs hooking.

Make sure that hipblastlt bytecodes are optional, too.
2025-01-18 00:59:17 +01:00
Steeve Morin
f228ec2edc
async: fix async hangs (#154)
Rework the libxev epoll backend to play nice with zigcoro based
scheduler.

In the process, load the PJRT plugins using `callBlocking` because it
can a bit slow sometimes.

Should have a slight performance increase, too.
2025-01-18 00:50:51 +01:00
Steeve Morin
6d67fcf399
async,runtimes: bump rocm, fixe cuda and rocm sandboxes, bump libxev (#153)
This PR is a collection of fixes related to CUDA and ROCm sandbox
discovery.

In the process, update the epoll libxev patch which caused 100% CPU on
linux.

While fixing CUDA add `xla_gpu_enable_latency_hiding_scheduler` as per
NVIDIA's recommendations.

Finally, bump cuDNN to 9.6.0 while we're here.

---------

Co-authored-by: Corentin Godeau <corentin@zml.ai>
2025-01-16 01:11:36 +01:00
Steeve Morin
5309a25420
async: vendor zigcoro and unify APIs (#152)
This PR merges zigcoro inside the repo and reworks its internals to play
well with `stdx.meta`.

It also adds `Channel.try_{send,recv}` methods as well as dynamically
sized channels (with a `comptime` capacity).

Although not used, it also introduces a `PoolStackAllocator` for
allocating coroutine stacks from a pool
2025-01-14 19:33:16 +01:00
Guillaume Wenzek
ec48976ce6
zml: revamp scatterSlices (#145)
Main issue with current `scatter` implementation is that it uses
broadcasting dims of `stablehlo.scatter`.
While nice in theory, the optimizer doesn't handle them well and they
often are unrolled into while loop.
Here I convert the batching dim to extra iotas indices.
2025-01-14 16:49:39 +01:00
Guillaume Wenzek
52f40144ce
fix #97 (#150)
Before we where using `module.op().writeBytecode(writer)` to compute the
hash of a model
but it crashes on some inputs, notably for unused variables.

So I used the text representation of the mlir.
2025-01-09 17:19:32 +01:00
Guillaume Wenzek
424bbc5dcb
zml: improve source locations (#148)
Put the name of the operator in the source location.
This seemed redundant at first because it's not needed when looking at
the stablehlo,
but it's nice to have it to understand the optimization passes.

I'm also including a few QoL:

* sdpa don't need a bias argument, attn_mask already play this role.
* gatherSlices should accept a shape literal `.{ .a = 5, .b = 128 }`
* Shape.outer to combine two shapes into the superset of their axes
* Tensor.all (since we already have Tensor.any)
* argMax doesn't need to specify the dtype, it can be inferred from the
size of the input tensors. If the user care about the dtype they can
manually convert.

This PR is stacked on top of #147
2025-01-09 17:18:20 +01:00
Guillaume Wenzek
016e152338
zml: HostBuffer.prettyPrint() (#147)
Add pretty printing of HostBuffer.

This will be leverage by the debug helper `x.print()`
It can also be used like this: `std.log.info("my buffer: {}",
.{host_buffer.pretty()})`
2025-01-09 17:17:53 +01:00
Guillaume Wenzek
56ac7b1b07
pjrt: Profiler fixes (#146)
* allow i64 resource id
* reserve memory when creating array lists
2025-01-08 09:50:14 +01:00
Guillaume Wenzek
b360a92f34
zml: rewrite simple transpose (#142)
Rewrite trivial transpose into reshapes.
Transpose over a 1-d axis are effectively no-op.
Normally the backend can detect this, but we observe that some backend
can get confused. So avoiding those extra transpose can help.

I also changed the default profiler size to 1_000_000 events because
above that the perfetto ui can't load the generated file anyway
2025-01-07 10:56:58 +01:00
Hugo Mano
80ec28f3c7
workspace: bump llvm / xla / stablehlo / pjrt plugins xla@20250103.0-5f1fe6a (#141) 2025-01-06 15:06:52 +01:00
Corentin Godeau
d0643ffd59
zml/module.zig: fix superfluous optional unwrapping (#140)
Closes #137
2025-01-02 14:12:31 +01:00
Guillaume Wenzek
cf135bd428
zml: Reorganize the compilation output folders (#135)
* create one folder per target, then one sub-folder per mlir module.
* pass this folder to `xla_dump_to` so that all the xla files arrive
there.
* explicitly save the mlir and the serialized executable there

I'm also exposing James profile conversion tool into ZML as
`profiler.dumpDataAsJson(file)`
2024-12-17 16:55:11 +01:00
Victor Moin
72c56a21bc
Tutorial updates for write_first_model.md (#136)
Quick doc fixes
2024-12-16 21:04:36 +01:00
Steeve Morin
e0041974d2
workspace: rework code completion using a custom build runner (#131)
Following the new release of the VSCode Zig plugin, rework the code
completion logic to leverage a build runner. This is much lighter,
faster and more reliable.
2024-12-16 13:20:27 +01:00
Guillaume Wenzek
e5c152c9f6
zml: add an arena to the CompilationContext (#130)
* add an arena to the CompilationContext

There was already an allocator in the CompilationContext, this PR makes
it into an arena and expose the arena allocator.
This allows ops that need to allocate to do so without abusing
`std.BoundedArray` with arbitrary bounds, and in a safe way.
For now little ops use it, which I believe should stay the case, because
most of the time you want to know how many tensor you're working with.
It'll also simplify the life of Zig new-comers

this include some **breaking changes** to `chunkAllowTrailing` and
`split` that are allocating

* upgrade a few `axis_: i64` into `axis_: anytype` to handle tags

* add a few TODO for the next version of the Tensor API

---------

Signed-off-by: Guillaume Wenzek <wzk@zml.ai>
2024-12-11 13:38:55 +01:00
Guillaume Wenzek
e8fa410537
pjrt/zml: expose client creation flags (#129)
The Pjrt api allows to passe string flags to the plugin on client
creation.
Since we don't like working with strings I've converted the strings
supported by the cuda plugin into a struct.
For now I've focused on the strings corresponding to the allocator
settings following #125 

More settings are not exposed ATM, will follow up with #132 

3d31c48c71/xla/pjrt/c/pjrt_c_api_gpu_internal.cc (L76-L86)

AFAICT the cpu plugin doesn't read flags at all.
Neuron doesn't have known flags, TPU requires more investigation.

**Breaking change**: `context.autoPlatform()` ->
`context.autoPlatform(.{})`
2024-12-11 13:38:04 +01:00
Guillaume Wenzek
05879b41fa
Fix test layer (#124)
* testLayer was silently broken by #115 : test + fix so we detect this
in the future.
* mapAlloc had a regression in #115 wrt struct with 0 size fields: test
+ fix

* add Python `venv` to gitignore
2024-12-11 11:04:38 +01:00
Guillaume Wenzek
c1543278e6
ci: build the examples directory (#121)
Run `bazel build //llama` in CI to make sure we don't accidentally break
it.
I'm also building all the examples target.

For reference it took 5 minutes to run this CI.
2024-12-11 10:56:20 +01:00
Allison Durham
88e0cc7da9
zml: Llama 3.2 (text only models) support (#117)
llama 3.2 1B and 3B do not have lm_head. 

I've added transposing over the embed_tokens as a replacement and set
lm_head as optional for backward compatibility with the other models.

I also added both repositories+executions to bazel and updated the
README.
2024-12-10 16:30:21 +01:00
Rene Schallner
754587f6bb
docs: move image in concepts.md (#128) 2024-12-09 19:48:10 +01:00
Jean-Baptiste Dalido
b2ef326620 readme: fix the banner url 2024-12-09 19:41:41 +01:00
Guillaume Wenzek
38ee9df735
update instructions (#123)
following #115, `prepare` doesn't alloc anymore, `ExeWithWeights` is
`ModuleExe`
2024-12-06 15:28:56 +01:00
Guillaume Wenzek
8cbfe48fe4
async: add intrusive queue (#122)
all code contributed by @steeve

* add intrusive queue
* change the constructor of Channel with default AsyncThread executor

---------

Co-authored-by: Steeve Morin <steeve@zml.ai>
2024-12-06 15:15:51 +01:00
Guillaume Wenzek
6578a3d11a
zml: emit func.call operations (#116)
Restore the machinery to emit individual functions and calls to them.

this should unblock composites, and generates cleaner MLIR in general

* functions are cached based on the input shapes including tags.
* donations/tags are propagated through function calls

---------

Co-authored-by: Corentin Godeau <corentin@zml.ai>
2024-12-05 17:45:07 +01:00
Guillaume Wenzek
d669bdc6bf
zml: cleanup module.zig (#115)
* compileInternal now works with function not with "module"
* BaseExe now allocates all the memory needed to call `pjrtExecute`
* `Exe` is the type variant of `BaseExe` and enforce the right Zig arg
are passed to `call`. `exe.prepare(model)` don't need to allocate
anymore, the allocation happened during compilation.
* public entry points like `compile`, `compileModel`, `compileFn` are
now at the top of the file
* the caching logic has been mostly inlined inside `compileInternal`
* `compile`, `compileFn`, `Exe`, `BaseExe` which are the user visible
API have been moved to `exe.zig`
* `CompilationContext` and all the compilation logic stays in
`module.zig`
2024-12-05 17:33:19 +01:00
Corentin Godeau
bed7a57ae4
examples/llama+neuron: enable buffer donation (#112)
Donate verything except the `token_index` buffer as it messes up the
output
2024-12-04 10:51:43 +01:00
Guillaume Wenzek
a6ea0bd905
zml: revisit rope implementation (#114)
The current implementation first compute the full cos/sin matrix for the
full sentence,
then slice into it to extract the positions for the current token.
This is generally optimized by the compiler so it's not a performance
issue, but it's a readability issue,
because it means calling `rope` requires first a call to `ropeCosSin`.

I rewrote this to directly produce the optimized IR where we only
compute rope for given offsets.
2024-12-02 14:51:36 +01:00
Guillaume Wenzek
8de3d19bb3
zml: fix float8 <-> float32 conversions, support for Tensor.constant(.{}, .{ .f8 = 1.0}) (#113)
Mostly:
* fix float8 <-> float32 conversions
* support for `Tensor.constant(.{}, .{ .f8 = 1.0})` 

Misc:
* fix small inconsistencies between different versions of sdpa
* better error message for broadcast
* bazelrc: --config=debug
2024-11-29 16:14:41 +01:00
Steeve Morin
02527de9b9
runtimes/cuda: fetch nccl from the pip package (#110)
Exactly as XLA does
2024-11-28 11:14:05 +01:00
Steeve Morin
6f6dfe9fe5
runtimes/cuda: rework sandboxing and dynamic symbol renaming (#109)
This change reworks the sandboxing and dynamic symbol renaming for the
cuda runtime.

Switch to the jax-cuda-pjrt pre-built PJRT plugin until we fix the pjrt
artifacts build. Most notably because our build of the plugin didn't
include the right compute capabilities.

Take the opportunity to bump the `cuda=12.6.2` `cudnn=9.5.1`.

Fixes #19
2024-11-27 19:39:17 +01:00
Hugo Mano
85685174c9
zml/pjrtx: use our own stablehlo version if plugin provided version is newer (#108)
Discovered use case:

- PJRT plugin attribute `stablehlo_current_version` is `1.7.8`
- `stablehloVersionFromCompatibilityRequirement(WEEK_12)` is `1.1.0`
-  `stablehloGetCurrentVersion` is `1.7.6`

The target version for artifact serialization is `1.7.8` but should be
`1.7.6`.

XLA ref:
2f99455cdf (diff-7f335a7892)
2024-11-27 18:29:04 +01:00
Hugo Mano
2f70fa4a00
workspace: log message before return NotFound (#107)
This allow us to find which NotFound is raised
2024-11-27 17:21:34 +01:00
Hugo Mano
db55475b17
examples/loader: missing stdx dep (#106) 2024-11-27 17:09:25 +01:00
Corentin Godeau
6a1183f8ad
zml/pjrtx: call bufferFromHostBuffer on the thread pool (#103)
This was causing very bad performance for weight loading on Neuron
because the calls are blocking
2024-11-26 18:57:59 +01:00
Corentin Godeau
58f5895d50
zml/platform: increase max device limit (#104)
Some platform (such as Neuron) can have up to 32 devices, one per core.
2024-11-26 18:57:21 +01:00
Steeve Morin
43c4348de9
runtimes/neuron: add support for AWS neuron / trainium / inferentia 2 (#102)
This PR adds support for AWS Neuron / Trainium 1 / Inferentia 2.

It can be enabled via the `--@zml//runtimes:neuron=true` flag.

Everything should work as expected.

---------

Co-authored-by: Corentin Godeau <corentin@zml.ai>
2024-11-26 14:20:47 +01:00
Guillaume Wenzek
09da897fe8
zml: fix memory usage of sdpaMemEfficient (#90)
use `zml.ops.while` instead of `zml.ops.for` the problem with the latter is that it concatenate all intermediary results, and thus using lot of memory unless the concat was optimized away which wasn't always the case
2024-11-26 13:57:27 +01:00
Hugo Mano
d366f00aed
workspace: bump llvm / xla / stablehlo / pjrt plugins and restore disabled tests xla@4663f04 (#98)
- workspace: new module versions added to Bazel registry:
  - deps/xla: 4663f04d26 (20241025.0-4663f04) 
This version introduces an upgraded overlay with Python repository initialization for the TSL extension. The changes were necessitated by this XLA commit: 19b4ecbe98 (diff-df5471accc) , which triggers a repository call. Python's hermetic version is provided via the .bazelrc file.
  - deps/stablehlo: f7f8e4e352 (20241021.0-1c0b606)  
  - deps/llvm: f7f8e4e352 (20241022.0-6c4267f)
 This version includes an upgraded module declaration with rules_python dependency added, as LLVM prepares its overlay for Bazel v8. Relevant LLVM changes here: 3903cb4695
  - deps/runtimes: use PJRT artifacts v3.0.0 (https://github.com/zml/pjrt-artifacts/releases/tag/v3.0.0)
- workspace: align used versions of rules_zig in the Bazel dependency graph
- pjrt: expose getPluginAttributes and getPluginAttribute private APIs and stablehloCurrentVersion public API
- pjrt: updated customCallRegistry function with the new FFI provided by the PJRT API GPU extension 
Minimal support for this feature was added, as this is not the main focus of the current PR. See related changes in XLA here: ef85a7bcc3
- dialects/mlir/stablehlo: expose getCurrentVersion API and fix broken stablehloVersionFromCompatibilityRequirement API
- zml/pjrtx: serialize IR to the PJRT plugin's supported HLO version. If the plugin attribute is unavailable, it falls back to the StableHLO's compatibility requirement version.
- zml/tensor: tests using batching dims for gather and scatter are now re-enabled
2024-11-25 15:53:25 +01:00
Steeve Morin
db88158518
async: rework async runtime to use coroutines and cross thread signaling (#100)
This mostly reworks the runtime to enable proper cross thread signaling
for `callBlocking`.

API changes include:
- `async_ -> asyncc`
- `await_ -> awaitt`
- proper type inference for non generic arguments even for generic
functions
- bump `libxev`
- use `epoll` on linux by default (can be comptime overridden through
`xev_options.linux_backend` in the root module)

Mostly, also, it's pretty fast.
2024-11-22 17:56:22 +01:00
Guillaume Wenzek
15b0cdc9ad
Fix crash in for_ (#99)
Contains two bug fixes

1. values should be pushed to their block before we open a new block,
otherwise this lead to us assigning values to the wrong block.
* to detect future instance of this bug, and misusage of ZML apis I've
added a few more asserts.
* now every mlir block we create is either "open" or "hermetic", and we
check that hermetic block don't reference variables from outside the
block.

2. in `for_` make sure `first_step` is always used. This is a workaround
for #97. For some reason you need two nested loops to trigger the bug.
Note that this used to work, so I'm not sure what changed in how we
generate MLIR or on MLIR side.

to enable this I had to use the `.pad(.{ ._0 = ... })` syntax. I feel
this could solve a lot of usability usage we have, without having to
introduce too many `.operation1d` operators.
2024-11-22 17:46:47 +01:00
Corentin Godeau
4d1d03cfaa
examples: fixes related to recent async changes (#96)
Recent changes to the `async` module reverted `asyncGeneric` to `asyncc`
but examples were not updated
2024-11-20 15:34:38 +01:00
Guillaume Wenzek
5c60687cde
zml: makeBlocks returns the inner tensor to propagate tags (#92)
Now zml.ops.makeBlock returns both the mlir.Block created and also the
Tensors returned by `func`.

The returned tensors should not be returned to the user,
because their `mlir.Value` must not escape the block that created them.
But their shapes/tags can be safely propagated further.

This makes `zml.ops.makeBlock` a bit more footgunny, but also it's a
utility function that most users shouldn't use. And it allows to use
tags created inside the block and apply them to the return value. In
particular tags and sharding info should now be propagated more
consistently for while loops, summation, reduceWindow, ...

I've also enabled tests to run on non-cpu platform.
this is useful to figure out if a specific platform has limitation in
it's pjrt support
2024-11-20 13:36:01 +01:00
Guillaume Wenzek
7f380a9882
fix a few compileError issues (#93)
fix a few `compileError` calls introduced by #89 . Zig compiler don't
catch those until you trigger a comptime error.
2024-11-20 13:26:44 +01:00
Rene Schallner
e99837a59f
Module-cache: increase max_pjrt_executable_size to 400MB (#95)
Some PJRT executables get really large (e.g. ca. 300MB). We want to be
able to load them.
2024-11-20 10:02:29 +01:00
Corentin Godeau
56c77707ac
third_party/modules/zig-protobuf: revert indentation changes (#94)
This was breaking old branches that didn't have these indentation
changes
2024-11-19 15:09:22 +01:00
James Deal
0ea2e65419
PJRT: Profiling Output to JSON (#79)
Implements conversion of PJRT protobuf profiling output to JSON format.
2024-11-18 11:32:42 -06:00
Steeve Morin
c65f07a123
zml: hotfixes for #89 (#91)
Due to miscommunication on my part, the tests were _not_ supposed to be
failing!
2024-11-15 18:03:00 +01:00
Steeve Morin
783a3cdeb8
zml: add stdx, rework async signature inference (#89)
This one is a chonky.

This commit adds a new `stdx` module with a few utility functions. It is
very far reaching. Forgive the size.

Also, clean the logs in the executables.
2024-11-15 17:12:07 +01:00
Guillaume Wenzek
28afd46728
zml: small cleanup (#88)
* add more scatterSlices testcases     
* zml: remove helpers.mapTensors in favor of zml.meta.map
* fix shape when for loop is fully unrolled
* zml.Tensor.pad should accept i64 to be compatible with dims
* allow to have array of tensor inside model struct
* split Buffer.asViewOf into asViewOfHostBuffer, asViewOfDeviceBuffer
2024-11-15 14:37:14 +01:00
Guillaume Wenzek
0be024db54
zml.nn: add dynamic sampling, supporting top-k, top-p, min-p settings (#87)
Given the output of the last layer of a LM with a `.voc` axis,
Compute indices for the next tokens, following the given sampling
strategy.
The dynamic sampling strategy is more expressive but top_p requires
computing the softmax.

Options are:

* top_k: only sample among the k top scoring tokens,
* max_top_k: limit a compilation time what is the max possible runtime
value for top_k, saving memory and compute by not having to fully sort
the tokens.
* top_p: only sample among top scoring tokens whose probabilities sum up
to top_p
* min_p: drop tokens whose probabilities are lower than a ratio of the
most likely token
2024-11-15 14:26:10 +01:00
Guillaume Wenzek
282cf58b3c
zml: iota default to .i32 (#86)
It's midly annoying to have to specify iota dtype everywhere to .i32.

In practice if people want to work with floating point range of number
they generally use arange.
Also `iota(x.shape(), 0).convert(.f32)` also work and isn't much work
than saying `iota(x.shape(), .f32, 0)`

For good measure I've added a fallback to .i64 when the given axis has
too many elements. I don't think this would cause much trouble, just
because the tensor that big aren't very common in the first place.
2024-11-15 14:05:36 +01:00
Guillaume Wenzek
ecef330aa1
stablehlo: add DotAlgorithm support to stablehlo.dot_general (#85)
In the API I merged the "precision" and "algorithm" attributes into a
union because according to the spec,
precision must be default when algorithm is set, and precision=default
means "fast" when algorithm is not set.

https://github.com/openxla/stablehlo/blob/main/docs/spec.md#dot_general

For now it's not exposed to users because the API is a bit limited. 
In particular only a subset of algorithm are supported by openxla, and
only a subset of that is supported by chips.

I was able to successfully change the accumulation type for dot on Cuda,
but want to experiment more before deciding what to expose.

---------

Co-authored-by: Corentin Godeau <corentin@zml.ai>
2024-11-15 14:00:18 +01:00
Guillaume Wenzek
d33208a10e
Add custom call support, and tensor.print() (#82)
* restore the integration with PJRT custom call extension
* add a generic `zmlHostBufferCallback` custom call that copies the
received tensor to the host, and call a user supplied callback on it.
* add a default "print" method on `Tensor` that print the runtime value
of a given tensor
* print itself has very little code, it's easy for a user to define
their own print if they don't like ours.

limitations:
* only supported on Cuda atm
* pre allocates a buffer on the host to copy the content of the device
buffer,
this buffer won't be freed. There is exactly one buffer per "print" call
in the IR.
* it uses cuda api, maybe I should be using `createViewOfDeviceBuffer`
to make it less cuda specific
2024-11-15 11:22:01 +01:00
Corentin Godeau
03feafa85a
examples: small cleanup (#84)
Changes were made to `asynk` and not applied everywhere
2024-11-15 10:23:11 +01:00
Guillaume Wenzek
c32c0da46c
pjrtx: error instead of panic (#81)
We currently have an issue with our version of openxla not being able to
serialize the new "batching_dim" attribute of gather/scatter.
This is a version mismatch we should fix, but also it's bound to happen
again, so since the compilation can fail, I'd rather return an error in
that case.
2024-11-12 22:23:49 +01:00
978 changed files with 234926 additions and 47397 deletions

14
.gitignore vendored

2999
MODULE.bazel.lock generated

File diff suppressed because it is too large Load diff

Some files were not shown because too many files have changed in this diff Show more