0
Fork 0
mirror of https://github.com/zml/zml.git synced 2026-11-16 19:11:09 +00:00
Commit graph

1,078 commits

Author SHA1 Message Date
Kevin Barre
0fd551e4d5
zml/zls: Add compatibility for aarch64-linux platform in zls_toolchain (#508) 2026-04-28 09:32:40 +02:00
elogir
5300905228
platforms/rocm: update amdgpuids URL and bump llvm version (#510)
This PR fixes two issues:

- Update `amdgpu.ids` URL to Gitlab, now that the old link is dead
- Bump LLVM version to `0.7.3`
2026-04-27 19:59:31 +02:00
Kevin Barre
29ad498772
zml/tpu: support padded KV-cache head dims in ragged paged attention (#507)
fix(mosaic_tpu): support padded KV-cache head dims in ragged paged
attention

Pass both logical and cache head dimensions through the Mosaic TPU paged
attention path so KV pages can remain 128-byte aligned without padding Q
or trimming the final output.

- add backend helpers and tests for Mosaic TPU kernel head-dim alignment
- keep query/output tensors and softmax scaling on the logical head dim
- allow cache head dims greater than or equal to query head dims in TPU
IR and kernel validation
- slice padded K/V vectors back to the logical head dim before attention
math
- fix the tensor pad call signature used by tensor padding
2026-04-27 10:56:57 +02:00
Guillaume Wenzek
9211a1770e
Remove BoundedArray.init (#506)
When looking at the new Sharding code I was triggered by the high number
of "try".

I then realized most of the try where traced back to `axes = try
.init(0)` which can never fail.
So I removed BoundedArray.init (aka Andrew mode)

It's generally unsafe because it makes `slice()` return undefined.;
I replaced`try init(0)` by `.empty`
I replaced `try init(n)` by `.fromSlice` or with `.{ .buffer =
undefined, .len = n }` to explicitly show the unsafeness

This leads to widespread changes because it means that most of the
sharding operations can't fail anymore.
2026-04-27 10:31:52 +02:00
Kevin Barre
6ee021faf9
zml/tpu: vendor ragged paged attention kernel for strided KV loads (#505)
Improve TPU ragged paged attention strided KV loading

- add the TPU ragged paged attention kernel to the repo and package it
with gen_ir
- switch gen_ir to import the vendored kernel instead of JAX's default
one
- update tpu_attention tensor axis tags to match the kernel's expected
shapes
2026-04-24 13:05:24 +02:00
Kevin Barre
bfb792342b
zml/io: do not fail .auto when application default can't be loaded (#502)
Co-authored-by: Copilot <copilot@github.com>
2026-04-23 17:43:37 +02:00
Benjamin Rabier
e79311ccf4
platforms/cpu: remove sha256: prefix from the pjrt artifact's sh256 (#499) 2026-04-23 11:17:40 +02:00
Kevin Barre
94d1b1582f
zml/tpu: implement pagedAttentionDense for fused K V cache (#489) zml-smi-v0.3.0
refactor: introduce KVCache union and wire dense/split layout through
pagedAttention

Adds a `KVCache` union type to `paged_attention.zig` with two variants:
- `.split { k, v }`: separate K and V page tensors (used by cuda_fa2,
cuda_fa3, triton)
- `.dense`: a single pre-fused K/V page tensor (used by mosaic_tpu)

Changes:

- `pagedAttention()` in `paged_attention.zig` now accepts a `kv_cache:
KVCache`
parameter alongside `k`/`v` (unused); dispatches `.split` to CUDA/Triton
  backends and the full `KVCache` union to `mosaic_tpu`.
- `mosaic_tpu.pagedAttention()` in `tpu_attention.zig` now takes
  `layout: paged_attention.KVCache` instead of separate k/v tensors, and
  passes `layout.dense` to the existing `prepareInputs()` helper.
- `KvCache.Layer` in `llmd/models.zig` is now a tagged union of `.dense`
(fused tensor) and `.split { k, v }`. `attentionLayout()` converts it to
`attention.KVCache`. The `.dense` update path in `Layer.update()` fuses
  K and V by inserting a `.kv` axis, concatenating, and merging `.hkv`
  before scattering into the page cache.
2026-04-22 16:48:30 +02:00
Hugo Mano
f80f925409
examples/llm: add profiling option (#497)
This allow use to easily trace what going on
2026-04-22 16:23:36 +02:00
Imran Alam
83ebe37e93
bin/zml-smi: Fix zml logo rendering in KDE Konsole (#494)
Specifying both `rows` and `cols` makes the zml logo in `zml-smi` render
correctly in KDE Konsole. Verified that the fix didn't cause any
regression in a terminal emulator where logo rendering was already
functional (`Ghostty`).

`BUILD.bazel` is also updated to run the `zml-smi`  tests.

**Before**
<img width="348" height="238" alt="image"
src="https://github.com/user-attachments/assets/7abc604d-14b9-4c0a-a561-9acde1d5359e"
/>
**With Fix**
<img width="348" height="162" alt="image"
src="https://github.com/user-attachments/assets/d93a7253-c1fc-4ff9-891b-2e0c6ec46a65"
/>

**Test Discovery Fix**

```
master $ bazel query 'kind(zig_test, //bin/zml-smi/...)'
INFO: Empty results

master $ git checkout -
Switched to branch 'fix/zml-smi-logo-size'

fix/zml-smi-logo-size $ bazel query 'kind(zig_test, //bin/zml-smi/...)'
//bin/zml-smi/tui:test
```
2026-04-22 12:39:14 +02:00
Corentin Kerisit
241aaf7d3b
platforms/cuda: add arm64 support (#493)
Co-authored-by: Steeve Morin <steeve@zml.ai>
2026-04-22 12:37:48 +02:00
Benjamin Rabier
fa3811fcd0
tokenizer/iree: Fix integer underflow (#492) 2026-04-21 15:09:21 +02:00
Benjamin Rabier
154a2cb641
workspace/xla: Use rolledback version of pjrt (#491)
Rolling back XLA to versions that work with triton 3.6
2026-04-21 15:06:39 +02:00
Hugo Mano
b25349b384
ci: bump ci actions (Node.js 20 actions are deprecated) (#490)
Use newer versions of several GitHub Actions. Also bump used Zig
version.
`mlugg/setup-zig` is still using Node.js v20 so warning still here but
only for this action. Check
https://codeberg.org/mlugg/setup-zig/pulls/55 in a near future in order
to update the action.
2026-04-21 14:27:00 +02:00
Benjamin Rabier
49c805f7e3
zml/attention/triton: thread-safe generateTtir (#488)
generateTtir isn't thread-safe as we keep a single process alive. Using
different processes didn't seem to be better from a quick try. So just
locking it with a mutex.
2026-04-20 19:28:49 +02:00
Benjamin Rabier
f258280ab8
platforms/cuda+rocm: update pjrt artifact (#487)
Update with the XLA patches for improved parallel compilation.

`//examples/llm` goes from 27s to 8s for llama3.1 8B compilation
2026-04-20 18:24:01 +02:00
Kevin Barre
c7f26eeeac
zml/attention: standardization of the mosaic ragged page attention structure (#486)
Aligns Mosaic TPU ragged paged-attention input/parameter naming with the
terminology used by other backends (e.g., block table + sequence
lengths)

Changes:
- Rename Mosaic TPU ragged attention inputs/parameters from
kv_lens/page_indices/cu_seqlens_q to
seq_lens/block_table/query_start_len.
2026-04-20 18:14:09 +02:00
Benjamin Rabier
c1a902b131
workspace/xla: update to nightly-2026-04-20 (#485) 2026-04-20 17:16:45 +02:00
Benjamin Rabier
df86cdfd96
zml/tokenizer/iree: Improve IREE tokenizer with upstream fix (#482)
I've reworked the Tokenizer API to actually work with all tokenizers and
not just IREE and moved the `Writer` part into the IREE tokenizer. I'm
now avoiding intermediate copies as much as possible now.

I'm also more defensive now to avoid the same infinite loop and put more
effort into providing appropriate output buffers to IREE. It fails with
empty output buffers and may return a `ResourceExhausted` if it's too
small. Also added a bunch of tests.

IREE tokenizer itself had a bug which I fixed here
https://github.com/iree-org/iree/pull/24150. In the meantime, I'm
pointing bazel to our fork.
2026-04-20 11:15:37 +02:00
Steeve Morin
3e3f2f7297 zml/tensor: for argxmax, mirror more closely emitted stablehlo for pattern matching
For instance, the tenstorrent plugin won't match it
2026-04-20 10:18:45 +02:00
Steeve Morin
da99fe93b5 platforms/rocm: bump ROCm=7.2.2 2026-04-17 16:48:08 +00:00
Kevin Barre
77bdfadc1a platforms/tpu: add ragged paged attention (#481) 2026-04-17 16:47:25 +02:00
Corentin Kerisit
45f65c7b97
workspace: expose translate-c so that downstream can use it (#484) 2026-04-17 14:54:14 +02:00
Corentin Kerisit
bd66cfa1f3
Update ZLS to 0.16.0 and rules_zig new providers (#483)
Bump rules_zig to latest zig_toolchain changes (for bootstrap)
Bump ZLS to 0.16.0
Amend ZLS completion rules and runners to match rules_zig new providers
2026-04-17 13:42:37 +02:00
Hugo Mano
8b12d2ef1e
bazel: make repo-wide cquery work across host and linux target analysis (#465)
This PR fixes several Bazel analysis regressions so both of these
commands work reliably:

- `bazel cquery //...`
- `bazel cquery --platforms=@zml//platforms:linux_amd64 //...`

After recent changes and the rebase, repo-wide cquery was failing for
three separate reasons:

- ZLS toolchains were over-constrained by target platform: updated ZLS
toolchain registration to constrain only the exec platform.
This avoids cross-target analysis failures when host-side completion
tooling is analyzed under a foreign target platform.
- OCI packaging targets were analyzed on host platforms where their base
images do not apply: marked Linux OCI image targets as Linux x86_64
compatible only. Also switched the Debian base image pull from Docker
Hub to the GCR mirror to avoid CI fetch failures when resolving
packaging images manifests due to rate limiting
- Triton Python deps were fetched for the host Python platform instead
of the Linux wheel platform the lockfile was generated for: restored
Linux x86_64 wheel resolution for `triton_py_deps` in `MODULE.bazel`.
This prevents macOS host analysis from attempting to fetch a
`triton==3.6.0` wheel for an unsupported host platform.


Also added a step in ci to prevent regression
2026-04-17 11:48:09 +02:00
Corentin Godeau
0c67102033
zml/module: provide an Io to the CompilationContext (#457)
This might come in handy in the future if we need to do some IO in the
compilation pipeline
2026-04-17 11:41:09 +02:00
Corentin Godeau
f23a8a14d3
zml/ops: improve metadata conversion (#474)
In #470 I introduced support for optional float in custom call metadata
but it was incomplete.

This PR cleans the code a bit and introduce support for all basic
optional types.
2026-04-17 11:35:58 +02:00
Benjamin Rabier
1e79469136
examples/llm: fix prompt tokenization (#480)
Tested with IREE and HF on all three models.
2026-04-16 11:09:47 +02:00
Corentin Kerisit
169f7a021b
workspace: bump llvm=0.7.1 (#479)
This fixes #467
2026-04-16 10:20:08 +02:00
Steeve Morin
185e75f884 workspace: bump zig=0.16.0
also bump translate-c and remove a genrule in arocc
2026-04-15 21:42:47 +02:00
Steeve Morin
b912b08c40 zml/attention/triton: sandbox zlib1g when running the triton codegen
else python fails in a container
2026-04-15 21:32:51 +02:00
Benjamin Rabier
91f58097cc
zml/tokenizer: use correct API for decoding with HF (#478)
The current API only worked with IREE. Reworked it to work with both HF
and IREE
2026-04-14 17:13:26 +02:00
Steeve Morin
2f11092deb workspace: correctly set zig mode on host tools
this was a typo
2026-04-14 13:10:33 +02:00
Steeve Morin
46862fa85a workspace: put overrides next to the dep
it's clearer, also, do a buildifier pass
2026-04-14 13:09:50 +02:00
Steeve Morin
e58c6af965 workspace: pin hermeticbuild/hermetic-llvm to git
this is until the release is done, this fixes duplicate symbols
between clang and zig compiler_rt
2026-04-14 13:07:54 +02:00
Corentin Godeau
35e8ae9082
zml/ops: fix comptime for optional float value in custom call metadata (#472) 2026-04-14 10:07:03 +02:00
Corentin Godeau
6ad3e26790
zml/attention: add option to set the softmax scaling (#470)
I added a `scale` field in the `AttentionOptions` struct that is passed
through the attention backend.

I had to add support for optional in the custom call metadata. For now
it only supports optional floats, we'll add support for other types
later.
2026-04-14 09:48:32 +02:00
Corentin Godeau
e47b7ea390
zml/nn: add proportional scaling option to RopeOpts and related tests (#471)
Co-authored-by: Kevin Barre <kevin@zml.ai>
2026-04-14 09:44:05 +02:00
Benjamin Rabier
d343b84fd8
zml/tokenizer: Re-introduce HF tokenizer (#469) 2026-04-14 09:42:02 +02:00
Benjamin Rabier
16fb132dec
zml/attention/triton: rename .num_pages_per_seq to .p (#466)
Using the same axis name as FlashAttention for consistency.
2026-04-13 15:16:18 +02:00
KB
bd113697d4
zml/attention: add Linux compatibility only for Triton dependencies (#464)
Co-authored-by: Kevin Barre <kevin@zml.ai>
2026-04-13 14:56:55 +02:00
Steeve Morin
8ec67ee5c3
zml: remove HuggingFace tokenizer (#463)
Following #416, the HuggingFace tokenizer is redundant.

This also removes the Rust code from the repo, as well as rules_rust and
rules_rs.
2026-04-13 12:42:44 +02:00
KB
8edf02d0c3
zml/chore: bump Zig to 0.16.0-dev.3133 and adapt codebase (#462)
Update Zig compiler to 0.16.0-dev.3133+5ec8e45f3 and adjust code
for breaking standard library changes:

- Replace ArrayListUnmanaged/ArrayList init with .empty
- Switch to unmanaged AutoArrayHashMap/StringArrayHashMap
- Use debug.GeneralPurposeAllocator (renamed upstream)
- Replace std.Io.File.ReadStreamingError with error.EndOfStream
- Refactor meta.visit to use structured visitor pattern
- Fix alignment allocation in mem.bufferizeInner

Also update third-party dependencies: flashattn 0.0.6-rc8, arocc,
uucode, libvaxis, and zigimg patches.

Co-authored-by: Kevin Barre <kevin@zml.ai>
2026-04-13 11:56:32 +02:00
Corentin Godeau
ac9bc32d89
zml/attention: add Triton Unified Attention backend (#417)
This PR introduces a Triton paged attention backend, unlocking ROCm
support.

It relies on a Python sandbox to generate the TTIR.

The major change is that the KVCache now needs to be per-layer for paged
attention. The `layer_index` is now superfluous and will probably be
removed in the future.
2026-04-13 11:39:08 +02:00
elogir
1362a2cc45
bin/zml-smi: fix release script (#460)
- bucket name in rclone config
- version name will be tag
2026-04-13 11:09:43 +02:00
Benjamin Rabier
177fddabeb
zml/attention & zml/ops: mark fa2 as sharding aware and fix implementation (#461)
Sharding aware case wasn't properly handled and I marked fa2 as sharding
unaware even though it is.
2026-04-11 10:18:44 +02:00
elogir
572ba04a0c
bin/zml-smi: fix mapfile (#458)
Replace mapfile with manual IFS read
2026-04-10 17:55:27 +02:00
Steeve Morin
e71c523605 workspace: swap strip linkopt to bazel flag
because --strip-all doesn't exist on macos
2026-04-10 17:30:09 +02:00
elogir
8701a54882
bin/zml-smi: add prometheus endpoint, json and macos host infos (#455)
This PR introduces:

- Prometheus metrics endpoint through `--prometheus` and optionally
`--prometheus-port`
- JSON output mode with `--json`
- macOS arm64 host information
- An install and release script
2026-04-10 17:26:09 +02:00
Steeve Morin
ef66f310ec workspace: bump rules_zig
to segregate exec platform only options
2026-04-09 10:23:11 +02:00