fix(mosaic_tpu): support padded KV-cache head dims in ragged paged
attention
Pass both logical and cache head dimensions through the Mosaic TPU paged
attention path so KV pages can remain 128-byte aligned without padding Q
or trimming the final output.
- add backend helpers and tests for Mosaic TPU kernel head-dim alignment
- keep query/output tensors and softmax scaling on the logical head dim
- allow cache head dims greater than or equal to query head dims in TPU
IR and kernel validation
- slice padded K/V vectors back to the logical head dim before attention
math
- fix the tensor pad call signature used by tensor padding
When looking at the new Sharding code I was triggered by the high number
of "try".
I then realized most of the try where traced back to `axes = try
.init(0)` which can never fail.
So I removed BoundedArray.init (aka Andrew mode)
It's generally unsafe because it makes `slice()` return undefined.;
I replaced`try init(0)` by `.empty`
I replaced `try init(n)` by `.fromSlice` or with `.{ .buffer =
undefined, .len = n }` to explicitly show the unsafeness
This leads to widespread changes because it means that most of the
sharding operations can't fail anymore.
Improve TPU ragged paged attention strided KV loading
- add the TPU ragged paged attention kernel to the repo and package it
with gen_ir
- switch gen_ir to import the vendored kernel instead of JAX's default
one
- update tpu_attention tensor axis tags to match the kernel's expected
shapes
refactor: introduce KVCache union and wire dense/split layout through
pagedAttention
Adds a `KVCache` union type to `paged_attention.zig` with two variants:
- `.split { k, v }`: separate K and V page tensors (used by cuda_fa2,
cuda_fa3, triton)
- `.dense`: a single pre-fused K/V page tensor (used by mosaic_tpu)
Changes:
- `pagedAttention()` in `paged_attention.zig` now accepts a `kv_cache:
KVCache`
parameter alongside `k`/`v` (unused); dispatches `.split` to CUDA/Triton
backends and the full `KVCache` union to `mosaic_tpu`.
- `mosaic_tpu.pagedAttention()` in `tpu_attention.zig` now takes
`layout: paged_attention.KVCache` instead of separate k/v tensors, and
passes `layout.dense` to the existing `prepareInputs()` helper.
- `KvCache.Layer` in `llmd/models.zig` is now a tagged union of `.dense`
(fused tensor) and `.split { k, v }`. `attentionLayout()` converts it to
`attention.KVCache`. The `.dense` update path in `Layer.update()` fuses
K and V by inserting a `.kv` axis, concatenating, and merging `.hkv`
before scattering into the page cache.
Specifying both `rows` and `cols` makes the zml logo in `zml-smi` render
correctly in KDE Konsole. Verified that the fix didn't cause any
regression in a terminal emulator where logo rendering was already
functional (`Ghostty`).
`BUILD.bazel` is also updated to run the `zml-smi` tests.
**Before**
<img width="348" height="238" alt="image"
src="https://github.com/user-attachments/assets/7abc604d-14b9-4c0a-a561-9acde1d5359e"
/>
**With Fix**
<img width="348" height="162" alt="image"
src="https://github.com/user-attachments/assets/d93a7253-c1fc-4ff9-891b-2e0c6ec46a65"
/>
**Test Discovery Fix**
```
master $ bazel query 'kind(zig_test, //bin/zml-smi/...)'
INFO: Empty results
master $ git checkout -
Switched to branch 'fix/zml-smi-logo-size'
fix/zml-smi-logo-size $ bazel query 'kind(zig_test, //bin/zml-smi/...)'
//bin/zml-smi/tui:test
```
Use newer versions of several GitHub Actions. Also bump used Zig
version.
`mlugg/setup-zig` is still using Node.js v20 so warning still here but
only for this action. Check
https://codeberg.org/mlugg/setup-zig/pulls/55 in a near future in order
to update the action.
generateTtir isn't thread-safe as we keep a single process alive. Using
different processes didn't seem to be better from a quick try. So just
locking it with a mutex.
Aligns Mosaic TPU ragged paged-attention input/parameter naming with the
terminology used by other backends (e.g., block table + sequence
lengths)
Changes:
- Rename Mosaic TPU ragged attention inputs/parameters from
kv_lens/page_indices/cu_seqlens_q to
seq_lens/block_table/query_start_len.
I've reworked the Tokenizer API to actually work with all tokenizers and
not just IREE and moved the `Writer` part into the IREE tokenizer. I'm
now avoiding intermediate copies as much as possible now.
I'm also more defensive now to avoid the same infinite loop and put more
effort into providing appropriate output buffers to IREE. It fails with
empty output buffers and may return a `ResourceExhausted` if it's too
small. Also added a bunch of tests.
IREE tokenizer itself had a bug which I fixed here
https://github.com/iree-org/iree/pull/24150. In the meantime, I'm
pointing bazel to our fork.
Bump rules_zig to latest zig_toolchain changes (for bootstrap)
Bump ZLS to 0.16.0
Amend ZLS completion rules and runners to match rules_zig new providers
This PR fixes several Bazel analysis regressions so both of these
commands work reliably:
- `bazel cquery //...`
- `bazel cquery --platforms=@zml//platforms:linux_amd64 //...`
After recent changes and the rebase, repo-wide cquery was failing for
three separate reasons:
- ZLS toolchains were over-constrained by target platform: updated ZLS
toolchain registration to constrain only the exec platform.
This avoids cross-target analysis failures when host-side completion
tooling is analyzed under a foreign target platform.
- OCI packaging targets were analyzed on host platforms where their base
images do not apply: marked Linux OCI image targets as Linux x86_64
compatible only. Also switched the Debian base image pull from Docker
Hub to the GCR mirror to avoid CI fetch failures when resolving
packaging images manifests due to rate limiting
- Triton Python deps were fetched for the host Python platform instead
of the Linux wheel platform the lockfile was generated for: restored
Linux x86_64 wheel resolution for `triton_py_deps` in `MODULE.bazel`.
This prevents macOS host analysis from attempting to fetch a
`triton==3.6.0` wheel for an unsupported host platform.
Also added a step in ci to prevent regression
In #470 I introduced support for optional float in custom call metadata
but it was incomplete.
This PR cleans the code a bit and introduce support for all basic
optional types.
I added a `scale` field in the `AttentionOptions` struct that is passed
through the attention backend.
I had to add support for optional in the custom call metadata. For now
it only supports optional floats, we'll add support for other types
later.
Update Zig compiler to 0.16.0-dev.3133+5ec8e45f3 and adjust code
for breaking standard library changes:
- Replace ArrayListUnmanaged/ArrayList init with .empty
- Switch to unmanaged AutoArrayHashMap/StringArrayHashMap
- Use debug.GeneralPurposeAllocator (renamed upstream)
- Replace std.Io.File.ReadStreamingError with error.EndOfStream
- Refactor meta.visit to use structured visitor pattern
- Fix alignment allocation in mem.bufferizeInner
Also update third-party dependencies: flashattn 0.0.6-rc8, arocc,
uucode, libvaxis, and zigimg patches.
Co-authored-by: Kevin Barre <kevin@zml.ai>
This PR introduces a Triton paged attention backend, unlocking ROCm
support.
It relies on a Python sandbox to generate the TTIR.
The major change is that the KVCache now needs to be per-layer for paged
attention. The `layer_index` is now superfluous and will probably be
removed in the future.
This PR introduces:
- Prometheus metrics endpoint through `--prometheus` and optionally
`--prometheus-port`
- JSON output mode with `--json`
- macOS arm64 host information
- An install and release script