Add `Builder.log1p` to Mosaic TPU kernel DSL
The MLIR math dialect already exposes `math.log1p`.
This change makes it available through the Mosaic TPU builder alongside
the already existing operations.
--> This avoids kernels constructind `math.log1p` operations manually
and makes the operation reusable for future DSL kernel translations.
A companion DSL-harness changes replaces existing manual `math.log1p`
construction with `Builder.log1p`.
Add MLIR composites, `scaledDot`, and a Linear path for NVFP4 weights
(unpack, optional activation quant on Blackwell, global scales). Skip
fused multi-source tensors in the generic loader so pack/`loadExecute`
can fill them (may be used later for fused QKV etc).
Implement paged_attention in pure stablehlo.
introduces `zml.ops.if` and `zml.ops.if2`
The first one is similar to `zml.ops.while` and requires a full struct
with two functions. it's a bit heavy to use, but it follows the
stablehlo API, and if one of the branch has side effects it will
correctly evaluate the side effect only in that branch.
`zml.ops.if2` is a simpler version that doesn't require to create
intermediary struct, but as a consequence, if you have side effect they
will always evaluate, and it relies on the compiler to correctly avoid
duplicated compute by moving relevant compute inside the branches.
(open to feedback about if/if2)
Simplify vfs architecture: merge `//zml/io` and `//zml/io/vfs`. They move to top level `@zml//vfs`
The PR also include two bug fix:
- S3: when no content length we were trying to read exactly 1M, while we
should read up to 1M
- GCP: restore the commented code for parsing bucket info
## Test plan:
```
bazel run --config=debug //examples/io:playground -- ls gs://gcp-public-data-landsat/
bazel run --config=debug //examples/io:playground -- tree s3://noaa-goes19/ABI-Flood-Day-Shapefiles/2025/08
```
Add unit test for paged attention, enforcing all our backends produce
the same outputs.
To simplify memory management of many small buffer provided in a struct
(like paged attention metadata buffers), I introduce
`zml.Buffer.deinitAll` backed by `zml.meta.visitFlatStruct`. Contrary to
og `zml.meta.visit` the `flatStruct` variant emit a compile error if the
input has a slice/ptr inside it. The idea is that since it enforces the
struct to be a trivial, it means the struct won't require complex
`deinit` and `zml.Buffer.deinitAll` is in fact everything you need to
`deinit`. This avoid previous footgun of `zml.io.unloadAll`.
This led to more improvement with `zml.Tensor.toMemoryAll` and
`zml.Tensor.onMemoryAll` that also leverage the new
`zml.meta.visitFlatStruct`
removes `anytype` from `zml.ops.while`, helping with type checking and
simplifying a bit the logic.
A `While` struct needs
* a State struct definition that captures all the loop state that will
change on each iteration
* a `cond(While, State) Tensor` function returning a scalar boolean
tensor
* a `body(While, State) State` function that preserves the state shapes.
example:
```
const While = struct {
// While struct can contain tensor and non-tensor fields.
limit: i64,
step: Tensor,
pub const State = struct {
i: zml.Tensor,
sum: zml.Tensor,
};
pub fn cond(self: @This(), state: State) Tensor {
return state.i.cmp(.LT, Tensor.scalar(self.limit, .i64));
}
pub fn body(self: @This(), state: State) State {
return .{
.i = state.i.add(self.step),
.sum = state.sum.add(self.step),
};
}
fn forward(i: Tensor, sum: Tensor) State {
return zml.ops.@"while"(
@This(),
.{ .limit = 10, .step = .scalar(1, .i64) },
.{ .i = i, .sum = sum },
);
}
};
```
The code is a bit more verbose for simple example, but for GatedDeltaNet
it's equivalent, and I have also used it for paged attention.
While working on paged attention I realize that
`zml.testing.expectClose` wasn't properly handling strided Slice.
To fix that I'm adding a `ContiguousItemsIterator` that loops over the
contiguous part of the slice.
Only used in test now, so no perf considerations.
## Context
Translate
https://github.com/vllm-project/tpu-inference/tree/main/tpu_inference/kernels/gdn/v3
kernels.
## Summary
- add `BufferedRefStrategy` callbacks for overriding `copyIn`, `waitIn`,
`copyOut`, and `waitOut` on individual `PipeSpec`s
- add the `input_output` buffer role, with separate receive/send
semaphores and four loop-carried slot counters: copy-in, wait-in,
copy-out, and wait-out (input-only refs use `copy_in_slot` and
`wait_in_slot` ; output-only refs use `copy_out_slot` and
`wait_out_slot`, `input_output` refs use all four)
- provide custom transfer callbacks with the references, buffer slots,
and semaphores needed to emit their DMA operations (`sourceRef`,
`windowRef`, `copyInSlot`, `waitInSlot`, `receiveSemaphoreAt`,
`copyOutSlot`, `waitOutSlot`, `sendSemaphoreAt`)
- fall back to the existing transfer implementation for callbacks that
are not overridden
## Validation
- all 7 `fused_conv1d_gdn_batched` sweeps produce matching Python and
Zig MLIR, covering tile size, state count, head counts, Q/K-to-V head
ratio, distinct head dimensions, and convolution kernel size
- all 19 existing `decoding_gdn-bt_8` sweeps continue to produce
matching Python and Zig MLIR
Pre-allocating 85% of memory for simple tests is overkill, we'll just
let the BFC grow as needed. That should hopefully limit the problem of
having concurrent CI runs.
This PR introduce a new option gpu_async_dispatch that set to true by
default the legacy pjrt API option: `use_tfrt_gpu_client`. This option
in xla creates an other thread per device to dispatch the thunk and
command buffers asynchronously on the gpu. It strongly reduces the cpu
overhead when the model is sharded across several devices.
This PR contains two main changes:
* A patch for the tokenizer to support the tokenizer config file
* A temporary workaround to speed-up loading of the model when executing
arbitrary executable at load time
Update the local neovim configuration to override only the fields that
needs to change (such as `cmd` for LSP) instead of replacing the user's
entire configuration.
The main difference is `vim.lsp.config("zls", { ... })` which merges the
provided table into the existing configuration, while
`vim.lsp.config["zls"] = { ... }` overwrite the LSP configuration for
ZLS.
I also fixed the same issue for conform.
Signed-off-by: dmehala <damien@zml.ai>
Small mistake happened while cleaning #586, I factorized code that
shouldn't have been factorized blindly.
This PR fixes a bug where tensor loaded using `Loader.load` were not
correctly sharded.
The old `zml.io.load` was plagued with problems. This PR is an attempt
at improving the state of things.
One major problem of `zml.io.load` was that it was not very composable.
It was either using it to load all the model or doing everything by
hand.
Another problem was that it didn't offer a way to apply transformations
at load time to the tensors stored in a safetensors file. For example,
it happens that for MoE models, instead of the experts weights to be
tightly packed in a nice 3D tensor, the weight file contain one tensor
for each expert. This is nice until your implementation required a 3D
tensor as input. Using the old `zml.io.load` API this resulted in very
convoluted way to do concatenation at load time. With this new API, we
offer convenience so that it can be handled nicely and as fast as
possible.
The main change is the introduction of the `Loader` struct. It's a
struct that you instantiate once and reuse whenever you want to load
tensors. It relies on `Binding`, which are a way to store which tensor
are required and how they need to be combined (if it's a complex binding
involving more than one source tensor).
One advantage of this `Loader` is that it's composable. Loading a struct
using `Loader.auto(Struct)` or loading each individual field of `Struct`
has essentially the same behavior. This solved the major problem of the
old solution.
For simplicity I kept the old model loading code almost identical, as
this new solution is a drop-in replacement.
# Update
After some discussions with @gwenzek I've changed my mind. Instead of
having a declarative API, we'll provide helper to do transformation
easily. The main changes that this PR now bring are:
* Introduce `Loader` struct
* `Loader` exposes `load` which is the composable replacement of
`zml.io.load`
* `Loader` exposes `loadExecute` that allows you to execute an
executable with the buffers loaded from the sources associated with a
tensor. This allows for example to concatenate tensors.
We'll continue to work on it, this is just a preliminary work that
allows us to do transformation at load time in LLMD.
I hit an issue with the plugin refusing to load on macos.
There was no information.
After adding this I saw that the the file was apparently not existing in
the bazel sandbox,
so I could reduce the bug to a Bazel issue.
# Bump intel oneapi and rules_distroless
The Debian package set is now also exposed separately through the
rules_distroless APT extension:
Now use `rules_distroless` to install the oneAPI packages in a
distroless container image.
So the platforms/oneapi/packages.lock.json file is now generated from
the platforms/oneapi/packages.yaml file, and the packages.yaml file is
now the source of truth for the oneAPI packages.
The MODULE.bazel contain rules_distroless patch
`third_party/rules_distroless/accept-nospace-version-constraints.patch`
because the rules_distroless APT extension does not support version
constraints with spaces, e.g. `>= 2023.2.0` is not supported, but
`>=2023.2.0` is supported.
A PR has been submitted to rules_distroless to fix this issue, but until
it is merged and released, the patch is needed in zml
https://github.com/bazel-contrib/rules_distroless/pull/229.
`_ROOT_MODULE_DIRECT_DEPS` does not need anymore to list the individual
oneAPI and Ubuntu package repositories.
`_ROOT_PACKAGES` is now in `_UBUNTU_PACKAGES` because all the package
share the same `_UBUNTU_STRIP_PREFIX`
## Tests
```
bazel run @apt_oneapi//:lock
```
```
ONEAPI_DEVICE_SELECTOR=level_zero:0 \
bazel run \
--config=release \
--@zml//platforms:cpu=false \
--@zml//platforms:oneapi=true \
//examples/llm -- \
--model=/var/models/meta-llama/Llama-3.1-8B-Instruct \
--topk=4 \
--prompt="Tell me a story about evariste galois"
```
__-> decode 26.248s · 34.9tok/s__
```
ONEAPI_DEVICE_SELECTOR=level_zero:0,1 \
bazel run \
--config=release \
--@zml//platforms:cpu=false \
--@zml//platforms:oneapi=true \
//examples/llm -- \
--model=/var/models/meta-llama/Llama-3.1-8B-Instruct \
--topk=4 \
--prompt="Tell me a story about evariste galois"
```
__-> decode 13.017s · 56.6tok/s__
It's easy to break `Buffer.from(x_tensor.shape(), data, sharding)` by
creating `x_tensor` in the model code with a specific sharding and you
pass a different one to `from`.
I think this should be a crash (like when you mix-up allocators in
init/deinit) and it should have a dedicated error message to guide the
developers.
We used to panic when PJRT compilation failed.
I now think it's better to return an error, since we introduced the WIP
Metal plugin that can fail on legitimate programs.
This will make it possible to run the ZML test suite on the Metal plugin
and seeing all failing tests.
```
bazel run --@zml//platforms:metal=true --@zml//platforms:cpu=false //zml:test
33/138 nn.decltest.sampleTokens...FAIL (Internal)
...
34/138 nn.decltest.sampleTokensDynamic...FAIL (Internal)
...
102/138 tensor.Tensor.decltest.convert...FAIL (Internal)
...
115/138 tensor.Tensor.decltest.argsort...FAIL (Internal)
```
Render the zml-smi text logo fallback from the shared ZML logo data,
including the shiny per-glyph colors. Add a lightweight //zml:logo
target so the TUI can depend on the logo art without pulling in //zml.
Verified with:
- bazel test //bin/zml-smi/tui:test
- bazel build //bin/zml-smi:zml-smi
When a DRM fd is duplicated or inherited through fork(), the same oneAPI
DRM client can appear multiple times under /proc/<pid>/fdinfo. Summing
each fdinfo entry double-counts GPU memory and engine time.
Parse drm-client-id and skip entries whose (device_idx, client_id) pair
has already been seen during the collection pass. Include the device in
the key because drm-client-id may be scoped per DRM device.
Add a focused test covering duplicate clients on the same device and
distinct clients across devices or client IDs.
---------
Co-authored-by: Kevin Barre <kevin@zml.ai>
## Summary
Fix IRFFT shape inference so it uses the requested FFT output length and
validates that the complex input length matches the corresponding RFFT
output length.
Fixes#411.
## Why
RFFT changes the innermost real dimension `n` into `n / 2 + 1` complex
bins. IRFFT receives those complex bins plus the requested real FFT
length, so its output shape should restore that requested length instead
of deriving a smaller value from the complex input dimension.
## Test verification (RED -> GREEN)
Command used for all runs:
```bash
npx -y @bazel/bazelisk@1.25.0 test //zml:test --test_output=errors
```
RED, with the new regression test before the fix:
```text
102/139 tensor.Tensor.test.Tensor.fft IRFFT restores RFFT last dimension...thread 1701155 panic: exact division produced remainder
INFO: Build completed, 1 test FAILED, 5355 total actions
//zml:test FAILED in 24.9s
Executed 1 out of 1 test: 1 fails locally.
```
GREEN, with the fix applied:
```text
INFO: Build completed successfully, 6 total actions
//zml:test PASSED in 8.1s
Executed 1 out of 1 test: 1 test passes.
```
Revert RED, with the production fix reverted while keeping the test:
```text
102/139 tensor.Tensor.test.Tensor.fft IRFFT restores RFFT last dimension...thread 1711292 panic: fft(.{ .kind = .IRFFT, .length = { 10 } }) expects tensor last dimensions to match given lengths, got 1 and 1
INFO: Build completed, 1 test FAILED, 6 total actions
//zml:test FAILED in 22.9s
Executed 1 out of 1 test: 1 fails locally.
```
## Additional checks
```text
zig fmt --check zml/tensor.zig
fmt_status=0
```
In the VFS router, a scheme-qualified path like `hf://owner/model/file`
is meant to be absolute: it should resolve from the scheme's root. But
`lookupDir` only treated it as absolute when the base directory was
`cwd`. When such a path is opened *relative to an already-open directory* of
the same backend, the router kept the directory's path and glued the new
path after it, duplicating it. So `hf://owner/model` +
`hf://owner/model/file` became `owner/model/owner/model/file`, and the
lookup failed with `FileNotFound`.
Fix: on `lookupDir`, any `scheme://` sub-path resolves from the scheme's root
regardless of the base directory, the same way an absolute POSIX path
already does. About four lines.
Regenerate the xprof uv lockfile to move xprof-nightly off the removed
2.23.2a20260505 build and onto 2.23.3a20260707.
Fixes Bazel fetch failures for //tools/xprof:xprof.
Use the standard minimum/power-of-two block_m sizing for paged attention
instead of the raw GQA group size. This keeps oneAPI decode tile sizing
valid for Triton IR generation.
## Summary
- replace the legacy `new_git_repository` alias with `git_repository`
for `arocc`, `libvaxis`, `linenoise`, and `translate-c`
## Why
`git_repository` is the canonical Bazel repository rule.
`new_git_repository` doesn't exist at bazel HEAD
A 1-chip TPU mesh collapses to rank 1 (only `link_x`), so the
unconditional `link_y` fold tripped InvalidPhysicalAxis. Only fold
`link_y` when the physical mesh actually has it.