0
Fork 0
mirror of https://github.com/zml/zml.git synced 2026-11-16 19:11:09 +00:00
Commit graph

1,078 commits

Author SHA1 Message Date
Raphael Nahoum
830c8facff
zml/nn: treat Linear global scales as dequant multipliers (#698)
Make `Linear.global_scale` / `input_global_scale` match checkpoint
conventions
2026-07-31 19:22:03 +02:00
Guillaume Wenzek
4976dfbeb3
bazel: use github.com instead of codeberg.org as reference (#695) 2026-07-31 14:59:58 +02:00
Olivier Boulant
57eabe72de
kernels/mosaic_tpu: add log1p builder operation (#691)
Add `Builder.log1p` to Mosaic TPU kernel DSL

The MLIR math dialect already exposes `math.log1p`. 
This change makes it available through the Mosaic TPU builder alongside
the already existing operations.

--> This avoids kernels constructind `math.log1p` operations manually
and makes the operation reusable for future DSL kernel translations.

A companion DSL-harness changes replaces existing manual `math.log1p`
construction with `Builder.log1p`.
2026-07-31 14:52:11 +02:00
Raphael Nahoum
14a0a8f507
zml: start of quantization support (#690)
Add MLIR composites, `scaledDot`, and a Linear path for NVFP4 weights
(unpack, optional activation quant on Blackwell, global scales). Skip
fused multi-source tensors in the generic loader so pack/`loadExecute`
can fill them (may be used later for fused QKV etc).
2026-07-31 11:02:07 +02:00
Guillaume Wenzek
648c5fd2ee
zml/attention: stablehlo paged attention (#688)
Implement paged_attention in pure stablehlo.

introduces `zml.ops.if` and `zml.ops.if2`
The first one is similar to `zml.ops.while` and requires a full struct
with two functions. it's a bit heavy to use, but it follows the
stablehlo API, and if one of the branch has side effects it will
correctly evaluate the side effect only in that branch.

`zml.ops.if2` is a simpler version that doesn't require to create
intermediary struct, but as a consequence, if you have side effect they
will always evaluate, and it relies on the compiler to correctly avoid
duplicated compute by moving relevant compute inside the branches.

(open to feedback  about if/if2)
2026-07-31 10:57:02 +02:00
Guillaume Wenzek
2b9320fe72
//zml/io/vfs: promote to //vfs (#680)
Simplify vfs architecture: merge `//zml/io` and `//zml/io/vfs`. They move to top level `@zml//vfs`

The PR also include two bug fix:
- S3: when no content length we were trying to read exactly 1M, while we
should read up to 1M
- GCP: restore the commented code for parsing bucket info

## Test plan:

```
bazel run --config=debug //examples/io:playground -- ls gs://gcp-public-data-landsat/
bazel run --config=debug //examples/io:playground -- tree s3://noaa-goes19/ABI-Flood-Day-Shapefiles/2025/08
```
2026-07-30 14:21:36 +02:00
Guillaume Wenzek
09a7a9130a
zml/attention: test paged attention (#681)
Add unit test for paged attention, enforcing all our backends produce
the same outputs.

To simplify memory management of many small buffer provided in a struct
(like paged attention metadata buffers), I introduce
`zml.Buffer.deinitAll` backed by `zml.meta.visitFlatStruct`. Contrary to
og `zml.meta.visit` the `flatStruct` variant emit a compile error if the
input has a slice/ptr inside it. The idea is that since it enforces the
struct to be a trivial, it means the struct won't require complex
`deinit` and `zml.Buffer.deinitAll` is in fact everything you need to
`deinit`. This avoid previous footgun of `zml.io.unloadAll`.

This led to more improvement with `zml.Tensor.toMemoryAll` and
`zml.Tensor.onMemoryAll` that also leverage the new
`zml.meta.visitFlatStruct`
2026-07-30 13:21:00 +02:00
Guillaume Wenzek
bdbfeb19ec
zml.ops: refactor while loop API (#686)
removes `anytype` from `zml.ops.while`, helping with type checking and
simplifying a bit the logic.
A `While` struct needs 
* a State struct definition that captures all the loop state that will
change on each iteration
* a `cond(While, State) Tensor` function returning a scalar boolean
tensor
* a `body(While, State) State` function that preserves the state shapes.

example:

```
const While = struct {
    // While struct can contain tensor and non-tensor fields.
    limit: i64,
    step: Tensor,

    pub const State = struct {
        i: zml.Tensor,
        sum: zml.Tensor,
    };

    pub fn cond(self: @This(), state: State) Tensor {
        return state.i.cmp(.LT, Tensor.scalar(self.limit, .i64));
    }

    pub fn body(self: @This(), state: State) State {
        return .{
            .i = state.i.add(self.step),
            .sum = state.sum.add(self.step),
        };
    }

    fn forward(i: Tensor, sum: Tensor) State {
        return zml.ops.@"while"(
            @This(),
            .{ .limit = 10, .step = .scalar(1, .i64) },
            .{ .i = i, .sum = sum },
        );
    }
};
```

The code is a bit more verbose for simple example, but for GatedDeltaNet
it's equivalent, and I have also used it for paged attention.
2026-07-30 11:29:42 +02:00
Benjamin Rabier
fdb6d5cd23
platforms/rocm: Update package.lock.json (#687) 2026-07-29 18:01:59 +02:00
Guillaume Wenzek
1affa657ae
zml/Slice: correctly handle strides (#685)
While working on paged attention I realize that
`zml.testing.expectClose` wasn't properly handling strided Slice.
To fix that I'm adding a `ContiguousItemsIterator` that loops over the
contiguous part of the slice.
Only used in test now, so no perf considerations.
2026-07-29 17:34:53 +02:00
Guillaume Wenzek
bf7e277f06
zml: simplify memory allocation during compilation (#682) 2026-07-28 18:34:52 +02:00
Louis Lechevalier
f4282a47ae
platforms/cuda: Update libpjrt cuda (#684)
Update of libpjrt cuda after rebase and patches update
2026-07-28 15:57:04 +02:00
Olivier Boulant
a0d0af2a8d
flashattn: harmonize zml.Tensor.zeroes() calls (#659)
Following https://github.com/zml/zml/pull/653
2026-07-23 11:17:28 +02:00
Benjamin Rabier
c6415035db
tools/xprof: Fix xprof after update (#664) 2026-07-23 10:54:22 +02:00
Olivier Boulant
d1d313cf67
mosaic_tpu: support custom buffered-ref transfer strategies (#676)
## Context

Translate
https://github.com/vllm-project/tpu-inference/tree/main/tpu_inference/kernels/gdn/v3
kernels.

## Summary

- add `BufferedRefStrategy` callbacks for overriding `copyIn`, `waitIn`,
`copyOut`, and `waitOut` on individual `PipeSpec`s
- add the `input_output` buffer role, with separate receive/send
semaphores and four loop-carried slot counters: copy-in, wait-in,
copy-out, and wait-out (input-only refs use `copy_in_slot` and
`wait_in_slot` ; output-only refs use `copy_out_slot` and
`wait_out_slot`, `input_output` refs use all four)
- provide custom transfer callbacks with the references, buffer slots,
and semaphores needed to emit their DMA operations (`sourceRef`,
`windowRef`, `copyInSlot`, `waitInSlot`, `receiveSemaphoreAt`,
`copyOutSlot`, `waitOutSlot`, `sendSemaphoreAt`)
- fall back to the existing transfer implementation for callbacks that
are not overridden

## Validation

- all 7 `fused_conv1d_gdn_batched` sweeps produce matching Python and
Zig MLIR, covering tile size, state count, head counts, Q/K-to-V head
ratio, distinct head dimensions, and convolution kernel size
- all 19 existing `decoding_gdn-bt_8` sweeps continue to produce
matching Python and Zig MLIR
2026-07-22 15:59:11 +02:00
Benjamin Rabier
252113095c
zml/testing: Do not pre-allocate memory for the zml.testing.env() (#679)
Pre-allocating 85% of memory for simple tests is overkill, we'll just
let the BFC grow as needed. That should hopefully limit the problem of
having concurrent CI runs.
2026-07-22 15:30:46 +02:00
Hugo Mano
3e589c9772
platforms/rocm: bump to rocm-jaxlib-v0.10.2 with TheRock 7.14 (#671)
Linked PR: https://github.com/zml/pjrt-artifacts/pull/96
2026-07-21 16:18:28 +02:00
Corentin Godeau
031b2a56a8
zml/io: add some feedback to loadExecute (#670)
It's a subotimal solution as we print a generic message but it makes the
progress behave as intended.
2026-07-21 09:13:57 +02:00
Louis Lechevalier
62a4d3c651
zml/platform: set async dispatch to true when platform is cuda (#673)
This PR introduce a new option gpu_async_dispatch that set to true by
default the legacy pjrt API option: `use_tfrt_gpu_client`. This option
in xla creates an other thread per device to dispatch the thunk and
command buffers asynchronously on the gpu. It strongly reduces the cpu
overhead when the model is sharded across several devices.
2026-07-20 20:57:36 +02:00
Louis Lechevalier
a35fac997e
platforms/cuda: Bump CUDA to 13.3.0 (#672)
Bump CUDA to 13.3.0 and CUDNN to 9.22.
2026-07-20 20:57:19 +02:00
Corentin Godeau
d57d851b63
zml: ground work to support Poolside's Laguna models (#669)
This PR contains two main changes:
* A patch for the tokenizer to support the tokenizer config file
* A temporary workaround to speed-up loading of the model when executing
arbitrary executable at load time
2026-07-20 13:31:54 +02:00
Damien Mehala
9ecd60f4d4
nvim: merge configurations table instead of overwriting (#611)
Update the local neovim configuration to override only the fields that
needs to change (such as `cmd` for LSP) instead of replacing the user's
entire configuration.

The main difference is `vim.lsp.config("zls", { ... })` which merges the
provided table into the existing configuration, while
`vim.lsp.config["zls"] = { ... }` overwrite the LSP configuration for
ZLS.

I also fixed the same issue for conform.

Signed-off-by: dmehala <damien@zml.ai>
2026-07-19 11:12:00 +02:00
Corentin Godeau
f4ffbbc017
zml/io: use correct shape when loading tensor (#665)
Small mistake happened while cleaning #586, I factorized code that
shouldn't have been factorized blindly.

This PR fixes a bug where tensor loaded using `Loader.load` were not
correctly sharded.
2026-07-17 22:27:55 +02:00
Benjamin Rabier
9c38bad5be
platforms/oneapi: update pjrt artifact with collectives fixes (#666) 2026-07-17 22:23:57 +02:00
Corentin Godeau
577d6452bf
zml/io: rework zml.io.load (#586)
The old `zml.io.load` was plagued with problems. This PR is an attempt
at improving the state of things.

One major problem of `zml.io.load` was that it was not very composable.
It was either using it to load all the model or doing everything by
hand.

Another problem was that it didn't offer a way to apply transformations
at load time to the tensors stored in a safetensors file. For example,
it happens that for MoE models, instead of the experts weights to be
tightly packed in a nice 3D tensor, the weight file contain one tensor
for each expert. This is nice until your implementation required a 3D
tensor as input. Using the old `zml.io.load` API this resulted in very
convoluted way to do concatenation at load time. With this new API, we
offer convenience so that it can be handled nicely and as fast as
possible.

The main change is the introduction of the `Loader` struct. It's a
struct that you instantiate once and reuse whenever you want to load
tensors. It relies on `Binding`, which are a way to store which tensor
are required and how they need to be combined (if it's a complex binding
involving more than one source tensor).

One advantage of this `Loader` is that it's composable. Loading a struct
using `Loader.auto(Struct)` or loading each individual field of `Struct`
has essentially the same behavior. This solved the major problem of the
old solution.

For simplicity I kept the old model loading code almost identical, as
this new solution is a drop-in replacement.

# Update

After some discussions with @gwenzek I've changed my mind. Instead of
having a declarative API, we'll provide helper to do transformation
easily. The main changes that this PR now bring are:
* Introduce `Loader` struct
* `Loader` exposes `load` which is the composable replacement of
`zml.io.load`
* `Loader` exposes `loadExecute` that allows you to execute an
executable with the buffers loaded from the sources associated with a
tensor. This allows for example to concatenate tensors.

We'll continue to work on it, this is just a preliminary work that
allows us to do transformation at load time in LLMD.
2026-07-17 18:41:18 +02:00
Guillaume Wenzek
d6489fbb84
zml/attention: implement batching in cu_fa2 backend (#641) 2026-07-17 14:10:41 +02:00
Guillaume Wenzek
787d8cafa3
zml: handle passthroug in zml.nn.rope (#662)
alternative to #660

---------

Co-authored-by: Corentin Godeau <corentin@zml.ai>
2026-07-17 12:21:19 +02:00
Benjamin Rabier
f29ed52ea8
platforms/oneapi: 4 GPUs fixes (#661)
Co-authored-by: Kevin Barre <kevin@zml.ai>
2026-07-17 11:18:53 +02:00
Guillaume Wenzek
1c65836a44
pjrt: better error message on dlopen failure (#657)
I hit an issue with the plugin refusing to load on macos.
There was no information.
After adding this I saw that the the file was apparently not existing in
the bazel sandbox,
so I could reduce the bug to a Bazel issue.
2026-07-16 17:44:50 +02:00
Guillaume Wenzek
2e05c383e7
zml: cleanup attention code (#653) 2026-07-16 10:11:08 +02:00
Corentin Godeau
2e1a07e7f1
stdx/json: fix potential invalid free (#631)
When we parsed from a std.json.Value we weren't correctly duping the
strings, potentially leading to invalid free.
2026-07-15 16:43:55 +02:00
Kevin Barre
24165146b9
platform/intel: bump oneapi=2026.1 and rules_distroless=0.8.0 (#644)
# Bump intel oneapi and rules_distroless

The Debian package set is now also exposed separately through the
rules_distroless APT extension:

Now use `rules_distroless` to install the oneAPI packages in a
distroless container image.

So the platforms/oneapi/packages.lock.json file is now generated from
the platforms/oneapi/packages.yaml file, and the packages.yaml file is
now the source of truth for the oneAPI packages.

The MODULE.bazel contain rules_distroless patch
`third_party/rules_distroless/accept-nospace-version-constraints.patch`
because the rules_distroless APT extension does not support version
constraints with spaces, e.g. `>= 2023.2.0` is not supported, but
`>=2023.2.0` is supported.

A PR has been submitted to rules_distroless to fix this issue, but until
it is merged and released, the patch is needed in zml
https://github.com/bazel-contrib/rules_distroless/pull/229.

`_ROOT_MODULE_DIRECT_DEPS` does not need anymore to list the individual
oneAPI and Ubuntu package repositories.

`_ROOT_PACKAGES` is now in `_UBUNTU_PACKAGES` because all the package
share the same `_UBUNTU_STRIP_PREFIX`

## Tests

```
bazel run @apt_oneapi//:lock
```

```
ONEAPI_DEVICE_SELECTOR=level_zero:0 \
    bazel run  \
    --config=release  \
    --@zml//platforms:cpu=false  \
    --@zml//platforms:oneapi=true   \
    //examples/llm --  \
    --model=/var/models/meta-llama/Llama-3.1-8B-Instruct  \
    --topk=4  \
    --prompt="Tell me a story about evariste galois"
```

__-> decode 26.248s · 34.9tok/s__

```
ONEAPI_DEVICE_SELECTOR=level_zero:0,1 \
    bazel run  \
    --config=release  \
    --@zml//platforms:cpu=false  \
    --@zml//platforms:oneapi=true   \
    //examples/llm --  \
    --model=/var/models/meta-llama/Llama-3.1-8B-Instruct  \
    --topk=4  \
    --prompt="Tell me a story about evariste galois"
```

__-> decode 13.017s · 56.6tok/s__
2026-07-15 16:21:56 +02:00
Corentin Godeau
16ea2e2718
zml/tensor: remove concatenate and stack limits (#656)
We can now rely on the compilation context to allocate when needed.
2026-07-15 14:27:50 +02:00
Guillaume Wenzek
dbf3834b29
zml: improve Buffer sharding related error messages (#655)
It's easy to break `Buffer.from(x_tensor.shape(), data, sharding)` by
creating `x_tensor` in the model code with a specific sharding and you
pass a different one to `from`.

I think this should be a crash (like when you mix-up allocators in
init/deinit) and it should have a dedicated error message to guide the
developers.
2026-07-15 14:17:22 +02:00
Guillaume Wenzek
ece52d46bf
zml: allow PJRT compilation to fail (#654)
We used to panic when PJRT compilation failed.

I now think it's better to return an error, since we introduced the WIP
Metal plugin that can fail on legitimate programs.
This will make it possible to run the ZML test suite on the Metal plugin
and seeing all failing tests.

```
bazel run --@zml//platforms:metal=true --@zml//platforms:cpu=false //zml:test

33/138 nn.decltest.sampleTokens...FAIL (Internal)
...
34/138 nn.decltest.sampleTokensDynamic...FAIL (Internal)
...
102/138 tensor.Tensor.decltest.convert...FAIL (Internal)
...
115/138 tensor.Tensor.decltest.argsort...FAIL (Internal)
```
2026-07-15 13:53:54 +02:00
Steeve Morin
715047e025
workspace: add LLMD issue template 2026-07-13 14:52:35 +02:00
Kevin Barre
ed2bd190e8
bin/zml-smi: reuse shared shiny logo art (#647)
Render the zml-smi text logo fallback from the shared ZML logo data,
including the shiny per-glyph colors. Add a lightweight //zml:logo
target so the TUI can depend on the logo art without pulling in //zml.

Verified with:
- bazel test //bin/zml-smi/tui:test
- bazel build //bin/zml-smi:zml-smi
2026-07-10 18:35:37 +02:00
Pierre Chapuis
6970e7923e
bin/zml-smi: fix memory & compute over-reporting for oneapi (#645)
When a DRM fd is duplicated or inherited through fork(), the same oneAPI
DRM client can appear multiple times under /proc/<pid>/fdinfo. Summing
each fdinfo entry double-counts GPU memory and engine time.

Parse drm-client-id and skip entries whose (device_idx, client_id) pair
has already been seen during the collection pass. Include the device in
the key because drm-client-id may be scoped per DRM device.

Add a focused test covering duplicate clients on the same device and
distinct clients across devices or client IDs.

---------

Co-authored-by: Kevin Barre <kevin@zml.ai>
2026-07-10 16:52:05 +02:00
Benjamin Rabier
87a211d3ea
zml/vfs/hf: Increased chunk size for HF (#646)
With only 16MiB chunks, I'm hitting the authenticated user rate limits
when loading Qwen3.6-27B sharded over two devices.
2026-07-10 13:21:30 +02:00
Ben Younes
a1b595043b
zml/tensor: fix IRFFT output shape (#642)
## Summary

Fix IRFFT shape inference so it uses the requested FFT output length and
validates that the complex input length matches the corresponding RFFT
output length.

Fixes #411.

## Why

RFFT changes the innermost real dimension `n` into `n / 2 + 1` complex
bins. IRFFT receives those complex bins plus the requested real FFT
length, so its output shape should restore that requested length instead
of deriving a smaller value from the complex input dimension.

## Test verification (RED -> GREEN)

Command used for all runs:

```bash
npx -y @bazel/bazelisk@1.25.0 test //zml:test --test_output=errors
```

RED, with the new regression test before the fix:

```text
102/139 tensor.Tensor.test.Tensor.fft IRFFT restores RFFT last dimension...thread 1701155 panic: exact division produced remainder
INFO: Build completed, 1 test FAILED, 5355 total actions
//zml:test                                                               FAILED in 24.9s
Executed 1 out of 1 test: 1 fails locally.
```

GREEN, with the fix applied:

```text
INFO: Build completed successfully, 6 total actions
//zml:test                                                               PASSED in 8.1s
Executed 1 out of 1 test: 1 test passes.
```

Revert RED, with the production fix reverted while keeping the test:

```text
102/139 tensor.Tensor.test.Tensor.fft IRFFT restores RFFT last dimension...thread 1711292 panic: fft(.{ .kind = .IRFFT, .length = { 10 } }) expects tensor last dimensions to match given lengths, got 1 and 1
INFO: Build completed, 1 test FAILED, 6 total actions
//zml:test                                                               FAILED in 22.9s
Executed 1 out of 1 test: 1 fails locally.
```

## Additional checks

```text
zig fmt --check zml/tensor.zig
fmt_status=0
```
2026-07-10 13:13:29 +02:00
Benjamin Rabier
0b4db7f108
zml/io/vfs: Retry on retryable errors (#639) 2026-07-10 11:55:38 +02:00
Adrien
dcad988918
zml/vfs: treat scheme:// paths as absolute in the router (#635)
In the VFS router, a scheme-qualified path like `hf://owner/model/file`
is meant to be absolute: it should resolve from the scheme's root. But
`lookupDir` only treated it as absolute when the base directory was
`cwd`. When such a path is opened *relative to an already-open directory* of
the same backend, the router kept the directory's path and glued the new
path after it, duplicating it. So `hf://owner/model` +
`hf://owner/model/file` became `owner/model/owner/model/file`, and the
lookup failed with `FileNotFound`.

Fix: on `lookupDir`, any `scheme://` sub-path resolves from the scheme's root
regardless of the base directory, the same way an absolute POSIX path
already does. About four lines.
2026-07-09 16:40:20 +02:00
Benjamin Rabier
19a15ca966
examples/llm: shiny ZML logo (#636)
Co-authored-by: Tristan Gautie <tristan@zml.ai>
2026-07-09 14:02:57 +02:00
Hugo Mano
5cacbe606e
platforms/intel: full hermetic sandboxing (#638)
Needed sandboxing for SDK less and distroless OCI image

---------

Co-authored-by: Kevin Barre <kevin@zml.ai>
2026-07-08 12:02:28 +02:00
Kevin Barre
8a89d73fb8
build/tools xprof: refresh xprof-nightly lock (#637)
Regenerate the xprof uv lockfile to move xprof-nightly off the removed
2.23.2a20260505 build and onto 2.23.3a20260707.

Fixes Bazel fetch failures for //tools/xprof:xprof.
2026-07-08 09:33:36 +02:00
Benjamin Rabier
6f4e8b8365
io/vfs: file handle race (#634) 2026-07-07 17:42:36 +02:00
Kevin Barre
75b820acad
zml/attention: pad block_m for oneAPI decode (#633)
Use the standard minimum/power-of-two block_m sizing for paged attention
instead of the raw GQA group size. This keeps oneAPI decode tile sizing
valid for Triton IR generation.
2026-07-06 17:12:45 +02:00
David Zbarsky
3b5530833c
workspace: replace deprecated new_git_repository (#629)
## Summary

- replace the legacy `new_git_repository` alias with `git_repository`
for `arocc`, `libvaxis`, `linenoise`, and `translate-c`

## Why

`git_repository` is the canonical Bazel repository rule.
`new_git_repository` doesn't exist at bazel HEAD
2026-07-04 21:40:33 +02:00
Kevin Barre
73cbeca7cf
platform/intel: update intel pjrt and limit the intel device to 2 (#628) 2026-07-03 17:02:23 +02:00
elogir
f9183137a1
examples/llm: fix single-chip TPU sharding crash (#627)
A 1-chip TPU mesh collapses to rank 1 (only `link_x`), so the
unconditional `link_y` fold tripped InvalidPhysicalAxis. Only fold
`link_y` when the physical mesh actually has it.
2026-07-03 14:32:41 +02:00