0
Fork 0
mirror of https://github.com/zml/zml.git synced 2026-11-16 19:11:09 +00:00
Commit graph

1,078 commits

Author SHA1 Message Date
Raphael Nahoum
24f8c2b986
examples/llm: sample only the last prompt token (#965)
The LM head normalized and projected every prompt position at prefill,
and the session read back the whole token buffer just to keep the last
token.

Every model now slices the hidden state at `active_length - 1` before
the norm, so prefill computes logits for a single position and returns a
single token. Decode writes the next token over the one it read, so it
allocates no buffer per step.

lfm2's `actual_seq_len` is renamed `active_length`, the name the other
models use.
2026-10-21 11:47:42 +02:00
Damien Mehala
21f5b1a55f
zml/kernels: vectorize KV load for triton paged attention (#971)
Make KV-cache strides compile-time constants so Triton can infer
alignment and simplify address calculations.

On Qwen3.5-35B-A3B, both 2D and 3D attention kernels now use 128-bit
vector loads for K/V, fetching eight bf16 values per instruction per
active thread instead of one.

Thanks @fior512 for your contribution 🙏

---------

Co-authored-by: fior512 <161968556+fior512@users.noreply.github.com>
2026-10-21 11:08:35 +02:00
Damien Mehala
0db5229cd4
zml/kernels: parallelize MoE expert block assignment (#961)
Assign blocks for multiple experts in parallel instead of processing
experts one at a time. Each lane writes the expert ids for its own block
range.

Reduces assignment cost from 7.6 us to 1.8 us per call.
2026-10-21 10:58:19 +02:00
Corentin Godeau
886a6b8a97
zml/kernels/paged_attention: add cute backend (#963)
This PR is the ground work that adds a `cute` backend for paged
attention.

It's not use by default yet because it's a very naive implementation
that is far from performant.

This PR also includes two targets that are useful when working on the
kernels.

The first one is `//zml/kernels/paged_attention/cute:correctness_test`.
As its name suggest, it's a correctness test that ensures the
computation done is correct and returns valid results. It's verified
against a CPU reference implementation (that is also checked against a
hardcoded truth). It relies on `Case` representing various real-life
scenarios.

The second one is `//zml/kernels/paged_attention/cute:benchmark`. It's
an executable that can be run and will output execution stats for
various `Case`. This is the target to use when working on the
performance of paged attention cute kernels.
2026-10-20 16:41:55 +02:00
Hugo Mano
6cd00a0d00
ci: add supported platforms with bumped and audited deps (#967) 2026-10-20 14:06:49 +02:00
Hugo Mano
b238e5e6e7
ci: update runs-on images and runners manifest (#968)
Needed configuration for an upcoming CI change. Also remove not used
anymore bzlcache over S3 since we use BuildBuddy.
2026-10-20 12:51:06 +02:00
Kevin Barre
e9685a3662
zml/bazel: simplify patchelf execution and fix RPATH removal (#916)
The `@patchelf` version 0.18.0 is more capable and can now construct
something like

```bash
patchelf \
  --add-needed somelib.so \
  --force-rpath \
  --set-rpath '/home/kevin/sdk/lib' \
  --rename-dynamic-symbols patched_tcc.symbols \
  --output patched_tcc/furiosa-tcc \
  somebinary
```

So now we do not need to spawn a shell and do all the cp, chmod, and the
successive patchelf command

Before 

```bash
remove_rpath = ["some/directory"]
```

`--remove-rpath` is a flag that `Removes the entire embedded search
path` and not an individual directory

will contruct something like `patchelf --remove-rpath some/directory
patched.so` but `--remove-rpath` do not take arguments

Switch to `remove_rpath = True` to ensure something like `patchelf
--remove-rpath --output patched.so original.so` the `--remove-rpath` is
now used just after the patchelf to ensure correct usage flag placement.
2026-10-20 11:49:41 +02:00
Damien Mehala
dd82a8a10b
third_party/zls: bump to 0.17.0 (#966) 2026-10-20 11:36:07 +02:00
Hugo Mano
1179f605f7
ci: remote build (#924)
Local runner growing disk usage will be handled in another PR.
2026-10-20 09:31:03 +02:00
Hugo Mano
a5dfa567d4
bazel: update to 9.3.0 (#964) 2026-10-20 09:30:12 +02:00
Arthur
231154ef14
zml/dtype: fix Value.as panicking for f4e2m1/f8e3m4/f8e4m3/f8e8m0 (#878)
`DataType.Value.as` converts a tagged-union `Value` to a native Zig type. The float branch of the inline switch lists which float dtypes it knows how to convert via `.toF32()`, but that list was never updated
when f4e2m1, f8e3m4, f8e4m3 (the plain variant, distinct from f8e4m3b11fnuz/fn/fnuz), and f8e8m0 were added to `DataType`. All four already implement `.toF32()` in zml/floats.zig, so calling `.as(f32)` on
a `Value` of any of these dtypes panics with "Unsupported conversion ...
-> f32" instead of returning the converted float.

The fix is a one-line addition of the four dtypes to the existing switch arm, the same pattern
as the other float dtypes already listed there.
2026-10-19 18:45:42 +02:00
Louis Lechevalier
94ead7094e
zml/platforms: bump ZML plugins and enable all reduce collective in cmd buffer (#959)
This enable Lamport all reduce collective if compute capabilities matched on CUDA and ROCm
2026-10-19 15:18:23 +02:00
Corentin Godeau
9c7448da3c
zml: re-structure moe kernels (#941)
Akin to #926, this PR introduces a restructuration of the moe kernels in
ZML.

No need to re-explaing the organizatiopn because it follows what has
been done in the mentioned PR.

Here is the detailed changes of this PR:
* Kernels reorganization. `zml/kernels/moe` is the place to put all the
moe implementation. There is one sub-folder per backend.
* Changed the expected shape and tags when calling the moe forward
function. It's now expecting a 2D tensor tagged `.{.t , .d}`.

Tests still passes without key modifications. The only small tweak that
was done is related to fp32 -> fp8 triton quantization on sm89. Due to
how triton lowers the convert, it was producing slightly incorrect
result. A small preparation was added to ensure the conversion is
identical on this platform.

**Future TODO**:
* Ideally, this conversion on sm89 should be gated so that we don't pay
the cost on other platforms. We don't have a way to detect the platform
in the triton `Builder` yet but it could probably be added
* Settle on a complete API that encodes all the scenario we could
encounter. One example of that is what is required for Deepseek V4.1
Flash, the moe gemms should be done in `mxfp8 x mxfp4` and we should
have a way to request that in a better way than `quantize_input: bool`.
2026-10-19 15:15:11 +02:00
Hugo Mano
c651a9de45
rbe: use GCR image registry instead of Docker Hub (#962)
GCR mirror is more reliable than fetching through the Docker
Hub
2026-10-19 12:45:42 +02:00
Damien Mehala
2a57062b29
zml/nn: softplus (#954) 2026-10-19 10:22:38 +02:00
Damien Mehala
2d516eeb78
zml/platforms: rollback to CUDA 13.3.0 (#955)
CUDA 13.4.2 ships PTAX 13.4.92, which miscompiles a valid atomic CAS in
some LLM models, causing crashses. The generated code uses a different
byte offset for the load and atomic update, so the two operations access
different memory locations.
2026-10-16 18:46:10 +02:00
Guillaume Wenzek
a7a4fcd3f8
zml/sharding: use logical meshes (#943)
## Motivation

This is the big sharding rewrite we need to achieve complex sharding.

Currently we always resolve a mesh logical axis to a mesh physical axis
(or several of them).
But on GPU we always have 1D topology, so we can't have 2D logical
meshes.
So this PR drops the logical -> physical axis bindings, and instead
store a logical -> devices binding.
This is more flexible, and has the nice side effect of simplifying
`Buffer.from`

On stablehlo side this changes how we generate the MLIR because we now
use logical meshes, and `manualComputation` uses logical axes.
(In shardy, GSPM is not affected AFAICT because it uses device ID
directly)

## Buffer.from

This also accelerates `Buffer.from` that I rewrote to be FAST.
Version | Fixed cost | Per device
-- | -- | --
origin/master | 109.6 ns | 35.39 ns
Rewrite | 35.8 ns | 4.38 ns
Improvement | 3.1× | 8.1×

## Creating mesh

I'm adding a new syntax to create LogicalMesh, to allow the more complex
meshes we want to do, and in particular, I now allow to force a specific
number of dimensions.
You can see it in action in `mnist.zig`. 

```
const dp_mp = try platform.registerMesh("DP-MP", try .init(
        .{ .data, .model },
        &.{ .low_bandwidth, .high_bandwidth },
        &.{ 2, -1 },
    ));
 
info: topology used: Mesh(name=DP-MP, partitions=4, replicas=1)
┌ Mappings (logical coordinates from physical coordinates):
├── data (low_bandwidth) -> (bus / 2)
└── model (high_bandwidth) -> (bus % 2)

┌ PhysicalMesh(platform=cpu shardable_axes={ .bus } num_devices=4)
└── [bus] 0..4 (tree)
    ├── Device(id=0, coords={ 0 })
    ├── Device(id=1, coords={ 1 })
    ├── Device(id=2, coords={ 2 })
    └── Device(id=3, coords={ 3 })

┌ Logical coordinates:
└── data=0
    └── model=0,1 -> devices=0,1
└── data=1
    └── model=0,1 -> devices=2,3
```

The previous syntax is still available, and ZML will align logical axes
to physical axes like we do before this PR.
This part as unit tests, and I checked it work as I expect on CUDA but I
would need to double check the logics still works on TPU/Neuron, because
there is a few platform specific branches in that code.

## Test plan

As part of the benchmark Codex generated a test suite with hard-coded
offsets, allowing to check that Buffer.from didn't change behavior
through the commits, checking we correctly compute each individual
shard.

TODO check that LLMD perf isn't degraded.
2026-10-16 15:27:30 +02:00
Damien Mehala
8329631da1
platforms/cuda: fix redist json checksum (#949) 2026-10-16 15:22:13 +02:00
Benjamin Rabier
c36b3f8f1e
platforms: Bump plugins for reduced Thunk Prepare/Initialize overhead (#946)
For DeepSeek 4.1-Flash on 4 GB300
```
|                                    |       current |     new |
|------------------------------------|--------------:|--------:|
| Start of XlaModule → cuGraphLaunch | 1,008 µs      | 132 µs  |
| ...of which on CPU                 | 896 µs        | 70 µs   |
| Prepare + Initialize               | ~790 µs       | ~2 µs   |
```
2026-10-16 13:43:03 +02:00
Guillaume Wenzek
7bb457ebeb
zml: refactor manualComputation (#925)
here I'm promoting a pattern I would like to extend to more of ZML:

function that takes `anytype` argument should do as little work as
possible and called a fully typed function.

I kept `manualComputation` signature to not break existing code,
I moved the code to `zml/ops/manual_computation.zig` to better
encapsulate the required helper functions.
(and `zml/ops.zig` is pretty big atm)

Similarly `Sharding.Partitioning.parse` now calls `const logical =
comptime Logical.parse(partitioning);` then `logical.applyUseAll(tags,
mesh, default_value);` which is fully typed.

I'm also documenting `manualComputation` better
2026-10-16 12:30:05 +02:00
Hugo Mano
bc9d6c2c8d
workspace: bump ZML plugins and CUDA/ROCm runtimes (#942)
Update XLA plugins to release 202610151602.227.1.e215eb02db73
for CPU, CUDA, ROCm, Metal, and Furiosa.

Updated CUDA to 13.4.3 and ROCm with TheRock 10.1.

Now we use CUDA compatibility libraries and NCCL 2.32.3 through NVIDIA's
APT repositories. Starting with CUDA 13.4's redistribution manifest
omits the generic cuda_compat package
(https://docs.nvidia.com/deploy/cuda-compatibility/why-cuda-compatibility.html).
So we resolve it through NVIDIA's APT repositories instead.
These repositories expose flat package indexes at separate x86_64
and SBSA URLs, so we extend rules_distroless to support flat sources.

For NCCL we leverage the same APT source workflow, replacing
separate cryptic wheel URLs.
2026-10-16 10:31:34 +02:00
Raphael Nahoum
e385d9c272
workspace: bump plugins to fix Metal GDN overflow (#940)
Metal GDN used to overflow on state indexes since the rework.
2026-10-15 14:45:20 +02:00
Raphael Nahoum
cddc9cf607
zml/io: load Furiosa weights through pinned host memory (#928)
Staging blocks are uninitialized `.host_pinned` buffers, and tensors are
sent with the direct writer. The plugin backs `.host_pinned` with SDK
pinned (dma-buf) memory, which the SDK recognizes by address and
transfers without pinning pages on every request.

Pinned memory comes from `/dev/dma_heap/system`, so the loading user
needs access to it.
2026-10-14 19:32:13 +02:00
Hugo Mano
016376f2f4
workspace: remove flashattn (#939) 2026-10-14 18:29:02 +02:00
Benjamin Rabier
fe25f16b46
zml/buffer: replace emptyShell by PjrtBufferMetadata.init (#935) 2026-10-14 18:09:21 +02:00
Hugo Mano
82e800dc5c
workspace: bump ZML plugins (#937)
This introduce the new tiling path support.
2026-10-14 15:18:04 +02:00
fior512
3b0f556a74
zml/nn: reducing sync in GDN decode (#932)
Fast-path inside the forward of
`GatedDeltaNet` in order to reduce syncs in decode path.

Master vs this commit:
- tok/s : 167 to 204
- cuStreamSynchronize call/tok : 37.24 to 1.24
- GPU idle/tok : 3504 us to 2240 us
2026-10-14 13:31:42 +02:00
Tristan Gautié
f83f5ece0b
zml/kernels/attention: Resolve TPU and NKI attention meshes by logical axis (#933)
We align TPU and NKI attention meshes expectation on the CuTe GPU
attention, which simply resolves the mesh from the logical axis.

This avoids expecting a hard-coded `model` mesh, which breaks in some
TPU workflows where the mesh is called `tp_mesh`.
2026-10-14 13:12:54 +02:00
Kevin Barre
a5c40b7614
zml/platform/furiosa: rename environment variables from to FURIOSA_* (#921) 2026-10-13 19:54:27 +02:00
Guillaume Wenzek
00a0af4a55
zig 0.17 (second attempt) (#923)
Bump for zig 0.17

Deps:
- uucode bumped to
1fb73433bb
- zigimg bumped to
734fa919fb
- libvaxis bumped to
14b7684272
- zio bumped to:
417757935e
- rules_zig upgraded to [version
0.17](https://github.com/hermeticbuild/rules_zig/releases/tag/v0.17.0)
- zls is using a commit from master, but they don't have yet a tagged
release so Codex did a bit of packaging. I confirmed it correctly
autocompletes on my machine
2026-10-13 17:11:38 +02:00
Corentin Godeau
8e95da3f2d
zml/kernels/paged_attention: add kernel documentation (#930) 2026-10-13 13:34:14 +02:00
Corentin Godeau
a0c358db12
zml/kernels/paged_attention: removed latent kvcache (#929)
It's not used anymore
2026-10-13 13:34:13 +02:00
Corentin Godeau
bd0886c3e5
zml: re-structure attention kernels (#926)
This PR introduces a restructuration of the kernels in ZML. This is done
to simplify and speedup bringup on new platforms.

From now on each "kernel" will have its own directory under
`zml/kernels`. "Kernel" is in quote because we need to think about it
being more of an "operation in a graph" than an actual kernel. For
example, `moe` will eventually be part of this `kernels` directory
despite being an operation potentially launching multiple actual kernels
on some hardware.

Also it's probable that in the future, this `kernels` directory will be
put outside of zml or, at least, properly separated so that we don't
have circular imports in the codebase. Currently, we have
`@import("../../../zml.zig")` in the kernel code and this is a recipe
for disaster.

On top of the reorganization, it also adds a level of indirection for
attention. Where previously `attention` was a generic term, we now have
`attention` and `flash_attention` because `flash_attention` is an
implementation of `attention`.

Here is a detailed summary of the changes introduced by this PR:
* Kernels reorganization. `flash_attention` and `paged_attention` are
now separated. Each folder has a subfolder for each backend we support.
`flash_attention` is also now a `BackendType` of `zml.attention` with
its own implementations.
* Removed `cuda_fa2` and `cuda_fa3` backend because they were obsolete.
They have been replaced by a `cute` backend that has been generated
automatically following the operation documentation, showing that this
approach of scoping kernels per DSL actually scales.
* Removed `attnd` support for now
2026-10-12 20:31:17 +02:00
Corentin Godeau
3a201a67fa
workspace: fix typo in nvim config (#927)
`mod` was not defined resulting in bazel file formatting being broken.
2026-10-12 17:55:27 +02:00
fior512
ecea83bda7
examples/llm: Qwen 3.5 Dense and MoE memory leak (#919)
Fix a memory leak in Qwen 3.5 Dense and MoE examples: 
`layer_index` was never freed.

Before (74562d5) / After
Dense 0.8B VRAM : 11079 MB vs 5461 MB
Dense 0.8B tok/s : 169 vs 198

tok/s increases because it reduces `cuMemCreate` calls from 26 to 2
per tokens
2026-10-12 11:08:52 +02:00
Hugo Mano
3500e88cf8
zml/moe: fit triton kernel launch config with LDS limit in mind (#920)
Temporary until we refactor zml.kernels
2026-10-09 17:49:32 +02:00
Corentin Godeau
c0992073fd
workspace: bump other plugins (#922) 2026-10-09 17:36:28 +02:00
Louis Lechevalier
8173a5b7f6
platforms/cuda: Bump zml CUDA plugin (#918) 2026-10-09 17:19:04 +02:00
Kevin Barre
74562d518d
zml/platform: Add furiosa platform (#871)
Co-authored-by: Raphael Nahoum <raphael@zml.ai>
2026-10-09 12:21:16 +02:00
Corentin Godeau
f82a474971
zml/moe: add backend compliance tests (#886)
This PR adds tests to the MoE backend. The comparison is done using a
StableHLO backend that has been tested against an oracle with known
value.

It's adding quite some time to the test so I don't know if we want to
keep that many tests. However, this is the only way we have to ensure
compliance across all backends and configuration.

Each backend is tested with the whole matrix it supports.

A nice side-effect of this PR is that we have a working implementation
of MoE that will work on all platforms. It's going to be slower but
allows for a quick platform bringup.
2026-10-08 16:40:59 +02:00
Benjamin Rabier
3972d4a011
zml/buffer: add Buffer.HostAccessible (#899)
`Buffer` placed in `.host_pinned` memory can be read by executables,
but the host can't address it. `Buffer.HostAccessible` gives the host
typed views over its allocations, to write inputs and read outputs in
place between executions, without staging copies. Like `Buffer`, it is
a value: executions read its `buffer` field or donate it through
`&buffer`, and `await` then picks up the handles they replaced.

- `init` takes a `Buffer.Donation`. On CPU, CUDA and ROCm, the devices
  holding a shard of an `.undonatable` buffer share one allocation: the
  first allocates it and the others read it through
  `createViewOfDeviceBuffer`, which now takes the memory the view
  belongs to; an external reference keeps the allocation alive.
  Elsewhere each device owns an allocation, and writes reach all of
  them. Each device owns its allocation of a `.donatable` buffer, so
  that no device reads another's output.
- `View(T)` addresses a row-major range of the array: `slice`, `get`,
  `set`, `fill`, `fillIota`, `copyFrom`, `copyTo`, `copyFromView`,
  `indexOfScalar`, `indexOfAny`, and `blocks`/`reverseBlocks`, which
  visit runs contiguous both in memory and in global order. Reads use
  the first replica of a region, and writes reach every replica.
  `constSlice` and `edit` expose a range as one slice: in place when
  every shard holds the whole array densely, through a scratch slice
  otherwise.

Addressing follows PJRT's layouts: row-major, strided, and tiled as
XLA's LayoutUtil::LinearIndexForNestedTiling expands them, including
TPU's `T(8,128)(2,1)` packing of bf16 and 8-bit arrays. The test
reference is a direct port of that function, checked against both
figures of https://openxla.org/xla/tiled_layout. Buffers whose shards
all hold the whole array densely, the case on every platform but TPU,
index elements directly.

The buffer is built for llmd's decode loop, which touches it thousands
of times per step:
- Reading a host pointer from PJRT takes and releases a hold on the
  buffer and allocates an external reference, about 0.9 us. Pointers
  are read at creation, and by `await` only when execution replaced the
  handles; `view` panics before that `await` instead of writing to a
  released allocation.
- The layout metadata and allocations take several KB. They are
  prepared once in a heap-allocated mapping, which views (24 B) and
  blocks borrow rather than copy.
On Qwen3.8-27B in llmd (TP=2 on GB300, batch 256, 8k context), this
takes host time between graph completion and the next Execute from
2.07 ms, when views copied the layout and donated buffers read their
pointers in every view, to 0.27-0.30 ms per step (bpftrace probes).

`initForTests` builds a buffer over host memory, for the tiled layouts
and shardings that no available platform produces. `Buffer.from` and
`Buffer.uninitialized` share the new `emptyShell` with
`HostAccessible.init` to compute the PJRT shape, placement and layout.
2026-10-08 14:52:10 +02:00
Tristan Gautié
734ec46230
workspace: bump ZML plugins to 202610081055.168.1.56e99704ada4 (#915) 2026-10-08 14:22:40 +02:00
Benjamin Rabier
728eedfbb3
zml/Compiler: Add COLLECTIVES_KERNEL to command buffers (#912)
We don't have a single command buffer in Deepseek since
`xla_gpu_experimental_use_collective_kernels=COLLECTIVE_KERNEL_ALL_REDUCE`
was added
2026-10-08 13:40:37 +02:00
Corentin Godeau
08391af750
zml/Compiler: remove conditionals from command buffer (#913)
Due to recent changes in XLA (introduced in
3cf18a01c7), the way child are
counted to decide if a conditional is expressed as a graph conditional
or kept outside has been changed.

With the new corrected way, the body of the conditional used in the
paged attention StableHLO backend was recorded as a child graph. The
problem is that when we are sharded, there are collectives inside this
child graph and XLA adds event wait/record around the actual collective
kernel. This is explicitly forbidden because
https://docs.nvidia.com/cuda/cuda-programming-guide/04-special-topics/cuda-graphs.html#conditional-node-body-graph-requirements
states:

> The graph can only contain kernel nodes, empty nodes, memcpy nodes,
memset nodes, child graph nodes, and conditional nodes.

The temporary solution is to remove `CONDITIONAL` from the
`xla_gpu_enable_command_buffer` XLA flag. In practice this reverts to
the old behavior we had when the counting was done differently.

Long term, I don't have a solution yet and. There is an environment
variable that solves it
[NCCL_GRAPH_MIXING_SUPPORT](https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/env.html#nccl-graph-mixing-support)
but I don't know if it's safe to disable it.

Maybe we could also check if the events introduced by NCCL are actually
required.
2026-10-08 12:33:34 +02:00
Raphael Nahoum
b774b7ad16
mlir/dialects: bind missing attributes and types through the C API (#905)
Some dialects (i.e. `gpu`) were missing the C bindings, this is now
done.
2026-10-08 10:39:26 +02:00
Guillaume Wenzek
edd3e0b4e1
zml/moe: avoid unecessary bitcast (#910)
let the partitioning information propagate down to manual computation
2026-10-07 22:13:30 +02:00
Olivier Boulant
daab01572f
platforms: switch LLVM libunwind packages from 23 to 22 (#907)
`apt.llvm.org` removed some links, breaking downloads from our lockfiles
and resulting in 404.

Switch CPU/CUDA to `llvm-toolchain-bookworm-22` and ROCm to
`llvm-toolchain-jammy-22`, and regenerate the lockfiles.

All test passed on:
- macOS
- 9985wx-5090x4
- gb300-2* 
- mi300

* See reverted commit
[f7eb1cb](https://github.com/zml/zml/pull/907/commits/f7eb1cbe3b715da465bd4e9cba8a2b071499b045)
@hugomano

---

It is a follow up from https://github.com/zml/zml/pull/875
2026-10-07 19:13:33 +02:00
Hugo Mano
b148f0b919
workspace: bump XLA + LLVM + IRs (#903) 2026-10-07 16:33:44 +02:00
Guillaume Wenzek
303ff3e6a4
Revert "zig 0.17 (#693)" (#909)
This reverts commit 5a18757368.

Our internal repos has a few zig patches, I need to also upgrade them
before we ugprade ZML
2026-10-07 11:34:51 +02:00
Benjamin Rabier
6c279b469e
zml/Compiler: add extra_device_memory_bytes (#843)
Adding `extra_device_memory_bytes` to the compilation options to have
XLA compute the accurate amount of memory left for the temporaries with
our engram hack.
2026-10-07 11:18:10 +02:00