The LM head normalized and projected every prompt position at prefill,
and the session read back the whole token buffer just to keep the last
token.
Every model now slices the hidden state at `active_length - 1` before
the norm, so prefill computes logits for a single position and returns a
single token. Decode writes the next token over the one it read, so it
allocates no buffer per step.
lfm2's `actual_seq_len` is renamed `active_length`, the name the other
models use.
Make KV-cache strides compile-time constants so Triton can infer
alignment and simplify address calculations.
On Qwen3.5-35B-A3B, both 2D and 3D attention kernels now use 128-bit
vector loads for K/V, fetching eight bf16 values per instruction per
active thread instead of one.
Thanks @fior512 for your contribution 🙏
---------
Co-authored-by: fior512 <161968556+fior512@users.noreply.github.com>
Assign blocks for multiple experts in parallel instead of processing
experts one at a time. Each lane writes the expert ids for its own block
range.
Reduces assignment cost from 7.6 us to 1.8 us per call.
This PR is the ground work that adds a `cute` backend for paged
attention.
It's not use by default yet because it's a very naive implementation
that is far from performant.
This PR also includes two targets that are useful when working on the
kernels.
The first one is `//zml/kernels/paged_attention/cute:correctness_test`.
As its name suggest, it's a correctness test that ensures the
computation done is correct and returns valid results. It's verified
against a CPU reference implementation (that is also checked against a
hardcoded truth). It relies on `Case` representing various real-life
scenarios.
The second one is `//zml/kernels/paged_attention/cute:benchmark`. It's
an executable that can be run and will output execution stats for
various `Case`. This is the target to use when working on the
performance of paged attention cute kernels.
The `@patchelf` version 0.18.0 is more capable and can now construct
something like
```bash
patchelf \
--add-needed somelib.so \
--force-rpath \
--set-rpath '/home/kevin/sdk/lib' \
--rename-dynamic-symbols patched_tcc.symbols \
--output patched_tcc/furiosa-tcc \
somebinary
```
So now we do not need to spawn a shell and do all the cp, chmod, and the
successive patchelf command
Before
```bash
remove_rpath = ["some/directory"]
```
`--remove-rpath` is a flag that `Removes the entire embedded search
path` and not an individual directory
will contruct something like `patchelf --remove-rpath some/directory
patched.so` but `--remove-rpath` do not take arguments
Switch to `remove_rpath = True` to ensure something like `patchelf
--remove-rpath --output patched.so original.so` the `--remove-rpath` is
now used just after the patchelf to ensure correct usage flag placement.
`DataType.Value.as` converts a tagged-union `Value` to a native Zig type. The float branch of the inline switch lists which float dtypes it knows how to convert via `.toF32()`, but that list was never updated
when f4e2m1, f8e3m4, f8e4m3 (the plain variant, distinct from f8e4m3b11fnuz/fn/fnuz), and f8e8m0 were added to `DataType`. All four already implement `.toF32()` in zml/floats.zig, so calling `.as(f32)` on
a `Value` of any of these dtypes panics with "Unsupported conversion ...
-> f32" instead of returning the converted float.
The fix is a one-line addition of the four dtypes to the existing switch arm, the same pattern
as the other float dtypes already listed there.
Akin to #926, this PR introduces a restructuration of the moe kernels in
ZML.
No need to re-explaing the organizatiopn because it follows what has
been done in the mentioned PR.
Here is the detailed changes of this PR:
* Kernels reorganization. `zml/kernels/moe` is the place to put all the
moe implementation. There is one sub-folder per backend.
* Changed the expected shape and tags when calling the moe forward
function. It's now expecting a 2D tensor tagged `.{.t , .d}`.
Tests still passes without key modifications. The only small tweak that
was done is related to fp32 -> fp8 triton quantization on sm89. Due to
how triton lowers the convert, it was producing slightly incorrect
result. A small preparation was added to ensure the conversion is
identical on this platform.
**Future TODO**:
* Ideally, this conversion on sm89 should be gated so that we don't pay
the cost on other platforms. We don't have a way to detect the platform
in the triton `Builder` yet but it could probably be added
* Settle on a complete API that encodes all the scenario we could
encounter. One example of that is what is required for Deepseek V4.1
Flash, the moe gemms should be done in `mxfp8 x mxfp4` and we should
have a way to request that in a better way than `quantize_input: bool`.
CUDA 13.4.2 ships PTAX 13.4.92, which miscompiles a valid atomic CAS in
some LLM models, causing crashses. The generated code uses a different
byte offset for the load and atomic update, so the two operations access
different memory locations.
## Motivation
This is the big sharding rewrite we need to achieve complex sharding.
Currently we always resolve a mesh logical axis to a mesh physical axis
(or several of them).
But on GPU we always have 1D topology, so we can't have 2D logical
meshes.
So this PR drops the logical -> physical axis bindings, and instead
store a logical -> devices binding.
This is more flexible, and has the nice side effect of simplifying
`Buffer.from`
On stablehlo side this changes how we generate the MLIR because we now
use logical meshes, and `manualComputation` uses logical axes.
(In shardy, GSPM is not affected AFAICT because it uses device ID
directly)
## Buffer.from
This also accelerates `Buffer.from` that I rewrote to be FAST.
Version | Fixed cost | Per device
-- | -- | --
origin/master | 109.6 ns | 35.39 ns
Rewrite | 35.8 ns | 4.38 ns
Improvement | 3.1× | 8.1×
## Creating mesh
I'm adding a new syntax to create LogicalMesh, to allow the more complex
meshes we want to do, and in particular, I now allow to force a specific
number of dimensions.
You can see it in action in `mnist.zig`.
```
const dp_mp = try platform.registerMesh("DP-MP", try .init(
.{ .data, .model },
&.{ .low_bandwidth, .high_bandwidth },
&.{ 2, -1 },
));
info: topology used: Mesh(name=DP-MP, partitions=4, replicas=1)
┌ Mappings (logical coordinates from physical coordinates):
├── data (low_bandwidth) -> (bus / 2)
└── model (high_bandwidth) -> (bus % 2)
┌ PhysicalMesh(platform=cpu shardable_axes={ .bus } num_devices=4)
└── [bus] 0..4 (tree)
├── Device(id=0, coords={ 0 })
├── Device(id=1, coords={ 1 })
├── Device(id=2, coords={ 2 })
└── Device(id=3, coords={ 3 })
┌ Logical coordinates:
└── data=0
└── model=0,1 -> devices=0,1
└── data=1
└── model=0,1 -> devices=2,3
```
The previous syntax is still available, and ZML will align logical axes
to physical axes like we do before this PR.
This part as unit tests, and I checked it work as I expect on CUDA but I
would need to double check the logics still works on TPU/Neuron, because
there is a few platform specific branches in that code.
## Test plan
As part of the benchmark Codex generated a test suite with hard-coded
offsets, allowing to check that Buffer.from didn't change behavior
through the commits, checking we correctly compute each individual
shard.
TODO check that LLMD perf isn't degraded.
here I'm promoting a pattern I would like to extend to more of ZML:
function that takes `anytype` argument should do as little work as
possible and called a fully typed function.
I kept `manualComputation` signature to not break existing code,
I moved the code to `zml/ops/manual_computation.zig` to better
encapsulate the required helper functions.
(and `zml/ops.zig` is pretty big atm)
Similarly `Sharding.Partitioning.parse` now calls `const logical =
comptime Logical.parse(partitioning);` then `logical.applyUseAll(tags,
mesh, default_value);` which is fully typed.
I'm also documenting `manualComputation` better
Update XLA plugins to release 202610151602.227.1.e215eb02db73
for CPU, CUDA, ROCm, Metal, and Furiosa.
Updated CUDA to 13.4.3 and ROCm with TheRock 10.1.
Now we use CUDA compatibility libraries and NCCL 2.32.3 through NVIDIA's
APT repositories. Starting with CUDA 13.4's redistribution manifest
omits the generic cuda_compat package
(https://docs.nvidia.com/deploy/cuda-compatibility/why-cuda-compatibility.html).
So we resolve it through NVIDIA's APT repositories instead.
These repositories expose flat package indexes at separate x86_64
and SBSA URLs, so we extend rules_distroless to support flat sources.
For NCCL we leverage the same APT source workflow, replacing
separate cryptic wheel URLs.
Staging blocks are uninitialized `.host_pinned` buffers, and tensors are
sent with the direct writer. The plugin backs `.host_pinned` with SDK
pinned (dma-buf) memory, which the SDK recognizes by address and
transfers without pinning pages on every request.
Pinned memory comes from `/dev/dma_heap/system`, so the loading user
needs access to it.
Fast-path inside the forward of
`GatedDeltaNet` in order to reduce syncs in decode path.
Master vs this commit:
- tok/s : 167 to 204
- cuStreamSynchronize call/tok : 37.24 to 1.24
- GPU idle/tok : 3504 us to 2240 us
We align TPU and NKI attention meshes expectation on the CuTe GPU
attention, which simply resolves the mesh from the logical axis.
This avoids expecting a hard-coded `model` mesh, which breaks in some
TPU workflows where the mesh is called `tp_mesh`.
This PR introduces a restructuration of the kernels in ZML. This is done
to simplify and speedup bringup on new platforms.
From now on each "kernel" will have its own directory under
`zml/kernels`. "Kernel" is in quote because we need to think about it
being more of an "operation in a graph" than an actual kernel. For
example, `moe` will eventually be part of this `kernels` directory
despite being an operation potentially launching multiple actual kernels
on some hardware.
Also it's probable that in the future, this `kernels` directory will be
put outside of zml or, at least, properly separated so that we don't
have circular imports in the codebase. Currently, we have
`@import("../../../zml.zig")` in the kernel code and this is a recipe
for disaster.
On top of the reorganization, it also adds a level of indirection for
attention. Where previously `attention` was a generic term, we now have
`attention` and `flash_attention` because `flash_attention` is an
implementation of `attention`.
Here is a detailed summary of the changes introduced by this PR:
* Kernels reorganization. `flash_attention` and `paged_attention` are
now separated. Each folder has a subfolder for each backend we support.
`flash_attention` is also now a `BackendType` of `zml.attention` with
its own implementations.
* Removed `cuda_fa2` and `cuda_fa3` backend because they were obsolete.
They have been replaced by a `cute` backend that has been generated
automatically following the operation documentation, showing that this
approach of scoping kernels per DSL actually scales.
* Removed `attnd` support for now
Fix a memory leak in Qwen 3.5 Dense and MoE examples:
`layer_index` was never freed.
Before (74562d5) / After
Dense 0.8B VRAM : 11079 MB vs 5461 MB
Dense 0.8B tok/s : 169 vs 198
tok/s increases because it reduces `cuMemCreate` calls from 26 to 2
per tokens
This PR adds tests to the MoE backend. The comparison is done using a
StableHLO backend that has been tested against an oracle with known
value.
It's adding quite some time to the test so I don't know if we want to
keep that many tests. However, this is the only way we have to ensure
compliance across all backends and configuration.
Each backend is tested with the whole matrix it supports.
A nice side-effect of this PR is that we have a working implementation
of MoE that will work on all platforms. It's going to be slower but
allows for a quick platform bringup.
`Buffer` placed in `.host_pinned` memory can be read by executables,
but the host can't address it. `Buffer.HostAccessible` gives the host
typed views over its allocations, to write inputs and read outputs in
place between executions, without staging copies. Like `Buffer`, it is
a value: executions read its `buffer` field or donate it through
`&buffer`, and `await` then picks up the handles they replaced.
- `init` takes a `Buffer.Donation`. On CPU, CUDA and ROCm, the devices
holding a shard of an `.undonatable` buffer share one allocation: the
first allocates it and the others read it through
`createViewOfDeviceBuffer`, which now takes the memory the view
belongs to; an external reference keeps the allocation alive.
Elsewhere each device owns an allocation, and writes reach all of
them. Each device owns its allocation of a `.donatable` buffer, so
that no device reads another's output.
- `View(T)` addresses a row-major range of the array: `slice`, `get`,
`set`, `fill`, `fillIota`, `copyFrom`, `copyTo`, `copyFromView`,
`indexOfScalar`, `indexOfAny`, and `blocks`/`reverseBlocks`, which
visit runs contiguous both in memory and in global order. Reads use
the first replica of a region, and writes reach every replica.
`constSlice` and `edit` expose a range as one slice: in place when
every shard holds the whole array densely, through a scratch slice
otherwise.
Addressing follows PJRT's layouts: row-major, strided, and tiled as
XLA's LayoutUtil::LinearIndexForNestedTiling expands them, including
TPU's `T(8,128)(2,1)` packing of bf16 and 8-bit arrays. The test
reference is a direct port of that function, checked against both
figures of https://openxla.org/xla/tiled_layout. Buffers whose shards
all hold the whole array densely, the case on every platform but TPU,
index elements directly.
The buffer is built for llmd's decode loop, which touches it thousands
of times per step:
- Reading a host pointer from PJRT takes and releases a hold on the
buffer and allocates an external reference, about 0.9 us. Pointers
are read at creation, and by `await` only when execution replaced the
handles; `view` panics before that `await` instead of writing to a
released allocation.
- The layout metadata and allocations take several KB. They are
prepared once in a heap-allocated mapping, which views (24 B) and
blocks borrow rather than copy.
On Qwen3.8-27B in llmd (TP=2 on GB300, batch 256, 8k context), this
takes host time between graph completion and the next Execute from
2.07 ms, when views copied the layout and donated buffers read their
pointers in every view, to 0.27-0.30 ms per step (bpftrace probes).
`initForTests` builds a buffer over host memory, for the tiled layouts
and shardings that no available platform produces. `Buffer.from` and
`Buffer.uninitialized` share the new `emptyShell` with
`HostAccessible.init` to compute the PJRT shape, placement and layout.
Due to recent changes in XLA (introduced in
3cf18a01c7), the way child are
counted to decide if a conditional is expressed as a graph conditional
or kept outside has been changed.
With the new corrected way, the body of the conditional used in the
paged attention StableHLO backend was recorded as a child graph. The
problem is that when we are sharded, there are collectives inside this
child graph and XLA adds event wait/record around the actual collective
kernel. This is explicitly forbidden because
https://docs.nvidia.com/cuda/cuda-programming-guide/04-special-topics/cuda-graphs.html#conditional-node-body-graph-requirements
states:
> The graph can only contain kernel nodes, empty nodes, memcpy nodes,
memset nodes, child graph nodes, and conditional nodes.
The temporary solution is to remove `CONDITIONAL` from the
`xla_gpu_enable_command_buffer` XLA flag. In practice this reverts to
the old behavior we had when the counting was done differently.
Long term, I don't have a solution yet and. There is an environment
variable that solves it
[NCCL_GRAPH_MIXING_SUPPORT](https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/env.html#nccl-graph-mixing-support)
but I don't know if it's safe to disable it.
Maybe we could also check if the events introduced by NCCL are actually
required.
Adding `extra_device_memory_bytes` to the compilation options to have
XLA compute the accurate amount of memory left for the temporaries with
our engram hack.