IREE writes a token's text whole or not at all, so `feedOne` asserted
whenever the caller's buffer was smaller than the token, e.g. Laguna's
415-byte whitespace tokens in a 128-byte buffer. The decoder now owns
its output buffer and grows it the way `decode` already did, so
`feedOne` and `finalize` no longer take a buffer.
Move manual regions out of flash attention kernels and use
`manualComputation2` for every shared attention backend.
Also fully replicate the embedding executable outputs in Qwen and Llama.
Partial graph constraints leave their batch and sequence dimensions
open, which is invalid at executable boundaries.
Fix two further blockers exposed by the Qwen run that we forgot to move
recently: softplus comparisons must use the input dtype and ROCm DNN
runfiles must include the MIOpen kernel archives.
The LM head normalized and projected every prompt position at prefill,
and the session read back the whole token buffer just to keep the last
token.
Every model now slices the hidden state at `active_length - 1` before
the norm, so prefill computes logits for a single position and returns a
single token. Decode writes the next token over the one it read, so it
allocates no buffer per step.
lfm2's `actual_seq_len` is renamed `active_length`, the name the other
models use.
This PR introduces a restructuration of the kernels in ZML. This is done
to simplify and speedup bringup on new platforms.
From now on each "kernel" will have its own directory under
`zml/kernels`. "Kernel" is in quote because we need to think about it
being more of an "operation in a graph" than an actual kernel. For
example, `moe` will eventually be part of this `kernels` directory
despite being an operation potentially launching multiple actual kernels
on some hardware.
Also it's probable that in the future, this `kernels` directory will be
put outside of zml or, at least, properly separated so that we don't
have circular imports in the codebase. Currently, we have
`@import("../../../zml.zig")` in the kernel code and this is a recipe
for disaster.
On top of the reorganization, it also adds a level of indirection for
attention. Where previously `attention` was a generic term, we now have
`attention` and `flash_attention` because `flash_attention` is an
implementation of `attention`.
Here is a detailed summary of the changes introduced by this PR:
* Kernels reorganization. `flash_attention` and `paged_attention` are
now separated. Each folder has a subfolder for each backend we support.
`flash_attention` is also now a `BackendType` of `zml.attention` with
its own implementations.
* Removed `cuda_fa2` and `cuda_fa3` backend because they were obsolete.
They have been replaced by a `cute` backend that has been generated
automatically following the operation documentation, showing that this
approach of scoping kernels per DSL actually scales.
* Removed `attnd` support for now
In Buffer code we already have explicit sharding, but not for tensor.
I'm adding a `zml.Sharding` to `.withPartitioning` to make it explicit
which mesh we are working with.
To simplify the migration, `.withPartitioning` in Tensor code can use
`.model` syntax to lookup a sharding by name.
This replace the previous logic of `selectSharding(shape)` which would
look up a sharding that matches the axes of the given partitioning which
could yield surprising behavior.
This also modifies how sharding are stored in zml.Shape.
Before it was storing a list of axis names + a tag (2 * 8 * 64bits), now
I'm storing a zml.Sharding (64 bits) and a compact PartitionArray (32
bits) that references the logical axis of the sharding by offset instead
of by name.
There is an impact on the `TensorRegistry` because it also needs to be
aware of the possible shardings, instead of delegating the matching
partition spec -> sharding to loading. This means error will be detected
sooner.
`store.createTensor` accepts `@enumLiteral()` to lookup the relevant
sharding.
This also simplifies `zml.Buffer` because the full sharding is now
stored in the shape, so it doesn't need to be passed independently. It
also means that it will propagate from the tensor who are graph inputs,
to the buffers that are eg inputs.
`Placement` logic also got a bit simpler.
Removing callOpts & runOpts to simplify the API and changing `io` to be
required for API clarity. I'm replacing `zml.testing.autoCall` with
`Exe.eval` which solves most needs in a simpler way IMHO
A binding created through `maybeCreateBinding` is computed from its
sources by an executable and must never be streamed from the checkpoint.
`Loader.defaultCallback` inferred that from `sources.len != 1`, which a
single-source binding fails resulting to the raw bytes being loaded and
producing wrong results
## Overview
* merge `dynamicSlice` and `slice` into `slices`
* merge `dynamicSlice1d` and `slice1d` into `slice`
* drop `choose`, `choose1d`
* handle singleton for dynamic slices too
* typed API so that it's easier to specify arguments
## Migration guide:
```diff
// new .slice accepts all old .slice1d inputs
- const x1 = x.slice1d(-1, .{ .start = 0, .end = half_dim });
+ const x1 = x.slice(-1, .{ .start = 0, .end = half_dim });
- k.dynamicSlice(.{ .layer = zml.Tensor.DynSlice{ .start = cache_index, .len = 1 } }).squeeze(.layer);
+ k.slice(.layer, .dynSingle(cache_index));
- k.dynamicSlice(.{
- .layer = zml.Tensor.DynSlice{ .start = cache_index, .len = 1 },
- }).squeeze(.layer).choose1d(.k, kv_head);
+ k.slice(.{ .layer, .k }, &.{ .dynSingle(cache_index), .single(kv_head) });
- o.choose1d(ax, 5),
+ o.slice(ax, .single(5))
```
Now the `Tensor.Slice` struct is bigger and now as a Tensor in it, which
may break user code if your model was storing a `Slice` inside it,
because now `zml.Bufferize(Model)` will contain one more `?Buffer`. You
can either pass an extra `null` to the `zml.Exe` or refactor to remove
`Slice` out of `Model`. I haven't seen this happen so far. The idea is
that `Slice` will mostly be a stack only struct, only created to call
`slice`
It's a bit of a mess to transfer ownership of the Exe to the runner as
that needs careful handling of defers. Fundamentally there is also no
particular reason why the Runner should own the Exe and it was not
consistent across zml.Exe and zml.TypedExe. Now it's always borrowed.
Renaming `zml.TypedExe` to `zml.FnExe` because it's shorter. I've
removed all the aliases `type LayerExe = zml.TypedExe(...)` it's useless
with ZLS, it's a lot easier to have `zml.FnExe(<function>)`
in LLMD and //examples/llm, we compile multiple executable, re-use the
args & results. We do all of this manually, in addition we don't have
any type-safety on the arguments/outputs which is a bit too easy to
misuse IMHO.
So adding two abstractions:
- Runner: Packs the args, exe and results together to avoid calling
`args.set`, `exe.run` and `results.fill` with similar arguments. Today
it owns the Exe. It doensn't strictly needs it, but it's simpler and
works for us for now.
- TypedExe: Type-safe version of an Exe, expecting a single input &
ouptut to have keyed arguments to have fully unambiguous arguments &
results mapping.
I've updated //example/llm with this new pattern.
Also adding an explicit `zml.module.CompileError`.
The old `zml.io.load` was plagued with problems. This PR is an attempt
at improving the state of things.
One major problem of `zml.io.load` was that it was not very composable.
It was either using it to load all the model or doing everything by
hand.
Another problem was that it didn't offer a way to apply transformations
at load time to the tensors stored in a safetensors file. For example,
it happens that for MoE models, instead of the experts weights to be
tightly packed in a nice 3D tensor, the weight file contain one tensor
for each expert. This is nice until your implementation required a 3D
tensor as input. Using the old `zml.io.load` API this resulted in very
convoluted way to do concatenation at load time. With this new API, we
offer convenience so that it can be handled nicely and as fast as
possible.
The main change is the introduction of the `Loader` struct. It's a
struct that you instantiate once and reuse whenever you want to load
tensors. It relies on `Binding`, which are a way to store which tensor
are required and how they need to be combined (if it's a complex binding
involving more than one source tensor).
One advantage of this `Loader` is that it's composable. Loading a struct
using `Loader.auto(Struct)` or loading each individual field of `Struct`
has essentially the same behavior. This solved the major problem of the
old solution.
For simplicity I kept the old model loading code almost identical, as
this new solution is a drop-in replacement.
# Update
After some discussions with @gwenzek I've changed my mind. Instead of
having a declarative API, we'll provide helper to do transformation
easily. The main changes that this PR now bring are:
* Introduce `Loader` struct
* `Loader` exposes `load` which is the composable replacement of
`zml.io.load`
* `Loader` exposes `loadExecute` that allows you to execute an
executable with the buffers loaded from the sources associated with a
tensor. This allows for example to concatenate tensors.
We'll continue to work on it, this is just a preliminary work that
allows us to do transformation at load time in LLMD.
- Bump Metal plugin version
- Drop the attention metadata fields from the llama and lfm2 Sessions. Prefill and decode now each build their own metadata inline and free it.
This PR splits Llama in `examples/llm` into separate `embed_tokens`,
`layer`, and `lm_head` kernels for both prefill and decode, and brings
LFM2 onto the same composed execution path. And done same work for Qwen
3.5 .
It removes the old LFM2 `--single` mode, reworks Llama KV-cache access
to use explicit layer indices, and reuses more decode-time state through
a persistent runner and prebuilt token index buffers.
It also tightens sharding/partitioning in LFM2 for multi-device runs,
adds clearer per-phase compile logging/program names, removes the unused
`--single` CLI flag.
This PR introduces the sharding as expert parallelism for MoE :
- Introduce a .experts logical sharding (required to express an other
semantic than TP (.model))
- Wrap the whole MoE impl into a manualComputation
- Implement a manual stableHlo all reduce since shardy can not infer
the correct communication after a manualComputation (blackbox)
- Partition qwen 3.5 moe example with TP for the rest of the model (non
MoE)
---------
Co-authored-by: loupicaaa <louis@zml.ai>
This PR simplifies the common replicated case while making sharding
intent explicit and use sharding by pointer ( sharding struct is quite
big: ~600 bytes).
Using pointers for shardings also avoids unnecessary sharding copies and
makes it easier to share stable sharding instances across compilation,
loading, and runtime paths.
- `Platform` owns the canonical replicated sharding
- `Buffer` APIs now take explicit sharding specs:
- `.replicated`
- `.{ .sharded = sharding_ptr }`
- `createTensor` / `maybeCreateTensor `now require explicit partitioning
instead of accepting null
- compilation/loading now use pointer-based sharding lists
This makes simple cases like mnist and benchmark much cleaner: callers
no longer need to manually construct and thread a replicated sharding
just to use the default case, but they still state placement explicitly
at the call site.
---------
Co-authored-by: Hugo Mano <hugo@zml.ai>
I'm doing a pass on the example code we have.
* llama/model.zig, qwen/model.zig: change layer_index from being a
tensor to being an integer.
It's clearer this way what is happening. Note that lfm doesn't have the
issue because the layer index is passed everywhere.
* tokenizePrompt: the code was doing unnecessary copies of data, I added
some helper to tokenizer.encoder to streamline the code by removing the
extra copies and allocations.
* `.stdout().writer() -> .stdout().writerStreaming()` this one is weird
but apparently using stdout by position isn't always allowed, in
particular when running the exe from within SublimeText.
* runDecode -> show case bake
I've reworked the Tokenizer API to actually work with all tokenizers and
not just IREE and moved the `Writer` part into the IREE tokenizer. I'm
now avoiding intermediate copies as much as possible now.
I'm also more defensive now to avoid the same infinite loop and put more
effort into providing appropriate output buffers to IREE. It fails with
empty output buffers and may return a `ResourceExhausted` if it's too
small. Also added a bunch of tests.
IREE tokenizer itself had a bug which I fixed here
https://github.com/iree-org/iree/pull/24150. In the meantime, I'm
pointing bazel to our fork.
Adding the [IREE
tokenizer](https://github.com/iree-org/iree/blob/main/runtime/src/iree/tokenizer/README.md).Being
made from the ground up with streaming (de-)tokenization in mind, I've
extended the APIs to expose a `std.Io.Writer` for the tokenization.
I've quickly benchmarked IREE's tokenizer vs HuggingFace's one with a
request I retrieved from a coding agent (opencode) in the middle of a
session. The request body was 360KB encoded in 95K tokens. I also did
another test with a similarly sized request with the same results as
shown hereafter.
I've tested two cases:
- oneshot: the template is rendered all at once and only then we encode
the whole string.
- stream: the template rendering and encoding are pipelined to avoid the
intermediate allocation.
There is a slight difference in the generated tokens between HuggingFace
and IREE. Some tokens like `introdu` were encoded as `i`, `n`, `t`, etc.
Decoding them returns the same text. It's very close in terms of total
token count:
- hf: 95779
- iree/stream: 95705
- iree/oneshot: 95587
| Model | IREE avg (oneshot / stream) ms | HuggingFace avg (ms) | Uplift
vs HF (oneshot / stream) |
|---|---:|---:|---|
| liquidai/LFM2.5-1.2B-Instruct | 8.5 / 7.3 | 40.5 | 4.8x / 5.5x |
| Qwen/Qwen3.5-9B | 6.0 / 8.2 | 59.4 | 9.9x / 7.2x |
| mistralai/Ministral-3-3B-Reasoning-2512 | 7.4 / 6.9 | 59.2 | 8x / 8.6x
|
It's unclear why the Qwen's chat template is somehow slower in the
stream case, maybe because there are more writes. Adding non-empty
buffer in the interface helps a bit, but tends to slow down mistral
case. So well
---------
Co-authored-by: Corentin Kerisit <corentin.kerisit@gmail.com>