0
Fork 0
mirror of https://github.com/zml/zml.git synced 2026-11-16 19:11:09 +00:00
Commit graph

17 commits

Author SHA1 Message Date
Raphael Nahoum
46bd32de2f zml/tokenizer: grow the decoder output to fit any token
IREE writes a token's text whole or not at all, so `feedOne` asserted
whenever the caller's buffer was smaller than the token, e.g. Laguna's
415-byte whitespace tokens in a 128-byte buffer. The decoder now owns
its output buffer and grows it the way `decode` already did, so
`feedOne` and `finalize` no longer take a buffer.
2026-11-09 10:56:20 +01:00
Raphael Nahoum
24f8c2b986
examples/llm: sample only the last prompt token (#965)
The LM head normalized and projected every prompt position at prefill,
and the session read back the whole token buffer just to keep the last
token.

Every model now slices the hidden state at `active_length - 1` before
the norm, so prefill computes logits for a single position and returns a
single token. Decode writes the next token over the one it read, so it
allocates no buffer per step.

lfm2's `actual_seq_len` is renamed `active_length`, the name the other
models use.
2026-10-21 11:47:42 +02:00
Corentin Godeau
bd0886c3e5
zml: re-structure attention kernels (#926)
This PR introduces a restructuration of the kernels in ZML. This is done
to simplify and speedup bringup on new platforms.

From now on each "kernel" will have its own directory under
`zml/kernels`. "Kernel" is in quote because we need to think about it
being more of an "operation in a graph" than an actual kernel. For
example, `moe` will eventually be part of this `kernels` directory
despite being an operation potentially launching multiple actual kernels
on some hardware.

Also it's probable that in the future, this `kernels` directory will be
put outside of zml or, at least, properly separated so that we don't
have circular imports in the codebase. Currently, we have
`@import("../../../zml.zig")` in the kernel code and this is a recipe
for disaster.

On top of the reorganization, it also adds a level of indirection for
attention. Where previously `attention` was a generic term, we now have
`attention` and `flash_attention` because `flash_attention` is an
implementation of `attention`.

Here is a detailed summary of the changes introduced by this PR:
* Kernels reorganization. `flash_attention` and `paged_attention` are
now separated. Each folder has a subfolder for each backend we support.
`flash_attention` is also now a `BackendType` of `zml.attention` with
its own implementations.
* Removed `cuda_fa2` and `cuda_fa3` backend because they were obsolete.
They have been replaced by a `cute` backend that has been generated
automatically following the operation documentation, showing that this
approach of scoping kernels per DSL actually scales.
* Removed `attnd` support for now
2026-10-12 20:31:17 +02:00
Guillaume Wenzek
b2f0e03043
zml/Tensor: explicit sharding in Tensor code (#868)
In Buffer code we already have explicit sharding, but not for tensor.

I'm adding a `zml.Sharding` to `.withPartitioning` to make it explicit
which mesh we are working with.
To simplify the migration, `.withPartitioning` in Tensor code can use
`.model` syntax to lookup a sharding by name.
This replace the previous logic of `selectSharding(shape)` which would
look up a sharding that matches the axes of the given partitioning which
could yield surprising behavior.

This also modifies how sharding are stored in zml.Shape.
Before it was storing a list of axis names + a tag (2 * 8 * 64bits), now
I'm storing a zml.Sharding (64 bits) and a compact PartitionArray (32
bits) that references the logical axis of the sharding by offset instead
of by name.

There is an impact on the `TensorRegistry` because it also needs to be
aware of the possible shardings, instead of delegating the matching
partition spec -> sharding to loading. This means error will be detected
sooner.

`store.createTensor` accepts `@enumLiteral()` to lookup the relevant
sharding.

This also simplifies `zml.Buffer` because the full sharding is now
stored in the shape, so it doesn't need to be passed independently. It
also means that it will propagate from the tensor who are graph inputs,
to the buffers that are eg inputs.

`Placement` logic also got a bit simpler.
2026-10-06 15:37:22 +02:00
Benjamin Rabier
c58d81a7b3
zml/exe: TypedExe (#697)
in LLMD and //examples/llm, we compile multiple executable, re-use the
args & results. We do all of this manually, in addition we don't have
any type-safety on the arguments/outputs which is a bit too easy to
misuse IMHO.

So adding two abstractions:
- Runner: Packs the args, exe and results together to avoid calling
`args.set`, `exe.run` and `results.fill` with similar arguments. Today
it owns the Exe. It doensn't strictly needs it, but it's simpler and
works for us for now.
- TypedExe: Type-safe version of an Exe, expecting a single input &
ouptut to have keyed arguments to have fully unambiguous arguments &
results mapping.

I've updated //example/llm with this new pattern.

Also adding an explicit `zml.module.CompileError`.
2026-08-18 16:36:38 +02:00
Guillaume Wenzek
d6489fbb84
zml/attention: implement batching in cu_fa2 backend (#641) 2026-07-17 14:10:41 +02:00
elogir
2057d68556
platforms/metal: bump plugin & remove metadata from struct (#620)
- Bump Metal plugin version
- Drop the attention metadata fields from the llama and lfm2 Sessions. Prefill and decode now each build their own metadata inline and free it.
2026-07-03 11:23:43 +02:00
Hugo Mano
950c80dee8
examples/llm: split Llama and Qwen 3.5 with LFM cleanup (#555)
This PR splits Llama in `examples/llm` into separate `embed_tokens`,
`layer`, and `lm_head` kernels for both prefill and decode, and brings
LFM2 onto the same composed execution path. And done same work for Qwen
3.5 .

It removes the old LFM2 `--single` mode, reworks Llama KV-cache access
to use explicit layer indices, and reuses more decode-time state through
a persistent runner and prebuilt token index buffers.

It also tightens sharding/partitioning in LFM2 for multi-device runs,
adds clearer per-phase compile logging/program names, removes the unused
`--single` CLI flag.
2026-05-28 10:10:29 +02:00
Guillaume Wenzek
0979289ffe
zml/sharding: pass sharding by pointer and simplify replicated usage (#514)
This PR simplifies the common replicated case while making sharding
intent explicit and use sharding by pointer ( sharding struct is quite
big: ~600 bytes).

Using pointers for shardings also avoids unnecessary sharding copies and
makes it easier to share stable sharding instances across compilation,
loading, and runtime paths.

- `Platform` owns the canonical replicated sharding
- `Buffer` APIs now take explicit sharding specs:
    - `.replicated`
    - `.{ .sharded = sharding_ptr }`
- `createTensor` / `maybeCreateTensor `now require explicit partitioning
instead of accepting null
- compilation/loading now use pointer-based sharding lists

This makes simple cases like mnist and benchmark much cleaner: callers
no longer need to manually construct and thread a replicated sharding
just to use the default case, but they still state placement explicitly
at the call site.

---------

Co-authored-by: Hugo Mano <hugo@zml.ai>
2026-05-06 12:46:15 +02:00
Guillaume Wenzek
039a638aeb
examples/llm: cleanups (#512)
I'm doing a pass on the example code we have.

* llama/model.zig, qwen/model.zig: change layer_index from being a
tensor to being an integer.
It's clearer this way what is happening. Note that lfm doesn't have the
issue because the layer index is passed everywhere.

* tokenizePrompt: the code was doing unnecessary copies of data, I added
some helper to tokenizer.encoder to streamline the code by removing the
extra copies and allocations.

* `.stdout().writer() -> .stdout().writerStreaming()` this one is weird
but apparently using stdout by position isn't always allowed, in
particular when running the exe from within SublimeText.

* runDecode -> show case bake
2026-04-29 10:38:19 +02:00
Benjamin Rabier
df86cdfd96
zml/tokenizer/iree: Improve IREE tokenizer with upstream fix (#482)
I've reworked the Tokenizer API to actually work with all tokenizers and
not just IREE and moved the `Writer` part into the IREE tokenizer. I'm
now avoiding intermediate copies as much as possible now.

I'm also more defensive now to avoid the same infinite loop and put more
effort into providing appropriate output buffers to IREE. It fails with
empty output buffers and may return a `ResourceExhausted` if it's too
small. Also added a bunch of tests.

IREE tokenizer itself had a bug which I fixed here
https://github.com/iree-org/iree/pull/24150. In the meantime, I'm
pointing bazel to our fork.
2026-04-20 11:15:37 +02:00
Benjamin Rabier
1e79469136
examples/llm: fix prompt tokenization (#480)
Tested with IREE and HF on all three models.
2026-04-16 11:09:47 +02:00
Benjamin Rabier
91f58097cc
zml/tokenizer: use correct API for decoding with HF (#478)
The current API only worked with IREE. Reworked it to work with both HF
and IREE
2026-04-14 17:13:26 +02:00
Benjamin Rabier
448084908a
zml/tokenizer: add IREE tokenizer (#416)
Adding the [IREE
tokenizer](https://github.com/iree-org/iree/blob/main/runtime/src/iree/tokenizer/README.md).Being
made from the ground up with streaming (de-)tokenization in mind, I've
extended the APIs to expose a `std.Io.Writer` for the tokenization.

I've quickly benchmarked IREE's tokenizer vs HuggingFace's one with a
request I retrieved from a coding agent (opencode) in the middle of a
session. The request body was 360KB encoded in 95K tokens. I also did
another test with a similarly sized request with the same results as
shown hereafter.

I've tested two cases:

- oneshot: the template is rendered all at once and only then we encode
the whole string.
- stream: the template rendering and encoding are pipelined to avoid the
intermediate allocation.

There is a slight difference in the generated tokens between HuggingFace
and IREE. Some tokens like `introdu` were encoded as `i`, `n`, `t`, etc.
Decoding them returns the same text. It's very close in terms of total
token count:

- hf: 95779
- iree/stream: 95705
- iree/oneshot: 95587

| Model | IREE avg (oneshot / stream) ms | HuggingFace avg (ms) | Uplift
vs HF (oneshot / stream) |
|---|---:|---:|---|
| liquidai/LFM2.5-1.2B-Instruct | 8.5 / 7.3 | 40.5 | 4.8x / 5.5x |
| Qwen/Qwen3.5-9B | 6.0 / 8.2 | 59.4 | 9.9x / 7.2x |
| mistralai/Ministral-3-3B-Reasoning-2512 | 7.4 / 6.9 | 59.2 | 8x / 8.6x
|


It's unclear why the Qwen's chat template is somehow slower in the
stream case, maybe because there are more writes. Adding non-empty
buffer in the interface helps a bit, but tends to slow down mistral
case. So well

---------

Co-authored-by: Corentin Kerisit <corentin.kerisit@gmail.com>
2026-04-07 16:42:55 +02:00
Benjamin Rabier
0d0d5020cf
example/llm: Further simplifications (#433) 2026-03-25 10:16:26 +01:00
Benjamin Rabier
8de2c42fe6
examples/llm: Simplifications (#430) 2026-03-24 18:16:25 +01:00
Benjamin Rabier
81299c67c8
examples/llm: Common model example (#425) 2026-03-24 16:55:29 +01:00