0
Fork 0
mirror of https://github.com/zml/zml.git synced 2026-11-16 19:11:09 +00:00
Commit graph

21 commits

Author SHA1 Message Date
Corentin Godeau
bd0886c3e5
zml: re-structure attention kernels (#926)
This PR introduces a restructuration of the kernels in ZML. This is done
to simplify and speedup bringup on new platforms.

From now on each "kernel" will have its own directory under
`zml/kernels`. "Kernel" is in quote because we need to think about it
being more of an "operation in a graph" than an actual kernel. For
example, `moe` will eventually be part of this `kernels` directory
despite being an operation potentially launching multiple actual kernels
on some hardware.

Also it's probable that in the future, this `kernels` directory will be
put outside of zml or, at least, properly separated so that we don't
have circular imports in the codebase. Currently, we have
`@import("../../../zml.zig")` in the kernel code and this is a recipe
for disaster.

On top of the reorganization, it also adds a level of indirection for
attention. Where previously `attention` was a generic term, we now have
`attention` and `flash_attention` because `flash_attention` is an
implementation of `attention`.

Here is a detailed summary of the changes introduced by this PR:
* Kernels reorganization. `flash_attention` and `paged_attention` are
now separated. Each folder has a subfolder for each backend we support.
`flash_attention` is also now a `BackendType` of `zml.attention` with
its own implementations.
* Removed `cuda_fa2` and `cuda_fa3` backend because they were obsolete.
They have been replaced by a `cute` backend that has been generated
automatically following the operation documentation, showing that this
approach of scoping kernels per DSL actually scales.
* Removed `attnd` support for now
2026-10-12 20:31:17 +02:00
Guillaume Wenzek
b2f0e03043
zml/Tensor: explicit sharding in Tensor code (#868)
In Buffer code we already have explicit sharding, but not for tensor.

I'm adding a `zml.Sharding` to `.withPartitioning` to make it explicit
which mesh we are working with.
To simplify the migration, `.withPartitioning` in Tensor code can use
`.model` syntax to lookup a sharding by name.
This replace the previous logic of `selectSharding(shape)` which would
look up a sharding that matches the axes of the given partitioning which
could yield surprising behavior.

This also modifies how sharding are stored in zml.Shape.
Before it was storing a list of axis names + a tag (2 * 8 * 64bits), now
I'm storing a zml.Sharding (64 bits) and a compact PartitionArray (32
bits) that references the logical axis of the sharding by offset instead
of by name.

There is an impact on the `TensorRegistry` because it also needs to be
aware of the possible shardings, instead of delegating the matching
partition spec -> sharding to loading. This means error will be detected
sooner.

`store.createTensor` accepts `@enumLiteral()` to lookup the relevant
sharding.

This also simplifies `zml.Buffer` because the full sharding is now
stored in the shape, so it doesn't need to be passed independently. It
also means that it will propagate from the tensor who are graph inputs,
to the buffers that are eg inputs.

`Placement` logic also got a bit simpler.
2026-10-06 15:37:22 +02:00
nzinci
5d0686de02
zml/safetensors: close model discovery handles (#758)
Model discovery leaves its safetensors entrypoint and repository handles
open after building the tensor registry, including when parsing fails.

Close handles owned by `fromRepo` and `fromPath`, and close repositories
at the existing `fromRepo` call sites.
2026-09-02 09:10:18 +02:00
Benjamin Rabier
12211f3765
zml/exe: borrowed Exe in Runner (#708)
It's a bit of a mess to transfer ownership of the Exe to the runner as
that needs careful handling of defers. Fundamentally there is also no
particular reason why the Runner should own the Exe and it was not
consistent across zml.Exe and zml.TypedExe. Now it's always borrowed.

Renaming `zml.TypedExe` to `zml.FnExe` because it's shorter. I've
removed all the aliases `type LayerExe = zml.TypedExe(...)` it's useless
with ZLS, it's a lot easier to have `zml.FnExe(<function>)`
2026-08-20 10:56:32 +02:00
Benjamin Rabier
c58d81a7b3
zml/exe: TypedExe (#697)
in LLMD and //examples/llm, we compile multiple executable, re-use the
args & results. We do all of this manually, in addition we don't have
any type-safety on the arguments/outputs which is a bit too easy to
misuse IMHO.

So adding two abstractions:
- Runner: Packs the args, exe and results together to avoid calling
`args.set`, `exe.run` and `results.fill` with similar arguments. Today
it owns the Exe. It doensn't strictly needs it, but it's simpler and
works for us for now.
- TypedExe: Type-safe version of an Exe, expecting a single input &
ouptut to have keyed arguments to have fully unambiguous arguments &
results mapping.

I've updated //example/llm with this new pattern.

Also adding an explicit `zml.module.CompileError`.
2026-08-18 16:36:38 +02:00
Guillaume Wenzek
d6489fbb84
zml/attention: implement batching in cu_fa2 backend (#641) 2026-07-17 14:10:41 +02:00
Benjamin Rabier
19a15ca966
examples/llm: shiny ZML logo (#636)
Co-authored-by: Tristan Gautie <tristan@zml.ai>
2026-07-09 14:02:57 +02:00
Olivier Boulant
1dbe39d3e8
zml/io/vfs: add parallel read to GCS and S3 (#604)
# Summary

This PR extends the support to chunked parallel read introduced
initially for HF in the VFS, to GCS and S3.

It:
- Adds a shared `zml/io/vfs/parallel_read.zig` helper for chunked
parallel HTTP range reads
- Add configurable read pool options for S3/GCS initialization
- Register the `gs://` VFS backend in `examples/llm`.


# Tests

## S3

We used [MINIO](https://hub.docker.com/r/minio/minio) docker image from
the `5090` GPU machine. From there, we mirror the
`meta-llama/Llama-3.1-8B-Instruct` model through the S3 server.

Then from the `4090` GPU machine, we run 

```bash
bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=s3://zml-test/Llama-3.1-8B-Instruct --prompt="hello, how are you ?"
```

It works both on 1 and 2 GPU. 

## GCS

In the same spirit, we put the `meta-llama/Llama-3.1-8B-Instruct` in a
GCS bucket we own. Then, on the `4090`, we run

```bash
bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct --prompt="hello, how are you ?"
```

It works both on 1 and 2 GPU. 

### Performance

As a reference, we run:
```bash
gcloud storage cp gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00001-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00002-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00003-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00004-of-00004.safetensors /tmp/gcs-gcloud-llama-shards/
```

It gives:
```
Completed files 4/4 | 15.0GiB/15.0GiB | 942.7MiB/s 
Average throughput: 918.8MiB/s
```

Using the VFS in various setups:

* `dma_chunks = 32`, `dma_chunk_size = 128`, `parallelism = 16`,
`chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64`

```bash
info (llama): Loaded weights [14.96GiB, 21.302s, 719.02MiB/s]
```

* `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`,
`chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64`

```bash
info (llama): Loaded weights [14.96GiB, 18.227s, 840.32MiB/s]
```

* `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`,
`chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64`

```bash
info (llama): Loaded weights [14.96GiB, 17.376s, 881.43MiB/s]
```

* `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`,
`chunk_size = 16 MiB`, `num_workers = 32`, `queue_capacity = 64`

```bash
info (llama): Loaded weights [14.96GiB, 17.28s, 886.36MiB/s]
```

* `dma_chunks = 32`, `dma_chunk_size = 512`, `parallelism = 16`,
`chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64`

```bash
info (llama): Loaded weights [14.96GiB, 18.658s, 820.90MiB/s]
```
2026-06-26 19:24:31 +02:00
Hugo Mano
1fead1c014
platforms/neuron: introduce NKI stack with SDPA kernel (#528)
**Summary**

This PR update to Neuron v2.30, introduces the Neuron NKI stack into ZML and wires it into the
LLM attention path.

The main goal is to make Neuron attention run through NKI kernels.
It adds the compiler/runtime plumbing needed to embed NKI kernels as
Neuron custom-native calls, then uses that path for SDPA-style
prefill/decode attention. Paged Attention will come later.

On inf2.8xlarge it runs :
- 25 tok/s for Llama 3.1 8B BF16
- 63.5 tok/s for LFM2.5-1.2B-Instruct BF16
- 19 tok/s for Qwen3.5-9B BF16
2026-06-16 10:57:00 +02:00
Hugo Mano
950c80dee8
examples/llm: split Llama and Qwen 3.5 with LFM cleanup (#555)
This PR splits Llama in `examples/llm` into separate `embed_tokens`,
`layer`, and `lm_head` kernels for both prefill and decode, and brings
LFM2 onto the same composed execution path. And done same work for Qwen
3.5 .

It removes the old LFM2 `--single` mode, reworks Llama KV-cache access
to use explicit layer indices, and reuses more decode-time state through
a persistent runner and prebuilt token index buffers.

It also tightens sharding/partitioning in LFM2 for multi-device runs,
adds clearer per-phase compile logging/program names, removes the unused
`--single` CLI flag.
2026-05-28 10:10:29 +02:00
Benjamin Rabier
ca8599aea0
zml/attention: attnd (#450)
Simple reference implementation of attnd client as a attention backend
relying on POSIX. I've tried various things with `sendmmsg`/`recvmmsg`
and `io_uring` out of curiosity but they weren't worth the extra code
here.

The integration as a attention backend is a bit clunky, we're still
allocating a KV cache and there are a few panics here and there because
the current initialization mechanism of attention backend doesn't work
for attnd, but that's something for later.
2026-05-12 17:27:37 +02:00
Guillaume Wenzek
039a638aeb
examples/llm: cleanups (#512)
I'm doing a pass on the example code we have.

* llama/model.zig, qwen/model.zig: change layer_index from being a
tensor to being an integer.
It's clearer this way what is happening. Note that lfm doesn't have the
issue because the layer index is passed everywhere.

* tokenizePrompt: the code was doing unnecessary copies of data, I added
some helper to tokenizer.encoder to streamline the code by removing the
extra copies and allocations.

* `.stdout().writer() -> .stdout().writerStreaming()` this one is weird
but apparently using stdout by position isn't always allowed, in
particular when running the exe from within SublimeText.

* runDecode -> show case bake
2026-04-29 10:38:19 +02:00
Hugo Mano
f80f925409
examples/llm: add profiling option (#497)
This allow use to easily trace what going on
2026-04-22 16:23:36 +02:00
Benjamin Rabier
f258280ab8
platforms/cuda+rocm: update pjrt artifact (#487)
Update with the XLA patches for improved parallel compilation.

`//examples/llm` goes from 27s to 8s for llama3.1 8B compilation
2026-04-20 18:24:01 +02:00
Corentin Godeau
ac9bc32d89
zml/attention: add Triton Unified Attention backend (#417)
This PR introduces a Triton paged attention backend, unlocking ROCm
support.

It relies on a Python sandbox to generate the TTIR.

The major change is that the KVCache now needs to be per-layer for paged
attention. The `layer_index` is now superfluous and will probably be
removed in the future.
2026-04-13 11:39:08 +02:00
Benjamin Rabier
448084908a
zml/tokenizer: add IREE tokenizer (#416)
Adding the [IREE
tokenizer](https://github.com/iree-org/iree/blob/main/runtime/src/iree/tokenizer/README.md).Being
made from the ground up with streaming (de-)tokenization in mind, I've
extended the APIs to expose a `std.Io.Writer` for the tokenization.

I've quickly benchmarked IREE's tokenizer vs HuggingFace's one with a
request I retrieved from a coding agent (opencode) in the middle of a
session. The request body was 360KB encoded in 95K tokens. I also did
another test with a similarly sized request with the same results as
shown hereafter.

I've tested two cases:

- oneshot: the template is rendered all at once and only then we encode
the whole string.
- stream: the template rendering and encoding are pipelined to avoid the
intermediate allocation.

There is a slight difference in the generated tokens between HuggingFace
and IREE. Some tokens like `introdu` were encoded as `i`, `n`, `t`, etc.
Decoding them returns the same text. It's very close in terms of total
token count:

- hf: 95779
- iree/stream: 95705
- iree/oneshot: 95587

| Model | IREE avg (oneshot / stream) ms | HuggingFace avg (ms) | Uplift
vs HF (oneshot / stream) |
|---|---:|---:|---|
| liquidai/LFM2.5-1.2B-Instruct | 8.5 / 7.3 | 40.5 | 4.8x / 5.5x |
| Qwen/Qwen3.5-9B | 6.0 / 8.2 | 59.4 | 9.9x / 7.2x |
| mistralai/Ministral-3-3B-Reasoning-2512 | 7.4 / 6.9 | 59.2 | 8x / 8.6x
|


It's unclear why the Qwen's chat template is somehow slower in the
stream case, maybe because there are more writes. Adding non-empty
buffer in the interface helps a bit, but tends to slow down mistral
case. So well

---------

Co-authored-by: Corentin Kerisit <corentin.kerisit@gmail.com>
2026-04-07 16:42:55 +02:00
Benjamin Rabier
9fa7768808
examples/llm: better single prompt UX + --topk (#438)
- Same UX as the chat
- Add `--topk` argument
2026-03-26 15:58:03 +01:00
Benjamin Rabier
268ef6e058
examples/llm: fix --help & relative/home paths (#437) 2026-03-25 15:14:22 +01:00
Benjamin Rabier
0d0d5020cf
example/llm: Further simplifications (#433) 2026-03-25 10:16:26 +01:00
Benjamin Rabier
8de2c42fe6
examples/llm: Simplifications (#430) 2026-03-24 18:16:25 +01:00
Benjamin Rabier
81299c67c8
examples/llm: Common model example (#425) 2026-03-24 16:55:29 +01:00