- Bump Metal plugin version
- Drop the attention metadata fields from the llama and lfm2 Sessions. Prefill and decode now each build their own metadata inline and free it.
This PR improves the `zml.nn.sampleTokensDynamic()` function. Most
notably:
* Convert parameters to input data type
* Support scalar/batched parameters
* Correctly use the temperature parameter
The IREE tokenizer was not producing the same output as the HF
tokenizer, potentially degrading the performance of the models. I put a
clanker to the task (because the tokenizer is already full clanked).
Outputs are now matching without any performance drop.
I also fixed an edge case issue on the `Writer` of the tokenizer.
Move the type out of `zml.compile` signature so that we can call
`io.async` more easily.
Sample code:
```
var layer_compile_future = io.async(zml.Compiler(TransformerLayer.forward).compile, .{
allocator,
io,
platform,
layer_compile_opts,
.{
model.model.layers[0],
hidden,
device_state.kv_cache.layers[0],
},
});
```
Previously, split tensors where allocated using an arena that would be
free at the end of the function scope. This commit uses the arena
context instead.
Signed-off-by: dmehala <damien@zml.ai>
## Summary
- Limit Zig cache sandbox mount flags to Linux, where linux-sandbox and
Bazel's Docker sandbox consume host path mounts.
- Remove the macOS mount entry because sandbox-exec uses a symlinked
execroot and ignores bind-mount settings.
- Mark `/tmp` and `/var/tmp` writable alongside the Linux mount pairs.
# Summary
This PR extends the support to chunked parallel read introduced
initially for HF in the VFS, to GCS and S3.
It:
- Adds a shared `zml/io/vfs/parallel_read.zig` helper for chunked
parallel HTTP range reads
- Add configurable read pool options for S3/GCS initialization
- Register the `gs://` VFS backend in `examples/llm`.
# Tests
## S3
We used [MINIO](https://hub.docker.com/r/minio/minio) docker image from
the `5090` GPU machine. From there, we mirror the
`meta-llama/Llama-3.1-8B-Instruct` model through the S3 server.
Then from the `4090` GPU machine, we run
```bash
bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=s3://zml-test/Llama-3.1-8B-Instruct --prompt="hello, how are you ?"
```
It works both on 1 and 2 GPU.
## GCS
In the same spirit, we put the `meta-llama/Llama-3.1-8B-Instruct` in a
GCS bucket we own. Then, on the `4090`, we run
```bash
bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct --prompt="hello, how are you ?"
```
It works both on 1 and 2 GPU.
### Performance
As a reference, we run:
```bash
gcloud storage cp gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00001-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00002-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00003-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00004-of-00004.safetensors /tmp/gcs-gcloud-llama-shards/
```
It gives:
```
Completed files 4/4 | 15.0GiB/15.0GiB | 942.7MiB/s
Average throughput: 918.8MiB/s
```
Using the VFS in various setups:
* `dma_chunks = 32`, `dma_chunk_size = 128`, `parallelism = 16`,
`chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64`
```bash
info (llama): Loaded weights [14.96GiB, 21.302s, 719.02MiB/s]
```
* `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`,
`chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64`
```bash
info (llama): Loaded weights [14.96GiB, 18.227s, 840.32MiB/s]
```
* `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`,
`chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64`
```bash
info (llama): Loaded weights [14.96GiB, 17.376s, 881.43MiB/s]
```
* `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`,
`chunk_size = 16 MiB`, `num_workers = 32`, `queue_capacity = 64`
```bash
info (llama): Loaded weights [14.96GiB, 17.28s, 886.36MiB/s]
```
* `dma_chunks = 32`, `dma_chunk_size = 512`, `parallelism = 16`,
`chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64`
```bash
info (llama): Loaded weights [14.96GiB, 18.658s, 820.90MiB/s]
```
part of highway XLA target selects which header to include based on
legacy --cpu flags but the rest not...
So even if we targeted linux amd64, the top part of the highway would be
used correctly, but the extension specific part would fallback to --cpu
darwin_arm64 and would not include hh_avx2.h resulting in compilation
errors.
The normal solution here is to use platform_mappings but since we pass
`--force_pic=true` in platforms, this takes precedence and the mappings
are ignored.
The temporary solution while https://github.com/openxla/xla/pull/44823
is merged, is to pass `--cpu` in all platforms we use for cross
compilation.
A few missing loads
Remove all flipped flags or deperacted flags from .bazelrc
Important:
- Keep autoload rules_cc and rules_python because XLA still has
transitive deps that are missing loads, PR in progress.
Updates to latest XLA which added support for bazel 8.x workspace and
fixed a bunch of things that made bazel 9 impossible.
3 problematic changes with this bump:
- PJRT headers were again incompatible with translate-c so a patch is
made and PR has been done in XLA
- PJRT introduced a proper type fo PJRT_Error so I used the InnerMixin
pattern for those.
- //tsl/platform/default package force evaluates Label("@cuda_nvcc")
even for CPU builds and we don't have this repo. I commented for now
until I find a solution on XLA side.
Bumped a few deps necessary for this and additional llvm deps that are
now evaluated as part of the llvm-project graph on XLA side (llvm_zlib
and llvm_zstd).
Next will be to investigate bazel 9 for real.
IREE's tokenizer only registered added tokens for pre-segmentation
matching if they were flagged special or normalized. Tokens that are
neither (e.g. <think> was dropped)
The fix is to match all added tokens
This PR introduces shardableDim helper in Sharding to handle the case
where : (dimension % number of sharding axes) != 0
Useful to stutter and shard correctly kv heads dim for example.
---------
Co-authored-by: Hugo Mano <hugo@zml.ai>
DirectMemoryWriter used to expose the next shard-sized slice as its writable
window. With HF/VFS this often produced tiny reads, even when adjacent tensor
bytes could be streamed through a larger direct-load window.
This PR changes direct loading to use an alias-window over the active primary
shard DMA buffer. The public `std.Io.Writer` buffer still points at real DMA
memory, but its visible length can coalesce across dispatch spans until a DMA
chunk, shard buffer, or tensor boundary is reached. On commit, the first active
primary segment stays zero-copy, while crossed spans and mirrors are scattered
before any buffer is submitted or rotated.
`DirectShardWriter` keeps ownership of DMA pool rotation, flip-flop events, and
PJRT transfer state. `DirectMemoryWriter` only walks the precomputed dispatch
spans with a small cursor.
Move the MoE allReduce inside the manual computation block.
Previously, zml.moe.forwardMoe placed allReduce outside the manual
computation block which caused XLA SPMD partitioning to throw the
following error with certain models.
"Cross-partition allreduce must be in (partial) manual partitioning mode."
**Summary**
This PR update to Neuron v2.30, introduces the Neuron NKI stack into ZML and wires it into the
LLM attention path.
The main goal is to make Neuron attention run through NKI kernels.
It adds the compiler/runtime plumbing needed to embed NKI kernels as
Neuron custom-native calls, then uses that path for SDPA-style
prefill/decode attention. Paged Attention will come later.
On inf2.8xlarge it runs :
- 25 tok/s for Llama 3.1 8B BF16
- 63.5 tok/s for LFM2.5-1.2B-Instruct BF16
- 19 tok/s for Qwen3.5-9B BF16
This PR add the TPU backend for MoE. It uses the GMM Expert parallelism
kernel to perform the grouped gemms on gate_up and down weights (these
weights must have a precise shape to run optimally).
This backend is currently optimized for small batches. A specialized
mosaic tpu kernel will be introduced to handle bigger ones.
This implementation come with few fixes concerning the TPU sharding and
the Qwen 3.5 moe model which uses the backend :
- TPU sharding : PJRT reports TPU device coordinates in a max rank form.
So it breaks folding. To avoid this we compute the Rank by not counting
where coords =0.
- Linear attention was reading padding token in the Qwen 3.5 model. So
we introduce masking of them.
n.b. : The kv head repetition to satisfy the sharding will be handled a
more generic way in a next PR
---------
Co-authored-by: Corentin Godeau <corentin@zml.ai>
Renamed `scope` to `span` to match OpenTelemetry convention I'm more
familiar with. Traceme doesn't seem to use `Scope` explicitly but talks
more about a scope object AFAIK.
Now `span` fails at compile time. if it cannot compute the appropriate
buffer size for the name. In that case the caller should use
`formatSpanName` which allocates. This makes `span` infaillible at
runtime.
This opt-in flag allows XLA to save the autotuning resulting from
previous runs in a folder, greatly speeding up the compilation.
It's not meant to be used at all times, so it's undocumented and opt-in.
This PR introduce the new GMM Expert parallelism kernel (basis grouped
gemm for moe on TPU). It induces some updates into the DSL mosaic TPU,
such as Builders helpers or Pipelines and schedulers :
- Buffer counts (for scheduling overlapping) > 2 allowed (until 7)
- source_reshape_shape reshaped at DMA slice/use sites to match Pallas
IR placement
- More grid dimensionality for pipeline (up to 4)