On cpu, neuron and metal, `toMemory` returned self. Donating a result to
a device buffer and its `toMemory` copy to a host buffer then called
`reuseBuffer` twice on the same tensor id: the second donation replaced
the first, and compilation panicked because both outputs aliased the
host buffer. This broke llmd async scheduling on Metal and CPU.
Return the same value under a new tensor id instead.
Bump plugins to
[202611132003.375.1.7d765f57a407](2026111320.375.1.7d765f57a407)
oneAPI: weight-only FP8 and FP4 (NVFP4, MXFP4) scaled dots now run in a
Triton kernel that reads the
weights in their stored format.
Separate dispatch from kernel generation. Use one factory to select BF16
and block-128 FP8 kernels by format and schedule, and reuse the shared
ActivationKind.
Pass the requested activation to both BF16 TCL MoE kernels and compute
tanh GELU on the gate before multiplying by the up projection. This lets
Gemma 4 compile and run with its required activation.
Verified both kernel paths against a CPU reference and confirmed
coherent Gemma 4 generation on two and four RNGD cards.
Replace nonlocal expert IDs with an in-range placeholder and zero their
routing weights before dispatch. This prevents the activation-stationary
and blockwise kernels from using -1 in gathers and scatters.
Verified BF16 outputs against the CPU reference and coherent
Qwen3.5-35B-A3B generation on two and four RNGD cards.
Last PR for manualComputation v2 which promote as default and only one
API for manual computation. Custom calls now require to be wrapped in a
manual computation too.
- Select TCL for plain E4M3FN block-128 experts with matching BF16 or
F32 scales. Preserve rejection of unsupported quantization.
- Quantize activations per block and apply checkpoint scales during
native projections. Keep BF16 materialization and F32 down and
routing accumulation.
- Reuse local expert routing and AllReduce for expert parallelism.
Choose expert-stationary for small FP8 token counts and use the
existing activation-stationary and blockwise recipes for larger
counts.
- Share FP8 and BF16 dispatch preparation while retaining the token
padding and BF16 SiLU rounding workarounds.
Refactor `zml/safetensors.zig`
* Drop metadata parsing
* refactor code to be more "object oriented", code is owned by the
`TensorRegistry`
* implement a new format `model.safetensors.full_index.json` that is
similar to `index.json` but actually has data inside it:
```
{
"metadata": {"num_tensors":96085,"total_size":510286023000},
"weight_map": {
"aligner.w1.bias": {"filename":"model-00001-of-00048.safetensors","shape":[5120],"data_offsets":[27384,37624],"dtype":"BF16"},
"aligner.w1.weight": {"filename":"model-00001-of-00048.safetensors","shape":[5120,9216],"data_offsets":[37624,94409464],"dtype":"BF16"},
"aligner.w2.bias": {"filename":"model-00001-of-00048.safetensors","shape":[5120],"data_offsets":[94409464,94419704],"dtype":"BF16"},
"aligner.w2.weight": {"filename":"model-00001-of-00048.safetensors","shape":[5120,5120],"data_offsets":[94419704,146848504],"dtype":"BF16"},
"vision.blocks.0.attn.wo.bias": {"filename":"model-00001-of-00048.safetensors","shape":[1024],"data_offsets":[146848504,146850552],"dtype":"BF16"},
"vision.blocks.0.attn.wo.weight": {"filename":"model-00001-of-00048.safetensors","shape":[1024,1024],"data_offsets":[146850552,148947704],"dtype":"BF16"},
"vision.blocks.0.attn.wqkv.bias": {"filename":"model-00001-of-00048.safetensors","shape":[3072],"data_offsets":[148947704,148953848],"dtype":"BF16"},
...
```
I followed conventions of `model.saftensors.index.json`
Replaces the `zml$flash_attn` custom call with Metal DSL kernels over
contiguous K/V:
- `fa_vec` + `fa_vec_reduce` for bf16 decode, at most 256 threads per
threadgroup.
- `fa_tiled` for prefill and f16.
- Batched queries work (the old path panicked). Other dtypes and head
dims fall back to sdpa.
Bumps the plugins to
[202611101651.363.1.d46c7471c419](2026111016.363.1.d46c7471c419)
Fixes Metal decode on M1/M2: `fa_vec_paged` launched up to 512 threads
per threadgroup, over the 384 those GPUs accept.
- A threadgroup holds up to 4 query heads of one KV head times position
splits, at most 256 threads.
- The split count follows the PJRT core count and the number of rows.
- Head dims 32 and 64 now use the vector kernel instead of the tiled
one.
llmd decode on M5 Max vs master: no change at short context, +4–27% at
8K, +12–74% at 24K, bs16 unchanged.
Head-dim-64 models: 2–7.6× at long context.
Should fix https://github.com/zml/zml/issues/1037 but needs some testing
on a real M1/M2
This PR enable the MegaMoe kernel support in llmd by :
- Making the symmetric workspace an arg of CuteOps
- Adding cuda.grid_constant arg to the DSL builder, enabling GPU kernels
to receive TMA tensormap
- Adapting the existing mxfp4 kernel (non mega) to the new shared
swizzling `interleaved_8`
`DirectMemoryWriter.drain` only wrote the first slice of data and
ignored the splat count. The `std.Io.Writer` contract is to write every
slice in order, then repeat the last one splat times.
- A one-byte splat advanced one byte per drain call.
- After a partial write, `splatBytesAll` passes an empty first slice
followed by the pattern. The old drain returned 0 for that forever, so
the fill never finished.
`drain` now writes every slice and repeats the last one splat times.
* Stopped using `zml.ops.CustomCall` because it creates cycles
impossible to break.
* renamed some `cutlass_flashinfer` to `flashinfer_cutlass` for
consistency
I didn't replace `CustomCall` because flashinfer_cutlass is the last
user and will be removed soon.
`zml/kernels/moe.zig` is not imported by backend implementation anymore.
I introduced `options.zig` to contain the shared options. I also
introduced `weights.zig` to expose the unpacking helpers.
MOE API currently relies on input sharding metadata or
partitioning choices made inside backend kernels. Ordinary graph
operations can leave that metadata unspecified, making the resulting
sharding depend on the backend and preceding operations.
Move the manual computation boundary into the shared
`zml.kernels.moe.forwardMoe` API using `manualComputation2`.
Callers provide a logical mesh and typed input/output partitioning specs.
- Quantize block FP8 activations to E4M3FN on Furiosa, so scaled dots
run natively. Other block sizes keep BF16 activations and XLA's
fallback.
- Pad paged attention query groups to a power of two with zero queries,
as TCC fails other group sizes. Marked as a workaround.
- Bump to
[202611051630.351.1.48d539621544](2026110516.351.1.48d539621544)
Paged attention currently relies on input sharding metadata or
partitioning choices made inside backend kernels. Ordinary graph
operations can leave that metadata unspecified, making the resulting
sharding depend on the backend and preceding operations.
Move the manual computation boundary into the shared `pagedAttention`
API using `manualComputation2`. Callers provide a logical mesh and typed
input/output partitioning specs.
Move manual regions out of flash attention kernels and use
`manualComputation2` for every shared attention backend.
Also fully replicate the embedding executable outputs in Qwen and Llama.
Partial graph constraints leave their batch and sequence dimensions
open, which is invalid at executable boundaries.
Fix two further blockers exposed by the Qwen run that we forgot to move
recently: softplus comparisons must use the input dtype and ROCm DNN
runfiles must include the MIOpen kernel archives.
The split-K/3D path writes partial results only for active queries, but
the reduction kernel processes inactive queries in the padded batch.
These queries have no valid results and cause invalid
computation/outputs.
This fix zeroes the ouput for inactive queries and return before before
accessing sequence metadata.
Resolves#1006
- **Work follows sequence lengths:** decode's compute no longer runs
over the block table's whole width, and flash prefill loops only over
the chunks in use.
- **One plan per call:** each call picks straight decode, the hybrid
loop or flash attention, following a calibration sweep based on what
compiles. The kernels might change once the compiler evolves.
- **Several chips:** each chip runs the kernels over its own heads
inside a manual computation.
avoid `stdx.BoundedArray` for tags, use `[]const Tag`, this allows to
simplify some logic and more explicitly separate comptime work from
runtime work.
It reduces binary sizes, preventing big functions to be duplicated
because of different tags.
we often use `withTags` to document the current shape, but it could also
hide bug by overriding previous tags.
I introduce `assertTags` that should be favored when possible
The name can appear in the IR and thanks to #1003 it shows in the web
view of the model making all compute blocks pretty obvious.
The screenshot below without names would be quite confusing,
but now we can see this is the QKV of first layer of a Llama model.
<img width="2211" height="1208" alt="image"
src="https://github.com/user-attachments/assets/1917edb0-a772-4a1d-9aa7-ad5aaf7b5260"
/>
Accept a logical mesh and typed input/output partition specs at manual
computation boundaries.
Will replace `zml.ops.manualComputation` in a future PR.
Ordinary graph results should have an unspecified layout until
constrained or inferred by the partitioner.
Previously, we could report replication simply because a tensor had no
mesh, even when its stored partition specs were open. Other paths
preserved selected operations’ sharding metadata, making the frontend’s
interpretation depend on how a tensor was produced.
This change gives those results a consistent open state while requiring
concrete layouts at executable boundaries.
Shardy explicitly defines the intended semantics: [“no sharding
attribute on a tensor is equivalent to a fully open tensor
sharding.”](https://openxla.org/shardy/sharding_representation#open)
Open dimensions remain available for further partitioning; closed
dimensions fix the specified layout.
Replication is therefore a constraint, whereas an open layout lets the
partitioner choose either sharding or replication:
This distinction also applies to unused mesh axes. Marking an axis
explicitly replicated prevents propagation from using it, even when a
tensor dimension is open. For example, `[{?}, {}]` must leave unused
axes available; adding replicated={"data", "model"} removes those
choices.
Keeping intermediates open lets the partitioner use constraints from
producers and consumers, with operation-specific rules for
transformations such as dot and reshape.
ZML initializes runfiles before `main` so its loaders can locate
packaged libraries and data.
When using `-fno-llvm -fincremental` and linking with the incremental
ELF linker, declaring the registration pointer in Zig produces incorrect
`.init_array` data.
See https://codeberg.org/ziglang/zig/issues/37453
Use a C constructor on Linux and macOS so the C compiler emits the
platform-specific startup registration, while Zig exports the existing
runfiles initialization function directly.
We can revert when https://codeberg.org/ziglang/zig/issues/37453 is
fixed.
With this fix, we bypass the wrong behavior in incremental ELF linker
and get closer to ZML compliance with zig incremental using their
incremental linker.
The registration is shared by the `use_cc_common_link=true` and `false`
paths. Keep the explicit linker retention so the constructor is included
even when its object is in a static archive.
Add helpers for TMA gather/prefetch, weight-stationary MMA,
shared-memory descriptors, packed FP32 arithmetic, async cluster stores,
and register limits.
Make CTA groups and barrier scopes explicit across TMA, block-scaled
MMA, TMEM, and synchronization, including two-CTA operations and
multicast completion. Consolidate named CTA barriers and update the MoE
and paged-attention callers.
Cover instruction operands, descriptor encoding, and MLIR round-trips in
builder tests.
This PR port the CuTe MoE for nvfp4 x nvfp4 from cutlass flashinfer into
zml. It uses the CuTe zig DSL. It is the first PR of a serie that aim to
replace the covering of flashinfer cutlass in ZML. The final goal is to
remove the flashinfer cutlass binding. It begins by removing the binding
of the bf16 kernel on sm90 (never used, we have triton).
Mistral Large 4's Tekken tokenizer uses positive lookahead to split
number runs into groups of three from the right. IREE's regex compiler
rejects this pattern, preventing the tokenizer from loading.
Patch IREE to recognize the exact isolated Split pattern, match complete
number runs with its DFA, and emit the original groups using Unicode
codepoint counts. Preserve progress when output fills during streaming
and finalization. Register the patch in ZML's IREE repository rule.
Treat RESOURCE_EXHAUSTED as normal encoder backpressure instead of
logging errors before retrying. Add regressions for numeric token IDs,
Unicode numbers, byte-at-a-time input, and output overflow.
- Turns preallocation back on by default on Metal, as before #979: not
preallocating seems to cause memory issues we haven't looked into yet.
- Bumps the PJRT plugins to `202610290951.309.1.785550677e75`.
Moves Metal paged attention from the `zml$paged_attn` custom call to
Metal DSL ports of the same kernels.
- `fa_vec_paged`, with a split-K reduce, handles causal bf16 decode.
`paged_attention_tiled` and `zero_padding_rows` handle everything else.
- The shared paged attention test now runs on Metal. Triton counts as
available only on CUDA, ROCm and oneAPI. StableHLO is the reference on
Metal for the causal case.
Adds a Zig DSL for Metal kernels over the AIR dialect, and moves the
Metal paged attention kernels onto it.
- `mlir/dialects/air`: a `thread` address space. Each op has an
`llvmBuilder` that calls its `air.*` builtin with Apple's attributes,
and the dialect's translation interface builds the `!air.kernel`
metadata, so lowering to LLVM IR is MLIR's own translation. The verifier
rejects what Apple's compiler can't take: bool, scalable and n-D
vectors, buffers of non-Metal scalars, scalar f32 converts, and
`air.kernel` on a declaration.
- `dsl/metal`: an MSL-shaped builder. It emits `air` plus the
`arith`/`cf`/`vector`/`llvm` IR that clang produces for MSL, and covers
MSL's math, SIMD-group, barrier and `simdgroup_matrix` functions.
- `zml.kernel.metal` / `ops.metal`: wrap a kernel into a
`__gpu$xla.gpu.air` custom call. XLA lowers it, compiles it to a
metallib and launches it.
Enable two flags for ROCm:
- multi output fusion: allows to search for fusion opportunity
generating different output.
- latency hiding scheduler: overlap async communication with
computation.
Bump to resolve a compiler bug encountered on LFM3.
LFM3 broadcasts one sigmoid gate per attention head. This generated code
of sigmoid introduced a BF16 conversion and conditional select. The
compiler then lower it to a scalar computation which had a
register-aligment bug. The fix introduced in 2.41.11
<5d92530a07>
enforces alignment.
The DSLs and the kernels no longer make ops or set dialect attributes by
name: each dialect package gets the builder or helper they were missing,
and every consumer goes through it. This now fix ZML core files as well
Makes kernels written in the DSLs read alike, without changing what they
emit.
- Every DSL target is `//dsl/<name>:builder`, and `dsl/common` is
imported as common.
- Scopes without values end with `close()`.
- The scf DSLs (triton, mosaic, cute, fly) share
`openFor`/`openIf`/`openIfElse`/`openWhile` in `dsl/common`. cuda_tile
keeps its own scopes with the same shapes.
- Scopes take tuples or arrays, and result types can be dtypes (.f32),
Values, or MLIR types.
- Typed constants are `constant(dtype, value)`, alongside constIndex and
constantLike.
- Comparisons and bitcasts have the same names in every DSL: Fly gains
`lt`/`eq`/…, and every DSL spells it `value.bitCast(dtype)`.
- One block stack and one `common.verify`, which logs the module that
fails to verify.
- READMEs updated.
By default rocprofiler log level is `warning`, which can produce
unnecessary log output. Now, we override this log level to error, unless
`ROCPROFILER_LOG_LEVEL` environment variable is already set.
zml: add TCL kernels and Furiosa attention and MoE backends
Introduce a typed TCL kernel DSL and MLIR bindings, and select TCL
automatically for Furiosa paged attention and BF16 SwiGLU MoE.
- Support native kernel calls, fusion, in-place outputs, device loops,
compiler configuration, and chip-aware layouts.
- Implement standard paged attention for decode, prefill, and mixed
batches, including causal/non-causal masks, sliding windows, custom
scales, and per-head sinks.
- Use BF16 contractions with FP32 normalization and accumulation.
Exclude unused KV entries before contractions and return zero for
inactive or fully masked queries.
- Reuse prepared metadata for the optimized decode path and process
longer contexts in page-aligned chunks.
- Add activation-stationary and blockwise fused-expert kernels.
- Fix StableHLO attention metadata for variable batch sizes and add
host-reference, emission, and hardware regression coverage.
- Bump the Furiosa sandbox to 202610221403.283.1.d9f4f68759e1.
---------
Co-authored-by: Raphael Nahoum <raphael@zml.ai>
Adds Apple's AIR dialect, for writing Metal kernels in MLIR.
- `mlir/dialects/air`: the AIR that Apple's Metal compiler emits for
MSL. Each op is named after its `air.*` intrinsic without the type
suffix (`fast_exp`, `max.u`, `simd_shuffle_xor`, `wg.barrier`,
`simdgroup_matrix_8x8_load`, ...), with fastmath flags on float ops.
Kernel arguments are described the way that compiler describes them:
builtins, buffers with access/noalias, and `air.arg_name`.
- `llvm`, `arith`: bind array and named struct types, linkage, loop
annotations, and integer overflow flags.
Splits the monolithic mlir/dialects library into one Bazel package per
dialect, then makes every package follow the same layout, C API and Zig
API. Each package owns its Zig bindings, its C API when upstream doesn't
provide one, and its tests. Dialect registration moves out of the core
mlir package, so a target only pulls in the dialects it actually uses.
The conventions are documented in `mlir/dialects/README.md`, with a
porting checklist and conformance checks.
Along the way:
- cute_ir is flattened, and its Cuda, CuteNVGPU and NVVM sub-dialects
become their own packages
- fly's ROCDL extension moves to fly_rocdl
- new packages for ROCDL and UB
- kernels/ is renamed to dsl/, since it holds the DSLs
- every package registers through its own registerDialect, and each DSL
through registerDialects, instead of dialect name strings
- the C and C++ follow the LLVM and MLIR C API conventions, formatted
with a new mlir/.clang-format
- BUILD files share one template
- accessors are named as nouns like in the rest of ZML
- each package tests itself in its own test.zig, including that every
bound operation is registered and that every wrapper method compiles
Fixes:
- memref atomic_rmw emitted the wrong kind, because the AtomicRMWKind
values didn't match the pinned LLVM
- the pinned Triton and Mosaic no longer have tt.cat and tpu.repeat, so
the DSL's cat and repeat now lower the way tl.cat and pltpu.repeat do
- the ROCDL bindings are completed: buffer address spaces, vector types,
waves_per_eu range
Breaking changes:
- register dialects through their package instead of
DialectHandle.fromString, and mlir.registerFuncExtensions becomes
func.registerExtensions
- accessors drop the get prefix, so getValue() becomes value()
- ttir.cat and mosaic_tpu.repeat are removed, and stablehlo pad and
mosaic_tpu sort take their options before the location
- zml.kernel.fly.rocdl becomes fly_rocdl
- Mesh.sdyMeshAttr takes a context and returns an attribute instead of
text
- func.func no longer prints public, its default visibility
---------
Co-authored-by: Steeve Morin <steeve@zml.ai>
Drafter scheduling in speculative decoding
(https://github.com/zml/monorepo/pull/685) can create empty slots with
empty kv content in full-attn layers of the drafter.
This fixes this issue by aligning the TPU logic with other platforms (eg
Triton) by avoiding to request DMA transfers of KV buffers for invalid
slots.
We fix a bug introduced in https://github.com/zml/zml/pull/925: the
output shapes passed to the local callback were left in their global
state and not translated any more to the local state.
Move sparse GQA into an standalone kernel rather than maintaining it as
a separate branch of the unified attention kernel. This separation
improves maintainability and keeps the unified attention kernel more
manageable, as previous performance optimizations had significantly
increased its complexity.
To reviewer: there's remaining work to merge the oneapi kernel into
unified attention and clean the kernels/kernels_sparse import.
The LM head normalized and projected every prompt position at prefill,
and the session read back the whole token buffer just to keep the last
token.
Every model now slices the hidden state at `active_length - 1` before
the norm, so prefill computes logits for a single position and returns a
single token. Decode writes the next token over the one it read, so it
allocates no buffer per step.
lfm2's `actual_seq_len` is renamed `active_length`, the name the other
models use.
Make KV-cache strides compile-time constants so Triton can infer
alignment and simplify address calculations.
On Qwen3.5-35B-A3B, both 2D and 3D attention kernels now use 128-bit
vector loads for K/V, fetching eight bf16 values per instruction per
active thread instead of one.
Thanks @fior512 for your contribution 🙏
---------
Co-authored-by: fior512 <161968556+fior512@users.noreply.github.com>
Assign blocks for multiple experts in parallel instead of processing
experts one at a time. Each lane writes the expert ids for its own block
range.
Reduces assignment cost from 7.6 us to 1.8 us per call.
This PR is the ground work that adds a `cute` backend for paged
attention.
It's not use by default yet because it's a very naive implementation
that is far from performant.
This PR also includes two targets that are useful when working on the
kernels.
The first one is `//zml/kernels/paged_attention/cute:correctness_test`.
As its name suggest, it's a correctness test that ensures the
computation done is correct and returns valid results. It's verified
against a CPU reference implementation (that is also checked against a
hardcoded truth). It relies on `Case` representing various real-life
scenarios.
The second one is `//zml/kernels/paged_attention/cute:benchmark`. It's
an executable that can be run and will output execution stats for
various `Case`. This is the target to use when working on the
performance of paged attention cute kernels.
The `@patchelf` version 0.18.0 is more capable and can now construct
something like
```bash
patchelf \
--add-needed somelib.so \
--force-rpath \
--set-rpath '/home/kevin/sdk/lib' \
--rename-dynamic-symbols patched_tcc.symbols \
--output patched_tcc/furiosa-tcc \
somebinary
```
So now we do not need to spawn a shell and do all the cp, chmod, and the
successive patchelf command
Before
```bash
remove_rpath = ["some/directory"]
```
`--remove-rpath` is a flag that `Removes the entire embedded search
path` and not an individual directory
will contruct something like `patchelf --remove-rpath some/directory
patched.so` but `--remove-rpath` do not take arguments
Switch to `remove_rpath = True` to ensure something like `patchelf
--remove-rpath --output patched.so original.so` the `--remove-rpath` is
now used just after the patchelf to ensure correct usage flag placement.
`DataType.Value.as` converts a tagged-union `Value` to a native Zig type. The float branch of the inline switch lists which float dtypes it knows how to convert via `.toF32()`, but that list was never updated
when f4e2m1, f8e3m4, f8e4m3 (the plain variant, distinct from f8e4m3b11fnuz/fn/fnuz), and f8e8m0 were added to `DataType`. All four already implement `.toF32()` in zml/floats.zig, so calling `.as(f32)` on
a `Value` of any of these dtypes panics with "Unsupported conversion ...
-> f32" instead of returning the converted float.
The fix is a one-line addition of the four dtypes to the existing switch arm, the same pattern
as the other float dtypes already listed there.
Akin to #926, this PR introduces a restructuration of the moe kernels in
ZML.
No need to re-explaing the organizatiopn because it follows what has
been done in the mentioned PR.
Here is the detailed changes of this PR:
* Kernels reorganization. `zml/kernels/moe` is the place to put all the
moe implementation. There is one sub-folder per backend.
* Changed the expected shape and tags when calling the moe forward
function. It's now expecting a 2D tensor tagged `.{.t , .d}`.
Tests still passes without key modifications. The only small tweak that
was done is related to fp32 -> fp8 triton quantization on sm89. Due to
how triton lowers the convert, it was producing slightly incorrect
result. A small preparation was added to ensure the conversion is
identical on this platform.
**Future TODO**:
* Ideally, this conversion on sm89 should be gated so that we don't pay
the cost on other platforms. We don't have a way to detect the platform
in the triton `Builder` yet but it could probably be added
* Settle on a complete API that encodes all the scenario we could
encounter. One example of that is what is required for Deepseek V4.1
Flash, the moe gemms should be done in `mxfp8 x mxfp4` and we should
have a way to request that in a better way than `quantize_input: bool`.
CUDA 13.4.2 ships PTAX 13.4.92, which miscompiles a valid atomic CAS in
some LLM models, causing crashses. The generated code uses a different
byte offset for the load and atomic update, so the two operations access
different memory locations.
## Motivation
This is the big sharding rewrite we need to achieve complex sharding.
Currently we always resolve a mesh logical axis to a mesh physical axis
(or several of them).
But on GPU we always have 1D topology, so we can't have 2D logical
meshes.
So this PR drops the logical -> physical axis bindings, and instead
store a logical -> devices binding.
This is more flexible, and has the nice side effect of simplifying
`Buffer.from`
On stablehlo side this changes how we generate the MLIR because we now
use logical meshes, and `manualComputation` uses logical axes.
(In shardy, GSPM is not affected AFAICT because it uses device ID
directly)
## Buffer.from
This also accelerates `Buffer.from` that I rewrote to be FAST.
Version | Fixed cost | Per device
-- | -- | --
origin/master | 109.6 ns | 35.39 ns
Rewrite | 35.8 ns | 4.38 ns
Improvement | 3.1× | 8.1×
## Creating mesh
I'm adding a new syntax to create LogicalMesh, to allow the more complex
meshes we want to do, and in particular, I now allow to force a specific
number of dimensions.
You can see it in action in `mnist.zig`.
```
const dp_mp = try platform.registerMesh("DP-MP", try .init(
.{ .data, .model },
&.{ .low_bandwidth, .high_bandwidth },
&.{ 2, -1 },
));
info: topology used: Mesh(name=DP-MP, partitions=4, replicas=1)
┌ Mappings (logical coordinates from physical coordinates):
├── data (low_bandwidth) -> (bus / 2)
└── model (high_bandwidth) -> (bus % 2)
┌ PhysicalMesh(platform=cpu shardable_axes={ .bus } num_devices=4)
└── [bus] 0..4 (tree)
├── Device(id=0, coords={ 0 })
├── Device(id=1, coords={ 1 })
├── Device(id=2, coords={ 2 })
└── Device(id=3, coords={ 3 })
┌ Logical coordinates:
└── data=0
└── model=0,1 -> devices=0,1
└── data=1
└── model=0,1 -> devices=2,3
```
The previous syntax is still available, and ZML will align logical axes
to physical axes like we do before this PR.
This part as unit tests, and I checked it work as I expect on CUDA but I
would need to double check the logics still works on TPU/Neuron, because
there is a few platform specific branches in that code.
## Test plan
As part of the benchmark Codex generated a test suite with hard-coded
offsets, allowing to check that Buffer.from didn't change behavior
through the commits, checking we correctly compute each individual
shard.
TODO check that LLMD perf isn't degraded.
here I'm promoting a pattern I would like to extend to more of ZML:
function that takes `anytype` argument should do as little work as
possible and called a fully typed function.
I kept `manualComputation` signature to not break existing code,
I moved the code to `zml/ops/manual_computation.zig` to better
encapsulate the required helper functions.
(and `zml/ops.zig` is pretty big atm)
Similarly `Sharding.Partitioning.parse` now calls `const logical =
comptime Logical.parse(partitioning);` then `logical.applyUseAll(tags,
mesh, default_value);` which is fully typed.
I'm also documenting `manualComputation` better
Update XLA plugins to release 202610151602.227.1.e215eb02db73
for CPU, CUDA, ROCm, Metal, and Furiosa.
Updated CUDA to 13.4.3 and ROCm with TheRock 10.1.
Now we use CUDA compatibility libraries and NCCL 2.32.3 through NVIDIA's
APT repositories. Starting with CUDA 13.4's redistribution manifest
omits the generic cuda_compat package
(https://docs.nvidia.com/deploy/cuda-compatibility/why-cuda-compatibility.html).
So we resolve it through NVIDIA's APT repositories instead.
These repositories expose flat package indexes at separate x86_64
and SBSA URLs, so we extend rules_distroless to support flat sources.
For NCCL we leverage the same APT source workflow, replacing
separate cryptic wheel URLs.
Staging blocks are uninitialized `.host_pinned` buffers, and tensors are
sent with the direct writer. The plugin backs `.host_pinned` with SDK
pinned (dma-buf) memory, which the SDK recognizes by address and
transfers without pinning pages on every request.
Pinned memory comes from `/dev/dma_heap/system`, so the loading user
needs access to it.
Fast-path inside the forward of
`GatedDeltaNet` in order to reduce syncs in decode path.
Master vs this commit:
- tok/s : 167 to 204
- cuStreamSynchronize call/tok : 37.24 to 1.24
- GPU idle/tok : 3504 us to 2240 us
We align TPU and NKI attention meshes expectation on the CuTe GPU
attention, which simply resolves the mesh from the logical axis.
This avoids expecting a hard-coded `model` mesh, which breaks in some
TPU workflows where the mesh is called `tp_mesh`.
This PR introduces a restructuration of the kernels in ZML. This is done
to simplify and speedup bringup on new platforms.
From now on each "kernel" will have its own directory under
`zml/kernels`. "Kernel" is in quote because we need to think about it
being more of an "operation in a graph" than an actual kernel. For
example, `moe` will eventually be part of this `kernels` directory
despite being an operation potentially launching multiple actual kernels
on some hardware.
Also it's probable that in the future, this `kernels` directory will be
put outside of zml or, at least, properly separated so that we don't
have circular imports in the codebase. Currently, we have
`@import("../../../zml.zig")` in the kernel code and this is a recipe
for disaster.
On top of the reorganization, it also adds a level of indirection for
attention. Where previously `attention` was a generic term, we now have
`attention` and `flash_attention` because `flash_attention` is an
implementation of `attention`.
Here is a detailed summary of the changes introduced by this PR:
* Kernels reorganization. `flash_attention` and `paged_attention` are
now separated. Each folder has a subfolder for each backend we support.
`flash_attention` is also now a `BackendType` of `zml.attention` with
its own implementations.
* Removed `cuda_fa2` and `cuda_fa3` backend because they were obsolete.
They have been replaced by a `cute` backend that has been generated
automatically following the operation documentation, showing that this
approach of scoping kernels per DSL actually scales.
* Removed `attnd` support for now
Fix a memory leak in Qwen 3.5 Dense and MoE examples:
`layer_index` was never freed.
Before (74562d5) / After
Dense 0.8B VRAM : 11079 MB vs 5461 MB
Dense 0.8B tok/s : 169 vs 198
tok/s increases because it reduces `cuMemCreate` calls from 26 to 2
per tokens
This PR adds tests to the MoE backend. The comparison is done using a
StableHLO backend that has been tested against an oracle with known
value.
It's adding quite some time to the test so I don't know if we want to
keep that many tests. However, this is the only way we have to ensure
compliance across all backends and configuration.
Each backend is tested with the whole matrix it supports.
A nice side-effect of this PR is that we have a working implementation
of MoE that will work on all platforms. It's going to be slower but
allows for a quick platform bringup.
`Buffer` placed in `.host_pinned` memory can be read by executables,
but the host can't address it. `Buffer.HostAccessible` gives the host
typed views over its allocations, to write inputs and read outputs in
place between executions, without staging copies. Like `Buffer`, it is
a value: executions read its `buffer` field or donate it through
`&buffer`, and `await` then picks up the handles they replaced.
- `init` takes a `Buffer.Donation`. On CPU, CUDA and ROCm, the devices
holding a shard of an `.undonatable` buffer share one allocation: the
first allocates it and the others read it through
`createViewOfDeviceBuffer`, which now takes the memory the view
belongs to; an external reference keeps the allocation alive.
Elsewhere each device owns an allocation, and writes reach all of
them. Each device owns its allocation of a `.donatable` buffer, so
that no device reads another's output.
- `View(T)` addresses a row-major range of the array: `slice`, `get`,
`set`, `fill`, `fillIota`, `copyFrom`, `copyTo`, `copyFromView`,
`indexOfScalar`, `indexOfAny`, and `blocks`/`reverseBlocks`, which
visit runs contiguous both in memory and in global order. Reads use
the first replica of a region, and writes reach every replica.
`constSlice` and `edit` expose a range as one slice: in place when
every shard holds the whole array densely, through a scratch slice
otherwise.
Addressing follows PJRT's layouts: row-major, strided, and tiled as
XLA's LayoutUtil::LinearIndexForNestedTiling expands them, including
TPU's `T(8,128)(2,1)` packing of bf16 and 8-bit arrays. The test
reference is a direct port of that function, checked against both
figures of https://openxla.org/xla/tiled_layout. Buffers whose shards
all hold the whole array densely, the case on every platform but TPU,
index elements directly.
The buffer is built for llmd's decode loop, which touches it thousands
of times per step:
- Reading a host pointer from PJRT takes and releases a hold on the
buffer and allocates an external reference, about 0.9 us. Pointers
are read at creation, and by `await` only when execution replaced the
handles; `view` panics before that `await` instead of writing to a
released allocation.
- The layout metadata and allocations take several KB. They are
prepared once in a heap-allocated mapping, which views (24 B) and
blocks borrow rather than copy.
On Qwen3.8-27B in llmd (TP=2 on GB300, batch 256, 8k context), this
takes host time between graph completion and the next Execute from
2.07 ms, when views copied the layout and donated buffers read their
pointers in every view, to 0.27-0.30 ms per step (bpftrace probes).
`initForTests` builds a buffer over host memory, for the tiled layouts
and shardings that no available platform produces. `Buffer.from` and
`Buffer.uninitialized` share the new `emptyShell` with
`HostAccessible.init` to compute the PJRT shape, placement and layout.
Due to recent changes in XLA (introduced in
3cf18a01c7), the way child are
counted to decide if a conditional is expressed as a graph conditional
or kept outside has been changed.
With the new corrected way, the body of the conditional used in the
paged attention StableHLO backend was recorded as a child graph. The
problem is that when we are sharded, there are collectives inside this
child graph and XLA adds event wait/record around the actual collective
kernel. This is explicitly forbidden because
https://docs.nvidia.com/cuda/cuda-programming-guide/04-special-topics/cuda-graphs.html#conditional-node-body-graph-requirements
states:
> The graph can only contain kernel nodes, empty nodes, memcpy nodes,
memset nodes, child graph nodes, and conditional nodes.
The temporary solution is to remove `CONDITIONAL` from the
`xla_gpu_enable_command_buffer` XLA flag. In practice this reverts to
the old behavior we had when the counting was done differently.
Long term, I don't have a solution yet and. There is an environment
variable that solves it
[NCCL_GRAPH_MIXING_SUPPORT](https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/env.html#nccl-graph-mixing-support)
but I don't know if it's safe to disable it.
Maybe we could also check if the events introduced by NCCL are actually
required.
Adding `extra_device_memory_bytes` to the compilation options to have
XLA compute the accurate amount of memory left for the temporaries with
our engram hack.
Follow-up to https://github.com/zml/zml/pull/888:
We introduce a new `.standard` entry to `AttentionMask` enum, that
accepts both flags `causal` and `sliding_window`. This makes the
interface more flexible, to allow cases with both non-causal _and_
sliding_window options.
In Buffer code we already have explicit sharding, but not for tensor.
I'm adding a `zml.Sharding` to `.withPartitioning` to make it explicit
which mesh we are working with.
To simplify the migration, `.withPartitioning` in Tensor code can use
`.model` syntax to lookup a sharding by name.
This replace the previous logic of `selectSharding(shape)` which would
look up a sharding that matches the axes of the given partitioning which
could yield surprising behavior.
This also modifies how sharding are stored in zml.Shape.
Before it was storing a list of axis names + a tag (2 * 8 * 64bits), now
I'm storing a zml.Sharding (64 bits) and a compact PartitionArray (32
bits) that references the logical axis of the sharding by offset instead
of by name.
There is an impact on the `TensorRegistry` because it also needs to be
aware of the possible shardings, instead of delegating the matching
partition spec -> sharding to loading. This means error will be detected
sooner.
`store.createTensor` accepts `@enumLiteral()` to lookup the relevant
sharding.
This also simplifies `zml.Buffer` because the full sharding is now
stored in the shape, so it doesn't need to be passed independently. It
also means that it will propagate from the tensor who are graph inputs,
to the buffers that are eg inputs.
`Placement` logic also got a bit simpler.
Replace `is_causal` and `sliding_window` fields in `AttentionOptions`
with a tagged union. This prepares the API for a subsequent commit that
will introduce sparse attention mask. A tagged union makes these
mutually exclusive behaviour explicit.
This PR removes a lot of slop in the MoE part of ZML, as well as some
global improvements.
Among the improvements:
* Unify activation functions and parameters across backends
* Unify activation computation across backends
* Replace error bubbling with explicit asserts (because we have no
meanignful way to handle them)
* Add a StableHLO MoE backend
Record donated input indices during compilation and destroy their
PJRT_Buffer handles once execution is enqueued for `zml.Exe.Runner`.
I'm not doing this automatically in `call()` because it's a bit
invasive. Not all the code uses the pattern of overwriting an input
buffer like `run(.{cache}, .{&cache})`, but most do. So I'm cleaning it
up on easy path and leave `call` as is for finer control.
Removing callOpts & runOpts to simplify the API and changing `io` to be
required for API clarity. I'm replacing `zml.testing.autoCall` with
`Exe.eval` which solves most needs in a simpler way IMHO
DirectShardWriter.init treated retrieveBuffer errors as unreachable and
leaked its transfer manager and buffer when a later step failed.
DirectMemoryWriter.init counted a shard writer as initialized even when
its init failed, so its errdefer deinitialized an undefined writer.
Allows placing all the threads/memory on the right numa nodes. It's
faster for a device to access memory on the same numa node and overall
the it's better to have all threads on the same numa node if all devices
are on the same one.
NVPTXAddressSpace is NVPTX specific. In LLVM, addressSpace is defined
for each target (nvptx, amdgpu)
The pointerType is general in llvm dialect by accepting an anytype.
- Implement the flashmla kernel for sm100 using the cute dsl in zig.
- Extend the cute dsl with specialized sm100 (tcgen05) ops.
- Introduces a new cute backend which calls the cute kernel.
- Uses a new cache for MLA (latent or quantized).
Even if `exec_compatible_with = ["@platforms//os:macos"],` is removed
from `platforms/metal/BUILD.bazel`, fetching the pinned Metal toolchain
on Linux currently fails:
```
ERROR: /home/oboulant/.cache/bazel/_bazel_oboulant/4ea653214b23f6728bdb9f5b40de2f50/external/zml+/bazel/http_aea_archive.bzl:20:13: An error occurred during the fetch of repository 'zml++http_aea_archive+metal_toolchain':
Traceback (most recent call last):
File "/home/oboulant/.cache/bazel/_bazel_oboulant/4ea653214b23f6728bdb9f5b40de2f50/external/zml+/bazel/http_aea_archive.bzl", line 43, column 19, in _http_aea_archive_impl
_require_macos(rctx)
File "/home/oboulant/.cache/bazel/_bazel_oboulant/4ea653214b23f6728bdb9f5b40de2f50/external/zml+/bazel/http_aea_archive.bzl", line 20, column 13, in _require_macos
fail("http_aea_archive is macOS-only; current repository OS is %r" % rctx.os.name)
Error in fail: http_aea_archive is macOS-only; current repository OS is "linux"
```
Add checksum-pinned `ipsw` and `7-Zip` binaries for Linux x86_64.
`bazel/http_aea_archive.bzl` now selects the right binaries according to
running host. It preserves the existing macOS extraction path.
Linux ARM64 remains uncovered.
Validation:
```bash
bazel build \
# selects compiler binaries from the extracted external repository \
# filegroup defined in platforms/metal/metal_toolchain.BUILD.bazel \
@metal_toolchain//:metal_compile_tools \
# selects the compiler’s runtime files \
# filegroup also defined in platforms/metal/metal_toolchain.BUILD.bazel \
@metal_toolchain//:metal_compile_runtime \
# copies compiler binaries and runtime files into the layout consumed by ZML \
//platforms/metal:metal_toolchain \
# selects macOS ARM64 as the build target \
--platforms=//platforms:macos_arm64 \
--lockfile_mode=off
```
---------
Co-authored-by: Corentin Kerisit <corentin.kerisit@gmail.com>
The error it fixes:
```bash
bazel build //llmd:upload_sentry_symbols_linux_amd64 //llmd:upload_sentry_symbols_linux_arm64 --config=remote
...
ERROR: no such package '@@zml++cuda_packages+llvm-libunwind1_linux_arm64//': java.io.IOException: Error downloading [https://apt.llvm.org/bookworm/pool/main/l/llvm-toolchain-23/llvm-libunwind1_23.1.2~++20260916013112+21a77e7bb6bf-1~exp1~20260916133123.74_arm64.deb] to /Users/oboulant/Library/Caches/bazel/_bazel_oboulant/50730f1e2ab11dae9d7f564452062ab5/external/zml++cuda_packages+llvm-libunwind1_linux_arm64/.downloaded.llvm-libunwind1_23.1.2~++20260916013112+21a77e7bb6bf-1~exp1~20260916133123.74_arm64.deb: GET returned 404 Not Found
...
```
The `platforms/cuda/packages.lock.json` was updated with `bazel run
@apt_cuda//:lock`.
This PR adds support for models downloaded via `hf download` without the
`--local-dir` flag.
This is done by keeping the given path instead of getting it from the
`std.Io.File`. The problem with `File.realpath` is that in the case of
models downloaded from HF, the files points to blobs whose name are hex
strings, thus losing the required file extension. Unfortunately, we rely
on the file extension to detect the kind of repository the path
represents.
I also added `//examples/io:playground` to the `//:completion` target so
that we have completion when working on it.
The changed code needs to round up the scale to the next power of two.
The math is:
```
(bits + 0x7fffff) & 0xff800000
```
Since 0x7fffff is \(2^{23}-1\):
- When \(m=0\), adding it fills the fraction bits without carrying into
the exponent.
- When \(m>0\), the addition carries exactly once into the exponent.
- The mask then clears all 23 fraction bits.
Too many problems for now in XLA:
- https://github.com/zml/zml/pull/863
- a bf16 dot failed
We have fixes incoming for those, but unclear whether it's enough
Refactoring of the MoE selection and add repacking contract required by
the model inits.
This solve a design problem in the model init :
- Before, backend selection required the resulting MoE after repacking
(dtype), and repacking required to know the selected backend.
- Now, backend selection is based on raw weights loaded from HF
(quantization scheme and dtype). Then it induces the packing.
Add useful structs for packing that are specific to MoE
(`ExpertsLayout`).
Introduces the CuTe mxfp4 moe backend. The kernels use the cute dsl and
are optimized (and so at the moment constraint to its shapes).
Introduces the cute launch host func and so the associated cuda dialect
in zml.
The CuTe dialects no longer build on a git checkout with a private
overlay.
- Every operation, type, attribute and enum of the compiler's cute and
cute_nvgpu dialects is typed, with its documentation, operand
constraints, traits and result inference. Nothing is kept as an opaque
payload, and the compiler's IR round-trips through it unchanged.
- The definitions are generated following `REGENERATING.md`, from the
release's Python bindings and by probing the compiler.
- The C API and the Zig builders follow: atom types take their
parameters instead of a text payload, constrained integers are one type,
and pointers take an address space enum.
Support normalization from HF JSON and typed normalizer sequences.
Changes:
- Patch IREE memory ownership and chunk-boundary bugs
- Add regression tests for IREE bugs.
Add explicit debug info in ZML.
Model author can now add `pushLocation` to mark specific sections of the
IR
---------
Co-authored-by: Guillaume Wenzek <gwenzek@users.noreply.github.com>
Update CPU, CUDA, Metal, and ROCm plugin references to release
202609221707.75.1.0cf3c5a71e94.
Default ROCm to one HIP graph stream and one hardware queue per device
to reduce synchronization overhead and throughput variability. Disable
rocprofiler queue interposition by default, while preserving explicit
environment overrides.
Restrict collective command-buffer capture to all-reduce and all-gather.
Other collective types execute outside HIP graphs.
With SKIP_TEMP, captured NCCL collectives can retain stale physical
mappings if VMM reclaims their backing under memory pressure, causing
silent corruption.
Enable user buffers to assign color 1 and receive the reclaim exemption:
https://github.com/openxla/xla/pull/46029
This enables runtime selection of the std.Io runtime. This also fixes
the compilation hangs by using zio.blockInPlace for CPU bound ops.
This may require some more `zio.blockInPlace`, so don't merge until
we're sure.
* `xla_gpu_experimental_use_collective_kernels=CONCURRENT` allows
parallel execution
* `xla_gpu_experimental_use_collective_kernels=SKIP_TEMP` avoids graph
updates
* `xla_gpu_experimental_use_collective_kernels` disables custom
collectives while we are looking for a fix
* `xla_gpu_enable_command_buffer` adds collectives to the graphs
in LTO, the linker would garbage collect that exported symbol since it
wouldn't be used inside the final binary
forcing it undefined via a link flag prevents it from being removed.
Allow block-32 scales to accompany E4M3FNUZ weights after load-time
normalization on CDNA3. The tensor layout and quantization scheme stay
unchanged, so existing Linear paths can use native FP8 conversion
without materializing software-decoded FN weights.
Add a [zio](https://github.com/lalinsky/zio) dependency.
We require ZML to be aware of which `std.Io` implementation it's running
because we use `threadlocal` to store the current `zml.Compiler`
instance when compiling a graph (to make the DSL pleasant to use).
This is our only `threadlocal` usage but it is one too much. So we must
make it compatible with `zio.TaskLocal`. Not a big deal, but this
requires opt-in from the user.
And so `zio` support can be opt-in enabled by setting the build flag
(default is `std`):
```
--@zml//zml:io_impl=zio
```
Sadly there is no `std.Io` response to `threadlocal` (or task local).
Hopefully this will be fixed in the future.
Note that this creates problem if using `std.Io.Evented`.
Introduce `zig_option` and `zig_options` rules to generate the bazel
equivalent of option modules plugged on bazel flags.
On `string` flags, if the flag has a `allowed_values`, the `string` is
converted into a `enum` in zig code.
Also, port platforms to this and remove the ugly `defines` hack to
comptime test for platforms enablement.
In the process, port `arocc` to the `zig_options`.
Two things, the second one falling out of the first.
**`fp8_block32`**: E4M3FN weights with one scale per 32x32 tile, the
layout
DeepSeek V4.1 ships. `quantizeBlockFp8` takes a block size and a scale
dtype;
the E8M0 path floors amax at 1e-4 and rounds the exponent up, so the
activation
side is MXFP8: one E8M0 scale per row per 32 contracted values. That
pairing is
what XLA's fp8_block_gemv arm needs to hand both scales to
`tt.dot_scaled`
(`Fp8BlockGemvScaleInMma` requires E8M0 on *both* sides); F32 activation
scales
would quietly fall to the per-block multiply. The Triton MoE path gets
the same
scheme at `block_size_k = 32`.
`Linear.forward` also stops rounding the activation to BF16 before
quantizing,
so amax is computed on the input as given.
**An explicit output dtype** on `scaledDot` and `Linear.forward`.
Inference
cannot answer what an FP8 x FP8 product should return: the accumulation
is F32
either way, so BF16 or F32 back is the caller's choice. `forwardScaled`
now
hands the caller's type to `scaledDot` rather than rounding through BF16
and
converting after. Narrow outputs are rejected.
Separately, `Linear.forward` converts the weight and bias to the
activation
dtype. `dotGeneral` asserts both operands share one, which broke the
moment a
model raised its activation to F32 against a BF16 weight; the wrappers
this API
replaced converted for exactly that reason.
**Breaking:** `Linear.forward` takes a second argument and `scaledDot` a
fifth.
---------
Co-authored-by: Raphael Nahoum <raphael@zml.ai>
`Buffer.createView` can be used to create a view from a host pinned
buffer and use it as if it was one the device. This avoids XLA
limitations of doing operations directly on host pinned buffers.
This only works on OneAPI and Cuda for now.
For other platform we would need a writer that allocates host pinned
uninitialized buffers for the shards and write directly to them.
The CPU plugin archives now ship the shared library under lib/, and
copy_to_directory outputs to sandbox/ instead of sandbox/lib. Look up
the sandbox directory in runfiles and join lib/ when building the
library path, matching metal, rocm and cuda.
Adds a Zig front end for NVIDIA's CuTe DSL, shaped after `cutlass.cute`,
plus the `__gpu$xla.gpu.cute` custom call that compiles and launches it
on CUDA.
- `mlir/dialects/cute_ir`: bindings for the `cute` and `cute_nvgpu`
dialects
- `zml.ops.cute`: emits the custom call with the textual module, `grid`
and `block`.
- `kernels/cute` + `zml.kernel.cute.Kernel`: builder, values, tensors
with coordinate `get`/`set`, thread/block indices, static smem, sync,
`scf`. See `kernels/cute/README.md`.
- bump `xprof-nightly`
---------
Co-authored-by: Steeve Morin <steeve@zml.ai>
Adds a Zig front end for the FlyDSL dialect, so AMD GPU kernels can be
written in Zig alongside the existing Mosaic, Triton and tile ones, and
dispatched the same way.
Types and layouts are built through the dialect's own C entry points
rather than by printing and reparsing assembly. That holds in both
directions: host-side layout literals become attributes directly, and
anything the dialect computes is read back through typed accessors.
Launch geometry is derived from the device rather than hard-coded, so
the same kernels run on both AMD architecture families and the tests
stay architecture agnostic. Everything builds and tests on any host, no
GPU needed.
Also carries one unrelated cleanup, dropping a needless null terminator
from the Triton and tile kernel strings.
## Summary
- `zml-smi --top` was stalling under load because each GPU and the
process list hit NVML on their own thread. Those calls serialize on the
driver RM lock, so one slow ioctl froze the whole TUI.
- All NVIDIA queries now run on a single poller.
`nvmlDeviceGetPcieThroughput` (it blocks while sampling) is only taken
on the first tick and every fourth tick after that; link speed/width
stay on the hot path.
- Process-query sizes above 4096 are ignored, and a failed per-process
util sample no longer skips the rest of that GPU.
---------
Co-authored-by: alhussein jamil <alhussein@zml.ai>
- The current Fp8 scaling wasn't properly finished. We didn't use the
scales at all.
- Adding a `quantize_input` option that GLM5.3-Flash uses
- `applyActivation` wasn't properly using the threshold for relu
- The bias was silently ignored for the MoE.
`makeFusedMoeConfig` was just adding extra complexity and we used `a`,
`b` and `c` everywhere. I think it's fine in the triton kernel itself,
but outside of it's just confusing IMHO.
I removed a bunch of useless parameters also that just end up returning
an `Unsupported` error.
Fix an issue when `padded_num_experts` computed doesn't land as a power of two. Notably for DeepSeek-v4 Pro with its 384 experts, divided by 32 warp is not of power of two, leading to a crash in the triton which expects a power of two number of experts.
Imitating `platform.cuda.computeCapability` as fp8 support is native for
e4m3fnuz on gfx942 but e4m3fn for everything else that has hardware
support AFAIK.
A third kernel language beside Triton and Mosaic TPU. It emits NVIDIA's
cuda_tile dialect and reaches the GPU through XLA's
`__gpu$xla.gpu.cuda_tile` custom call.
- `third_party/cuda_tile`: cuda-tile v13.3.3
- `mlir/dialects/cuda_tile`: every type and enum through the dialect's C
API, one constructor per op (all 100, in Ops.td order), and a small C
shim for the TileView interface the C API does not expose.
- `kernels/cuda_tile`: views, tokens, for/if/loop, reduce/scan (single
and variadic), atomics, MMA, every elementwise op.
- `zml.kernel.cuda_tile.Kernel` and `ops.cudaTile`: the custom call,
with a grid and no num_warps/num_stages (those are optimization_hints in
the IR).
Every CUDA gate used to parse the compute capability string itself. They
now share `platform.cuda.computeCapability`, which returns null off CUDA
and offers `.is` / `.atLeast` / `.sm`.
Also folds the NVFP4 scale clamp into its reduce to remove one launch
per projection, and is equivalent.
The fork keeps upstream's flag defaults, so zml turns on what each
target actually wants:
- the Tile IR scaled-dot backend, subchannel dequantize fusion and
Triton multi-output fusion on CUDA
- Metal AIR fast math off, since it lets the AIR compiler assume no NaNs
and is_finite(NaN) comes back true.
Fixing an issue raised in https://github.com/zml/zml/pull/754, to bound
the GDN recurrence in the Qwen3.5 example model to the number of active
tokens.
We also make qwen3.5 and qwen3.5-moe slightly more consistent.
By default, XLA assumes 80% of the memory to be available during
compilation. It wasn't an issue as long as we had one executable per
layer, because most of the weights weren't parts of the inputs, but for
a full forward executable we exceed that 80% threshold in llmd with the
cache being sized to use almost everything up to the BFC limit. So XLA
ends up generating an executable the least possible temporary buffers
even though it has more than that.
Make unpackFp4 accept the packed source axis explicitly instead of
requiring the hardcoded .kw tag.
Updates the Linear and MoE callers and adds regression coverage for
expanding and renaming the packed axis.
Simplify `manualComputation` by merging `inputs` and `body_context` into a single `input` argument.
Internally, collect all and shard all tensors (including `?Tensor`) from `input` before invoking the body function.
Give each concurrent RunsOn job a unique routing label using the
workflow run ID, run attempt, and job name. This prevents runners
provisioned for one matrix job from being assigned to another compatible
job.
- [Runner
stealing](https://runs-on.com/docs/maintenance/troubleshooting/#runner-stealing)
---
Replace the deprecated image-level `disk: large` setting with
`volume=80gb` on each RunsOn job. Storage belongs to the runner
configuration rather than the custom image definition.
- [RunsOn storage labels](https://runs-on.com/docs/runners/labels/#disk)
- [Custom runners and images](https://runs-on.com/docs/runners/custom/)
A binding created through `maybeCreateBinding` is computed from its
sources by an executable and must never be streamed from the checkpoint.
`Loader.defaultCallback` inferred that from `sources.len != 1`, which a
single-source binding fails resulting to the raw bytes being loaded and
producing wrong results
Model discovery leaves its safetensors entrypoint and repository handles
open after building the tensor registry, including when parsing fails.
Close handles owned by `fromRepo` and `fromPath`, and close repositories
at the existing `fromRepo` call sites.
`forwardMoe` is effectively consuming a `gate_up` and `down` linear MLP,
it "just" only applies parts of it.
I've renamed:
```
Quant -> Quantization
QuantSchem -> Quantization.Scheme
TensorScale -> Quantization.GlobalScale
Direction { multiplier, divisor } -> Operation { multiply, divide }
```
and moved it outside of zml.nn.Linear
This makes it easier in LLMD to just load zml.nn.Linear
`Tensor.triangularSolve` currently rejects batched inputs and derives
the result shape from the coefficient matrix.
Allow tensors with rank 2 or greater and preserve the right-hand side's
shape in the result.
This was found while working on a separate Qwen3.5 GDN optimization.
Tests:
- `bazel test //zml:test`
- `bazel build //zml`
in #738 I accidentally introduced implicit donation `fn foo(x: Tensor)
Tensor { return x; }` would not make a copy of the input tensor but
return the same pointers.
I'm not sure an implicit copies is better than an implicit aliasing, but
fixing since this is an un-planned breaking change.
Also fixing a compiler error for `.items(bool)` after #724.
Note: this is an argument for snapshot testing: the CI should check if a
given PR changes the emitted IR for a given model.
The XLA API assumes that all sub-bytes data is given to it unpacked.
But when you read safetensors files the data is packed.
My proposition is the following: when we have a zml.fp4 tensor/buffer we
say to pjrt it's a u8 buffer instead.
Since PJRT infers the input/output shape from stablehlo I had to
introduce some bitcast at the edge of the compilation:
- fp4 input are declared as u8, but I insert bitcast operand before
calling the zig code, so the model code only see the fp4 tensors.
- when the model code returns a fp4 output, I bitcast to u8 before
exiting the final compilation
Breaking changes: none
Currently fp4 can't cross the pjrt boundary, so all fp4 code already
manually lie about the data being u8, and it will keep working. In the
future this code will be able to be "upgraded" by explicitly tagging the
input as fp4 and remove the "unpacking" code from their model code.
Note: the tracking XLA issue seems pretty cold
https://github.com/openxla/xla/issues/16795
* use multiArrayList in `emitMlirFunc` to store information about
input/output tensors.
* merge OutputInfo and InputInfo: The code is collecting same info for
input/output, some code was merge.
`emitMlirFunc` is now clearly split between the function specific
wrapper, and the part that works only with slices of tensor info.
* zml.Sharding: clarify error sources, added more error messages. Most
`zml.Sharding` function can fail because they resolve the sharding on
the fly. Changed some APIs to explicitly take a resolved sharding,
leaving only OOM errors to handle. I've put a TODO for you @hugomano
about GSPMD sharding that can still fails pretty late when generating
the attribute. I feel this should be checked when constructing the
Partitionning.
* add Tensor.Id instead of old `usize` this allows accidental confusion
or operation on type id.
* improve Compiler.Scope API: `pushBlock` now returns the `scope` this
remove the old pattern of `pushBlock(); const scope = currentScope();`
Investigation has been done with the help of a clanker. The problem
might manifest on other blackwell hardware but for the time being,
simply gate to GB300.
@elogir If you can add some details
GLM 5.2 represents each MLA query/key head as 512 latent dimensions
followed by 64 RoPE dimensions. All 576 dimensions participate in the
attention score, but the value is only the 512-dimensional latent
prefix.
Previously, sparse MLA accumulated a 576-dimensional context and sliced
off the RoPE suffix afterward. Return the 512-dimensional value context
directly instead. This also reduces the padded value accumulator from
1024 to 512 elements.
Make the value width explicit so the same implementation also supports
models such as DeepSeek V4, where the RoPE dimensions are included in
the value.
Keys and values share the same latent cache. Remove the redundant value
pointer and strides, reuse the loaded key tile for value accumulation,
and remove paging arguments that are no longer needed after converting
top-k indices to physical cache positions.
Add a grouped partial-reduction path for the high split counts that
improve occupancy on MI300, and tune the shape-based launch selection
more aggressively for GB300.
Migration instructions:
zml.module.CompilationContext -> zml.Compiler
zml.module.CompileError -> zml.Compiler.Error
zml.module.CompilationOptions -> zml.Compiler.Options
zml.module.compile -> zml.Platform.compiler or zml.Compiler.compile
Breaking changes: none, I've kept a small compatibility layer at the top
level, with upgrade instructions.
We can remove it in a second time.
## Overview
* merge `dynamicSlice` and `slice` into `slices`
* merge `dynamicSlice1d` and `slice1d` into `slice`
* drop `choose`, `choose1d`
* handle singleton for dynamic slices too
* typed API so that it's easier to specify arguments
## Migration guide:
```diff
// new .slice accepts all old .slice1d inputs
- const x1 = x.slice1d(-1, .{ .start = 0, .end = half_dim });
+ const x1 = x.slice(-1, .{ .start = 0, .end = half_dim });
- k.dynamicSlice(.{ .layer = zml.Tensor.DynSlice{ .start = cache_index, .len = 1 } }).squeeze(.layer);
+ k.slice(.layer, .dynSingle(cache_index));
- k.dynamicSlice(.{
- .layer = zml.Tensor.DynSlice{ .start = cache_index, .len = 1 },
- }).squeeze(.layer).choose1d(.k, kv_head);
+ k.slice(.{ .layer, .k }, &.{ .dynSingle(cache_index), .single(kv_head) });
- o.choose1d(ax, 5),
+ o.slice(ax, .single(5))
```
Now the `Tensor.Slice` struct is bigger and now as a Tensor in it, which
may break user code if your model was storing a `Slice` inside it,
because now `zml.Bufferize(Model)` will contain one more `?Buffer`. You
can either pass an extra `null` to the `zml.Exe` or refactor to remove
`Slice` out of `Model`. I haven't seen this happen so far. The idea is
that `Slice` will mostly be a stack only struct, only created to call
`slice`
Latent `KvCache.update` currently passes one tuple to `scatterSlices`,
so the branch fails to compile when instantiated.
Pass the three arguments directly and add a latent-cache compile
regression test.
Tests:
- `bazel test --jobs=2 //zml:test`
- `bazel build --jobs=2 //zml`
This PR add the support of nvfp4 x nvfp4 moe backend on sm120 / sm100
(natively supported nvfp4 hardware). It :
- retrieves the released artifact cutlass moe flashinfer :
https://github.com/zml/flashinfer/releases/tag/cutlass-moe-v0.1.0
- loads the .so
- implements the nvfp4 x nvfp4 backend (added to existing triton and
mosaic tpu).
It's a bit of a mess to transfer ownership of the Exe to the runner as
that needs careful handling of defers. Fundamentally there is also no
particular reason why the Runner should own the Exe and it was not
consistent across zml.Exe and zml.TypedExe. Now it's always borrowed.
Renaming `zml.TypedExe` to `zml.FnExe` because it's shorter. I've
removed all the aliases `type LayerExe = zml.TypedExe(...)` it's useless
with ZLS, it's a lot easier to have `zml.FnExe(<function>)`
`QuantScheme` classifies a checkpoint's scheme from the weight and scale
shapes rather than from key names, and `Linear.forward` serves it:
| scheme | stored as |
|---|---|
| NVFP4 | `f4e2m1`, or two values per byte in `u8` |
| FP8 per-channel | `f8e4m3fn` + `[N, 1]` bf16 |
| FP8 block-128 | `f8e4m3fn` + `[N/128, K/128]` bf16 |
Emits `__metal$moe_gemm` / `$f4` / `$f8` so a MoE layer can run bf16,
NVFP4 or DeepSeek block-FP8 (`f8e4m3fn` weights + bf16 128×128 scales)
`forwardMoe` takes two new optional arguments, `w1_global_scale` and
`w2_global_scale`. The per-expert global (`weight_scale_2`) is applied
in-graph after the GEMM.
in LLMD and //examples/llm, we compile multiple executable, re-use the
args & results. We do all of this manually, in addition we don't have
any type-safety on the arguments/outputs which is a bit too easy to
misuse IMHO.
So adding two abstractions:
- Runner: Packs the args, exe and results together to avoid calling
`args.set`, `exe.run` and `results.fill` with similar arguments. Today
it owns the Exe. It doensn't strictly needs it, but it's simpler and
works for us for now.
- TypedExe: Type-safe version of an Exe, expecting a single input &
ouptut to have keyed arguments to have fully unambiguous arguments &
results mapping.
I've updated //example/llm with this new pattern.
Also adding an explicit `zml.module.CompileError`.
platforms/oneapi: bundle oneDNN and TBB with PJRT
Add oneDNN 2026.0 and TBB 2023.1 to the hermetic oneAPI packages.
the pjrj plugin need it for the mistral models type with the support of
the FP8 scaled matmul with libdnn.
platforms/oneapi: update PJRT_ONEAPI_RELEASE and SHA256 for new artifact
version
Sometimes there is a race where the input buffers get deinited at the
end of loadExecute before the executable call finishes, which can
corrupt data.
Waiting on the call closes that race, but it also prevents concurrency,
so we may need another way to keep the buffers alive without blocking.
Fix two PJRT error-handling cases that could allow execution to continue
after a backend failure.
- Use the same error conversion and destruction path for ready-event and
callback errors.
- Reject a successful `PJRT_Buffer_OpaqueDeviceMemoryDataPointer` result
when `device_memory_ptr` is null.
Add `Builder.log1p` to Mosaic TPU kernel DSL
The MLIR math dialect already exposes `math.log1p`.
This change makes it available through the Mosaic TPU builder alongside
the already existing operations.
--> This avoids kernels constructind `math.log1p` operations manually
and makes the operation reusable for future DSL kernel translations.
A companion DSL-harness changes replaces existing manual `math.log1p`
construction with `Builder.log1p`.
Add MLIR composites, `scaledDot`, and a Linear path for NVFP4 weights
(unpack, optional activation quant on Blackwell, global scales). Skip
fused multi-source tensors in the generic loader so pack/`loadExecute`
can fill them (may be used later for fused QKV etc).
Implement paged_attention in pure stablehlo.
introduces `zml.ops.if` and `zml.ops.if2`
The first one is similar to `zml.ops.while` and requires a full struct
with two functions. it's a bit heavy to use, but it follows the
stablehlo API, and if one of the branch has side effects it will
correctly evaluate the side effect only in that branch.
`zml.ops.if2` is a simpler version that doesn't require to create
intermediary struct, but as a consequence, if you have side effect they
will always evaluate, and it relies on the compiler to correctly avoid
duplicated compute by moving relevant compute inside the branches.
(open to feedback about if/if2)
Simplify vfs architecture: merge `//zml/io` and `//zml/io/vfs`. They move to top level `@zml//vfs`
The PR also include two bug fix:
- S3: when no content length we were trying to read exactly 1M, while we
should read up to 1M
- GCP: restore the commented code for parsing bucket info
## Test plan:
```
bazel run --config=debug //examples/io:playground -- ls gs://gcp-public-data-landsat/
bazel run --config=debug //examples/io:playground -- tree s3://noaa-goes19/ABI-Flood-Day-Shapefiles/2025/08
```
Add unit test for paged attention, enforcing all our backends produce
the same outputs.
To simplify memory management of many small buffer provided in a struct
(like paged attention metadata buffers), I introduce
`zml.Buffer.deinitAll` backed by `zml.meta.visitFlatStruct`. Contrary to
og `zml.meta.visit` the `flatStruct` variant emit a compile error if the
input has a slice/ptr inside it. The idea is that since it enforces the
struct to be a trivial, it means the struct won't require complex
`deinit` and `zml.Buffer.deinitAll` is in fact everything you need to
`deinit`. This avoid previous footgun of `zml.io.unloadAll`.
This led to more improvement with `zml.Tensor.toMemoryAll` and
`zml.Tensor.onMemoryAll` that also leverage the new
`zml.meta.visitFlatStruct`
removes `anytype` from `zml.ops.while`, helping with type checking and
simplifying a bit the logic.
A `While` struct needs
* a State struct definition that captures all the loop state that will
change on each iteration
* a `cond(While, State) Tensor` function returning a scalar boolean
tensor
* a `body(While, State) State` function that preserves the state shapes.
example:
```
const While = struct {
// While struct can contain tensor and non-tensor fields.
limit: i64,
step: Tensor,
pub const State = struct {
i: zml.Tensor,
sum: zml.Tensor,
};
pub fn cond(self: @This(), state: State) Tensor {
return state.i.cmp(.LT, Tensor.scalar(self.limit, .i64));
}
pub fn body(self: @This(), state: State) State {
return .{
.i = state.i.add(self.step),
.sum = state.sum.add(self.step),
};
}
fn forward(i: Tensor, sum: Tensor) State {
return zml.ops.@"while"(
@This(),
.{ .limit = 10, .step = .scalar(1, .i64) },
.{ .i = i, .sum = sum },
);
}
};
```
The code is a bit more verbose for simple example, but for GatedDeltaNet
it's equivalent, and I have also used it for paged attention.
While working on paged attention I realize that
`zml.testing.expectClose` wasn't properly handling strided Slice.
To fix that I'm adding a `ContiguousItemsIterator` that loops over the
contiguous part of the slice.
Only used in test now, so no perf considerations.
## Context
Translate
https://github.com/vllm-project/tpu-inference/tree/main/tpu_inference/kernels/gdn/v3
kernels.
## Summary
- add `BufferedRefStrategy` callbacks for overriding `copyIn`, `waitIn`,
`copyOut`, and `waitOut` on individual `PipeSpec`s
- add the `input_output` buffer role, with separate receive/send
semaphores and four loop-carried slot counters: copy-in, wait-in,
copy-out, and wait-out (input-only refs use `copy_in_slot` and
`wait_in_slot` ; output-only refs use `copy_out_slot` and
`wait_out_slot`, `input_output` refs use all four)
- provide custom transfer callbacks with the references, buffer slots,
and semaphores needed to emit their DMA operations (`sourceRef`,
`windowRef`, `copyInSlot`, `waitInSlot`, `receiveSemaphoreAt`,
`copyOutSlot`, `waitOutSlot`, `sendSemaphoreAt`)
- fall back to the existing transfer implementation for callbacks that
are not overridden
## Validation
- all 7 `fused_conv1d_gdn_batched` sweeps produce matching Python and
Zig MLIR, covering tile size, state count, head counts, Q/K-to-V head
ratio, distinct head dimensions, and convolution kernel size
- all 19 existing `decoding_gdn-bt_8` sweeps continue to produce
matching Python and Zig MLIR
Pre-allocating 85% of memory for simple tests is overkill, we'll just
let the BFC grow as needed. That should hopefully limit the problem of
having concurrent CI runs.
This PR introduce a new option gpu_async_dispatch that set to true by
default the legacy pjrt API option: `use_tfrt_gpu_client`. This option
in xla creates an other thread per device to dispatch the thunk and
command buffers asynchronously on the gpu. It strongly reduces the cpu
overhead when the model is sharded across several devices.
This PR contains two main changes:
* A patch for the tokenizer to support the tokenizer config file
* A temporary workaround to speed-up loading of the model when executing
arbitrary executable at load time
Update the local neovim configuration to override only the fields that
needs to change (such as `cmd` for LSP) instead of replacing the user's
entire configuration.
The main difference is `vim.lsp.config("zls", { ... })` which merges the
provided table into the existing configuration, while
`vim.lsp.config["zls"] = { ... }` overwrite the LSP configuration for
ZLS.
I also fixed the same issue for conform.
Signed-off-by: dmehala <damien@zml.ai>
Small mistake happened while cleaning #586, I factorized code that
shouldn't have been factorized blindly.
This PR fixes a bug where tensor loaded using `Loader.load` were not
correctly sharded.
The old `zml.io.load` was plagued with problems. This PR is an attempt
at improving the state of things.
One major problem of `zml.io.load` was that it was not very composable.
It was either using it to load all the model or doing everything by
hand.
Another problem was that it didn't offer a way to apply transformations
at load time to the tensors stored in a safetensors file. For example,
it happens that for MoE models, instead of the experts weights to be
tightly packed in a nice 3D tensor, the weight file contain one tensor
for each expert. This is nice until your implementation required a 3D
tensor as input. Using the old `zml.io.load` API this resulted in very
convoluted way to do concatenation at load time. With this new API, we
offer convenience so that it can be handled nicely and as fast as
possible.
The main change is the introduction of the `Loader` struct. It's a
struct that you instantiate once and reuse whenever you want to load
tensors. It relies on `Binding`, which are a way to store which tensor
are required and how they need to be combined (if it's a complex binding
involving more than one source tensor).
One advantage of this `Loader` is that it's composable. Loading a struct
using `Loader.auto(Struct)` or loading each individual field of `Struct`
has essentially the same behavior. This solved the major problem of the
old solution.
For simplicity I kept the old model loading code almost identical, as
this new solution is a drop-in replacement.
# Update
After some discussions with @gwenzek I've changed my mind. Instead of
having a declarative API, we'll provide helper to do transformation
easily. The main changes that this PR now bring are:
* Introduce `Loader` struct
* `Loader` exposes `load` which is the composable replacement of
`zml.io.load`
* `Loader` exposes `loadExecute` that allows you to execute an
executable with the buffers loaded from the sources associated with a
tensor. This allows for example to concatenate tensors.
We'll continue to work on it, this is just a preliminary work that
allows us to do transformation at load time in LLMD.
I hit an issue with the plugin refusing to load on macos.
There was no information.
After adding this I saw that the the file was apparently not existing in
the bazel sandbox,
so I could reduce the bug to a Bazel issue.
# Bump intel oneapi and rules_distroless
The Debian package set is now also exposed separately through the
rules_distroless APT extension:
Now use `rules_distroless` to install the oneAPI packages in a
distroless container image.
So the platforms/oneapi/packages.lock.json file is now generated from
the platforms/oneapi/packages.yaml file, and the packages.yaml file is
now the source of truth for the oneAPI packages.
The MODULE.bazel contain rules_distroless patch
`third_party/rules_distroless/accept-nospace-version-constraints.patch`
because the rules_distroless APT extension does not support version
constraints with spaces, e.g. `>= 2023.2.0` is not supported, but
`>=2023.2.0` is supported.
A PR has been submitted to rules_distroless to fix this issue, but until
it is merged and released, the patch is needed in zml
https://github.com/bazel-contrib/rules_distroless/pull/229.
`_ROOT_MODULE_DIRECT_DEPS` does not need anymore to list the individual
oneAPI and Ubuntu package repositories.
`_ROOT_PACKAGES` is now in `_UBUNTU_PACKAGES` because all the package
share the same `_UBUNTU_STRIP_PREFIX`
## Tests
```
bazel run @apt_oneapi//:lock
```
```
ONEAPI_DEVICE_SELECTOR=level_zero:0 \
bazel run \
--config=release \
--@zml//platforms:cpu=false \
--@zml//platforms:oneapi=true \
//examples/llm -- \
--model=/var/models/meta-llama/Llama-3.1-8B-Instruct \
--topk=4 \
--prompt="Tell me a story about evariste galois"
```
__-> decode 26.248s · 34.9tok/s__
```
ONEAPI_DEVICE_SELECTOR=level_zero:0,1 \
bazel run \
--config=release \
--@zml//platforms:cpu=false \
--@zml//platforms:oneapi=true \
//examples/llm -- \
--model=/var/models/meta-llama/Llama-3.1-8B-Instruct \
--topk=4 \
--prompt="Tell me a story about evariste galois"
```
__-> decode 13.017s · 56.6tok/s__
It's easy to break `Buffer.from(x_tensor.shape(), data, sharding)` by
creating `x_tensor` in the model code with a specific sharding and you
pass a different one to `from`.
I think this should be a crash (like when you mix-up allocators in
init/deinit) and it should have a dedicated error message to guide the
developers.
We used to panic when PJRT compilation failed.
I now think it's better to return an error, since we introduced the WIP
Metal plugin that can fail on legitimate programs.
This will make it possible to run the ZML test suite on the Metal plugin
and seeing all failing tests.
```
bazel run --@zml//platforms:metal=true --@zml//platforms:cpu=false //zml:test
33/138 nn.decltest.sampleTokens...FAIL (Internal)
...
34/138 nn.decltest.sampleTokensDynamic...FAIL (Internal)
...
102/138 tensor.Tensor.decltest.convert...FAIL (Internal)
...
115/138 tensor.Tensor.decltest.argsort...FAIL (Internal)
```
Render the zml-smi text logo fallback from the shared ZML logo data,
including the shiny per-glyph colors. Add a lightweight //zml:logo
target so the TUI can depend on the logo art without pulling in //zml.
Verified with:
- bazel test //bin/zml-smi/tui:test
- bazel build //bin/zml-smi:zml-smi
When a DRM fd is duplicated or inherited through fork(), the same oneAPI
DRM client can appear multiple times under /proc/<pid>/fdinfo. Summing
each fdinfo entry double-counts GPU memory and engine time.
Parse drm-client-id and skip entries whose (device_idx, client_id) pair
has already been seen during the collection pass. Include the device in
the key because drm-client-id may be scoped per DRM device.
Add a focused test covering duplicate clients on the same device and
distinct clients across devices or client IDs.
---------
Co-authored-by: Kevin Barre <kevin@zml.ai>
## Summary
Fix IRFFT shape inference so it uses the requested FFT output length and
validates that the complex input length matches the corresponding RFFT
output length.
Fixes#411.
## Why
RFFT changes the innermost real dimension `n` into `n / 2 + 1` complex
bins. IRFFT receives those complex bins plus the requested real FFT
length, so its output shape should restore that requested length instead
of deriving a smaller value from the complex input dimension.
## Test verification (RED -> GREEN)
Command used for all runs:
```bash
npx -y @bazel/bazelisk@1.25.0 test //zml:test --test_output=errors
```
RED, with the new regression test before the fix:
```text
102/139 tensor.Tensor.test.Tensor.fft IRFFT restores RFFT last dimension...thread 1701155 panic: exact division produced remainder
INFO: Build completed, 1 test FAILED, 5355 total actions
//zml:test FAILED in 24.9s
Executed 1 out of 1 test: 1 fails locally.
```
GREEN, with the fix applied:
```text
INFO: Build completed successfully, 6 total actions
//zml:test PASSED in 8.1s
Executed 1 out of 1 test: 1 test passes.
```
Revert RED, with the production fix reverted while keeping the test:
```text
102/139 tensor.Tensor.test.Tensor.fft IRFFT restores RFFT last dimension...thread 1711292 panic: fft(.{ .kind = .IRFFT, .length = { 10 } }) expects tensor last dimensions to match given lengths, got 1 and 1
INFO: Build completed, 1 test FAILED, 6 total actions
//zml:test FAILED in 22.9s
Executed 1 out of 1 test: 1 fails locally.
```
## Additional checks
```text
zig fmt --check zml/tensor.zig
fmt_status=0
```
In the VFS router, a scheme-qualified path like `hf://owner/model/file`
is meant to be absolute: it should resolve from the scheme's root. But
`lookupDir` only treated it as absolute when the base directory was
`cwd`. When such a path is opened *relative to an already-open directory* of
the same backend, the router kept the directory's path and glued the new
path after it, duplicating it. So `hf://owner/model` +
`hf://owner/model/file` became `owner/model/owner/model/file`, and the
lookup failed with `FileNotFound`.
Fix: on `lookupDir`, any `scheme://` sub-path resolves from the scheme's root
regardless of the base directory, the same way an absolute POSIX path
already does. About four lines.
Regenerate the xprof uv lockfile to move xprof-nightly off the removed
2.23.2a20260505 build and onto 2.23.3a20260707.
Fixes Bazel fetch failures for //tools/xprof:xprof.
Use the standard minimum/power-of-two block_m sizing for paged attention
instead of the raw GQA group size. This keeps oneAPI decode tile sizing
valid for Triton IR generation.
## Summary
- replace the legacy `new_git_repository` alias with `git_repository`
for `arocc`, `libvaxis`, `linenoise`, and `translate-c`
## Why
`git_repository` is the canonical Bazel repository rule.
`new_git_repository` doesn't exist at bazel HEAD
A 1-chip TPU mesh collapses to rank 1 (only `link_x`), so the
unconditional `link_y` fold tripped InvalidPhysicalAxis. Only fold
`link_y` when the physical mesh actually has it.
- Bump Metal plugin version
- Drop the attention metadata fields from the llama and lfm2 Sessions. Prefill and decode now each build their own metadata inline and free it.
This PR improves the `zml.nn.sampleTokensDynamic()` function. Most
notably:
* Convert parameters to input data type
* Support scalar/batched parameters
* Correctly use the temperature parameter
The IREE tokenizer was not producing the same output as the HF
tokenizer, potentially degrading the performance of the models. I put a
clanker to the task (because the tokenizer is already full clanked).
Outputs are now matching without any performance drop.
I also fixed an edge case issue on the `Writer` of the tokenizer.
Move the type out of `zml.compile` signature so that we can call
`io.async` more easily.
Sample code:
```
var layer_compile_future = io.async(zml.Compiler(TransformerLayer.forward).compile, .{
allocator,
io,
platform,
layer_compile_opts,
.{
model.model.layers[0],
hidden,
device_state.kv_cache.layers[0],
},
});
```
Previously, split tensors where allocated using an arena that would be
free at the end of the function scope. This commit uses the arena
context instead.
Signed-off-by: dmehala <damien@zml.ai>
## Summary
- Limit Zig cache sandbox mount flags to Linux, where linux-sandbox and
Bazel's Docker sandbox consume host path mounts.
- Remove the macOS mount entry because sandbox-exec uses a symlinked
execroot and ignores bind-mount settings.
- Mark `/tmp` and `/var/tmp` writable alongside the Linux mount pairs.
# Summary
This PR extends the support to chunked parallel read introduced
initially for HF in the VFS, to GCS and S3.
It:
- Adds a shared `zml/io/vfs/parallel_read.zig` helper for chunked
parallel HTTP range reads
- Add configurable read pool options for S3/GCS initialization
- Register the `gs://` VFS backend in `examples/llm`.
# Tests
## S3
We used [MINIO](https://hub.docker.com/r/minio/minio) docker image from
the `5090` GPU machine. From there, we mirror the
`meta-llama/Llama-3.1-8B-Instruct` model through the S3 server.
Then from the `4090` GPU machine, we run
```bash
bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=s3://zml-test/Llama-3.1-8B-Instruct --prompt="hello, how are you ?"
```
It works both on 1 and 2 GPU.
## GCS
In the same spirit, we put the `meta-llama/Llama-3.1-8B-Instruct` in a
GCS bucket we own. Then, on the `4090`, we run
```bash
bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct --prompt="hello, how are you ?"
```
It works both on 1 and 2 GPU.
### Performance
As a reference, we run:
```bash
gcloud storage cp gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00001-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00002-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00003-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00004-of-00004.safetensors /tmp/gcs-gcloud-llama-shards/
```
It gives:
```
Completed files 4/4 | 15.0GiB/15.0GiB | 942.7MiB/s
Average throughput: 918.8MiB/s
```
Using the VFS in various setups:
* `dma_chunks = 32`, `dma_chunk_size = 128`, `parallelism = 16`,
`chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64`
```bash
info (llama): Loaded weights [14.96GiB, 21.302s, 719.02MiB/s]
```
* `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`,
`chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64`
```bash
info (llama): Loaded weights [14.96GiB, 18.227s, 840.32MiB/s]
```
* `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`,
`chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64`
```bash
info (llama): Loaded weights [14.96GiB, 17.376s, 881.43MiB/s]
```
* `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`,
`chunk_size = 16 MiB`, `num_workers = 32`, `queue_capacity = 64`
```bash
info (llama): Loaded weights [14.96GiB, 17.28s, 886.36MiB/s]
```
* `dma_chunks = 32`, `dma_chunk_size = 512`, `parallelism = 16`,
`chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64`
```bash
info (llama): Loaded weights [14.96GiB, 18.658s, 820.90MiB/s]
```
part of highway XLA target selects which header to include based on
legacy --cpu flags but the rest not...
So even if we targeted linux amd64, the top part of the highway would be
used correctly, but the extension specific part would fallback to --cpu
darwin_arm64 and would not include hh_avx2.h resulting in compilation
errors.
The normal solution here is to use platform_mappings but since we pass
`--force_pic=true` in platforms, this takes precedence and the mappings
are ignored.
The temporary solution while https://github.com/openxla/xla/pull/44823
is merged, is to pass `--cpu` in all platforms we use for cross
compilation.
A few missing loads
Remove all flipped flags or deperacted flags from .bazelrc
Important:
- Keep autoload rules_cc and rules_python because XLA still has
transitive deps that are missing loads, PR in progress.
Updates to latest XLA which added support for bazel 8.x workspace and
fixed a bunch of things that made bazel 9 impossible.
3 problematic changes with this bump:
- PJRT headers were again incompatible with translate-c so a patch is
made and PR has been done in XLA
- PJRT introduced a proper type fo PJRT_Error so I used the InnerMixin
pattern for those.
- //tsl/platform/default package force evaluates Label("@cuda_nvcc")
even for CPU builds and we don't have this repo. I commented for now
until I find a solution on XLA side.
Bumped a few deps necessary for this and additional llvm deps that are
now evaluated as part of the llvm-project graph on XLA side (llvm_zlib
and llvm_zstd).
Next will be to investigate bazel 9 for real.
IREE's tokenizer only registered added tokens for pre-segmentation
matching if they were flagged special or normalized. Tokens that are
neither (e.g. <think> was dropped)
The fix is to match all added tokens
This PR introduces shardableDim helper in Sharding to handle the case
where : (dimension % number of sharding axes) != 0
Useful to stutter and shard correctly kv heads dim for example.
---------
Co-authored-by: Hugo Mano <hugo@zml.ai>
DirectMemoryWriter used to expose the next shard-sized slice as its writable
window. With HF/VFS this often produced tiny reads, even when adjacent tensor
bytes could be streamed through a larger direct-load window.
This PR changes direct loading to use an alias-window over the active primary
shard DMA buffer. The public `std.Io.Writer` buffer still points at real DMA
memory, but its visible length can coalesce across dispatch spans until a DMA
chunk, shard buffer, or tensor boundary is reached. On commit, the first active
primary segment stays zero-copy, while crossed spans and mirrors are scattered
before any buffer is submitted or rotated.
`DirectShardWriter` keeps ownership of DMA pool rotation, flip-flop events, and
PJRT transfer state. `DirectMemoryWriter` only walks the precomputed dispatch
spans with a small cursor.
Move the MoE allReduce inside the manual computation block.
Previously, zml.moe.forwardMoe placed allReduce outside the manual
computation block which caused XLA SPMD partitioning to throw the
following error with certain models.
"Cross-partition allreduce must be in (partial) manual partitioning mode."
**Summary**
This PR update to Neuron v2.30, introduces the Neuron NKI stack into ZML and wires it into the
LLM attention path.
The main goal is to make Neuron attention run through NKI kernels.
It adds the compiler/runtime plumbing needed to embed NKI kernels as
Neuron custom-native calls, then uses that path for SDPA-style
prefill/decode attention. Paged Attention will come later.
On inf2.8xlarge it runs :
- 25 tok/s for Llama 3.1 8B BF16
- 63.5 tok/s for LFM2.5-1.2B-Instruct BF16
- 19 tok/s for Qwen3.5-9B BF16
This PR add the TPU backend for MoE. It uses the GMM Expert parallelism
kernel to perform the grouped gemms on gate_up and down weights (these
weights must have a precise shape to run optimally).
This backend is currently optimized for small batches. A specialized
mosaic tpu kernel will be introduced to handle bigger ones.
This implementation come with few fixes concerning the TPU sharding and
the Qwen 3.5 moe model which uses the backend :
- TPU sharding : PJRT reports TPU device coordinates in a max rank form.
So it breaks folding. To avoid this we compute the Rank by not counting
where coords =0.
- Linear attention was reading padding token in the Qwen 3.5 model. So
we introduce masking of them.
n.b. : The kv head repetition to satisfy the sharding will be handled a
more generic way in a next PR
---------
Co-authored-by: Corentin Godeau <corentin@zml.ai>
Renamed `scope` to `span` to match OpenTelemetry convention I'm more
familiar with. Traceme doesn't seem to use `Scope` explicitly but talks
more about a scope object AFAIK.
Now `span` fails at compile time. if it cannot compute the appropriate
buffer size for the name. In that case the caller should use
`formatSpanName` which allocates. This makes `span` infaillible at
runtime.
This opt-in flag allows XLA to save the autotuning resulting from
previous runs in a folder, greatly speeding up the compilation.
It's not meant to be used at all times, so it's undocumented and opt-in.
This PR introduce the new GMM Expert parallelism kernel (basis grouped
gemm for moe on TPU). It induces some updates into the DSL mosaic TPU,
such as Builders helpers or Pipelines and schedulers :
- Buffer counts (for scheduling overlapping) > 2 allowed (until 7)
- source_reshape_shape reshaped at DMA slice/use sites to match Pallas
IR placement
- More grid dimensionality for pipeline (up to 4)
`Buffer.from` is in the hot path everytime we want to send smth to the
GPU.
Sometimes we can overlap it with GPU compute but not always, and
sometimes it's quite complicated to do so.
I've rewrote Sharding.Placement to be device agnostic instead of device
specific.
This allow to reuse most of the logic for multiple devices.
According to LLVM MCA (with script included here), we got from:
```
Iterations: 100
Instructions: 324200
Total Cycles: 175191
Total uOps: 369700
```
to
```
Iterations: 100
Instructions: 162900
Total Cycles: 85585
Total uOps: 182600
```
So about twice as fast for this part of the code. I would need some help
to confirm the numbers on TPU.
(Once the PR is approved, I'll remove the mca script as well as the
build.zig)
---------
Co-authored-by: Hugo Mano <hugo@zml.ai>
Add a oneAPI backend for `zml-smi` that detects Intel DRM render devices
and reports GPU telemetry from sysfs, hwmon, and fdinfo.
The new backend collects utilization, memory usage, power, temperature,
clocks, PCIe link data, and per-process usage for Intel GPUs. It is
wired into device detection, Bazel builds, Prometheus metrics, and the
TUI, including Intel GPU imagery and GPU detail/process table support.
Also clean up related TUI handling for GPU clock/PCIe fields and process
table lifetime management.
Fix the triton attention kernel by avoiding the case where `TILE_SIZE ==
BLOCK_SIZE`. Now we enforce masking this way : `const tile_mask: Value =
seq_offset.lt(max_seq_prefix_len);`.
Co-authored-by: loupicaaa <louis@zml.ai>
This PR splits Llama in `examples/llm` into separate `embed_tokens`,
`layer`, and `lm_head` kernels for both prefill and decode, and brings
LFM2 onto the same composed execution path. And done same work for Qwen
3.5 .
It removes the old LFM2 `--single` mode, reworks Llama KV-cache access
to use explicit layer indices, and reuses more decode-time state through
a persistent runner and prebuilt token index buffers.
It also tightens sharding/partitioning in LFM2 for multi-device runs,
adds clearer per-phase compile logging/program names, removes the unused
`--single` CLI flag.
This PR introduces the sharding as expert parallelism for MoE :
- Introduce a .experts logical sharding (required to express an other
semantic than TP (.model))
- Wrap the whole MoE impl into a manualComputation
- Implement a manual stableHlo all reduce since shardy can not infer
the correct communication after a manualComputation (blackbox)
- Partition qwen 3.5 moe example with TP for the rest of the model (non
MoE)
---------
Co-authored-by: loupicaaa <louis@zml.ai>
Keeping `devices_in_canonical_order: []Device` in the `PhysicalMesh` and
inside `Sharding.Data` I've changed `physical: PhysicalMesh` to
`physical: *const PhysicalMesh` to have the platform own it now that
PhysicalMesh has a `deinit()`.
Add experimental Intel GPU backend
Add oneAPI as a ZML platform target with Bazel flags, module extension
repositories, and PJRT runtime packaging for Linux Intel GPUs.
- package libpjrt_oneapi with oneAPI runtime libraries, Level Zero
loader,
libsycl, and a zmlxoneapi dlopen shim for versioned shared libraries
- register the oneAPI platform loader and detect available render
devices
before loading the packaged PJRT plugin from runfiles
- thread the oneAPI target through platform selection, sharding,
placement,
buffers, execution, memory, IO, tensor behavior, and XLA compile flags
- document oneAPI prerequisites and example LLM usage
- update Bazel module lockfile entries for the new runtime dependencies
---------
Co-authored-by: Steeve Morin <steeve@zml.ai>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Steeve Morin <steeve@9960x-b70x2.tailb57cfe.ts.net>
This PR changes the following:
- Name-based `output_operand_aliases` for MosaicTPU and Triton DSLs
- New `kernels/mosaic_tpu/pipeline.zig`, a Zig port of `pltpu.emit_pipeline`
- Some fixes to MosaicTPU
`Placement` is 27KiB big, it led to stack allocation failure with
gigantic struct sizes (~400KiB) in llmd. So changing the shards to be
allocated on the heap with functions to avoid allocations when we don't
need it (`format`, etc.).
This PR bumps rules_zig back to upstream/main where we now have all the
necessary
features to enable zig local actions while the reste on remote.
It also adds the necessary configurable to actually enable this.
Additionally, since rules_zig now ships ZLS support, we remove zls
entirely from zml.
### How the Bazel resolution works
Bazel resolves execution platforms and toolchains during analysis for
each
target that requires toolchains. If a rule defines exe, resolution
happens separately for the default target execution group and for each
`exec_group`;
actions assigned to an exec group use that group’s selected execution
platform
and toolchains.
Execution platforms come from an ordered set. In this repository’s
remote
configuration, the relevant order is:
1. `//:rbe_platform`, supplied by
`--extra_execution_platforms=@zml//:rbe_platform`
2. `//bazel/zig_toolchain_config:local_execution_platform`, registered
by this PR
3. the host platform, included automatically by Bazel
Earlier platforms are selected when they can satisfy the requested
toolchains.
For each target or exec group, Bazel takes the required toolchain types,
the
target platform, the ordered execution platforms, and the ordered
registered
toolchains. It filters execution platforms by execution constraints,
then tries
to find a matching toolchain for every required toolchain type. A
toolchain
matches when:
- its `exec_compatible_with` constraints are present on the candidate
execution
platform
- its `target_compatible_with` constraints are present on the target
platform
- its `target_settings` match the current configuration
Any execution platform missing a required toolchain is rejected. The
first
remaining execution platform wins, and the first matching toolchain for
each type
is selected.
### How this makes Zig local-only
This PR defines a custom execution constraint:
- `//bazel/zig_toolchain_config:local_zig_execution`
It then defines a host-derived execution platform:
- `//bazel/zig_toolchain_config:local_execution_platform`
That platform inherits the host OS/CPU constraints from
`@platforms//host`, adds
`local_zig_execution`, and sets:
```starlark
exec_properties = {
"no-remote-exec": "1",
}
```
The zig.toolchain(...) module extension call now passes:
extra_exec_compatible_with =
["//bazel/zig_toolchain_config:local_zig_execution"]
rules_zig appends that extra execution constraint to every generated Zig
SDK
toolchain. As a result, when rules_zig resolves
@rules_zig//zig:toolchain_type
for its zig execution group:
- the remote execution platform is considered first, but it does not
have
local_zig_execution, so it cannot match the Zig SDK toolchain
- the custom local execution platform has both the host OS/CPU
constraints and
local_zig_execution, so it matches
- the selected platform carries no-remote-exec, so Zig actions execute
locally
even in remote builds
Other toolchains are not constrained by local_zig_execution, so they can
continue
to resolve to the remote execution platform first. This keeps the build
remote by
default while carving out Zig actions as local-only.
---------
Co-authored-by: Steeve Morin <steeve@zml.ai>
Change made in response to the following error:
```
zml/testing.zig:299:43: error: root source file struct 'meta' has no member named 'FnArgs'
const ArgsT = stdx.meta.Tail(stdx.meta.FnArgs(forward));
~~~~~~~~~^~~~~~~
stdx/meta.zig:1:1: note: struct declared here
```
List of changes:
* Removed `Context`
* Simplified metadata to not contain scratch buffers. They are created
uninitialized on the fly instead (thanks to #535)
* Added helper for memory movement (host_pinned -> device) (thanks to
#534)
* FA2/FA3 don't read the `max_seqlen_q` from host anymore. This was
error prone. Instead rely on a fixed value for now. (It's error prone
because it's baked in the command buffer and can result in wrong value
if the value changes but the command buffer is not recorded again).
This one slipped from #537
It worked because if there is no translate-c toolchain, zig uses zig
translate-c.
But we want to respect user choice here too.
### Overview
Context: Modern models utilize a MLP called Mixture of Experts (MoE). In
a MoE, the MLP is organized into "experts," where specific sub-tensors
are specialized to process particular linguistic patterns or elements.
During inference, the hidden states of each token are routed to the most
relevant experts, ensuring that matrix multiplications (matmul) are
performed only on the active parameters.
Problem: Due to the sparse and dynamic nature of MoE where token
assignment is determined at runtime via routing, treating an MoE layer
as a standard matmul results in significant performance degradation.
Standard kernels are not optimized for the irregular memory access
patterns in sparse routing.
Approach: MoE is implemented as a primitive in the ZML operation set.
The generic MoE Backend is designed to support various specialized
kernels tailored to specific hardware and weight precisions. The primary
implementation currently features a Triton MoE backend optimized for
NVIDIA Hopper architecture using BF16 and FP8 weights. This
implementation is utilized in the Qwen 3.5 MoE model.
### Features
**Generic MoE Backend (zml/moe/moe.zig)**
- Integrate specialized MoE backends (e.g., Triton).
- Select the optimal backend based on hardware and precision.
- Implements essential utilities and structures for registration,
parameter loading, and metadata management.
- The forwardMoe() function acts as the entry point to invoke the
appropriate backend.
**Triton MoE Backend (zml/moe/triton.zig)**
- Generates TTIR (Triton Intermediate Representation) at compile time.
- Leverages Triton's custom calls exposed via XLA.
Execution Flow:
1. Select the optimal configuration for the fused MoE kernel.
2. Determine token distribution: use naive assignment if the
distribution is sparse; otherwise, sort and align tokens for efficiency.
3. Quantization: quantize activations if using FP8 weights.
4. Fused Kernel: execute the fused MoE kernel for the gate and
up-projection layers.
5. Post-Processing: split the output and apply the activation function.
6. Quantization : re-quantize activations for the next stage if required
by the FP8 path.
7. Fused Kernel: execute the fused MoE kernel for the down-projection
layer.
8. Aggregation: sum the Top-K results for each token.
Qwen 3.5 MoE
- Fully utilizes the Triton MoE backend.
- BF16 weights implementation (also tested with FP8 but not implemented
here)
---------
Co-authored-by: loupicaaa <louis@zml.ai>
Co-authored-by: Raphael NAHOUM <raphael@zml.ai>
This PR exposes the XLA-specific custom calls that allow to create
uninitialized tensors (thus not paying the cost of a kernel to
essentially do a memset).
The `stablehlo` part of it was already done, so I added a wrapper in
`tensor.zig`
It is now required with newer plugin to annotate the executable
arguments with their location.
Like `Tensor.reuseBuffer()`, you can now use `Tensor.onMemory()` to
specify on which memory a tensor is located.
Simple reference implementation of attnd client as a attention backend
relying on POSIX. I've tried various things with `sendmmsg`/`recvmmsg`
and `io_uring` out of curiosity but they weren't worth the extra code
here.
The integration as a attention backend is a bit clunky, we're still
allocating a KV cache and there are a few panics here and there because
the current initialization mechanism of attention backend doesn't work
for attnd, but that's something for later.
This PR introduces the following:
- Wire up the LLVM dialect so the Triton builder can emit
`llvm.intr.assume` so we can match
[`triton.language.assume`](https://triton-lang.org/main/python-api/generated/triton.language.assume.html#triton.language.assume)
- Thread `tt.num_stages` through `scf.for` for pipelining hints
- Add an inline-return mode to `ReturnIfScope`. Not necessary but better
for IR matching with Python reference
- Build DSL kernels IR in a new MLIR context
Also fixes i1 extension to use `extui` instead of `extsi`.
- bump Neuron dependencies in `platforms/neuron/requirements.lock.txt`
- `libneuronxla` `2.2.15515.0+50c26cbd` -> `2.2.16408.0+50c26cbd`
- `neuronx-cc` `2.23.6484.0+3b612583` -> `2.24.8799.0+6f62ff7c`
- make `neuronx-cc` invocation more robust by fixing path resolution,
releasing the Python GIL during compilation, checking the compiler
process exit status, and propagating NEFF wrapping/read
failures
- resolve PJRT devices and memories by PJRT handles instead of assuming
stable numeric ids during platform initialization, including Neuron
cases where memory ids are reused across devices
- avoid unsupported Neuron synchronization/event paths by always
requesting execution events on Neuron, deinitializing them without
awaiting, and removing the explicit await from the sharding
example
- avoid unsupported Neuron runtime paths by:
- falling back to host-backed buffer initialization for
`Buffer.uninitialized`
- using passthrough DMA allocation on Neuron
- skipping `Tensor.print()` custom calls on Neuron
- add NRT-backed Neuron physical mesh topology construction, including
chip/core placement, ring/torus/point-to-point topology inference, and
Neuron local-hardware-id based shard placement
- let the sharding example pick the default partitioner from the current
target
Add `compute_on_host` to force host offloading and merging the
`customCallInternal` and `typeCustomCall` function into one to avoid
duplication. I've slightly simplified the signature of some of the
functions I wrote initially to avoid passing the types explicitly.
Two fixes to the `write_first_model.md` tutorial:
- `allocator` on line 221 is deinitialized, however, `io` is not. `io`
should be deinitialized as well.
- `sharding` on line 216, is an undeclared identifier. It should be
replaced with `.replicated`, which is used in the full code example.
This PR modernizes ZML profiling in four areas:
- replaces the old C++ `xspace_to_perfetto` tool with a native Zig
exporter built on upb
- integrates trace export directly into `zml/profiler.zig`
- introduces a scoped tracing API in `zml.tracer` and wires it through
examples
- adds platform-specific annotation support and Bazel-native profiler
wrappers for CUDA and ROCm
The result is a simpler profiling flow, better large-trace behavior, and
a more consistent experience across Linux CUDA, Linux ROCm, and macOS.
This PR simplifies the common replicated case while making sharding
intent explicit and use sharding by pointer ( sharding struct is quite
big: ~600 bytes).
Using pointers for shardings also avoids unnecessary sharding copies and
makes it easier to share stable sharding instances across compilation,
loading, and runtime paths.
- `Platform` owns the canonical replicated sharding
- `Buffer` APIs now take explicit sharding specs:
- `.replicated`
- `.{ .sharded = sharding_ptr }`
- `createTensor` / `maybeCreateTensor `now require explicit partitioning
instead of accepting null
- compilation/loading now use pointer-based sharding lists
This makes simple cases like mnist and benchmark much cleaner: callers
no longer need to manually construct and thread a replicated sharding
just to use the default case, but they still state placement explicitly
at the call site.
---------
Co-authored-by: Hugo Mano <hugo@zml.ai>
the previous nightly we depend on is not on pypi anymore.
I wanted to switch to stable release but 2.23 isn't public yet:
https://pypi.org/project/xprof/#history
This change fixes profile timestamp handling so saved xplane.pb data is
post-processed like JAX/XLA’s `ProfilerSession`, and also cleans up the
Zig/C++ FFI used by the profile export bridge.
Before this, we were writing raw profiler output from the PJRT extension
directly to `profiling.xplane.pb`, which bypassed XLA’s single-host
post-processing step done by JAX that we can find here:
efac222ebf/tsl/profiler/lib/profiler_session.cc (L100)
That made ZML behave differently from JAX, including wrong timeline
origins in downstream viewers like XProf.
To fix that, we now run
`tsl::profiler::PostProcessSingleHostXSpace(...)` before writing the
protobuf, and we treat that post-processed xplane.pb as the source for
subsequent Perfetto conversion.
We also register start/stop timestamps in `zml/profiler.zig` so they
match XLA’s expectations for session timestamps with `real` clock.
While touching this path, I tightened the FFI boundary as well.
Error text returned from the xspace export bridge is now allocated with
the provided Zig allocator instead of malloc, C receives a pointer to
the `zig_allocator wrapper` to avoid lifetime issues from passing
allocator state by value, and error propagation in
`tools/xspace_to_perfetto_lib.cc `now uses `zig_slice` outputs directly
instead of building std::string error objects just to copy them back
out. This removes mixed allocation ownership between C++ and Zig and
fixes the allocator lifetime bug that could crash across the FFI
boundary.
Drops the Python build-time dependency for Triton and Mosaic TPU
kernels. ZML now emits TTIR and Mosaic TPU IR directly from Zig, against
the same MLIR context as the rest of the compiler. So no more JAX,
Triton at build time and a lighter sandbox.
Split into two commits:
- **`c72f44884` — Add Triton and Mosaic Zig DSL.** Additive: IR builders
(`kernels/triton/`, `kernels/mosaic_tpu/`), upstream dialect bindings
(`ttir`, `mosaic_tpu`, plus `arith`/`scf`/`math`/`memref`/`vector`/
`affine`/`cf`), and the high-level
`zml.kernel.{triton,mosaic_tpu}.Kernel(Config, spec)` declarative form.
Comes with author guides in each `README.md`.
- **`cb61e529f` — Remove Python and replace with Zig kernels.** Migrates
the two production kernels (`unified_attention` for Triton,
`ragged_paged` for Mosaic TPU) and deletes everything Python related in
the sandbox and what was generating the IR.
---------
Co-authored-by: Steeve Morin <steeve@zml.ai>
This PR removes `optimal="1"` from the `[const std::string&]` out
typemap in `zml/tokenizer/sentencepiece/sentencepiece.i`.
SWIG cannot apply the requested optimal code path for the `%extend
std::string { data() }` wrapper and emitted `Warning 474: ` during
SwigC. Removing the unsupported hint preserves the generated behavior
while eliminating the warning.
**Reproduce**
```
$ bazel build //examples/llm
[...]
zml/tokenizer/sentencepiece/sentencepiece.i:41: Warning 474: Method std::string::data() const usage of the optimal attribute ignored
zml/tokenizer/sentencepiece/sentencepiece.i:25: Warning 474: in the out typemap as the following cannot be used to generate optimal code: cppresult = (std::string *) &std_string_data__SWIG((std::string const *)arg1);
[...]
```
**Validation**
* Rebuilt `//examples/llm:llm` and a clean
* Confirmed `Warning 474: ` no longer appears
* Confirmed build still succeeds
* Warning 524 remains and is not addressed in this PR
This suppresses SWIG Warning 524 in `bazel/swig.bzl`.
The custom SWIG rule intentionally parses C++ interfaces while
generating a C ABI wrapper for Zig consumption (in
[sentencepiece.zig](zml/tokenizer/sentencepiece/sentencepiece.zig)), so
this warning is expected noise from an intentional -c backend selection
rather than a build problem.
**Reproduce**
```
$ bazel build //examples/llm
[...]
SWIG:1: Warning 524: Experimental target language. Target language C specified by -c is an experimental language. See the 'Target Languages' section in the Introduction chapter of the SWIG documentation.
[...]
```
**Validation**
* Built `//examples/llm:llm` after a clean
* Confirmed Warning 524 no longer appears
* Build still succeeds without changing generated binding behavior
I'm doing a pass on the example code we have.
* llama/model.zig, qwen/model.zig: change layer_index from being a
tensor to being an integer.
It's clearer this way what is happening. Note that lfm doesn't have the
issue because the layer index is passed everywhere.
* tokenizePrompt: the code was doing unnecessary copies of data, I added
some helper to tokenizer.encoder to streamline the code by removing the
extra copies and allocations.
* `.stdout().writer() -> .stdout().writerStreaming()` this one is weird
but apparently using stdout by position isn't always allowed, in
particular when running the exe from within SublimeText.
* runDecode -> show case bake
fix(mosaic_tpu): support padded KV-cache head dims in ragged paged
attention
Pass both logical and cache head dimensions through the Mosaic TPU paged
attention path so KV pages can remain 128-byte aligned without padding Q
or trimming the final output.
- add backend helpers and tests for Mosaic TPU kernel head-dim alignment
- keep query/output tensors and softmax scaling on the logical head dim
- allow cache head dims greater than or equal to query head dims in TPU
IR and kernel validation
- slice padded K/V vectors back to the logical head dim before attention
math
- fix the tensor pad call signature used by tensor padding
When looking at the new Sharding code I was triggered by the high number
of "try".
I then realized most of the try where traced back to `axes = try
.init(0)` which can never fail.
So I removed BoundedArray.init (aka Andrew mode)
It's generally unsafe because it makes `slice()` return undefined.;
I replaced`try init(0)` by `.empty`
I replaced `try init(n)` by `.fromSlice` or with `.{ .buffer =
undefined, .len = n }` to explicitly show the unsafeness
This leads to widespread changes because it means that most of the
sharding operations can't fail anymore.
Improve TPU ragged paged attention strided KV loading
- add the TPU ragged paged attention kernel to the repo and package it
with gen_ir
- switch gen_ir to import the vendored kernel instead of JAX's default
one
- update tpu_attention tensor axis tags to match the kernel's expected
shapes
refactor: introduce KVCache union and wire dense/split layout through
pagedAttention
Adds a `KVCache` union type to `paged_attention.zig` with two variants:
- `.split { k, v }`: separate K and V page tensors (used by cuda_fa2,
cuda_fa3, triton)
- `.dense`: a single pre-fused K/V page tensor (used by mosaic_tpu)
Changes:
- `pagedAttention()` in `paged_attention.zig` now accepts a `kv_cache:
KVCache`
parameter alongside `k`/`v` (unused); dispatches `.split` to CUDA/Triton
backends and the full `KVCache` union to `mosaic_tpu`.
- `mosaic_tpu.pagedAttention()` in `tpu_attention.zig` now takes
`layout: paged_attention.KVCache` instead of separate k/v tensors, and
passes `layout.dense` to the existing `prepareInputs()` helper.
- `KvCache.Layer` in `llmd/models.zig` is now a tagged union of `.dense`
(fused tensor) and `.split { k, v }`. `attentionLayout()` converts it to
`attention.KVCache`. The `.dense` update path in `Layer.update()` fuses
K and V by inserting a `.kv` axis, concatenating, and merging `.hkv`
before scattering into the page cache.
Specifying both `rows` and `cols` makes the zml logo in `zml-smi` render
correctly in KDE Konsole. Verified that the fix didn't cause any
regression in a terminal emulator where logo rendering was already
functional (`Ghostty`).
`BUILD.bazel` is also updated to run the `zml-smi` tests.
**Before**
<img width="348" height="238" alt="image"
src="https://github.com/user-attachments/assets/7abc604d-14b9-4c0a-a561-9acde1d5359e"
/>
**With Fix**
<img width="348" height="162" alt="image"
src="https://github.com/user-attachments/assets/d93a7253-c1fc-4ff9-891b-2e0c6ec46a65"
/>
**Test Discovery Fix**
```
master $ bazel query 'kind(zig_test, //bin/zml-smi/...)'
INFO: Empty results
master $ git checkout -
Switched to branch 'fix/zml-smi-logo-size'
fix/zml-smi-logo-size $ bazel query 'kind(zig_test, //bin/zml-smi/...)'
//bin/zml-smi/tui:test
```
Use newer versions of several GitHub Actions. Also bump used Zig
version.
`mlugg/setup-zig` is still using Node.js v20 so warning still here but
only for this action. Check
https://codeberg.org/mlugg/setup-zig/pulls/55 in a near future in order
to update the action.
generateTtir isn't thread-safe as we keep a single process alive. Using
different processes didn't seem to be better from a quick try. So just
locking it with a mutex.
Aligns Mosaic TPU ragged paged-attention input/parameter naming with the
terminology used by other backends (e.g., block table + sequence
lengths)
Changes:
- Rename Mosaic TPU ragged attention inputs/parameters from
kv_lens/page_indices/cu_seqlens_q to
seq_lens/block_table/query_start_len.
I've reworked the Tokenizer API to actually work with all tokenizers and
not just IREE and moved the `Writer` part into the IREE tokenizer. I'm
now avoiding intermediate copies as much as possible now.
I'm also more defensive now to avoid the same infinite loop and put more
effort into providing appropriate output buffers to IREE. It fails with
empty output buffers and may return a `ResourceExhausted` if it's too
small. Also added a bunch of tests.
IREE tokenizer itself had a bug which I fixed here
https://github.com/iree-org/iree/pull/24150. In the meantime, I'm
pointing bazel to our fork.
Bump rules_zig to latest zig_toolchain changes (for bootstrap)
Bump ZLS to 0.16.0
Amend ZLS completion rules and runners to match rules_zig new providers
This PR fixes several Bazel analysis regressions so both of these
commands work reliably:
- `bazel cquery //...`
- `bazel cquery --platforms=@zml//platforms:linux_amd64 //...`
After recent changes and the rebase, repo-wide cquery was failing for
three separate reasons:
- ZLS toolchains were over-constrained by target platform: updated ZLS
toolchain registration to constrain only the exec platform.
This avoids cross-target analysis failures when host-side completion
tooling is analyzed under a foreign target platform.
- OCI packaging targets were analyzed on host platforms where their base
images do not apply: marked Linux OCI image targets as Linux x86_64
compatible only. Also switched the Debian base image pull from Docker
Hub to the GCR mirror to avoid CI fetch failures when resolving
packaging images manifests due to rate limiting
- Triton Python deps were fetched for the host Python platform instead
of the Linux wheel platform the lockfile was generated for: restored
Linux x86_64 wheel resolution for `triton_py_deps` in `MODULE.bazel`.
This prevents macOS host analysis from attempting to fetch a
`triton==3.6.0` wheel for an unsupported host platform.
Also added a step in ci to prevent regression
In #470 I introduced support for optional float in custom call metadata
but it was incomplete.
This PR cleans the code a bit and introduce support for all basic
optional types.
I added a `scale` field in the `AttentionOptions` struct that is passed
through the attention backend.
I had to add support for optional in the custom call metadata. For now
it only supports optional floats, we'll add support for other types
later.
Update Zig compiler to 0.16.0-dev.3133+5ec8e45f3 and adjust code
for breaking standard library changes:
- Replace ArrayListUnmanaged/ArrayList init with .empty
- Switch to unmanaged AutoArrayHashMap/StringArrayHashMap
- Use debug.GeneralPurposeAllocator (renamed upstream)
- Replace std.Io.File.ReadStreamingError with error.EndOfStream
- Refactor meta.visit to use structured visitor pattern
- Fix alignment allocation in mem.bufferizeInner
Also update third-party dependencies: flashattn 0.0.6-rc8, arocc,
uucode, libvaxis, and zigimg patches.
Co-authored-by: Kevin Barre <kevin@zml.ai>
This PR introduces a Triton paged attention backend, unlocking ROCm
support.
It relies on a Python sandbox to generate the TTIR.
The major change is that the KVCache now needs to be per-layer for paged
attention. The `layer_index` is now superfluous and will probably be
removed in the future.
This PR introduces:
- Prometheus metrics endpoint through `--prometheus` and optionally
`--prometheus-port`
- JSON output mode with `--json`
- macOS arm64 host information
- An install and release script
This pull request adds support for Google Cloud Storage (GCS) to the
virtual file system (VFS) module, alongside some build configuration
updates. The main changes introduce the new `GCS` backend, make it
available in the VFS API, and register it in the example application.
Authentication is handled like other GCS libraries via the
`GOOGLE_APPLICATION_CREDENTIALS` env or application default account
files.
**IMPORTANT NOTE**: Only Authorized User and Metadata Server auth are
implemented. Service Account auth isn't implemented yet.
Adding the [IREE
tokenizer](https://github.com/iree-org/iree/blob/main/runtime/src/iree/tokenizer/README.md).Being
made from the ground up with streaming (de-)tokenization in mind, I've
extended the APIs to expose a `std.Io.Writer` for the tokenization.
I've quickly benchmarked IREE's tokenizer vs HuggingFace's one with a
request I retrieved from a coding agent (opencode) in the middle of a
session. The request body was 360KB encoded in 95K tokens. I also did
another test with a similarly sized request with the same results as
shown hereafter.
I've tested two cases:
- oneshot: the template is rendered all at once and only then we encode
the whole string.
- stream: the template rendering and encoding are pipelined to avoid the
intermediate allocation.
There is a slight difference in the generated tokens between HuggingFace
and IREE. Some tokens like `introdu` were encoded as `i`, `n`, `t`, etc.
Decoding them returns the same text. It's very close in terms of total
token count:
- hf: 95779
- iree/stream: 95705
- iree/oneshot: 95587
| Model | IREE avg (oneshot / stream) ms | HuggingFace avg (ms) | Uplift
vs HF (oneshot / stream) |
|---|---:|---:|---|
| liquidai/LFM2.5-1.2B-Instruct | 8.5 / 7.3 | 40.5 | 4.8x / 5.5x |
| Qwen/Qwen3.5-9B | 6.0 / 8.2 | 59.4 | 9.9x / 7.2x |
| mistralai/Ministral-3-3B-Reasoning-2512 | 7.4 / 6.9 | 59.2 | 8x / 8.6x
|
It's unclear why the Qwen's chat template is somehow slower in the
stream case, maybe because there are more writes. Adding non-empty
buffer in the interface helps a bit, but tends to slow down mistral
case. So well
---------
Co-authored-by: Corentin Kerisit <corentin.kerisit@gmail.com>
Today, creating a custom call is a bit tedious and error prone. The
primary goal of this PR is to only accept structs as
input/output/attributes and rely more strongly on the names everywhere
when creating as custom call. I've migrated the `fa2` custom to provide
an example. Can migrate the rest.
- Custom call relies on the position of the input/outputs/attributes to
be in the correct order to work. The `output_operand_aliases` also maps
the outputs to the input field position.
- If the custom call is "sharding aware", as in it knows it works on a
single partition on multi devices setup, one must use
`zml.ops.manualComputation` which is a bit tedious to set up.
- `flashattn.zig` has a bunch of logic to register said custom call but
also to map inputs into something more usable which can be shared.
- both `zml.ops.customCall` and `zml.ops.manualComputation` tend to
accept to many different types IMHO which makes it a bit hard to follow.
But well, to be fair not sure how much simpler my code is. I do
constrain myself to only accepting simple structs for the generic types.
I've re-used the existing `manualComputationInternal` and
`customCallInteral` for this new API without touching them. The only
exception is the `output_operand_alises` where I needed a bit more
flexibility.
This allows setting DT_RPATH instead of DT_RUNPATH. DT_RPATH takes
precendence to LD_LIBRARY_PATH.
Sometimes LD_LIBRARY_PATH is useful, sometimes it's a pain. Using this
enables us at least to precede it.
Handle Qwen3.5 full-attention K/V heads like vLLM when
num_key_value_heads < tp: replicate K/V heads across the model mesh
instead of sharding the KV head axis.
**Motivation**
Qwen3.5 9B has 4 KV heads. On 8-way TP, sharding the KV head axis over
`.model` is invalid because the dimension is not divisible by the
tensor-parallel degree.
This is a placement issue, not a model-shape issue. The model still has
4 logical KV heads, so the right fix is to replicate those heads across
ranks rather than materialize fake extra heads.
This also matches cost modeling better: replication represents
duplicated placement of the same logical data, while repeating would
incorrectly make the tensor look larger than the model actually defines.
**Changes**
- Add a policy helper to detect when KV heads should be replicated
instead of model-sharded.
- Apply that policy to projected K/V tensors in Qwen3.5 self-attention.
- Apply the same policy to KV cache shape partitioning.
- Thread model partition count through the Qwen3.5 inference compilation
path so cache placement follows the same rule.
**Why replicate instead of repeat**
Replicating:
- preserves the logical model shape
- matches grouped-query attention semantics
- keeps cost modeling honest by representing placement, not model
expansion
Repeating:
- would change the graph from 4 KV heads to 8 KV heads
- would increase KV cache size and bandwidth for no new information
- would distort cost modeling by turning a placement choice into fake
tensor growth
**Result**
Qwen3.5 now supports TP setups such as 8 GPUs where KV head count is
smaller than the tensor-parallel degree.
This PR introduces **zml-smi**, a tool to monitor metrics of any
platform supported by ZML, featuring but not limited to:
- real time `--top` mode
- per process tracking
- responsive layout
---------
Co-authored-by: Steeve Morin <steeve@zml.ai>
This PR introduces a script to merge `amdgpu.ids` from two sources: Mesa
and ROCm.
Both provide GPU ID-to-name mappings, but each may contain entries the
other lacks. For example, Mesa includes `AMD Radeon 8060S Graphics`
while ROCm does not, and ROCm lists newer GPUs absent from Mesa.
The script merges the two sources together, with ROCm entries taking
priority in case of conflicts.
Fix ROCm runtime hermeticity in the packaged LLM sandbox by patching transitive ROCm libraries to use zmlxrocm_dlopen and by adding missing libdw/liblzma/libbz2 runtime deps for distroless execution. Also update the ROCm packaging to 7.2.1 and refresh the pinned libpjrt_rocm
```
4cpus:
total: 2m56s
ci-cpu: 2m14
ci-neuron: 2m07s
ci-tpu: 1m58s
8cpus:
total: 2m26s
ci-cpu: 1m37s
ci-neuron: 1m47s
ci-tpu: 1m36s
16cpus (mi8-flex):
total: 2m07s
ci-cpu: 1m24s
ci-neuron: 1m26s
ci-tpu: 1m19s
16cpus (c8i-flex):
total: 1m58s
ci-cpu: 1m17s
ci-neuron: 1m15s
ci-tpu: 1m17s
32cpus:
total: 1m48s
ci-cpu: 1m05s
ci-neuron: 1m06s
ci-tpu: 1m02s
```
RunsOn tends to prefer m8i instance (memory-optimzed) rather than c8i
ones for some reason, so forced the family with `c8+c7`, maybe we could
just use `c8`. But not sure if it's worth being that selective. RunsOn
selects the best there is.
Fixes:
- Added `target_compatible_with = ["@platforms//os:linux"],` for neuron
& rocm
- Updated the zig in neuron
- Updated neuron to use Python 3.12, it's at best a mess to handle two
different python version with `build //...`
- renamed `third_party/flashattn/BUILD.bazel` to
`third_party/flashattn/flashattention.BUILD.bazel` as it's not part of
this module.
- Used HTTPS & fallback URL for swig, sometimes we would end up with an
incorrect archive (wrong sha256)
- Using `--@rules_python//python/config_settings:bootstrap_impl=script`
for hermetic Python build, it relied on the system Python otherwise.
- Used a separate pip extension for neuron to avoid the extra pip args
to leak to other installations.
- Fix `zml/testing` `fn autoCall` function
---------
Co-authored-by: Hugo Mano <hugo@zml.ai>
Move profiling behind a zml-owned primitive instead of exposing PJRT
details in user code. Add the XLA/TSL Bazel deps needed to convert
collected XSpace protobufs into Perfetto traces with XLA/TLS APIs
instead of having a custom fallible converter.
This allows the ZLS server to work.
Before this fix, `bazel run //:completion.print_build_config` would fail
into
```bash
ERROR: no such package 'examples/transfer-benchmark': BUILD file not found in any of the following directories. Add a BUILD file to a directory to mark it as a package.
- examples/transfer-benchmark
```
Files provided by bazel may not exist when zls starts, so
canonicalization would fail and break zls. Now we fall back to the
provided path instead. Furthermore, to make it a bit more robust, if any
part of the post process/canonicalization fails, we return the provided
input string instead of failing.
I'm now also using the same struct definition as ZLS instead of
manipulating `std.json.Value`.
We improve the defer pattern in the Llama example to:
- avoid double deinit of `prefill_exe` and `decode_exe` by cleaning at
future-definition time,
- using defer instead of errdefer for future-allocated resources cleanup
in the main function scope,
- unload buffers from the arena allocator.
This pull request includes several improvements and bug fixes across the
codebase, focusing on memory handling, correctness in tensor testing,
and API consistency. The most significant changes are grouped below:
* Fixed a bug in `zml/testing.zig`'s `testlayer` by updating the
callback signature to take a mutable context and increment the index
correctly, ensuring tensors are correctly initialized.
* Improved memory management for expected output slices in `testlayer`
by switching to `zml.Slice.alloc` for allocation, using `readSliceAll`,
and freeing only the data instead of the whole slice, removing the
potential alignment problems.
* Enhanced test logging to include the tolerance value in success
messages for better diagnostics.
* Increased the evaluation branch quota in `MapRestrict` to support more
complex compile-time type mapping, preventing quota exceeded errors.
* Updated the `fromCStruct` function in `pjrt/pjrt.zig` to accept a
`meta.Struct`.
* Improved handling of output operand aliases in
`mlir/dialects/stablehlo/stablehlo.zig::custom_call` by passing
`output_tuple_indices` for multi-result functions, increasing
correctness in custom call attribute construction.
This pull request includes several improvements and bug fixes across the
codebase, focusing on memory handling, correctness in tensor testing,
and API consistency. The most significant changes are grouped below:
### Tensor Testing and Memory Management Improvements
* Fixed a bug in `zml/testing.zig`'s `testlayer` by updating the
callback signature to take a mutable context and increment the index
correctly, ensuring tensors are correctly initialized.
* Improved memory management for expected output slices in `testlayer`
by switching to `zml.Slice.alloc` for allocation, using `readSliceAll`,
and freeing only the data instead of the whole slice, removing the
potential alignment problems.
* Enhanced test logging to include the tolerance value in success
messages for better diagnostics.
### API and Code Quality Enhancements
* Increased the evaluation branch quota in `MapRestrict` to support more
complex compile-time type mapping, preventing quota exceeded errors.
* Updated the `fromCStruct` function in `pjrt/pjrt.zig` to accept a
`meta.Struct`.
### StableHLO Output Operand Alias Handling
* Improved handling of output operand aliases in
`mlir/dialects/stablehlo/stablehlo.zig::custom_call` by passing
`output_tuple_indices` for multi-result functions, increasing
correctness in custom call attribute construction.
Platform is destined to hold more things, since it's effectively an
object
that is instanciated once per program. Make it a pointer and pass it via
const pointers around.
runfiles are process-wide in space and time.
so it makes sense to keep them around
because they are bound to the process, we use the smp_allocator and
except the memory to be kept around during the lifetime of the process.
* Removed `zml.Context` for `zml.init()` and `zml.deinit()`
* `zml.Platform` is responsible for loading the PJRT api when
initialized. This replace the previous behavior of trying to load all
enabled runtimes.
Use the new zigopt global setting from rules_zig to set -fllvm to all
zig copmile actions.
The Zig self-hosted backend is more fragile that we thought before. It
fails in numerous cases, in debug, but also in release sometimes. USe
LLVM backend until it is more stable.
zig 0.15.2 changed the meaning of `reader.takeDelimiterExclusive` to not
consume the separator. I rewrote the code to use
`takeDelimiterInclusive` since we want to consume the delimiter.
0.15.2 made this test fail with:
112/117 aio.torch.pickle.test.parse protocol 0...error(zml/aio): Unknow
pickle operator @enumFromInt(10), note we are only supporting pickle
protocol up to version 5.
cc @gwenzek
Zig modules that want to access the "c" module needs to have a non empty
CcCompilationContext for now.
So basically depending on an empty cc_library is not enough, it needs to
contain something (defines or headers or includes or wathever)
This PR uses the new release of rules_zig where we upstreamed all the
work from the zml rules_zig fork.
Not much changes from the POV of the public API except for `copts`
becoming `zigopts`.
Also, the global `c` module is only available to Zig modules that depend
on a non empty `cc_library`.
This is behavior that can probably be modified in `rules_zig` later on.
Finally, ZLS support had to be completely rethought to comply with the
change in shape of ZigModuleInfo providers and subsequent rules.
This PR is blocked until `rules_zig` exposes `ZigModuleInfo` for
`zig_binaries`.
This implement support for OpenAI "gpt-oss" models.
It's mostly a take over of #292 branch, but MoE code is kept inside
examples folder cause it's still a bit warm.
In particular the routing logic can break apart when there are undefined
memory in the inputs, where it wasn't an issue for normal MLP.
I only tested with the 20B version.
Revamp the `gather` api to match `scatter`
(prerequesite of #329 )
## Gather new api
Now the syntax is to give a named list of indices. Each name must
correspond to one axis in the gathered tensor.
This match `scatter`, and avoid the user to have to concatenate indices
in the same order than the input axes,
which is error prone cause any screw up will compile but index at the
wrong offsets.
For 1 axis it doesn't change much:
```
// before
self.weight.withTags(.{ .voc, .d }).gather(.{.voc}, .{ idx }, .{});
// after
self.weight.withTags(.{ .voc, .d }).gather(.{ .voc = idx }, .{});
```
But for two it helps a lot:
```
// before
self.weight.withTags(.{ .voc, .d }).gather(.{.voc, .d}, .{ .concatenate(&.{idx_voc, idx_d}, .last)}, .{});
// after
self.weight.withTags(.{ .voc, .d }).gather(.{ .voc = idx_voc, .d = idx_d }, .{});
```
Also adds a `gather_` that offers the same apis but with axis position
instead of names.
This one is more suitable inside library functions when dealing with
unknown names, making it complicated to create the "indices" struct.
I think we could generalize this pattern of offering "non-tagged" api
with an underscore suffix, simplifying the job of LSP.
## topk new API
I realized that most of the code calling topk was renaming the output
axis.
The idea is that logprob.topk(.voc) doesn't return you a tensor with voc
entry but only top_voc entries.
For the values it doesn't make much a difference but for the indices
it's embarrassing and it break usages of gather/scatter.
Now the user of `topK` can either use the unnamed api and pass a number,
of if using the name api must give a new name for the indexed axis.
## Yarn
Support Yarn rope embeddings. It's equivalent to the "llama3" embedding
except they do the interpolation in time space instead of freq space.
Adding the following missing data types:
- u2
- i1
- i2
- f4e2m1fn
- f8e3m4
- f8e4m3
- f8e8m0fnu
(ground truth is PJRT)
I've also realized the code in from/to Float32 wasn't correct and was
truncating to zero to often.
Hence the rewrite and bug fix.
CPU buffers behave differently from Cuda/Rocm/TPU buffers,
and in particular they are addressable from main memory.
This was annoying and prevented us to add some safety checks that
weren't valid for CPU.
I'm adding a tag to buffers so that we know which target they refer to.
Also there is a bug in PJRT unitialized which doesn't respected the
memory. I've opened https://github.com/openxla/xla/pull/31292
but in the meantime I've a workaround in ZML that uses the regular
.fromHostBuffer.
(I've also a patched version of PJRT plugins in
https://github.com/zml/pjrt-artifacts/pull/41 but not merged yet)
We already assign a unique tensor id to each tensor object created by a
BufferStore.
But we weren't using the id to load find the corresponding buffer in the
store, instead relying on the name.
This is allow to decouple the shape of the model in the struct vs the
shape of the model inside the safetensor file.
I also moved the id logic out of Tensor and into BufferStore.
I also updated llama/main.zig to use the new logic.
Bonus:
* fix to `zml.call` tag hashing leading to different MLIR for every
recompile.
* expose Cpu device count to control the number of CPU "devices"
Resuming work from @hugomano #244
This is still a bit experimental.
The core API is here, but there are some caveats.
ZML should help with copying buffers to host memory (which is often
needed) but I'm not sure the current implementation using PJRT is good:
- does it allow graph capture
- what's the overhead ?
We should probably use directly cuMemcpy here instead of going through
PJRT.
Current implem should work with all plugins though.
This fix `tensor.print()`
I've also made a slightly more interesting examples/callback that
manually calls a cuda kernel. (there are other way to achieve this but
this shows you can do whatever you want with `zml.callback`
Note:
the feature was originally named `custom_call` but it was conflicting
with existing bindings for xla builtin custom calls,
that are fully handled by the backend.
So I renamed it to `callback`
---------
Co-authored-by: Hugo Mano <hugo@zml.ai>
- Temporarily disable libnvptxcompiler usage since it's not available in
the PjRT CUDA plugin v13.0
- Add nvshmem in the sandbox, required by the PjRT CUDA plugin
- Deduplicate workflows (introduced by
57fd9cda27)
- Fix linking issue on Linux since upb was introduced
- Fix duplicated `_start` symbol for some tests targets
- Temporarily use `-c opt` instead of `--config=debug` to bypass static
archive bug
Fix#316, close#304
- Bump platform specific Zig code that doesn't get built in CI.
- Restore pytorch loader, using `std.fs.File` instead of `async.File`,
allowing to reuse `std.zip`
- Update `zig fmt` in CI to `0.15.1` and ran it on the codebase
(third_party excluded)
- Remove `stdx.io` which was pointing to remove
`std.io.BufferededReader`.
The Neuron target is still broken though:
`bazel build //runtimes/neuron:libneuronxla`
As well as the Cuda plugin in `--config=debug` mode:
`bazel build --config=debug //runtimes/cuda:zmlxcuda`
Much of the work was done by @gwenzek
Sadly we drop the torch loader in the process, to be added back later !
---------
Co-authored-by: Guillaume Wenzek <wzk@zml.ai>
This PR fixes the neuron runtime with the following:
Proxy the PJRT Api method to enforce the client struct sizes since the
neuron PJRT plugin doesn't use `>=` but `==` to assert them, breaking
PJRT compatibility guarantees.
Fixes https://github.com/aws-neuron/aws-neuron-sdk/issues/1095
Reimplement `libneuronxla` in Zig to control neuronx-cc sandboxing and
invocation.
Implement a python bootstrapper in Zig to create a full blown
`neuronx-cc` executable, avoiding the infamous chicken and egg problem
of python executables boostrapping when sandboxed (due to fixed path
shebangs).
---------
Co-authored-by: Corentin Kerisit <corentin.kerisit@gmail.com>
This PR intends to remove XLA as a bzlmod and transfer it as a non
bzlmod dep.
This is because this module will never be upstreamed as is, so keep it
private.
Also, we fetch llvm-raw and stablehlo from it, which is fine.
While there, dummify the various local_config XLA symbols to please the
imports, as we don't use those parts in ZML itself.
Closes#305
The difference with other sandboxing PRs is that we also define a
trampoline PJRT so file which will prove itself useful to circumvent
PJRT version mismatch because of
https://github.com/aws-neuron/aws-neuron-sdk/issues/1095
Additionally, we artificially create a empty repository so that we can
sandbox the distroless deps inside the neuron module extension.
---------
Co-authored-by: EC2 Default User <ec2-user@ip-172-31-64-93.ec2.internal>
remove usingnamespace from libxev and zig-protobuf
relevant PRs:
* libxev:
7f803181b1
* zig-protobuf: https://github.com/zml/zig-protobuf/pull/1/files
This means we can now build ZML with `-fincremental` (in debug mode only
and linux x86-64)
---------
Signed-off-by: Guillaume Wenzek <wzk@zml.ai>
ROCm counterpart of #275
The most important thing with this change is that ROCm dependencies
(along with the PJRT plugin) must be loaded in the main thread.
This is because some ROCm dependencies (amdhip64) use Thread Local
Storage in the destructor of static C++ instances.
Because all dependencies are unloaded at exit of the main program,
destructors are run on whatever thread exited (in general the main
thread), and any access to the TLS will except values from a different
threads....
This PR introduces CUDA sandboxing.
This essentially means:
- Removing all CUDA related dependencies off the leaf binary.
- Sandboxing the whole CUDA dependency graph.
- Mark dlopen direct dependencies of the PJRT shared library as NEEDED.
- Set the RPATH to the sandbox for all libraries that have sandbox
dependencies.
- Load the PJRT plugin from the sandbox.
And in the case of CUDA specifically for now:
- Enable usage of weak CUDA symbols without linking against CUDA
directly.
This is the non-controversial part of #262
It contains:
* the new `zig_srcs` helper,
* bug fix related to sources in rules_zig,
* cleanups of zml/BUILD.bazel
* test fixes in async/coro.zig (those tests weren't run in CI but now
are)
* disabled nemo and yaml models loader (don't compile anymore since
0.14.0 bump)
The first stack of each new coroutines were holding not zeroed and hold
random memory data, which get popped in rbp during first coroutine stack
swap.
Which made unwinding random or even SIGSEGV when _Unwind_Backtrace was
used (during C++ exception for instance).
Zeroing the first part of the stack makes unwinding stop gracefully when
it encounters $rbp=0x00.
- Use runs-on s3 cache
- Use mlugg/setup-zig@v2 with ZML mirror instead of deprecated and unreachable goto-bus-stop/setup-zig@v2
- Setup Bazel with bazel-contrib/setup-bazel@0.15.0, leverage disk cache and bazelrc flags
- Leverage Bazel `manual` tag to filter out unneeded targets for `//...`
Co-authored-by: Jean-Baptiste Dalido <jb@zml.ai>
* `Buffer.from` is blocking by default
* `Buffer.from` takes options to control the loading This includes
awaiting for the buffer or to which memory the buffer is copied
this remove the need to pass `undefined` to `exe.call` when the
corresponding `.forward` function expects integer/float arguments that
have been inlined in the executable.
This was not needed and could just create confusion.
Now you may end up having to pass void to `exe.call` if the executable
doesn't have any runtime argument,
but it's still better than passing `undefined`.
The only place that was using the non-buffer parts of a `zml.Bufferized`
object was the infamous `zml.testing.compileAndCall`.
So I had to introduce a `zml.testing.BufferizedWithArgs` which is only
used for this function.
I put it under the `zml.testing` namespace to make it clear it's only
for this testing utility.
Note that because `compileAndCall` is strongly typed, you generally
don't need to explicitly mention `BufferizedWithArgs`
and the can just write the arguments in place and get help from ZLS.
* buffer/hostbuffer utils
* more precise f32 -> bf16 conversion
* visit should not read irrelevant fields
* HostBuffer.data -> HostBuffer.bytes()
* zml.aio loadBuffers: type inference
* store expected input shapes in the executable
* zml.meta.visit support std.BoundedArray
* stdx.json.TaggedUnion
* protection against undefined float hyperparams
* avoid meta.TagPayload which can be quite slow
* improve logging
This PR bumps rules_zig to add command line to the zig compilation in
order to enable debug and release configs from the bazel command line
(and bazelrc).
Most notably, this is useful for passing `-mcpu` flags to Zig
(especially for SIMD support like AVX512).
While we're there, add `stdx.time.Duration.hz()` and
`stdx.math.roundeven()`.
currently using async.LogFn inside async runtime itself panics.
So I'm avoiding that by forcing async.zig logging to go through
std.log.defaultLog which is synchronous.
you can reproduce the crash by changing log level to .debug
The extension checks that the ZLS path is absolute, else it will run
which to find it in the PATH.
Add a heading / so that the path looks absolute. This works because we
use ${workspaceFolder}.
This enables making sure versions are the same across the board.
Also, import `triton` repo for later use.
---------
Co-authored-by: Hugo Mano <hugo@zml.ai>
I tried to remove usingnamespace from ZML and it's deps.
Even though `-fincremental` works now, it doesn't provide improved
compilation speed.
I guess because our compilation time are dominated by the linking of all
our C++ deps, and the linking isn't incremental ATM.
I split this in 3 pull requests,
I find the code a bit uglier everywhere, not sure if it's worth it.
* floats.zig this PR
* mlir.zig #204
* asyncio.zig #205
This PR updates the `rules_cc` dependency from `0.1.0` to `0.1.1` and
the Bazel version from `8.0.1` to `8.1.1`.
### Changes:
1. Updated `rules_cc` version in `zml/MODULE.bazel` from 0.1.0 to 0.1.1:
```bazel
bazel_dep(name = "rules_cc", version = "0.1.1")
```
2. Updated the Bazel version in `zml/examples/.bazelversion` from
`8.0.1` to `8.1.1`.
### Context:
1. Initially, running `bazel run -c opt //mnist` failed due to a Bazel
version mismatch (required `8.0.1` but the currently brew-installable
version is `8.1.1`). This was resolved by updating `.bazelversion`.
2. After fixing the Bazel version, the build failed again this time due
to a yanked version of `rules_cc` (`0.1.0`).
> ERROR: Error computing the main repository mapping: Yanked version
detected in your resolved dependency graph: rules_cc@0.1.0, for the
reason: rules_cc 0.1.0 is yanked due to incompatible change (prematurely
removing cc_proto_library from defs.bzl), please upgrade to 0.1.1.
> Yanked versions may contain serious vulnerabilities and should not be
used. To fix this, use a bazel_dep on a newer version of this module. To
continue using this version, allow it using the --allow_yanked_versions
flag or the BZLMOD_ALLOW_YANKED_VERSIONS env variable.
3. Updating `rules_cc` to `0.1.1` resolved this issue.
### Testing:
1. Verified the update by running:
```zsh
git clone https://github.com/jaco-bro/zml
cd zml/examples
bazel run -c opt //mnist
```
2. The build and execution succeeded without errors.
```
info(mnist): ✅ RECOGNIZED DIGIT:
+-------------+
| ### |
| # # |
| # # |
| # # |
| # # |
| # # |
| ### |
+-------------+
```
Modern bert implementation from Victor: @vctrmn
Tested with https://huggingface.co/answerdotai/ModernBERT-base
I started from his PR #149 and cleaned up a bit the code to make it more
what I consider "idiomatic ZML" even if it's still a bit in flux since
we are so young.
---------
Co-authored-by: vctrmn <victormoin.vm@gmail.com>
Co-authored-by: Rene Schallner <rene@zml.ai>
Ensure that our Bazel setup for the framework and examples work through
queries, builds and tests.
Also fix broken code and deps in the codebase.
As of today only Neuron is broken (cf:
https://github.com/zml/zml/pull/111)
Currently if you used the FnCache (through zml.call) on a function with
non tensor fields you would get a struct with undefined memory.
This wasn't nice. So now we copy the value of the function returned by
the first zml.call, and use this as base for future calls, reusing
non-tensor fields, and adding correct mlir.values to the previous
tensors.
This is easy because now zml.call only supports fixed size structs (no
slice/pointer)
---------
Co-authored-by: Corentin Godeau <corentin@zml.ai>
Current implementation of llama is discarding the first outputted token,
and feeding the last prompt token a second time in the model.
Due to how self attention works this wasn't too disturbing for attention
but it was technically wrong.
Also it was doing a bit extra compute for nothing.
---------
Co-authored-by: Corentin Godeau <corentin@zml.ai>
This is a follow up on #162
* restore homemade tokenizer, integrate it with the new zml/tokenizer
library. This will only be used for working with tinyllama, I could also
make it work to support GGUF if needed (couple lines of code).
* this allows us to use the original tinyllama files from karpathy repo
which generates actual sentences. This is important when working on
modeling because subtle bugs can degrade the output quality, which is
not detectable if the output is crap to begin with.
* fix casing of zml/tokenizer api: fromFile, tokenToId. Let's stick to
the Zig style guide. Function written in Rust follows rust convention,
but the Zig bindings should follow Zig convention.
* make tokenizer/main.zig a proper tokenizer test tool reusing
llama/test_tokenizer.zig
~~use native_binary instead of cc_binary~~
Reverted because apparently native_binary break the runfiles and
therefore the cuda sandbox
This PR fixes issue #170 , making Llama run again on CUDA and RoCM.
- Latest Llama changes had introduced an unnecessary batching dimension:
fixed
- Ops returning indices return i32 yet we now use u32 as token indices
-> dtype conversion requred: fixed
Tested on CUDA.
Sometimes we require prebuilt PJRT plugins from outside (neuron,
TPU...), and their `SONAME` are wrong.
Let's ensure all the plugins have their soname proper.
# HF tokenizers & Llama improvements
1. Added HuggingFace tokenizer and SentencePiece bindings, removed old
tokenizers
2. Improved Llama example
## 1. zml/tokenizer: add HuggingFace tokenizers binding, remove old
tokenizers
`zml.Tokenizer` now uses the new HuggingFace tokenizer bindings when
loading a `tokenizer.json`.
To move fast, we dropped the `tinyllama`, `gguf`, and `sentencepiece`
tokenizers.
## 2. examples/llama
Changes to the llama example:
- config.json support -> less cmdline flags
- switched to [zig-clap](https://github.com/Hejsil/zig-clap) for cmdline
args parsing
- using HuggingFace tokenizer
- re-worked and simplified the code
- single token prediction
- removed esoteric models from BUILD.bazel
- default container now uses Llama-3.2-1B-Instruct
---------
Co-authored-by: Steeve Morin <steeve@zml.ai>
Co-authored-by: Corentin Godeau <corentin.godeau@zml.ai>
Co-authored-by: Corentin Kerisit <corentin.kerisit@gmail.com>
The reason for the upgrade is because the //:lock target of the apt
repository checks for bazel marker files to write the lockfile at the
right relative path from the root module.
In 0.3.8, only WORKSPACE and WORKSPACE.bazel files were checked as
marker files, and since we are full bzlmod with a single MODULE.bazel
marker file, the run target would fail like so:
```bash
> bazel run @neuron_bookworm//:lock
Writing lockfile to runtimes/neuron/packages.lock.json
Error: neither WORKSPACE nor WORKSPACE.bazel file was found
```
0.4.2 brings that support.
# Tests
I only tested `cd examples; bazel run //mnist:image` to validate that it
still works. Let me know what else is needed.
* tinyllama tokenizer: rewrite "<0x0A>" token to "\n"
* HostBuffer.isContiguous: don't get triggerd by 1 dim axes
* HostBuffer.slice1d: improve error messages
* module.zig: just print where the .mlir file is instead of cluttering
the screen. most of the time the full IR is not very useful anyway
* setFlags: correctly handle comptime int/float
* tokenizer.zig: don't panic when detokenizing high number just use
`<oob>` instead.
* Buffer.constant -> speedup creation up to 2.5G/s on cuda.
* Bump XLA to 20250122.0-cc075be
* Use nvptx compiler and nvlink with nvjitlink support
* Warn user if cuda path detected in LD_LIBRARY_PATH
* Revert CUDA sandbox fix added here: 6d67fcf and fixed by openxla/xla@a04b262
Fix#134 , superseed #133
Basically the llama3 tokenizer.json has the following config:
```
"model": {
"type": "BPE",
"dropout": null,
"unk_token": null,
"continuing_subword_prefix": null,
"end_of_word_suffix": null,
"fuse_unk": false,
"byte_fallback": false,
...
}
```
this it technically invalid because it says the model has no byte
fallback and no unknown token either.
This means that unknown emoji can't be converted to bytes nor to the
special unknown token.
This ended up in tokenizer.zig using the value of MAX_INT for the UNK
token then panicking for out of bound.
So I've added some logic to recognize that in the case of GPT2 encoded
vocab (like llama3) that actually contains individual bytes, we should
use the "byte_fallback" algorithm even if disabled.
I've also added a Llama3 "well known" normalizer because HF tokenizers
doesn't add a dummy prefix for llama3 tokenizer.
Technically this is still not a correct implementation of llama3
tokenizer because we don't do the regex based splitting they do. But
even for quite long snippets I tested it now produces exact same tokens.
Rework the libxev epoll backend to play nice with zigcoro based
scheduler.
In the process, load the PJRT plugins using `callBlocking` because it
can a bit slow sometimes.
Should have a slight performance increase, too.
This PR is a collection of fixes related to CUDA and ROCm sandbox
discovery.
In the process, update the epoll libxev patch which caused 100% CPU on
linux.
While fixing CUDA add `xla_gpu_enable_latency_hiding_scheduler` as per
NVIDIA's recommendations.
Finally, bump cuDNN to 9.6.0 while we're here.
---------
Co-authored-by: Corentin Godeau <corentin@zml.ai>
This PR merges zigcoro inside the repo and reworks its internals to play
well with `stdx.meta`.
It also adds `Channel.try_{send,recv}` methods as well as dynamically
sized channels (with a `comptime` capacity).
Although not used, it also introduces a `PoolStackAllocator` for
allocating coroutine stacks from a pool
Main issue with current `scatter` implementation is that it uses
broadcasting dims of `stablehlo.scatter`.
While nice in theory, the optimizer doesn't handle them well and they
often are unrolled into while loop.
Here I convert the batching dim to extra iotas indices.
Before we where using `module.op().writeBytecode(writer)` to compute the
hash of a model
but it crashes on some inputs, notably for unused variables.
So I used the text representation of the mlir.
Put the name of the operator in the source location.
This seemed redundant at first because it's not needed when looking at
the stablehlo,
but it's nice to have it to understand the optimization passes.
I'm also including a few QoL:
* sdpa don't need a bias argument, attn_mask already play this role.
* gatherSlices should accept a shape literal `.{ .a = 5, .b = 128 }`
* Shape.outer to combine two shapes into the superset of their axes
* Tensor.all (since we already have Tensor.any)
* argMax doesn't need to specify the dtype, it can be inferred from the
size of the input tensors. If the user care about the dtype they can
manually convert.
This PR is stacked on top of #147
Add pretty printing of HostBuffer.
This will be leverage by the debug helper `x.print()`
It can also be used like this: `std.log.info("my buffer: {}",
.{host_buffer.pretty()})`
Rewrite trivial transpose into reshapes.
Transpose over a 1-d axis are effectively no-op.
Normally the backend can detect this, but we observe that some backend
can get confused. So avoiding those extra transpose can help.
I also changed the default profiler size to 1_000_000 events because
above that the perfetto ui can't load the generated file anyway
* create one folder per target, then one sub-folder per mlir module.
* pass this folder to `xla_dump_to` so that all the xla files arrive
there.
* explicitly save the mlir and the serialized executable there
I'm also exposing James profile conversion tool into ZML as
`profiler.dumpDataAsJson(file)`
Following the new release of the VSCode Zig plugin, rework the code
completion logic to leverage a build runner. This is much lighter,
faster and more reliable.
* add an arena to the CompilationContext
There was already an allocator in the CompilationContext, this PR makes
it into an arena and expose the arena allocator.
This allows ops that need to allocate to do so without abusing
`std.BoundedArray` with arbitrary bounds, and in a safe way.
For now little ops use it, which I believe should stay the case, because
most of the time you want to know how many tensor you're working with.
It'll also simplify the life of Zig new-comers
this include some **breaking changes** to `chunkAllowTrailing` and
`split` that are allocating
* upgrade a few `axis_: i64` into `axis_: anytype` to handle tags
* add a few TODO for the next version of the Tensor API
---------
Signed-off-by: Guillaume Wenzek <wzk@zml.ai>
The Pjrt api allows to passe string flags to the plugin on client
creation.
Since we don't like working with strings I've converted the strings
supported by the cuda plugin into a struct.
For now I've focused on the strings corresponding to the allocator
settings following #125
More settings are not exposed ATM, will follow up with #1323d31c48c71/xla/pjrt/c/pjrt_c_api_gpu_internal.cc (L76-L86)
AFAICT the cpu plugin doesn't read flags at all.
Neuron doesn't have known flags, TPU requires more investigation.
**Breaking change**: `context.autoPlatform()` ->
`context.autoPlatform(.{})`
* testLayer was silently broken by #115 : test + fix so we detect this
in the future.
* mapAlloc had a regression in #115 wrt struct with 0 size fields: test
+ fix
* add Python `venv` to gitignore
Run `bazel build //llama` in CI to make sure we don't accidentally break
it.
I'm also building all the examples target.
For reference it took 5 minutes to run this CI.
llama 3.2 1B and 3B do not have lm_head.
I've added transposing over the embed_tokens as a replacement and set
lm_head as optional for backward compatibility with the other models.
I also added both repositories+executions to bazel and updated the
README.
all code contributed by @steeve
* add intrusive queue
* change the constructor of Channel with default AsyncThread executor
---------
Co-authored-by: Steeve Morin <steeve@zml.ai>
Restore the machinery to emit individual functions and calls to them.
this should unblock composites, and generates cleaner MLIR in general
* functions are cached based on the input shapes including tags.
* donations/tags are propagated through function calls
---------
Co-authored-by: Corentin Godeau <corentin@zml.ai>
* compileInternal now works with function not with "module"
* BaseExe now allocates all the memory needed to call `pjrtExecute`
* `Exe` is the type variant of `BaseExe` and enforce the right Zig arg
are passed to `call`. `exe.prepare(model)` don't need to allocate
anymore, the allocation happened during compilation.
* public entry points like `compile`, `compileModel`, `compileFn` are
now at the top of the file
* the caching logic has been mostly inlined inside `compileInternal`
* `compile`, `compileFn`, `Exe`, `BaseExe` which are the user visible
API have been moved to `exe.zig`
* `CompilationContext` and all the compilation logic stays in
`module.zig`
The current implementation first compute the full cos/sin matrix for the
full sentence,
then slice into it to extract the positions for the current token.
This is generally optimized by the compiler so it's not a performance
issue, but it's a readability issue,
because it means calling `rope` requires first a call to `ropeCosSin`.
I rewrote this to directly produce the optimized IR where we only
compute rope for given offsets.
Mostly:
* fix float8 <-> float32 conversions
* support for `Tensor.constant(.{}, .{ .f8 = 1.0})`
Misc:
* fix small inconsistencies between different versions of sdpa
* better error message for broadcast
* bazelrc: --config=debug
This change reworks the sandboxing and dynamic symbol renaming for the
cuda runtime.
Switch to the jax-cuda-pjrt pre-built PJRT plugin until we fix the pjrt
artifacts build. Most notably because our build of the plugin didn't
include the right compute capabilities.
Take the opportunity to bump the `cuda=12.6.2` `cudnn=9.5.1`.
Fixes#19
Discovered use case:
- PJRT plugin attribute `stablehlo_current_version` is `1.7.8`
- `stablehloVersionFromCompatibilityRequirement(WEEK_12)` is `1.1.0`
- `stablehloGetCurrentVersion` is `1.7.6`
The target version for artifact serialization is `1.7.8` but should be
`1.7.6`.
XLA ref:
2f99455cdf (diff-7f335a7892)
This PR adds support for AWS Neuron / Trainium 1 / Inferentia 2.
It can be enabled via the `--@zml//runtimes:neuron=true` flag.
Everything should work as expected.
---------
Co-authored-by: Corentin Godeau <corentin@zml.ai>
use `zml.ops.while` instead of `zml.ops.for` the problem with the latter is that it concatenate all intermediary results, and thus using lot of memory unless the concat was optimized away which wasn't always the case
- workspace: new module versions added to Bazel registry:
- deps/xla: 4663f04d26 (20241025.0-4663f04)
This version introduces an upgraded overlay with Python repository initialization for the TSL extension. The changes were necessitated by this XLA commit: 19b4ecbe98 (diff-df5471accc) , which triggers a repository call. Python's hermetic version is provided via the .bazelrc file.
- deps/stablehlo: f7f8e4e352 (20241021.0-1c0b606)
- deps/llvm: f7f8e4e352 (20241022.0-6c4267f)
This version includes an upgraded module declaration with rules_python dependency added, as LLVM prepares its overlay for Bazel v8. Relevant LLVM changes here: 3903cb4695
- deps/runtimes: use PJRT artifacts v3.0.0 (https://github.com/zml/pjrt-artifacts/releases/tag/v3.0.0)
- workspace: align used versions of rules_zig in the Bazel dependency graph
- pjrt: expose getPluginAttributes and getPluginAttribute private APIs and stablehloCurrentVersion public API
- pjrt: updated customCallRegistry function with the new FFI provided by the PJRT API GPU extension
Minimal support for this feature was added, as this is not the main focus of the current PR. See related changes in XLA here: ef85a7bcc3
- dialects/mlir/stablehlo: expose getCurrentVersion API and fix broken stablehloVersionFromCompatibilityRequirement API
- zml/pjrtx: serialize IR to the PJRT plugin's supported HLO version. If the plugin attribute is unavailable, it falls back to the StableHLO's compatibility requirement version.
- zml/tensor: tests using batching dims for gather and scatter are now re-enabled
This mostly reworks the runtime to enable proper cross thread signaling
for `callBlocking`.
API changes include:
- `async_ -> asyncc`
- `await_ -> awaitt`
- proper type inference for non generic arguments even for generic
functions
- bump `libxev`
- use `epoll` on linux by default (can be comptime overridden through
`xev_options.linux_backend` in the root module)
Mostly, also, it's pretty fast.
Contains two bug fixes
1. values should be pushed to their block before we open a new block,
otherwise this lead to us assigning values to the wrong block.
* to detect future instance of this bug, and misusage of ZML apis I've
added a few more asserts.
* now every mlir block we create is either "open" or "hermetic", and we
check that hermetic block don't reference variables from outside the
block.
2. in `for_` make sure `first_step` is always used. This is a workaround
for #97. For some reason you need two nested loops to trigger the bug.
Note that this used to work, so I'm not sure what changed in how we
generate MLIR or on MLIR side.
to enable this I had to use the `.pad(.{ ._0 = ... })` syntax. I feel
this could solve a lot of usability usage we have, without having to
introduce too many `.operation1d` operators.
Now zml.ops.makeBlock returns both the mlir.Block created and also the
Tensors returned by `func`.
The returned tensors should not be returned to the user,
because their `mlir.Value` must not escape the block that created them.
But their shapes/tags can be safely propagated further.
This makes `zml.ops.makeBlock` a bit more footgunny, but also it's a
utility function that most users shouldn't use. And it allows to use
tags created inside the block and apply them to the return value. In
particular tags and sharding info should now be propagated more
consistently for while loops, summation, reduceWindow, ...
I've also enabled tests to run on non-cpu platform.
this is useful to figure out if a specific platform has limitation in
it's pjrt support
This one is a chonky.
This commit adds a new `stdx` module with a few utility functions. It is
very far reaching. Forgive the size.
Also, clean the logs in the executables.
* add more scatterSlices testcases
* zml: remove helpers.mapTensors in favor of zml.meta.map
* fix shape when for loop is fully unrolled
* zml.Tensor.pad should accept i64 to be compatible with dims
* allow to have array of tensor inside model struct
* split Buffer.asViewOf into asViewOfHostBuffer, asViewOfDeviceBuffer
Given the output of the last layer of a LM with a `.voc` axis,
Compute indices for the next tokens, following the given sampling
strategy.
The dynamic sampling strategy is more expressive but top_p requires
computing the softmax.
Options are:
* top_k: only sample among the k top scoring tokens,
* max_top_k: limit a compilation time what is the max possible runtime
value for top_k, saving memory and compute by not having to fully sort
the tokens.
* top_p: only sample among top scoring tokens whose probabilities sum up
to top_p
* min_p: drop tokens whose probabilities are lower than a ratio of the
most likely token
It's midly annoying to have to specify iota dtype everywhere to .i32.
In practice if people want to work with floating point range of number
they generally use arange.
Also `iota(x.shape(), 0).convert(.f32)` also work and isn't much work
than saying `iota(x.shape(), .f32, 0)`
For good measure I've added a fallback to .i64 when the given axis has
too many elements. I don't think this would cause much trouble, just
because the tensor that big aren't very common in the first place.
In the API I merged the "precision" and "algorithm" attributes into a
union because according to the spec,
precision must be default when algorithm is set, and precision=default
means "fast" when algorithm is not set.
https://github.com/openxla/stablehlo/blob/main/docs/spec.md#dot_general
For now it's not exposed to users because the API is a bit limited.
In particular only a subset of algorithm are supported by openxla, and
only a subset of that is supported by chips.
I was able to successfully change the accumulation type for dot on Cuda,
but want to experiment more before deciding what to expose.
---------
Co-authored-by: Corentin Godeau <corentin@zml.ai>
* restore the integration with PJRT custom call extension
* add a generic `zmlHostBufferCallback` custom call that copies the
received tensor to the host, and call a user supplied callback on it.
* add a default "print" method on `Tensor` that print the runtime value
of a given tensor
* print itself has very little code, it's easy for a user to define
their own print if they don't like ours.
limitations:
* only supported on Cuda atm
* pre allocates a buffer on the host to copy the content of the device
buffer,
this buffer won't be freed. There is exactly one buffer per "print" call
in the IR.
* it uses cuda api, maybe I should be using `createViewOfDeviceBuffer`
to make it less cuda specific
We currently have an issue with our version of openxla not being able to
serialize the new "batching_dim" attribute of gather/scatter.
This is a version mismatch we should fix, but also it's bound to happen
again, so since the compilation can fail, I'd rather return an error in
that case.
In an async scenario, it is very easy to footgun yourself and block the
reactor, especially with Events.
Reintroduce pjrtx that deals with the specifics.
This implements the `async` module using a thread pool. This is now the
default.
Implementations can be swapped using the
`@zml//async:impl=[threaded,zigcoro]` flag.
Because io_uring is not enabled in the default seccomp profile, some
container implementations will block on io_uring calls. Use the threaded
implementation to work around this, for now.
For sharded safetensor files, we already have a separate json file we
need to read.
Let's allow writing metadata there, to avoid having a separate json file
with the model config.
- triangular: zeroes out the upper right part of a matrix. Options
control how much is actually zeroes with offset from the diagonal.
- toDiagonal: equivalent to pytorch "diag_embed". The base case is
converting a vector into a diagonal matrix. We already have a `diag`
operator doing this, but the name was wrong cause in general `diag` does
the reverse operation of extracting the diagonal from a matrix into a
vector. Also it was using old style ZML, and was a bit too restrictive
on input shapes.
- elu: one more activation function. it's getting a bit out of hand, I've put it under zml.nn.Activation
This PR adds the "cumulative sum" operator.
This is based on what jax does, which is leveraging reduce window
operator,
where for each n in [0..N] you compute the some of elements [0..n]
I'm not sure how this get optimized in complex programs, to be
investigated.
**BREAKING CHANGES**:
* `reduceWindow` is still pretty hard to used but I merged the
`padding_shape` and `padding_value` into one operator and made the
signature more explicit.
* this also lead me to change a bit how `maxPool1D` and `maxPool2D` take
arguments, using proper tuples instead of slices that need to be of
specific len.
* `pad` now take options using
* "tagged syntax":`x.pad(0, .{ .a = .{ .low = 1, .high = -1 }});`
* or the "AOS syntax": `x.pad(0, &.{.{}, { .low = 1, .high = -1 }});`
* the "SOA syntax" is **REMOVED**: `x.pad(0, .{ .low = &.{1}, .high =
&.{-1} });`
Add a tiny abstraction layer between ZML and PJRT.
Most notably, to implement custom logic to decide wether or not we
should/can load a PJRT plugin.
For instance, `libtpu` will block the program trying to reach the GCP
metadata server, which doesn't exist outside GCP.
This also allows for comptime detection of wether a runtime is linked
with the program, and its dependencies.
If we start bazel built binaries inside a ZML process, the environment
has to be configured properly for the sub processes to find their
runfiles properly.
Do it process wide at context start.
Some runtimes do not implement `PJRT_Event`. For those, we must dispatch
on our own thread pool for buffer copies and wait on events only if the
runtime returns them.
If there are no events, assume all operations are synchronous.
`asyncc` is arguably nicer to write than `async_`, so use that.
Also, add an explicit `Generic` version that can work with `anytype`
functions (but on which type inference will be disabled).
Currently, in ROCm, we use
`__attribute__((ifunc("_zml_rocm_resolve_dlopen")))` to make sure
`dlsym(RTLD_NEXT, "dlopen")` will resolve to the real `dlopen` and not
the always failing stub.
This is a bit brittle.
Instead, leverage `patchelf`'s `rename_dynamic_symbols` feature to
import a custom `dlopen` implementation, and forward to the real one
normally.
For `silu`, `logistic` is more precise than `sigmoid` since the first
one is bound to StableHLO while the second one is a custom
implementation. Implemented `logistic` implementation in `sigmoid`
and aliased `logistic` to `sigmoid`.
* add support for `Tensor.concatenate(&.{x, y}, .d)`
* start removing some `broadcastLeft` usage
* use f32 to compute head scaling constant in transformer. We should
always compute constant with max precision.
* use more complicated test cases with long integers
* add tests for protocol 0. Pickle as evolve a lot since protocol 0 was
there,
and it's unlikely we receive such pickle, I thought it was justified
since a big part of the code is related to protocol 0 anyway.
* add tests to the "zipped pickle"
* simplify structs hierarchy -> `PickleData` has been merged with
`Parser` into `torch.File`.
* rename `torch/value.zig` -> `torch/py_object.zig`
* intermediary results: []pickle.Op, []py_object.AnyPy, should not be
stored inside objects but passed around function calls.
* we should use buffered reader to parse the pickle and the zip headers
* python Dict should not be represented by a singleton seq of seq of seq
of kv, just a single seq with keys and values following each other is
enough (like in pickle protocol)
* kwargs should not be mixed with args
* added harder pytorch test with a raw tensor and a `torch.nn.Conv2d`
module
* simplified a bit lifecylcle of `BufferStore` by passing the files
during init.
Part of ongoing improvements to the torch loader
* rename zml.aio.Value -> zml.aio.Metadata
* simplify the types zml.aio.Metadata can have
* pickle `parse` function moved to `pickle.zig`
* add dedicated test case
* fix small bug in parse where `short_binstring` was converted to
`bin_string`
* pickle `evaluate` function now returns `[]const zml.aio.torch.Value`
note: I didn't modified the core logic.
`testLayer` was broken because it wasn't directly used by `llama.zig`.
Remove the uneeded "compile_options" argument,
update the testing logic to the new sharded output.
We had issues with the CodeLlama tokenizer.
https://huggingface.co/codellama/CodeLlama-7b-hf
Looking at the config, it's actually a "sentencepiece" tokenizer, that
uses the special "▁" token to represent spaces.
```
"normalizer": {
"type": "Sequence",
"normalizers": [
{
"type": "Prepend",
"prepend": "▁"
},
{
"type": "Replace",
"pattern": {
"String": " "
},
"content": "▁"
}
]
},
```
I've added a `Normalizer.fromHfJson` that reads this json and tries to
convert this style to our options.
This config can express more things than we can, so it's a best effort.
I've also made the magic space token configurable, requiring adding a
small buffer in Normalizer to store it.
There is a small change that can have importance, is that now in most
cases the setting "remove_extra_whitespaces" will not be set, meaning
that user input will be tokenized differently in face of double spaces.
I generally don't think this is a good practice, but it respect HF
defaults.
I've also added a `debug` boolean on the tokenizer struct that shows the
individual merges of tokens.
It helped me previously when debugging tokenizer issues.
Before you needed to explicitly uncomment some code to use it, now it's
behind a runtime boolean.
---------
Co-authored-by: Rene Schallner <rene@zml.ai>
safetensor loader was segfaulting when loading metadata with an empty
json array.
I also rewrote some of the torch loader code to improve the readability.
Notably every slice in `ops.zig` is now const. I've also realized that
our strategy to load Pytorch tensor is not future proof see the warning
at: https://pytorch.org/docs/stable/storage.html
- Introduce sharding hints from the user code to specify how operations
should be split across user's devices mesh leveraging the HLO sharding
annotations
- Remove options arguments in `zml.module` including
`opts.add_donations_attributes`: it's for the function caching because inner function can't do donation, so we don't want to put the donation attributes there but for now we it's removed and we will do it differently next time around
The end goal is to allow zml users to use our async test runner in their
project.
This approach works but is a bit hacky because it relies on the fact
that `test_runner.zig` is part of the `zml` library
otherwise `zig_test` doesn't correctly put the test_runner into the
build sandbox.
I made this dependency explicit, but it was already the case because of
the glob pattern used in zml library.
And I think `test_runner` should be `zig_library` in the first place,
and not a filegroup.
See https://github.com/zml/rules_zig/issues/2 for follow up.
Indicate that it's also the standard location used by the CLI, and add
the environment variable, too.
---------
Co-authored-by: Rene Schallner <rene@renerocks.ai>
pjrtx.zig was mean to be a ZML specific wrapper over pjrt.zig.
But pjrt.zig in its current form is pretty usable without wrapper,
the extra layer of indirection is IMHO confusing because it mostly
transfer the calls directly to pjrt.zig but make some call suspending.
I moved the few helpers left in pjrtx.zig where they are used, ie
`pjrtx.Buffer.bufferFromHostBuffer`-> in `buffer.Buffer.from`
`pjrtx.Client.compile` -> in `module.compileModuleToPjrtExecutable`
`pjrtx.Event.await` -> `zml.Platform.awaitEvent()`
This also fix a bug in `pjrtx.Event.await` which was blocking for the
event before suspending for the event.
Some tokenizers have bytes encoded in hex like this: "<0x40>".
This break the tokenization algorithm because the input text will
contain "@" not "<0x40>",
and if the input contains "<0x40>" it needs to not be treated as a
single byte.
So we replace byte fallbacks strings, by their corresponding character.
This enables the normal tokenization algorithm to work.
This PR also implement the main byte fallback logic, that split unknow
utf8 codepoint into bytes.
---------
Co-authored-by: cryptodeal <james@zml.ai>
First rough support for in-process sharding.
The goal of sharding is to allow leveraging several GPUs to collaborate
to run one big model.
Current implementation:
* added `platform.sharding()` describing how the set of devices managed
by the current process should be used.
* added a `_sharding_info` to shape.
* A tensor / shape axis can be marked as needing sharding using
`.withSharding(.{ .a })` or `.withSharding(.{ 1 })`.
* When using `populateModelWeights` tensors with sharded axes, will be
populated by sharded buffers.
* when generating the MLIR bytecode for a ZML function the **input**
arguments, will see their `_sharding_info` converted into
`mhlo.sharding` attributes