Splits the monolithic mlir/dialects library into one Bazel package per
dialect, then makes every package follow the same layout, C API and Zig
API. Each package owns its Zig bindings, its C API when upstream doesn't
provide one, and its tests. Dialect registration moves out of the core
mlir package, so a target only pulls in the dialects it actually uses.
The conventions are documented in `mlir/dialects/README.md`, with a
porting checklist and conformance checks.
Along the way:
- cute_ir is flattened, and its Cuda, CuteNVGPU and NVVM sub-dialects
become their own packages
- fly's ROCDL extension moves to fly_rocdl
- new packages for ROCDL and UB
- kernels/ is renamed to dsl/, since it holds the DSLs
- every package registers through its own registerDialect, and each DSL
through registerDialects, instead of dialect name strings
- the C and C++ follow the LLVM and MLIR C API conventions, formatted
with a new mlir/.clang-format
- BUILD files share one template
- accessors are named as nouns like in the rest of ZML
- each package tests itself in its own test.zig, including that every
bound operation is registered and that every wrapper method compiles
Fixes:
- memref atomic_rmw emitted the wrong kind, because the AtomicRMWKind
values didn't match the pinned LLVM
- the pinned Triton and Mosaic no longer have tt.cat and tpu.repeat, so
the DSL's cat and repeat now lower the way tl.cat and pltpu.repeat do
- the ROCDL bindings are completed: buffer address spaces, vector types,
waves_per_eu range
Breaking changes:
- register dialects through their package instead of
DialectHandle.fromString, and mlir.registerFuncExtensions becomes
func.registerExtensions
- accessors drop the get prefix, so getValue() becomes value()
- ttir.cat and mosaic_tpu.repeat are removed, and stablehlo pad and
mosaic_tpu sort take their options before the location
- zml.kernel.fly.rocdl becomes fly_rocdl
- Mesh.sdyMeshAttr takes a context and returns an attribute instead of
text
- func.func no longer prints public, its default visibility
---------
Co-authored-by: Steeve Morin <steeve@zml.ai>
In Buffer code we already have explicit sharding, but not for tensor.
I'm adding a `zml.Sharding` to `.withPartitioning` to make it explicit
which mesh we are working with.
To simplify the migration, `.withPartitioning` in Tensor code can use
`.model` syntax to lookup a sharding by name.
This replace the previous logic of `selectSharding(shape)` which would
look up a sharding that matches the axes of the given partitioning which
could yield surprising behavior.
This also modifies how sharding are stored in zml.Shape.
Before it was storing a list of axis names + a tag (2 * 8 * 64bits), now
I'm storing a zml.Sharding (64 bits) and a compact PartitionArray (32
bits) that references the logical axis of the sharding by offset instead
of by name.
There is an impact on the `TensorRegistry` because it also needs to be
aware of the possible shardings, instead of delegating the matching
partition spec -> sharding to loading. This means error will be detected
sooner.
`store.createTensor` accepts `@enumLiteral()` to lookup the relevant
sharding.
This also simplifies `zml.Buffer` because the full sharding is now
stored in the shape, so it doesn't need to be passed independently. It
also means that it will propagate from the tensor who are graph inputs,
to the buffers that are eg inputs.
`Placement` logic also got a bit simpler.
Record donated input indices during compilation and destroy their
PJRT_Buffer handles once execution is enqueued for `zml.Exe.Runner`.
I'm not doing this automatically in `call()` because it's a bit
invasive. Not all the code uses the pattern of overwriting an input
buffer like `run(.{cache}, .{&cache})`, but most do. So I'm cleaning it
up on easy path and leave `call` as is for finer control.
Removing callOpts & runOpts to simplify the API and changing `io` to be
required for API clarity. I'm replacing `zml.testing.autoCall` with
`Exe.eval` which solves most needs in a simpler way IMHO
Two things, the second one falling out of the first.
**`fp8_block32`**: E4M3FN weights with one scale per 32x32 tile, the
layout
DeepSeek V4.1 ships. `quantizeBlockFp8` takes a block size and a scale
dtype;
the E8M0 path floors amax at 1e-4 and rounds the exponent up, so the
activation
side is MXFP8: one E8M0 scale per row per 32 contracted values. That
pairing is
what XLA's fp8_block_gemv arm needs to hand both scales to
`tt.dot_scaled`
(`Fp8BlockGemvScaleInMma` requires E8M0 on *both* sides); F32 activation
scales
would quietly fall to the per-block multiply. The Triton MoE path gets
the same
scheme at `block_size_k = 32`.
`Linear.forward` also stops rounding the activation to BF16 before
quantizing,
so amax is computed on the input as given.
**An explicit output dtype** on `scaledDot` and `Linear.forward`.
Inference
cannot answer what an FP8 x FP8 product should return: the accumulation
is F32
either way, so BF16 or F32 back is the caller's choice. `forwardScaled`
now
hands the caller's type to `scaledDot` rather than rounding through BF16
and
converting after. Narrow outputs are rejected.
Separately, `Linear.forward` converts the weight and bias to the
activation
dtype. `dotGeneral` asserts both operands share one, which broke the
moment a
model raised its activation to F32 against a BF16 weight; the wrappers
this API
replaced converted for exactly that reason.
**Breaking:** `Linear.forward` takes a second argument and `scaledDot` a
fifth.
---------
Co-authored-by: Raphael Nahoum <raphael@zml.ai>
Migration instructions:
zml.module.CompilationContext -> zml.Compiler
zml.module.CompileError -> zml.Compiler.Error
zml.module.CompilationOptions -> zml.Compiler.Options
zml.module.compile -> zml.Platform.compiler or zml.Compiler.compile
Breaking changes: none, I've kept a small compatibility layer at the top
level, with upgrade instructions.
We can remove it in a second time.
**Summary**
This PR update to Neuron v2.30, introduces the Neuron NKI stack into ZML and wires it into the
LLM attention path.
The main goal is to make Neuron attention run through NKI kernels.
It adds the compiler/runtime plumbing needed to embed NKI kernels as
Neuron custom-native calls, then uses that path for SDPA-style
prefill/decode attention. Paged Attention will come later.
On inf2.8xlarge it runs :
- 25 tok/s for Llama 3.1 8B BF16
- 63.5 tok/s for LFM2.5-1.2B-Instruct BF16
- 19 tok/s for Qwen3.5-9B BF16
Two fixes to the `write_first_model.md` tutorial:
- `allocator` on line 221 is deinitialized, however, `io` is not. `io`
should be deinitialized as well.
- `sharding` on line 216, is an undeclared identifier. It should be
replaced with `.replicated`, which is used in the full code example.
This PR modernizes ZML profiling in four areas:
- replaces the old C++ `xspace_to_perfetto` tool with a native Zig
exporter built on upb
- integrates trace export directly into `zml/profiler.zig`
- introduces a scoped tracing API in `zml.tracer` and wires it through
examples
- adds platform-specific annotation support and Bazel-native profiler
wrappers for CUDA and ROCm
The result is a simpler profiling flow, better large-trace behavior, and
a more consistent experience across Linux CUDA, Linux ROCm, and macOS.
This PR simplifies the common replicated case while making sharding
intent explicit and use sharding by pointer ( sharding struct is quite
big: ~600 bytes).
Using pointers for shardings also avoids unnecessary sharding copies and
makes it easier to share stable sharding instances across compilation,
loading, and runtime paths.
- `Platform` owns the canonical replicated sharding
- `Buffer` APIs now take explicit sharding specs:
- `.replicated`
- `.{ .sharded = sharding_ptr }`
- `createTensor` / `maybeCreateTensor `now require explicit partitioning
instead of accepting null
- compilation/loading now use pointer-based sharding lists
This makes simple cases like mnist and benchmark much cleaner: callers
no longer need to manually construct and thread a replicated sharding
just to use the default case, but they still state placement explicitly
at the call site.
---------
Co-authored-by: Hugo Mano <hugo@zml.ai>
CPU buffers behave differently from Cuda/Rocm/TPU buffers,
and in particular they are addressable from main memory.
This was annoying and prevented us to add some safety checks that
weren't valid for CPU.
I'm adding a tag to buffers so that we know which target they refer to.
Also there is a bug in PJRT unitialized which doesn't respected the
memory. I've opened https://github.com/openxla/xla/pull/31292
but in the meantime I've a workaround in ZML that uses the regular
.fromHostBuffer.
(I've also a patched version of PJRT plugins in
https://github.com/zml/pjrt-artifacts/pull/41 but not merged yet)
This PR bumps rules_zig to add command line to the zig compilation in
order to enable debug and release configs from the bazel command line
(and bazelrc).
Most notably, this is useful for passing `-mcpu` flags to Zig
(especially for SIMD support like AVX512).
While we're there, add `stdx.time.Duration.hz()` and
`stdx.math.roundeven()`.
# HF tokenizers & Llama improvements
1. Added HuggingFace tokenizer and SentencePiece bindings, removed old
tokenizers
2. Improved Llama example
## 1. zml/tokenizer: add HuggingFace tokenizers binding, remove old
tokenizers
`zml.Tokenizer` now uses the new HuggingFace tokenizer bindings when
loading a `tokenizer.json`.
To move fast, we dropped the `tinyllama`, `gguf`, and `sentencepiece`
tokenizers.
## 2. examples/llama
Changes to the llama example:
- config.json support -> less cmdline flags
- switched to [zig-clap](https://github.com/Hejsil/zig-clap) for cmdline
args parsing
- using HuggingFace tokenizer
- re-worked and simplified the code
- single token prediction
- removed esoteric models from BUILD.bazel
- default container now uses Llama-3.2-1B-Instruct
---------
Co-authored-by: Steeve Morin <steeve@zml.ai>
Co-authored-by: Corentin Godeau <corentin.godeau@zml.ai>
Co-authored-by: Corentin Kerisit <corentin.kerisit@gmail.com>
The Pjrt api allows to passe string flags to the plugin on client
creation.
Since we don't like working with strings I've converted the strings
supported by the cuda plugin into a struct.
For now I've focused on the strings corresponding to the allocator
settings following #125
More settings are not exposed ATM, will follow up with #1323d31c48c71/xla/pjrt/c/pjrt_c_api_gpu_internal.cc (L76-L86)
AFAICT the cpu plugin doesn't read flags at all.
Neuron doesn't have known flags, TPU requires more investigation.
**Breaking change**: `context.autoPlatform()` ->
`context.autoPlatform(.{})`
This PR adds support for AWS Neuron / Trainium 1 / Inferentia 2.
It can be enabled via the `--@zml//runtimes:neuron=true` flag.
Everything should work as expected.
---------
Co-authored-by: Corentin Godeau <corentin@zml.ai>
This mostly reworks the runtime to enable proper cross thread signaling
for `callBlocking`.
API changes include:
- `async_ -> asyncc`
- `await_ -> awaitt`
- proper type inference for non generic arguments even for generic
functions
- bump `libxev`
- use `epoll` on linux by default (can be comptime overridden through
`xev_options.linux_backend` in the root module)
Mostly, also, it's pretty fast.
`asyncc` is arguably nicer to write than `async_`, so use that.
Also, add an explicit `Generic` version that can work with `anytype`
functions (but on which type inference will be disabled).
Indicate that it's also the standard location used by the CLI, and add
the environment variable, too.
---------
Co-authored-by: Rene Schallner <rene@renerocks.ai>
**Changes**
- zml_utils: fix `AttributeError: type object 'ActivationCollector' has
no attribute 'CollectionOver'`
- zml_utils: remove dead code and comments
- docs(howto_torch2zml): add explanation on why `output` variable can be
`None`