This enables runtime selection of the std.Io runtime. This also fixes
the compilation hangs by using zio.blockInPlace for CPU bound ops.
This may require some more `zio.blockInPlace`, so don't merge until
we're sure.
* `xla_gpu_experimental_use_collective_kernels=CONCURRENT` allows
parallel execution
* `xla_gpu_experimental_use_collective_kernels=SKIP_TEMP` avoids graph
updates
* `xla_gpu_experimental_use_collective_kernels` disables custom
collectives while we are looking for a fix
* `xla_gpu_enable_command_buffer` adds collectives to the graphs
in LTO, the linker would garbage collect that exported symbol since it
wouldn't be used inside the final binary
forcing it undefined via a link flag prevents it from being removed.
Allow block-32 scales to accompany E4M3FNUZ weights after load-time
normalization on CDNA3. The tensor layout and quantization scheme stay
unchanged, so existing Linear paths can use native FP8 conversion
without materializing software-decoded FN weights.
Add a [zio](https://github.com/lalinsky/zio) dependency.
We require ZML to be aware of which `std.Io` implementation it's running
because we use `threadlocal` to store the current `zml.Compiler`
instance when compiling a graph (to make the DSL pleasant to use).
This is our only `threadlocal` usage but it is one too much. So we must
make it compatible with `zio.TaskLocal`. Not a big deal, but this
requires opt-in from the user.
And so `zio` support can be opt-in enabled by setting the build flag
(default is `std`):
```
--@zml//zml:io_impl=zio
```
Sadly there is no `std.Io` response to `threadlocal` (or task local).
Hopefully this will be fixed in the future.
Note that this creates problem if using `std.Io.Evented`.
Introduce `zig_option` and `zig_options` rules to generate the bazel
equivalent of option modules plugged on bazel flags.
On `string` flags, if the flag has a `allowed_values`, the `string` is
converted into a `enum` in zig code.
Also, port platforms to this and remove the ugly `defines` hack to
comptime test for platforms enablement.
In the process, port `arocc` to the `zig_options`.
Two things, the second one falling out of the first.
**`fp8_block32`**: E4M3FN weights with one scale per 32x32 tile, the
layout
DeepSeek V4.1 ships. `quantizeBlockFp8` takes a block size and a scale
dtype;
the E8M0 path floors amax at 1e-4 and rounds the exponent up, so the
activation
side is MXFP8: one E8M0 scale per row per 32 contracted values. That
pairing is
what XLA's fp8_block_gemv arm needs to hand both scales to
`tt.dot_scaled`
(`Fp8BlockGemvScaleInMma` requires E8M0 on *both* sides); F32 activation
scales
would quietly fall to the per-block multiply. The Triton MoE path gets
the same
scheme at `block_size_k = 32`.
`Linear.forward` also stops rounding the activation to BF16 before
quantizing,
so amax is computed on the input as given.
**An explicit output dtype** on `scaledDot` and `Linear.forward`.
Inference
cannot answer what an FP8 x FP8 product should return: the accumulation
is F32
either way, so BF16 or F32 back is the caller's choice. `forwardScaled`
now
hands the caller's type to `scaledDot` rather than rounding through BF16
and
converting after. Narrow outputs are rejected.
Separately, `Linear.forward` converts the weight and bias to the
activation
dtype. `dotGeneral` asserts both operands share one, which broke the
moment a
model raised its activation to F32 against a BF16 weight; the wrappers
this API
replaced converted for exactly that reason.
**Breaking:** `Linear.forward` takes a second argument and `scaledDot` a
fifth.
---------
Co-authored-by: Raphael Nahoum <raphael@zml.ai>
`Buffer.createView` can be used to create a view from a host pinned
buffer and use it as if it was one the device. This avoids XLA
limitations of doing operations directly on host pinned buffers.
This only works on OneAPI and Cuda for now.
For other platform we would need a writer that allocates host pinned
uninitialized buffers for the shards and write directly to them.
The CPU plugin archives now ship the shared library under lib/, and
copy_to_directory outputs to sandbox/ instead of sandbox/lib. Look up
the sandbox directory in runfiles and join lib/ when building the
library path, matching metal, rocm and cuda.
Adds a Zig front end for NVIDIA's CuTe DSL, shaped after `cutlass.cute`,
plus the `__gpu$xla.gpu.cute` custom call that compiles and launches it
on CUDA.
- `mlir/dialects/cute_ir`: bindings for the `cute` and `cute_nvgpu`
dialects
- `zml.ops.cute`: emits the custom call with the textual module, `grid`
and `block`.
- `kernels/cute` + `zml.kernel.cute.Kernel`: builder, values, tensors
with coordinate `get`/`set`, thread/block indices, static smem, sync,
`scf`. See `kernels/cute/README.md`.
- bump `xprof-nightly`
---------
Co-authored-by: Steeve Morin <steeve@zml.ai>
Adds a Zig front end for the FlyDSL dialect, so AMD GPU kernels can be
written in Zig alongside the existing Mosaic, Triton and tile ones, and
dispatched the same way.
Types and layouts are built through the dialect's own C entry points
rather than by printing and reparsing assembly. That holds in both
directions: host-side layout literals become attributes directly, and
anything the dialect computes is read back through typed accessors.
Launch geometry is derived from the device rather than hard-coded, so
the same kernels run on both AMD architecture families and the tests
stay architecture agnostic. Everything builds and tests on any host, no
GPU needed.
Also carries one unrelated cleanup, dropping a needless null terminator
from the Triton and tile kernel strings.
## Summary
- `zml-smi --top` was stalling under load because each GPU and the
process list hit NVML on their own thread. Those calls serialize on the
driver RM lock, so one slow ioctl froze the whole TUI.
- All NVIDIA queries now run on a single poller.
`nvmlDeviceGetPcieThroughput` (it blocks while sampling) is only taken
on the first tick and every fourth tick after that; link speed/width
stay on the hot path.
- Process-query sizes above 4096 are ignored, and a failed per-process
util sample no longer skips the rest of that GPU.
---------
Co-authored-by: alhussein jamil <alhussein@zml.ai>
- The current Fp8 scaling wasn't properly finished. We didn't use the
scales at all.
- Adding a `quantize_input` option that GLM5.3-Flash uses
- `applyActivation` wasn't properly using the threshold for relu
- The bias was silently ignored for the MoE.
`makeFusedMoeConfig` was just adding extra complexity and we used `a`,
`b` and `c` everywhere. I think it's fine in the triton kernel itself,
but outside of it's just confusing IMHO.
I removed a bunch of useless parameters also that just end up returning
an `Unsupported` error.
Fix an issue when `padded_num_experts` computed doesn't land as a power of two. Notably for DeepSeek-v4 Pro with its 384 experts, divided by 32 warp is not of power of two, leading to a crash in the triton which expects a power of two number of experts.
Imitating `platform.cuda.computeCapability` as fp8 support is native for
e4m3fnuz on gfx942 but e4m3fn for everything else that has hardware
support AFAIK.
A third kernel language beside Triton and Mosaic TPU. It emits NVIDIA's
cuda_tile dialect and reaches the GPU through XLA's
`__gpu$xla.gpu.cuda_tile` custom call.
- `third_party/cuda_tile`: cuda-tile v13.3.3
- `mlir/dialects/cuda_tile`: every type and enum through the dialect's C
API, one constructor per op (all 100, in Ops.td order), and a small C
shim for the TileView interface the C API does not expose.
- `kernels/cuda_tile`: views, tokens, for/if/loop, reduce/scan (single
and variadic), atomics, MMA, every elementwise op.
- `zml.kernel.cuda_tile.Kernel` and `ops.cudaTile`: the custom call,
with a grid and no num_warps/num_stages (those are optimization_hints in
the IR).
Every CUDA gate used to parse the compute capability string itself. They
now share `platform.cuda.computeCapability`, which returns null off CUDA
and offers `.is` / `.atLeast` / `.sm`.
Also folds the NVFP4 scale clamp into its reduce to remove one launch
per projection, and is equivalent.
The fork keeps upstream's flag defaults, so zml turns on what each
target actually wants:
- the Tile IR scaled-dot backend, subchannel dequantize fusion and
Triton multi-output fusion on CUDA
- Metal AIR fast math off, since it lets the AIR compiler assume no NaNs
and is_finite(NaN) comes back true.