0
Fork 0
mirror of https://github.com/zml/zml.git synced 2026-11-16 19:11:09 +00:00
Commit graph

71 commits

Author SHA1 Message Date
Guillaume Wenzek
a7a4fcd3f8
zml/sharding: use logical meshes (#943)
## Motivation

This is the big sharding rewrite we need to achieve complex sharding.

Currently we always resolve a mesh logical axis to a mesh physical axis
(or several of them).
But on GPU we always have 1D topology, so we can't have 2D logical
meshes.
So this PR drops the logical -> physical axis bindings, and instead
store a logical -> devices binding.
This is more flexible, and has the nice side effect of simplifying
`Buffer.from`

On stablehlo side this changes how we generate the MLIR because we now
use logical meshes, and `manualComputation` uses logical axes.
(In shardy, GSPM is not affected AFAICT because it uses device ID
directly)

## Buffer.from

This also accelerates `Buffer.from` that I rewrote to be FAST.
Version | Fixed cost | Per device
-- | -- | --
origin/master | 109.6 ns | 35.39 ns
Rewrite | 35.8 ns | 4.38 ns
Improvement | 3.1× | 8.1×

## Creating mesh

I'm adding a new syntax to create LogicalMesh, to allow the more complex
meshes we want to do, and in particular, I now allow to force a specific
number of dimensions.
You can see it in action in `mnist.zig`. 

```
const dp_mp = try platform.registerMesh("DP-MP", try .init(
        .{ .data, .model },
        &.{ .low_bandwidth, .high_bandwidth },
        &.{ 2, -1 },
    ));
 
info: topology used: Mesh(name=DP-MP, partitions=4, replicas=1)
┌ Mappings (logical coordinates from physical coordinates):
├── data (low_bandwidth) -> (bus / 2)
└── model (high_bandwidth) -> (bus % 2)

┌ PhysicalMesh(platform=cpu shardable_axes={ .bus } num_devices=4)
└── [bus] 0..4 (tree)
    ├── Device(id=0, coords={ 0 })
    ├── Device(id=1, coords={ 1 })
    ├── Device(id=2, coords={ 2 })
    └── Device(id=3, coords={ 3 })

┌ Logical coordinates:
└── data=0
    └── model=0,1 -> devices=0,1
└── data=1
    └── model=0,1 -> devices=2,3
```

The previous syntax is still available, and ZML will align logical axes
to physical axes like we do before this PR.
This part as unit tests, and I checked it work as I expect on CUDA but I
would need to double check the logics still works on TPU/Neuron, because
there is a few platform specific branches in that code.

## Test plan

As part of the benchmark Codex generated a test suite with hard-coded
offsets, allowing to check that Buffer.from didn't change behavior
through the commits, checking we correctly compute each individual
shard.

TODO check that LLMD perf isn't degraded.
2026-10-16 15:27:30 +02:00
Guillaume Wenzek
7bb457ebeb
zml: refactor manualComputation (#925)
here I'm promoting a pattern I would like to extend to more of ZML:

function that takes `anytype` argument should do as little work as
possible and called a fully typed function.

I kept `manualComputation` signature to not break existing code,
I moved the code to `zml/ops/manual_computation.zig` to better
encapsulate the required helper functions.
(and `zml/ops.zig` is pretty big atm)

Similarly `Sharding.Partitioning.parse` now calls `const logical =
comptime Logical.parse(partitioning);` then `logical.applyUseAll(tags,
mesh, default_value);` which is fully typed.

I'm also documenting `manualComputation` better
2026-10-16 12:30:05 +02:00
Guillaume Wenzek
00a0af4a55
zig 0.17 (second attempt) (#923)
Bump for zig 0.17

Deps:
- uucode bumped to
1fb73433bb
- zigimg bumped to
734fa919fb
- libvaxis bumped to
14b7684272
- zio bumped to:
417757935e
- rules_zig upgraded to [version
0.17](https://github.com/hermeticbuild/rules_zig/releases/tag/v0.17.0)
- zls is using a commit from master, but they don't have yet a tagged
release so Codex did a bit of packaging. I confirmed it correctly
autocompletes on my machine
2026-10-13 17:11:38 +02:00
Guillaume Wenzek
303ff3e6a4
Revert "zig 0.17 (#693)" (#909)
This reverts commit 5a18757368.

Our internal repos has a few zig patches, I need to also upgrade them
before we ugprade ZML
2026-10-07 11:34:51 +02:00
Guillaume Wenzek
5a18757368
zig 0.17 (#693)
Bump for zig 0.17

Deps:
- uucode bumped to
1fb73433bb
- zio bumped to:
417757935e
- rules_zig upgraded to [version
0.17](https://github.com/hermeticbuild/rules_zig/releases/tag/v0.17.0)

There is one Codex generated patch:
- zigimg: minor fixes to allow zig 0.17 to compile it, until they
officially support
2026-10-07 11:08:06 +02:00
Guillaume Wenzek
b2f0e03043
zml/Tensor: explicit sharding in Tensor code (#868)
In Buffer code we already have explicit sharding, but not for tensor.

I'm adding a `zml.Sharding` to `.withPartitioning` to make it explicit
which mesh we are working with.
To simplify the migration, `.withPartitioning` in Tensor code can use
`.model` syntax to lookup a sharding by name.
This replace the previous logic of `selectSharding(shape)` which would
look up a sharding that matches the axes of the given partitioning which
could yield surprising behavior.

This also modifies how sharding are stored in zml.Shape.
Before it was storing a list of axis names + a tag (2 * 8 * 64bits), now
I'm storing a zml.Sharding (64 bits) and a compact PartitionArray (32
bits) that references the logical axis of the sharding by offset instead
of by name.

There is an impact on the `TensorRegistry` because it also needs to be
aware of the possible shardings, instead of delegating the matching
partition spec -> sharding to loading. This means error will be detected
sooner.

`store.createTensor` accepts `@enumLiteral()` to lookup the relevant
sharding.

This also simplifies `zml.Buffer` because the full sharding is now
stored in the shape, so it doesn't need to be passed independently. It
also means that it will propagate from the tensor who are graph inputs,
to the buffers that are eg inputs.

`Placement` logic also got a bit simpler.
2026-10-06 15:37:22 +02:00
Benjamin Rabier
177f81f21b
stdx: Add cpu.zig with cpu utilities (#902) 2026-10-05 16:48:54 +02:00
Guillaume Wenzek
e1356ee4ed
zml: Compiler.pushLoc (#846)
Add explicit debug info in ZML.
Model author can now add `pushLocation` to mark specific sections of the
IR

---------

Co-authored-by: Guillaume Wenzek <gwenzek@users.noreply.github.com>
2026-09-23 00:17:41 +02:00
Guillaume Wenzek
2e5150eb0f
stdx/json: {Union,TaggedUnion}Helpers (#770) 2026-09-09 09:01:10 +02:00
Guillaume Wenzek
a46e4cac06
zml: handle packed fp4 on CPU side (#724)
The XLA API assumes that all sub-bytes data is given to it unpacked.
But when you read safetensors files the data is packed.

My proposition is the following: when we have a zml.fp4 tensor/buffer we
say to pjrt it's a u8 buffer instead.
Since PJRT infers the input/output shape from stablehlo I had to
introduce some bitcast at the edge of the compilation:
- fp4 input are declared as u8, but I insert bitcast operand before
calling the zig code, so the model code only see the fp4 tensors.
- when the model code returns a fp4 output, I bitcast to u8 before
exiting the final compilation

Breaking changes: none
Currently fp4 can't cross the pjrt boundary, so all fp4 code already
manually lie about the data being u8, and it will keep working. In the
future this code will be able to be "upgraded" by explicitly tagging the
input as fp4 and remove the "unpacking" code from their model code.

Note: the tracking XLA issue seems pretty cold
https://github.com/openxla/xla/issues/16795
2026-08-31 13:21:52 +02:00
Guillaume Wenzek
bdbfeb19ec
zml.ops: refactor while loop API (#686)
removes `anytype` from `zml.ops.while`, helping with type checking and
simplifying a bit the logic.
A `While` struct needs 
* a State struct definition that captures all the loop state that will
change on each iteration
* a `cond(While, State) Tensor` function returning a scalar boolean
tensor
* a `body(While, State) State` function that preserves the state shapes.

example:

```
const While = struct {
    // While struct can contain tensor and non-tensor fields.
    limit: i64,
    step: Tensor,

    pub const State = struct {
        i: zml.Tensor,
        sum: zml.Tensor,
    };

    pub fn cond(self: @This(), state: State) Tensor {
        return state.i.cmp(.LT, Tensor.scalar(self.limit, .i64));
    }

    pub fn body(self: @This(), state: State) State {
        return .{
            .i = state.i.add(self.step),
            .sum = state.sum.add(self.step),
        };
    }

    fn forward(i: Tensor, sum: Tensor) State {
        return zml.ops.@"while"(
            @This(),
            .{ .limit = 10, .step = .scalar(1, .i64) },
            .{ .i = i, .sum = sum },
        );
    }
};
```

The code is a bit more verbose for simple example, but for GatedDeltaNet
it's equivalent, and I have also used it for paged attention.
2026-07-30 11:29:42 +02:00
Corentin Godeau
2e1a07e7f1
stdx/json: fix potential invalid free (#631)
When we parsed from a std.json.Value we weren't correctly duping the
strings, potentially leading to invalid free.
2026-07-15 16:43:55 +02:00
elogir
8c03af9987
stdx/flags: Update typeInfo tag to Zig 0.16 .@"union". (#565) 2026-06-01 17:51:42 +02:00
Guillaume Wenzek
807a9116d4
zml: make Buffer.from faster (#556)
`Buffer.from` is in the hot path everytime we want to send smth to the
GPU.
Sometimes we can overlap it with GPU compute but not always, and
sometimes it's quite complicated to do so.

I've rewrote Sharding.Placement to be device agnostic instead of device
specific.
This allow to reuse most of the logic for multiple devices.

According to LLVM MCA (with script included here), we got from:

```
Iterations:        100
Instructions:      324200
Total Cycles:      175191
Total uOps:        369700
```

to 

```
    Iterations:        100
    Instructions:      162900
    Total Cycles:      85585
    Total uOps:        182600
```

So about twice as fast for this part of the code. I would need some help
to confirm the numbers on TPU.


(Once the PR is approved, I'll remove the mca script as well as the
build.zig)

---------

Co-authored-by: Hugo Mano <hugo@zml.ai>
2026-06-01 10:51:32 +02:00
elogir
144f3eb0af
platforms/neuron: bump toolchain and stabilize support (#476)
- bump Neuron dependencies in `platforms/neuron/requirements.lock.txt`
- `libneuronxla` `2.2.15515.0+50c26cbd` -> `2.2.16408.0+50c26cbd`
- `neuronx-cc` `2.23.6484.0+3b612583` -> `2.24.8799.0+6f62ff7c`
- make `neuronx-cc` invocation more robust by fixing path resolution,
releasing the Python GIL during compilation, checking the compiler
process exit status, and propagating NEFF wrapping/read
failures
- resolve PJRT devices and memories by PJRT handles instead of assuming
stable numeric ids during platform initialization, including Neuron
cases where memory ids are reused across devices
- avoid unsupported Neuron synchronization/event paths by always
requesting execution events on Neuron, deinitializing them without
awaiting, and removing the explicit await from the sharding
example
- avoid unsupported Neuron runtime paths by:
- falling back to host-backed buffer initialization for
`Buffer.uninitialized`
- using passthrough DMA allocation on Neuron
- skipping `Tensor.print()` custom calls on Neuron
- add NRT-backed Neuron physical mesh topology construction, including
chip/core placement, ring/torus/point-to-point topology inference, and
Neuron local-hardware-id based shard placement
- let the sharding example pick the default partitioner from the current
target
2026-05-12 09:58:32 +02:00
Guillaume Wenzek
5d94f9b83c
mlir: add shardy dialect (#526)
Add bindings to shardy dialect,
remove the string generation code from zml.Sharding, and replace it with
mlir calls.
2026-05-11 09:37:05 +02:00
Guillaume Wenzek
9211a1770e
Remove BoundedArray.init (#506)
When looking at the new Sharding code I was triggered by the high number
of "try".

I then realized most of the try where traced back to `axes = try
.init(0)` which can never fail.
So I removed BoundedArray.init (aka Andrew mode)

It's generally unsafe because it makes `slice()` return undefined.;
I replaced`try init(0)` by `.empty`
I replaced `try init(n)` by `.fromSlice` or with `.{ .buffer =
undefined, .len = n }` to explicitly show the unsafeness

This leads to widespread changes because it means that most of the
sharding operations can't fail anymore.
2026-04-27 10:31:52 +02:00
Benjamin Rabier
268ef6e058
examples/llm: fix --help & relative/home paths (#437) 2026-03-25 15:14:22 +01:00
Hugo Mano
2f175540bf
ci: rework pipeline + add remote with buildbuddy
Co-authored-by: Benjamin Rabier <benjamin@zml.ai>
2026-03-23 16:07:40 +01:00
Steeve Morin
78546350af stdx: remove meta.Signature
has proven unstable on Zig nightlies and mostly needed since std.Io
anyway.
2026-03-21 17:33:45 +01:00
Benjamin Rabier
ba8134f4a2
stdx/json: consistent serialization of Union, TaggedUnion and NeverNull (#412) 2026-03-17 13:58:31 +01:00
Corentin Godeau
70e4c9953d
Bump zig to 0.16.0-dev.2722+f16eb18ce (#409)
Co-authored-by: Hugo Mano <hugo@zml.ai>
2026-03-10 15:58:48 +01:00
Hugo Mano
abde0a04dd
zml: sharding (#407)
Co-authored-by: Benjamin Rabier <benjamin@zml.ai>
2026-03-10 11:17:57 +01:00
Hugo Mano
a9f2687237
zml/io/vfs: fixes for Zig 0.16.0-dev.2611+f996d2866 + new S3 VFS (#388) 2026-02-19 10:48:15 +01:00
Steeve Morin
dcbf5f2e27 workspace: update to 0.16.0-dev.2611+f996d2866, add remote exec
also:
- use toolchains_llvm_bootstrapped for macOS builds
2026-02-16 12:05:01 +01:00
Steeve Morin
7832fd26ac stdx: rework stdx.Io.LimitedGroup to use atomics and conditions 2026-01-28 18:12:15 +00:00
Steeve Morin
a8154fb5de zml: upgrade meta.TupleRange to Zig 0.16 2026-01-28 18:07:06 +00:00
Steeve Morin
1b644887b7 bazel,platforms: centralize runfiles loading
runfiles are process-wide in space and time.
so it makes sense to keep them around

because they are bound to the process, we use the smp_allocator and
except the memory to be kept around during the lifetime of the process.
2026-01-19 18:14:06 +01:00
Hugo Mano
3c96903568 Refactor llama and VFS to new CLI and IO APIs 2026-01-15 15:19:10 +01:00
Steeve Morin
6ee968e033 WIP 2026-01-14 18:08:56 +01:00
Hugo Mano
25a5ee10f6 rework zml.io.* 2026-01-14 18:04:55 +01:00
Steeve Morin
3112f1138a stdx: add utf32 helpers 2026-01-14 18:00:22 +01:00
Steeve Morin
ab62c536e2 runtimes/neuron: wip 2026-01-11 16:07:33 +01:00
Steeve Morin
a1ce0a387b runtimes: remove usage of std.fs, add std.Io 2026-01-05 23:39:33 +00:00
Steeve Morin
bfcdf27527 stdx: add time.hzFloat 2026-01-05 09:07:18 +00:00
Steeve Morin
8e9a28fc2c stdx: mirror new std layout 2026-01-04 19:37:42 +00:00
Corentin Godeau
ed1e690578 Small fix in stdx 2025-12-15 16:10:37 +01:00
Corentin Godeau
03b0b5726d Ported benchmark examples to v2 2025-12-12 15:25:01 +01:00
Corentin Godeau
715b0f745a Commented Writer in stdx.BoundedArray
It was causing compile issue in tests
2025-12-09 14:59:14 +01:00
Corentin Godeau
86b5bb84df Various small fixes 2025-12-08 17:46:17 +01:00
Corentin Godeau
8e0e987968 Made it working on Zig 0.16 2025-12-05 16:42:28 +01:00
Guillaume Wenzek
876ca22ff2
add patches to http_deb_archive (#338)
add patches to http_deb_archive
add documentation for command_buffer_compatible
pinToCore
Optional head dim for Llama 3.1
2025-10-15 15:06:58 +02:00
Corentin Kerisit
36d583dd58
workspace: use upstreamed rules_zig instead of our fork (#340)
This PR uses the new release of rules_zig where we upstreamed all the
work from the zml rules_zig fork.

Not much changes from the POV of the public API except for `copts`
becoming `zigopts`.

Also, the global `c` module is only available to Zig modules that depend
on a non empty `cc_library`.
This is behavior that can probably be modified in `rules_zig` later on.

Finally, ZLS support had to be completely rethought to comply with the
change in shape of ZigModuleInfo providers and subsequent rules.

This PR is blocked until `rules_zig` exposes `ZigModuleInfo` for
`zig_binaries`.
2025-10-13 14:34:33 +02:00
Guillaume Wenzek
e9b42f0ca8
examples: gpt oss (#329)
This implement support for OpenAI "gpt-oss" models. 

It's mostly a take over of #292 branch, but MoE code is kept inside
examples folder cause it's still a bit warm.
In particular the routing logic can break apart when there are undefined
memory in the inputs, where it wasn't an issue for normal MLP.

I only tested with the 20B version.
2025-09-25 14:03:37 +02:00
Guillaume Wenzek
1480f77a55
zml: add the platform tag inside the Buffer (#326)
CPU buffers behave differently from Cuda/Rocm/TPU buffers,
and in particular they are addressable from main memory.
This was annoying and prevented us to add some safety checks that
weren't valid for CPU.

I'm adding a tag to buffers so that we know which target they refer to.

Also there is a bug in PJRT unitialized which doesn't respected the
memory. I've opened https://github.com/openxla/xla/pull/31292
but in the meantime I've a workaround in ZML that uses the regular
.fromHostBuffer.
(I've also a patched version of PJRT plugins in
https://github.com/zml/pjrt-artifacts/pull/41 but not merged yet)
2025-09-18 15:35:25 +02:00
Guillaume Wenzek
dda7f6c3f0
zml: remove old writer interface APIs (#324)
We still had a few old reader/writer left over.
I think we are good now
2025-09-18 11:36:23 +02:00
Guillaume Wenzek
988f2711c5
zml: loadById (#322)
We already assign a unique tensor id to each tensor object created by a
BufferStore.
But we weren't using the id to load find the corresponding buffer in the
store, instead relying on the name.

This is allow to decouple the shape of the model in the struct vs the
shape of the model inside the safetensor file.
I also moved the id logic out of Tensor and into BufferStore.

I also updated llama/main.zig to use the new logic. 

Bonus:
* fix to `zml.call` tag hashing leading to different MLIR for every
recompile.
* expose  Cpu device count to control the number of CPU "devices"
2025-09-11 10:28:16 +02:00
Guillaume Wenzek
3f95bdd900
zml: callback api (#319)
Resuming work from @hugomano #244 
This is still a bit experimental.

The core API is here, but there are some caveats.

ZML should help with copying buffers to host memory (which is often
needed) but I'm not sure the current implementation using PJRT is good:
- does it allow graph capture
- what's the overhead ?
We should probably use directly cuMemcpy here instead of going through
PJRT.
Current implem should work with all plugins though.

This fix `tensor.print()`

I've also made a slightly more interesting examples/callback that
manually calls a cuda kernel. (there are other way to achieve this but
this shows you can do whatever you want with `zml.callback`

Note: 
the feature was originally named `custom_call` but it was conflicting
with existing bindings for xla builtin custom calls,
that are fully handled by the backend.
So I renamed it to `callback`

---------

Co-authored-by: Hugo Mano <hugo@zml.ai>
2025-09-04 16:57:34 +02:00
Hugo Mano
5d6c04ec69
ci: revamp commit workflow (#320)
- Deduplicate workflows (introduced by
57fd9cda27)
- Fix linking issue on Linux since upb was introduced
- Fix duplicated `_start` symbol for some tests targets
- Temporarily use `-c opt` instead of `--config=debug` to bypass static
archive bug
2025-09-03 14:23:22 +02:00
Guillaume Wenzek
8994a37792
bump runtimes/* code for 0.15 (#318)
Fix #316, close #304 

- Bump platform specific Zig code that doesn't get built in CI.
- Restore pytorch loader, using `std.fs.File` instead of `async.File`,
allowing to reuse `std.zip`
- Update `zig fmt` in CI to `0.15.1` and ran it on the codebase
(third_party excluded)
- Remove `stdx.io` which was pointing to remove
`std.io.BufferededReader`.

The Neuron target is still broken though:
`bazel build //runtimes/neuron:libneuronxla`

As well as the Cuda plugin in `--config=debug` mode:
 `bazel build --config=debug //runtimes/cuda:zmlxcuda`
2025-09-02 15:16:09 +02:00