In Buffer code we already have explicit sharding, but not for tensor.
I'm adding a `zml.Sharding` to `.withPartitioning` to make it explicit
which mesh we are working with.
To simplify the migration, `.withPartitioning` in Tensor code can use
`.model` syntax to lookup a sharding by name.
This replace the previous logic of `selectSharding(shape)` which would
look up a sharding that matches the axes of the given partitioning which
could yield surprising behavior.
This also modifies how sharding are stored in zml.Shape.
Before it was storing a list of axis names + a tag (2 * 8 * 64bits), now
I'm storing a zml.Sharding (64 bits) and a compact PartitionArray (32
bits) that references the logical axis of the sharding by offset instead
of by name.
There is an impact on the `TensorRegistry` because it also needs to be
aware of the possible shardings, instead of delegating the matching
partition spec -> sharding to loading. This means error will be detected
sooner.
`store.createTensor` accepts `@enumLiteral()` to lookup the relevant
sharding.
This also simplifies `zml.Buffer` because the full sharding is now
stored in the shape, so it doesn't need to be passed independently. It
also means that it will propagate from the tensor who are graph inputs,
to the buffers that are eg inputs.
`Placement` logic also got a bit simpler.
The XLA API assumes that all sub-bytes data is given to it unpacked.
But when you read safetensors files the data is packed.
My proposition is the following: when we have a zml.fp4 tensor/buffer we
say to pjrt it's a u8 buffer instead.
Since PJRT infers the input/output shape from stablehlo I had to
introduce some bitcast at the edge of the compilation:
- fp4 input are declared as u8, but I insert bitcast operand before
calling the zig code, so the model code only see the fp4 tensors.
- when the model code returns a fp4 output, I bitcast to u8 before
exiting the final compilation
Breaking changes: none
Currently fp4 can't cross the pjrt boundary, so all fp4 code already
manually lie about the data being u8, and it will keep working. In the
future this code will be able to be "upgraded" by explicitly tagging the
input as fp4 and remove the "unpacking" code from their model code.
Note: the tracking XLA issue seems pretty cold
https://github.com/openxla/xla/issues/16795
removes `anytype` from `zml.ops.while`, helping with type checking and
simplifying a bit the logic.
A `While` struct needs
* a State struct definition that captures all the loop state that will
change on each iteration
* a `cond(While, State) Tensor` function returning a scalar boolean
tensor
* a `body(While, State) State` function that preserves the state shapes.
example:
```
const While = struct {
// While struct can contain tensor and non-tensor fields.
limit: i64,
step: Tensor,
pub const State = struct {
i: zml.Tensor,
sum: zml.Tensor,
};
pub fn cond(self: @This(), state: State) Tensor {
return state.i.cmp(.LT, Tensor.scalar(self.limit, .i64));
}
pub fn body(self: @This(), state: State) State {
return .{
.i = state.i.add(self.step),
.sum = state.sum.add(self.step),
};
}
fn forward(i: Tensor, sum: Tensor) State {
return zml.ops.@"while"(
@This(),
.{ .limit = 10, .step = .scalar(1, .i64) },
.{ .i = i, .sum = sum },
);
}
};
```
The code is a bit more verbose for simple example, but for GatedDeltaNet
it's equivalent, and I have also used it for paged attention.
This implement support for OpenAI "gpt-oss" models.
It's mostly a take over of #292 branch, but MoE code is kept inside
examples folder cause it's still a bit warm.
In particular the routing logic can break apart when there are undefined
memory in the inputs, where it wasn't an issue for normal MLP.
I only tested with the 20B version.
We already assign a unique tensor id to each tensor object created by a
BufferStore.
But we weren't using the id to load find the corresponding buffer in the
store, instead relying on the name.
This is allow to decouple the shape of the model in the struct vs the
shape of the model inside the safetensor file.
I also moved the id logic out of Tensor and into BufferStore.
I also updated llama/main.zig to use the new logic.
Bonus:
* fix to `zml.call` tag hashing leading to different MLIR for every
recompile.
* expose Cpu device count to control the number of CPU "devices"
Resuming work from @hugomano #244
This is still a bit experimental.
The core API is here, but there are some caveats.
ZML should help with copying buffers to host memory (which is often
needed) but I'm not sure the current implementation using PJRT is good:
- does it allow graph capture
- what's the overhead ?
We should probably use directly cuMemcpy here instead of going through
PJRT.
Current implem should work with all plugins though.
This fix `tensor.print()`
I've also made a slightly more interesting examples/callback that
manually calls a cuda kernel. (there are other way to achieve this but
this shows you can do whatever you want with `zml.callback`
Note:
the feature was originally named `custom_call` but it was conflicting
with existing bindings for xla builtin custom calls,
that are fully handled by the backend.
So I renamed it to `callback`
---------
Co-authored-by: Hugo Mano <hugo@zml.ai>
Much of the work was done by @gwenzek
Sadly we drop the torch loader in the process, to be added back later !
---------
Co-authored-by: Guillaume Wenzek <wzk@zml.ai>