## Motivation
This is the big sharding rewrite we need to achieve complex sharding.
Currently we always resolve a mesh logical axis to a mesh physical axis
(or several of them).
But on GPU we always have 1D topology, so we can't have 2D logical
meshes.
So this PR drops the logical -> physical axis bindings, and instead
store a logical -> devices binding.
This is more flexible, and has the nice side effect of simplifying
`Buffer.from`
On stablehlo side this changes how we generate the MLIR because we now
use logical meshes, and `manualComputation` uses logical axes.
(In shardy, GSPM is not affected AFAICT because it uses device ID
directly)
## Buffer.from
This also accelerates `Buffer.from` that I rewrote to be FAST.
Version | Fixed cost | Per device
-- | -- | --
origin/master | 109.6 ns | 35.39 ns
Rewrite | 35.8 ns | 4.38 ns
Improvement | 3.1× | 8.1×
## Creating mesh
I'm adding a new syntax to create LogicalMesh, to allow the more complex
meshes we want to do, and in particular, I now allow to force a specific
number of dimensions.
You can see it in action in `mnist.zig`.
```
const dp_mp = try platform.registerMesh("DP-MP", try .init(
.{ .data, .model },
&.{ .low_bandwidth, .high_bandwidth },
&.{ 2, -1 },
));
info: topology used: Mesh(name=DP-MP, partitions=4, replicas=1)
┌ Mappings (logical coordinates from physical coordinates):
├── data (low_bandwidth) -> (bus / 2)
└── model (high_bandwidth) -> (bus % 2)
┌ PhysicalMesh(platform=cpu shardable_axes={ .bus } num_devices=4)
└── [bus] 0..4 (tree)
├── Device(id=0, coords={ 0 })
├── Device(id=1, coords={ 1 })
├── Device(id=2, coords={ 2 })
└── Device(id=3, coords={ 3 })
┌ Logical coordinates:
└── data=0
└── model=0,1 -> devices=0,1
└── data=1
└── model=0,1 -> devices=2,3
```
The previous syntax is still available, and ZML will align logical axes
to physical axes like we do before this PR.
This part as unit tests, and I checked it work as I expect on CUDA but I
would need to double check the logics still works on TPU/Neuron, because
there is a few platform specific branches in that code.
## Test plan
As part of the benchmark Codex generated a test suite with hard-coded
offsets, allowing to check that Buffer.from didn't change behavior
through the commits, checking we correctly compute each individual
shard.
TODO check that LLMD perf isn't degraded.
here I'm promoting a pattern I would like to extend to more of ZML:
function that takes `anytype` argument should do as little work as
possible and called a fully typed function.
I kept `manualComputation` signature to not break existing code,
I moved the code to `zml/ops/manual_computation.zig` to better
encapsulate the required helper functions.
(and `zml/ops.zig` is pretty big atm)
Similarly `Sharding.Partitioning.parse` now calls `const logical =
comptime Logical.parse(partitioning);` then `logical.applyUseAll(tags,
mesh, default_value);` which is fully typed.
I'm also documenting `manualComputation` better
`Buffer.from` is in the hot path everytime we want to send smth to the
GPU.
Sometimes we can overlap it with GPU compute but not always, and
sometimes it's quite complicated to do so.
I've rewrote Sharding.Placement to be device agnostic instead of device
specific.
This allow to reuse most of the logic for multiple devices.
According to LLVM MCA (with script included here), we got from:
```
Iterations: 100
Instructions: 324200
Total Cycles: 175191
Total uOps: 369700
```
to
```
Iterations: 100
Instructions: 162900
Total Cycles: 85585
Total uOps: 182600
```
So about twice as fast for this part of the code. I would need some help
to confirm the numbers on TPU.
(Once the PR is approved, I'll remove the mca script as well as the
build.zig)
---------
Co-authored-by: Hugo Mano <hugo@zml.ai>
When looking at the new Sharding code I was triggered by the high number
of "try".
I then realized most of the try where traced back to `axes = try
.init(0)` which can never fail.
So I removed BoundedArray.init (aka Andrew mode)
It's generally unsafe because it makes `slice()` return undefined.;
I replaced`try init(0)` by `.empty`
I replaced `try init(n)` by `.fromSlice` or with `.{ .buffer =
undefined, .len = n }` to explicitly show the unsafeness
This leads to widespread changes because it means that most of the
sharding operations can't fail anymore.
Much of the work was done by @gwenzek
Sadly we drop the torch loader in the process, to be added back later !
---------
Co-authored-by: Guillaume Wenzek <wzk@zml.ai>