0
Fork 0
mirror of https://github.com/zml/zml.git synced 2026-11-16 19:11:09 +00:00
zml/stdx/bounded_array.zig

415 lines
15 KiB

zml/sharding: use logical meshes (#943) ## Motivation This is the big sharding rewrite we need to achieve complex sharding. Currently we always resolve a mesh logical axis to a mesh physical axis (or several of them). But on GPU we always have 1D topology, so we can't have 2D logical meshes. So this PR drops the logical -> physical axis bindings, and instead store a logical -> devices binding. This is more flexible, and has the nice side effect of simplifying `Buffer.from` On stablehlo side this changes how we generate the MLIR because we now use logical meshes, and `manualComputation` uses logical axes. (In shardy, GSPM is not affected AFAICT because it uses device ID directly) ## Buffer.from This also accelerates `Buffer.from` that I rewrote to be FAST. Version | Fixed cost | Per device -- | -- | -- origin/master | 109.6 ns | 35.39 ns Rewrite | 35.8 ns | 4.38 ns Improvement | 3.1× | 8.1× ## Creating mesh I'm adding a new syntax to create LogicalMesh, to allow the more complex meshes we want to do, and in particular, I now allow to force a specific number of dimensions. You can see it in action in `mnist.zig`. ``` const dp_mp = try platform.registerMesh("DP-MP", try .init( .{ .data, .model }, &.{ .low_bandwidth, .high_bandwidth }, &.{ 2, -1 }, )); info: topology used: Mesh(name=DP-MP, partitions=4, replicas=1) ┌ Mappings (logical coordinates from physical coordinates): ├── data (low_bandwidth) -> (bus / 2) └── model (high_bandwidth) -> (bus % 2) ┌ PhysicalMesh(platform=cpu shardable_axes={ .bus } num_devices=4) └── [bus] 0..4 (tree) ├── Device(id=0, coords={ 0 }) ├── Device(id=1, coords={ 1 }) ├── Device(id=2, coords={ 2 }) └── Device(id=3, coords={ 3 }) ┌ Logical coordinates: └── data=0 └── model=0,1 -> devices=0,1 └── data=1 └── model=0,1 -> devices=2,3 ``` The previous syntax is still available, and ZML will align logical axes to physical axes like we do before this PR. This part as unit tests, and I checked it work as I expect on CUDA but I would need to double check the logics still works on TPU/Neuron, because there is a few platform specific branches in that code. ## Test plan As part of the benchmark Codex generated a test suite with hard-coded offsets, allowing to check that Buffer.from didn't change behavior through the commits, checking we correctly compute each individual shard. TODO check that LLMD perf isn't degraded.
2026-10-16 15:27:30 +02:00
zml/sharding: use logical meshes (#943) ## Motivation This is the big sharding rewrite we need to achieve complex sharding. Currently we always resolve a mesh logical axis to a mesh physical axis (or several of them). But on GPU we always have 1D topology, so we can't have 2D logical meshes. So this PR drops the logical -> physical axis bindings, and instead store a logical -> devices binding. This is more flexible, and has the nice side effect of simplifying `Buffer.from` On stablehlo side this changes how we generate the MLIR because we now use logical meshes, and `manualComputation` uses logical axes. (In shardy, GSPM is not affected AFAICT because it uses device ID directly) ## Buffer.from This also accelerates `Buffer.from` that I rewrote to be FAST. Version | Fixed cost | Per device -- | -- | -- origin/master | 109.6 ns | 35.39 ns Rewrite | 35.8 ns | 4.38 ns Improvement | 3.1× | 8.1× ## Creating mesh I'm adding a new syntax to create LogicalMesh, to allow the more complex meshes we want to do, and in particular, I now allow to force a specific number of dimensions. You can see it in action in `mnist.zig`. ``` const dp_mp = try platform.registerMesh("DP-MP", try .init( .{ .data, .model }, &.{ .low_bandwidth, .high_bandwidth }, &.{ 2, -1 }, )); info: topology used: Mesh(name=DP-MP, partitions=4, replicas=1) ┌ Mappings (logical coordinates from physical coordinates): ├── data (low_bandwidth) -> (bus / 2) └── model (high_bandwidth) -> (bus % 2) ┌ PhysicalMesh(platform=cpu shardable_axes={ .bus } num_devices=4) └── [bus] 0..4 (tree) ├── Device(id=0, coords={ 0 }) ├── Device(id=1, coords={ 1 }) ├── Device(id=2, coords={ 2 }) └── Device(id=3, coords={ 3 }) ┌ Logical coordinates: └── data=0 └── model=0,1 -> devices=0,1 └── data=1 └── model=0,1 -> devices=2,3 ``` The previous syntax is still available, and ZML will align logical axes to physical axes like we do before this PR. This part as unit tests, and I checked it work as I expect on CUDA but I would need to double check the logics still works on TPU/Neuron, because there is a few platform specific branches in that code. ## Test plan As part of the benchmark Codex generated a test suite with hard-coded offsets, allowing to check that Buffer.from didn't change behavior through the commits, checking we correctly compute each individual shard. TODO check that LLMD perf isn't degraded.
2026-10-16 15:27:30 +02:00