2024-09-17 11:11:18 +02:00
|
|
|
|
2025-04-08 16:28:39 +02:00
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
2025-04-08 16:28:39 +02:00
|
|
|
|
|
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
zml/sharding: use logical meshes (#943)
## Motivation
This is the big sharding rewrite we need to achieve complex sharding.
Currently we always resolve a mesh logical axis to a mesh physical axis
(or several of them).
But on GPU we always have 1D topology, so we can't have 2D logical
meshes.
So this PR drops the logical -> physical axis bindings, and instead
store a logical -> devices binding.
This is more flexible, and has the nice side effect of simplifying
`Buffer.from`
On stablehlo side this changes how we generate the MLIR because we now
use logical meshes, and `manualComputation` uses logical axes.
(In shardy, GSPM is not affected AFAICT because it uses device ID
directly)
## Buffer.from
This also accelerates `Buffer.from` that I rewrote to be FAST.
Version | Fixed cost | Per device
-- | -- | --
origin/master | 109.6 ns | 35.39 ns
Rewrite | 35.8 ns | 4.38 ns
Improvement | 3.1× | 8.1×
## Creating mesh
I'm adding a new syntax to create LogicalMesh, to allow the more complex
meshes we want to do, and in particular, I now allow to force a specific
number of dimensions.
You can see it in action in `mnist.zig`.
```
const dp_mp = try platform.registerMesh("DP-MP", try .init(
.{ .data, .model },
&.{ .low_bandwidth, .high_bandwidth },
&.{ 2, -1 },
));
info: topology used: Mesh(name=DP-MP, partitions=4, replicas=1)
┌ Mappings (logical coordinates from physical coordinates):
├── data (low_bandwidth) -> (bus / 2)
└── model (high_bandwidth) -> (bus % 2)
┌ PhysicalMesh(platform=cpu shardable_axes={ .bus } num_devices=4)
└── [bus] 0..4 (tree)
├── Device(id=0, coords={ 0 })
├── Device(id=1, coords={ 1 })
├── Device(id=2, coords={ 2 })
└── Device(id=3, coords={ 3 })
┌ Logical coordinates:
└── data=0
└── model=0,1 -> devices=0,1
└── data=1
└── model=0,1 -> devices=2,3
```
The previous syntax is still available, and ZML will align logical axes
to physical axes like we do before this PR.
This part as unit tests, and I checked it work as I expect on CUDA but I
would need to double check the logics still works on TPU/Neuron, because
there is a few platform specific branches in that code.
## Test plan
As part of the benchmark Codex generated a test suite with hard-coded
offsets, allowing to check that Buffer.from didn't change behavior
through the commits, checking we correctly compute each individual
shard.
TODO check that LLMD perf isn't degraded.
2026-10-16 15:27:30 +02:00
|
|
|
|
|
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-01-15 18:20:15 +00:00
|
|
|
|
2026-01-11 09:48:15 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
2026-01-11 09:48:15 +01:00
|
|
|
|
2026-10-13 17:11:38 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-01-11 09:48:15 +01:00
|
|
|
|
2026-10-13 17:11:38 +02:00
|
|
|
|
2026-01-11 09:48:15 +01:00
|
|
|
|
|
|
|
|
|
2026-10-13 17:11:38 +02:00
|
|
|
|
2026-01-11 09:48:15 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-12-05 14:11:24 +01:00
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
zml/sharding: use logical meshes (#943)
## Motivation
This is the big sharding rewrite we need to achieve complex sharding.
Currently we always resolve a mesh logical axis to a mesh physical axis
(or several of them).
But on GPU we always have 1D topology, so we can't have 2D logical
meshes.
So this PR drops the logical -> physical axis bindings, and instead
store a logical -> devices binding.
This is more flexible, and has the nice side effect of simplifying
`Buffer.from`
On stablehlo side this changes how we generate the MLIR because we now
use logical meshes, and `manualComputation` uses logical axes.
(In shardy, GSPM is not affected AFAICT because it uses device ID
directly)
## Buffer.from
This also accelerates `Buffer.from` that I rewrote to be FAST.
Version | Fixed cost | Per device
-- | -- | --
origin/master | 109.6 ns | 35.39 ns
Rewrite | 35.8 ns | 4.38 ns
Improvement | 3.1× | 8.1×
## Creating mesh
I'm adding a new syntax to create LogicalMesh, to allow the more complex
meshes we want to do, and in particular, I now allow to force a specific
number of dimensions.
You can see it in action in `mnist.zig`.
```
const dp_mp = try platform.registerMesh("DP-MP", try .init(
.{ .data, .model },
&.{ .low_bandwidth, .high_bandwidth },
&.{ 2, -1 },
));
info: topology used: Mesh(name=DP-MP, partitions=4, replicas=1)
┌ Mappings (logical coordinates from physical coordinates):
├── data (low_bandwidth) -> (bus / 2)
└── model (high_bandwidth) -> (bus % 2)
┌ PhysicalMesh(platform=cpu shardable_axes={ .bus } num_devices=4)
└── [bus] 0..4 (tree)
├── Device(id=0, coords={ 0 })
├── Device(id=1, coords={ 1 })
├── Device(id=2, coords={ 2 })
└── Device(id=3, coords={ 3 })
┌ Logical coordinates:
└── data=0
└── model=0,1 -> devices=0,1
└── data=1
└── model=0,1 -> devices=2,3
```
The previous syntax is still available, and ZML will align logical axes
to physical axes like we do before this PR.
This part as unit tests, and I checked it work as I expect on CUDA but I
would need to double check the logics still works on TPU/Neuron, because
there is a few platform specific branches in that code.
## Test plan
As part of the benchmark Codex generated a test suite with hard-coded
offsets, allowing to check that Buffer.from didn't change behavior
through the commits, checking we correctly compute each individual
shard.
TODO check that LLMD perf isn't degraded.
2026-10-16 15:27:30 +02:00
|
|
|
|
2026-08-17 15:16:33 +02:00
|
|
|
|
|
|
|
|
|
zml/sharding: use logical meshes (#943)
## Motivation
This is the big sharding rewrite we need to achieve complex sharding.
Currently we always resolve a mesh logical axis to a mesh physical axis
(or several of them).
But on GPU we always have 1D topology, so we can't have 2D logical
meshes.
So this PR drops the logical -> physical axis bindings, and instead
store a logical -> devices binding.
This is more flexible, and has the nice side effect of simplifying
`Buffer.from`
On stablehlo side this changes how we generate the MLIR because we now
use logical meshes, and `manualComputation` uses logical axes.
(In shardy, GSPM is not affected AFAICT because it uses device ID
directly)
## Buffer.from
This also accelerates `Buffer.from` that I rewrote to be FAST.
Version | Fixed cost | Per device
-- | -- | --
origin/master | 109.6 ns | 35.39 ns
Rewrite | 35.8 ns | 4.38 ns
Improvement | 3.1× | 8.1×
## Creating mesh
I'm adding a new syntax to create LogicalMesh, to allow the more complex
meshes we want to do, and in particular, I now allow to force a specific
number of dimensions.
You can see it in action in `mnist.zig`.
```
const dp_mp = try platform.registerMesh("DP-MP", try .init(
.{ .data, .model },
&.{ .low_bandwidth, .high_bandwidth },
&.{ 2, -1 },
));
info: topology used: Mesh(name=DP-MP, partitions=4, replicas=1)
┌ Mappings (logical coordinates from physical coordinates):
├── data (low_bandwidth) -> (bus / 2)
└── model (high_bandwidth) -> (bus % 2)
┌ PhysicalMesh(platform=cpu shardable_axes={ .bus } num_devices=4)
└── [bus] 0..4 (tree)
├── Device(id=0, coords={ 0 })
├── Device(id=1, coords={ 1 })
├── Device(id=2, coords={ 2 })
└── Device(id=3, coords={ 3 })
┌ Logical coordinates:
└── data=0
└── model=0,1 -> devices=0,1
└── data=1
└── model=0,1 -> devices=2,3
```
The previous syntax is still available, and ZML will align logical axes
to physical axes like we do before this PR.
This part as unit tests, and I checked it work as I expect on CUDA but I
would need to double check the logics still works on TPU/Neuron, because
there is a few platform specific branches in that code.
## Test plan
As part of the benchmark Codex generated a test suite with hard-coded
offsets, allowing to check that Buffer.from didn't change behavior
through the commits, checking we correctly compute each individual
shard.
TODO check that LLMD perf isn't degraded.
2026-10-16 15:27:30 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-08-17 15:16:33 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-09-02 15:16:09 +02:00
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-02-03 10:59:39 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-07-15 14:18:15 +02:00
|
|
|
|
2026-02-03 10:59:39 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
2026-02-12 21:08:46 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-07-16 17:44:50 +02:00
|
|
|
|
2026-02-12 21:08:46 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
2025-07-15 14:18:15 +02:00
|
|
|
|
|
|
|
|
|
2026-07-16 17:44:50 +02:00
|
|
|
|
2025-07-15 14:18:15 +02:00
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
2026-02-12 21:00:05 +01:00
|
|
|
|
2026-07-02 14:01:18 +02:00
|
|
|
|
2026-07-16 17:44:50 +02:00
|
|
|
|
2026-07-02 14:01:18 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-02-12 21:00:05 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-08-27 14:11:22 +02:00
|
|
|
|
2026-07-02 14:01:18 +02:00
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-07-02 14:01:18 +02:00
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-01-11 09:48:15 +01:00
|
|
|
|
2025-08-27 14:11:22 +02:00
|
|
|
|
2025-03-14 09:49:23 +01:00
|
|
|
|
|
|
|
|
|
2026-10-13 17:11:38 +02:00
|
|
|
|
2025-03-14 09:49:23 +01:00
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
2026-01-11 09:48:15 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
zml/sharding: use logical meshes (#943)
## Motivation
This is the big sharding rewrite we need to achieve complex sharding.
Currently we always resolve a mesh logical axis to a mesh physical axis
(or several of them).
But on GPU we always have 1D topology, so we can't have 2D logical
meshes.
So this PR drops the logical -> physical axis bindings, and instead
store a logical -> devices binding.
This is more flexible, and has the nice side effect of simplifying
`Buffer.from`
On stablehlo side this changes how we generate the MLIR because we now
use logical meshes, and `manualComputation` uses logical axes.
(In shardy, GSPM is not affected AFAICT because it uses device ID
directly)
## Buffer.from
This also accelerates `Buffer.from` that I rewrote to be FAST.
Version | Fixed cost | Per device
-- | -- | --
origin/master | 109.6 ns | 35.39 ns
Rewrite | 35.8 ns | 4.38 ns
Improvement | 3.1× | 8.1×
## Creating mesh
I'm adding a new syntax to create LogicalMesh, to allow the more complex
meshes we want to do, and in particular, I now allow to force a specific
number of dimensions.
You can see it in action in `mnist.zig`.
```
const dp_mp = try platform.registerMesh("DP-MP", try .init(
.{ .data, .model },
&.{ .low_bandwidth, .high_bandwidth },
&.{ 2, -1 },
));
info: topology used: Mesh(name=DP-MP, partitions=4, replicas=1)
┌ Mappings (logical coordinates from physical coordinates):
├── data (low_bandwidth) -> (bus / 2)
└── model (high_bandwidth) -> (bus % 2)
┌ PhysicalMesh(platform=cpu shardable_axes={ .bus } num_devices=4)
└── [bus] 0..4 (tree)
├── Device(id=0, coords={ 0 })
├── Device(id=1, coords={ 1 })
├── Device(id=2, coords={ 2 })
└── Device(id=3, coords={ 3 })
┌ Logical coordinates:
└── data=0
└── model=0,1 -> devices=0,1
└── data=1
└── model=0,1 -> devices=2,3
```
The previous syntax is still available, and ZML will align logical axes
to physical axes like we do before this PR.
This part as unit tests, and I checked it work as I expect on CUDA but I
would need to double check the logics still works on TPU/Neuron, because
there is a few platform specific branches in that code.
## Test plan
As part of the benchmark Codex generated a test suite with hard-coded
offsets, allowing to check that Buffer.from didn't change behavior
through the commits, checking we correctly compute each individual
shard.
TODO check that LLMD perf isn't degraded.
2026-10-16 15:27:30 +02:00
|
|
|
|
2026-01-11 16:00:38 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-01-11 09:48:15 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
zml/sharding: use logical meshes (#943)
## Motivation
This is the big sharding rewrite we need to achieve complex sharding.
Currently we always resolve a mesh logical axis to a mesh physical axis
(or several of them).
But on GPU we always have 1D topology, so we can't have 2D logical
meshes.
So this PR drops the logical -> physical axis bindings, and instead
store a logical -> devices binding.
This is more flexible, and has the nice side effect of simplifying
`Buffer.from`
On stablehlo side this changes how we generate the MLIR because we now
use logical meshes, and `manualComputation` uses logical axes.
(In shardy, GSPM is not affected AFAICT because it uses device ID
directly)
## Buffer.from
This also accelerates `Buffer.from` that I rewrote to be FAST.
Version | Fixed cost | Per device
-- | -- | --
origin/master | 109.6 ns | 35.39 ns
Rewrite | 35.8 ns | 4.38 ns
Improvement | 3.1× | 8.1×
## Creating mesh
I'm adding a new syntax to create LogicalMesh, to allow the more complex
meshes we want to do, and in particular, I now allow to force a specific
number of dimensions.
You can see it in action in `mnist.zig`.
```
const dp_mp = try platform.registerMesh("DP-MP", try .init(
.{ .data, .model },
&.{ .low_bandwidth, .high_bandwidth },
&.{ 2, -1 },
));
info: topology used: Mesh(name=DP-MP, partitions=4, replicas=1)
┌ Mappings (logical coordinates from physical coordinates):
├── data (low_bandwidth) -> (bus / 2)
└── model (high_bandwidth) -> (bus % 2)
┌ PhysicalMesh(platform=cpu shardable_axes={ .bus } num_devices=4)
└── [bus] 0..4 (tree)
├── Device(id=0, coords={ 0 })
├── Device(id=1, coords={ 1 })
├── Device(id=2, coords={ 2 })
└── Device(id=3, coords={ 3 })
┌ Logical coordinates:
└── data=0
└── model=0,1 -> devices=0,1
└── data=1
└── model=0,1 -> devices=2,3
```
The previous syntax is still available, and ZML will align logical axes
to physical axes like we do before this PR.
This part as unit tests, and I checked it work as I expect on CUDA but I
would need to double check the logics still works on TPU/Neuron, because
there is a few platform specific branches in that code.
## Test plan
As part of the benchmark Codex generated a test suite with hard-coded
offsets, allowing to check that Buffer.from didn't change behavior
through the commits, checking we correctly compute each individual
shard.
TODO check that LLMD perf isn't degraded.
2026-10-16 15:27:30 +02:00
|
|
|
|
2026-01-11 09:48:15 +01:00
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
zml/sharding: use logical meshes (#943)
## Motivation
This is the big sharding rewrite we need to achieve complex sharding.
Currently we always resolve a mesh logical axis to a mesh physical axis
(or several of them).
But on GPU we always have 1D topology, so we can't have 2D logical
meshes.
So this PR drops the logical -> physical axis bindings, and instead
store a logical -> devices binding.
This is more flexible, and has the nice side effect of simplifying
`Buffer.from`
On stablehlo side this changes how we generate the MLIR because we now
use logical meshes, and `manualComputation` uses logical axes.
(In shardy, GSPM is not affected AFAICT because it uses device ID
directly)
## Buffer.from
This also accelerates `Buffer.from` that I rewrote to be FAST.
Version | Fixed cost | Per device
-- | -- | --
origin/master | 109.6 ns | 35.39 ns
Rewrite | 35.8 ns | 4.38 ns
Improvement | 3.1× | 8.1×
## Creating mesh
I'm adding a new syntax to create LogicalMesh, to allow the more complex
meshes we want to do, and in particular, I now allow to force a specific
number of dimensions.
You can see it in action in `mnist.zig`.
```
const dp_mp = try platform.registerMesh("DP-MP", try .init(
.{ .data, .model },
&.{ .low_bandwidth, .high_bandwidth },
&.{ 2, -1 },
));
info: topology used: Mesh(name=DP-MP, partitions=4, replicas=1)
┌ Mappings (logical coordinates from physical coordinates):
├── data (low_bandwidth) -> (bus / 2)
└── model (high_bandwidth) -> (bus % 2)
┌ PhysicalMesh(platform=cpu shardable_axes={ .bus } num_devices=4)
└── [bus] 0..4 (tree)
├── Device(id=0, coords={ 0 })
├── Device(id=1, coords={ 1 })
├── Device(id=2, coords={ 2 })
└── Device(id=3, coords={ 3 })
┌ Logical coordinates:
└── data=0
└── model=0,1 -> devices=0,1
└── data=1
└── model=0,1 -> devices=2,3
```
The previous syntax is still available, and ZML will align logical axes
to physical axes like we do before this PR.
This part as unit tests, and I checked it work as I expect on CUDA but I
would need to double check the logics still works on TPU/Neuron, because
there is a few platform specific branches in that code.
## Test plan
As part of the benchmark Codex generated a test suite with hard-coded
offsets, allowing to check that Buffer.from didn't change behavior
through the commits, checking we correctly compute each individual
shard.
TODO check that LLMD perf isn't degraded.
2026-10-16 15:27:30 +02:00
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
zml/sharding: use logical meshes (#943)
## Motivation
This is the big sharding rewrite we need to achieve complex sharding.
Currently we always resolve a mesh logical axis to a mesh physical axis
(or several of them).
But on GPU we always have 1D topology, so we can't have 2D logical
meshes.
So this PR drops the logical -> physical axis bindings, and instead
store a logical -> devices binding.
This is more flexible, and has the nice side effect of simplifying
`Buffer.from`
On stablehlo side this changes how we generate the MLIR because we now
use logical meshes, and `manualComputation` uses logical axes.
(In shardy, GSPM is not affected AFAICT because it uses device ID
directly)
## Buffer.from
This also accelerates `Buffer.from` that I rewrote to be FAST.
Version | Fixed cost | Per device
-- | -- | --
origin/master | 109.6 ns | 35.39 ns
Rewrite | 35.8 ns | 4.38 ns
Improvement | 3.1× | 8.1×
## Creating mesh
I'm adding a new syntax to create LogicalMesh, to allow the more complex
meshes we want to do, and in particular, I now allow to force a specific
number of dimensions.
You can see it in action in `mnist.zig`.
```
const dp_mp = try platform.registerMesh("DP-MP", try .init(
.{ .data, .model },
&.{ .low_bandwidth, .high_bandwidth },
&.{ 2, -1 },
));
info: topology used: Mesh(name=DP-MP, partitions=4, replicas=1)
┌ Mappings (logical coordinates from physical coordinates):
├── data (low_bandwidth) -> (bus / 2)
└── model (high_bandwidth) -> (bus % 2)
┌ PhysicalMesh(platform=cpu shardable_axes={ .bus } num_devices=4)
└── [bus] 0..4 (tree)
├── Device(id=0, coords={ 0 })
├── Device(id=1, coords={ 1 })
├── Device(id=2, coords={ 2 })
└── Device(id=3, coords={ 3 })
┌ Logical coordinates:
└── data=0
└── model=0,1 -> devices=0,1
└── data=1
└── model=0,1 -> devices=2,3
```
The previous syntax is still available, and ZML will align logical axes
to physical axes like we do before this PR.
This part as unit tests, and I checked it work as I expect on CUDA but I
would need to double check the logics still works on TPU/Neuron, because
there is a few platform specific branches in that code.
## Test plan
As part of the benchmark Codex generated a test suite with hard-coded
offsets, allowing to check that Buffer.from didn't change behavior
through the commits, checking we correctly compute each individual
shard.
TODO check that LLMD perf isn't degraded.
2026-10-16 15:27:30 +02:00
|
|
|
|
2026-01-11 09:48:15 +01:00
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
2026-01-11 09:48:15 +01:00
|
|
|
|
|
|
|
|
|
zml/sharding: use logical meshes (#943)
## Motivation
This is the big sharding rewrite we need to achieve complex sharding.
Currently we always resolve a mesh logical axis to a mesh physical axis
(or several of them).
But on GPU we always have 1D topology, so we can't have 2D logical
meshes.
So this PR drops the logical -> physical axis bindings, and instead
store a logical -> devices binding.
This is more flexible, and has the nice side effect of simplifying
`Buffer.from`
On stablehlo side this changes how we generate the MLIR because we now
use logical meshes, and `manualComputation` uses logical axes.
(In shardy, GSPM is not affected AFAICT because it uses device ID
directly)
## Buffer.from
This also accelerates `Buffer.from` that I rewrote to be FAST.
Version | Fixed cost | Per device
-- | -- | --
origin/master | 109.6 ns | 35.39 ns
Rewrite | 35.8 ns | 4.38 ns
Improvement | 3.1× | 8.1×
## Creating mesh
I'm adding a new syntax to create LogicalMesh, to allow the more complex
meshes we want to do, and in particular, I now allow to force a specific
number of dimensions.
You can see it in action in `mnist.zig`.
```
const dp_mp = try platform.registerMesh("DP-MP", try .init(
.{ .data, .model },
&.{ .low_bandwidth, .high_bandwidth },
&.{ 2, -1 },
));
info: topology used: Mesh(name=DP-MP, partitions=4, replicas=1)
┌ Mappings (logical coordinates from physical coordinates):
├── data (low_bandwidth) -> (bus / 2)
└── model (high_bandwidth) -> (bus % 2)
┌ PhysicalMesh(platform=cpu shardable_axes={ .bus } num_devices=4)
└── [bus] 0..4 (tree)
├── Device(id=0, coords={ 0 })
├── Device(id=1, coords={ 1 })
├── Device(id=2, coords={ 2 })
└── Device(id=3, coords={ 3 })
┌ Logical coordinates:
└── data=0
└── model=0,1 -> devices=0,1
└── data=1
└── model=0,1 -> devices=2,3
```
The previous syntax is still available, and ZML will align logical axes
to physical axes like we do before this PR.
This part as unit tests, and I checked it work as I expect on CUDA but I
would need to double check the logics still works on TPU/Neuron, because
there is a few platform specific branches in that code.
## Test plan
As part of the benchmark Codex generated a test suite with hard-coded
offsets, allowing to check that Buffer.from didn't change behavior
through the commits, checking we correctly compute each individual
shard.
TODO check that LLMD perf isn't degraded.
2026-10-16 15:27:30 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-01-11 15:59:43 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-01-11 10:17:39 +01:00
|
|
|
|
2026-01-11 15:59:43 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-01-11 10:17:39 +01:00
|
|
|
|
|
|
|
|
|
2026-01-11 15:59:43 +01:00
|
|
|
|
2026-01-11 10:17:39 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-01-11 15:59:43 +01:00
|
|
|
|
2026-01-11 10:31:05 +01:00
|
|
|
|
2026-01-11 10:17:39 +01:00
|
|
|
|
2026-01-11 15:59:43 +01:00
|
|
|
|
|
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-11-15 11:22:01 +01:00
|
|
|
|
2025-02-04 11:11:02 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-10-13 17:11:38 +02:00
|
|
|
|
2025-02-04 11:11:02 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-03-10 11:17:57 +01:00
|
|
|
|
2026-01-11 10:17:39 +01:00
|
|
|
|
|
|
|
|
|
2026-10-13 17:11:38 +02:00
|
|
|
|
2026-01-11 10:17:39 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-11-25 15:53:25 +01:00
|
|
|
|
2025-02-04 11:11:02 +01:00
|
|
|
|
2024-11-25 15:53:25 +01:00
|
|
|
|
|
|
|
|
|
2025-06-06 11:24:21 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-09-04 16:57:34 +02:00
|
|
|
|
2026-01-15 18:20:15 +00:00
|
|
|
|
|
|
|
|
|
2024-11-15 11:22:01 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-11-25 15:53:25 +01:00
|
|
|
|
2026-03-10 11:17:57 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-11-25 15:53:25 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-03-10 11:17:57 +01:00
|
|
|
|
2024-11-25 15:53:25 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-02-04 11:11:02 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-06-25 14:17:46 +02:00
|
|
|
|
|
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
2026-06-25 14:17:46 +02:00
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-06-25 14:17:46 +02:00
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
2026-10-13 17:11:38 +02:00
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-06-25 14:17:46 +02:00
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-02-04 11:11:02 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-01-11 09:48:15 +01:00
|
|
|
|
2025-02-04 11:11:02 +01:00
|
|
|
|
2025-12-15 18:55:43 +01:00
|
|
|
|
2025-02-04 11:11:02 +01:00
|
|
|
|
2026-01-11 09:48:15 +01:00
|
|
|
|
2026-01-11 10:17:39 +01:00
|
|
|
|
2025-12-15 18:55:43 +01:00
|
|
|
|
2026-10-13 17:11:38 +02:00
|
|
|
|
2026-01-11 09:48:15 +01:00
|
|
|
|
2025-02-04 11:11:02 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-12-18 16:07:17 +01:00
|
|
|
|
2025-02-04 11:11:02 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-10-13 17:11:38 +02:00
|
|
|
|
2025-02-04 11:11:02 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-06-06 11:24:21 +02:00
|
|
|
|
|
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-01-11 10:17:39 +01:00
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-01-11 22:23:25 +01:00
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-01-11 22:23:25 +01:00
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-01-11 22:23:25 +01:00
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-01-02 22:14:00 +00:00
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
2026-01-11 09:48:15 +01:00
|
|
|
|
|
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
2026-01-11 09:48:15 +01:00
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
2026-01-11 09:48:15 +01:00
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-01-02 22:14:00 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
2025-05-20 15:08:35 +02:00
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-03-10 11:17:57 +01:00
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
2025-09-18 15:35:25 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
2024-11-01 17:38:14 +01:00
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
2025-05-20 15:08:35 +02:00
|
|
|
|
2026-10-13 17:11:38 +02:00
|
|
|
|
2026-01-11 10:17:39 +01:00
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
2026-01-11 10:17:39 +01:00
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
2026-03-10 11:17:57 +01:00
|
|
|
|
2026-10-13 17:11:38 +02:00
|
|
|
|
2026-01-11 09:56:47 +01:00
|
|
|
|
2026-06-25 14:17:46 +02:00
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
2024-11-01 17:38:14 +01:00
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
2024-11-01 17:38:14 +01:00
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-09-16 17:57:40 +02:00
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-02-10 10:11:48 +01:00
|
|
|
|
zml/buffer: add Buffer.HostAccessible (#899)
`Buffer` placed in `.host_pinned` memory can be read by executables,
but the host can't address it. `Buffer.HostAccessible` gives the host
typed views over its allocations, to write inputs and read outputs in
place between executions, without staging copies. Like `Buffer`, it is
a value: executions read its `buffer` field or donate it through
`&buffer`, and `await` then picks up the handles they replaced.
- `init` takes a `Buffer.Donation`. On CPU, CUDA and ROCm, the devices
holding a shard of an `.undonatable` buffer share one allocation: the
first allocates it and the others read it through
`createViewOfDeviceBuffer`, which now takes the memory the view
belongs to; an external reference keeps the allocation alive.
Elsewhere each device owns an allocation, and writes reach all of
them. Each device owns its allocation of a `.donatable` buffer, so
that no device reads another's output.
- `View(T)` addresses a row-major range of the array: `slice`, `get`,
`set`, `fill`, `fillIota`, `copyFrom`, `copyTo`, `copyFromView`,
`indexOfScalar`, `indexOfAny`, and `blocks`/`reverseBlocks`, which
visit runs contiguous both in memory and in global order. Reads use
the first replica of a region, and writes reach every replica.
`constSlice` and `edit` expose a range as one slice: in place when
every shard holds the whole array densely, through a scratch slice
otherwise.
Addressing follows PJRT's layouts: row-major, strided, and tiled as
XLA's LayoutUtil::LinearIndexForNestedTiling expands them, including
TPU's `T(8,128)(2,1)` packing of bf16 and 8-bit arrays. The test
reference is a direct port of that function, checked against both
figures of https://openxla.org/xla/tiled_layout. Buffers whose shards
all hold the whole array densely, the case on every platform but TPU,
index elements directly.
The buffer is built for llmd's decode loop, which touches it thousands
of times per step:
- Reading a host pointer from PJRT takes and releases a hold on the
buffer and allocates an external reference, about 0.9 us. Pointers
are read at creation, and by `await` only when execution replaced the
handles; `view` panics before that `await` instead of writing to a
released allocation.
- The layout metadata and allocations take several KB. They are
prepared once in a heap-allocated mapping, which views (24 B) and
blocks borrow rather than copy.
On Qwen3.8-27B in llmd (TP=2 on GB300, batch 256, 8k context), this
takes host time between graph completion and the next Execute from
2.07 ms, when views copied the layout and donated buffers read their
pointers in every view, to 0.27-0.30 ms per step (bpftrace probes).
`initForTests` builds a buffer over host memory, for the tiled layouts
and shardings that no available platform produces. `Buffer.from` and
`Buffer.uninitialized` share the new `emptyShell` with
`HostAccessible.init` to compute the PJRT shape, placement and layout.
2026-10-08 14:52:10 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-08-27 14:11:22 +02:00
|
|
|
|
|
|
|
|
|
2025-02-04 11:11:02 +01:00
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
2025-06-06 11:24:21 +02:00
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-09-16 17:57:40 +02:00
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
2026-10-13 17:11:38 +02:00
|
|
|
|
2026-01-11 09:56:47 +01:00
|
|
|
|
|
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
2026-01-11 09:56:47 +01:00
|
|
|
|
zml/buffer: add Buffer.HostAccessible (#899)
`Buffer` placed in `.host_pinned` memory can be read by executables,
but the host can't address it. `Buffer.HostAccessible` gives the host
typed views over its allocations, to write inputs and read outputs in
place between executions, without staging copies. Like `Buffer`, it is
a value: executions read its `buffer` field or donate it through
`&buffer`, and `await` then picks up the handles they replaced.
- `init` takes a `Buffer.Donation`. On CPU, CUDA and ROCm, the devices
holding a shard of an `.undonatable` buffer share one allocation: the
first allocates it and the others read it through
`createViewOfDeviceBuffer`, which now takes the memory the view
belongs to; an external reference keeps the allocation alive.
Elsewhere each device owns an allocation, and writes reach all of
them. Each device owns its allocation of a `.donatable` buffer, so
that no device reads another's output.
- `View(T)` addresses a row-major range of the array: `slice`, `get`,
`set`, `fill`, `fillIota`, `copyFrom`, `copyTo`, `copyFromView`,
`indexOfScalar`, `indexOfAny`, and `blocks`/`reverseBlocks`, which
visit runs contiguous both in memory and in global order. Reads use
the first replica of a region, and writes reach every replica.
`constSlice` and `edit` expose a range as one slice: in place when
every shard holds the whole array densely, through a scratch slice
otherwise.
Addressing follows PJRT's layouts: row-major, strided, and tiled as
XLA's LayoutUtil::LinearIndexForNestedTiling expands them, including
TPU's `T(8,128)(2,1)` packing of bf16 and 8-bit arrays. The test
reference is a direct port of that function, checked against both
figures of https://openxla.org/xla/tiled_layout. Buffers whose shards
all hold the whole array densely, the case on every platform but TPU,
index elements directly.
The buffer is built for llmd's decode loop, which touches it thousands
of times per step:
- Reading a host pointer from PJRT takes and releases a hold on the
buffer and allocates an external reference, about 0.9 us. Pointers
are read at creation, and by `await` only when execution replaced the
handles; `view` panics before that `await` instead of writing to a
released allocation.
- The layout metadata and allocations take several KB. They are
prepared once in a heap-allocated mapping, which views (24 B) and
blocks borrow rather than copy.
On Qwen3.8-27B in llmd (TP=2 on GB300, batch 256, 8k context), this
takes host time between graph completion and the next Execute from
2.07 ms, when views copied the layout and donated buffers read their
pointers in every view, to 0.27-0.30 ms per step (bpftrace probes).
`initForTests` builds a buffer over host memory, for the tiled layouts
and shardings that no available platform produces. `Buffer.from` and
`Buffer.uninitialized` share the new `emptyShell` with
`HostAccessible.init` to compute the PJRT shape, placement and layout.
2026-10-08 14:52:10 +02:00
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-02-04 11:11:02 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-09-18 15:35:25 +02:00
|
|
|
|
2025-02-04 11:11:02 +01:00
|
|
|
|
2025-09-02 15:16:09 +02:00
|
|
|
|
2025-02-04 11:11:02 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-06-06 11:24:21 +02:00
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
2025-02-04 11:11:02 +01:00
|
|
|
|
2026-01-11 10:17:39 +01:00
|
|
|
|
2025-02-04 11:11:02 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-06-06 11:24:21 +02:00
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
2025-02-04 11:11:02 +01:00
|
|
|
|
2026-01-11 10:17:39 +01:00
|
|
|
|
2025-06-06 11:24:21 +02:00
|
|
|
|
2025-02-04 11:11:02 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-01-11 10:17:39 +01:00
|
|
|
|
2025-02-04 11:11:02 +01:00
|
|
|
|
2026-01-11 10:17:39 +01:00
|
|
|
|
2026-01-11 09:56:47 +01:00
|
|
|
|
2026-06-25 14:17:46 +02:00
|
|
|
|
2025-02-04 11:11:02 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-06-19 09:11:53 +02:00
|
|
|
|
2026-03-10 11:17:57 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-10-13 17:11:38 +02:00
|
|
|
|
2026-03-10 11:17:57 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-06-19 09:11:53 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-09-18 15:35:25 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-06-19 09:11:53 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-01-11 09:56:47 +01:00
|
|
|
|
2026-10-13 17:11:38 +02:00
|
|
|
|
2026-01-11 09:56:47 +01:00
|
|
|
|
|
|
|
|
|
2026-06-25 14:17:46 +02:00
|
|
|
|
2025-06-19 09:11:53 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-10-05 11:15:57 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-01-11 22:23:25 +01:00
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-04-22 11:29:48 +02:00
|
|
|
|
2025-09-18 15:35:25 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-04-22 11:29:48 +02:00
|
|
|
|
|
|
|
|
|
2025-06-18 18:32:15 +02:00
|
|
|
|
2026-01-11 10:26:29 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-02-03 10:59:39 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-01-11 10:22:12 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-02-09 12:01:22 +01:00
|
|
|
|
2026-01-11 10:22:12 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-06-18 18:32:15 +02:00
|
|
|
|
zml/sharding: use logical meshes (#943)
## Motivation
This is the big sharding rewrite we need to achieve complex sharding.
Currently we always resolve a mesh logical axis to a mesh physical axis
(or several of them).
But on GPU we always have 1D topology, so we can't have 2D logical
meshes.
So this PR drops the logical -> physical axis bindings, and instead
store a logical -> devices binding.
This is more flexible, and has the nice side effect of simplifying
`Buffer.from`
On stablehlo side this changes how we generate the MLIR because we now
use logical meshes, and `manualComputation` uses logical axes.
(In shardy, GSPM is not affected AFAICT because it uses device ID
directly)
## Buffer.from
This also accelerates `Buffer.from` that I rewrote to be FAST.
Version | Fixed cost | Per device
-- | -- | --
origin/master | 109.6 ns | 35.39 ns
Rewrite | 35.8 ns | 4.38 ns
Improvement | 3.1× | 8.1×
## Creating mesh
I'm adding a new syntax to create LogicalMesh, to allow the more complex
meshes we want to do, and in particular, I now allow to force a specific
number of dimensions.
You can see it in action in `mnist.zig`.
```
const dp_mp = try platform.registerMesh("DP-MP", try .init(
.{ .data, .model },
&.{ .low_bandwidth, .high_bandwidth },
&.{ 2, -1 },
));
info: topology used: Mesh(name=DP-MP, partitions=4, replicas=1)
┌ Mappings (logical coordinates from physical coordinates):
├── data (low_bandwidth) -> (bus / 2)
└── model (high_bandwidth) -> (bus % 2)
┌ PhysicalMesh(platform=cpu shardable_axes={ .bus } num_devices=4)
└── [bus] 0..4 (tree)
├── Device(id=0, coords={ 0 })
├── Device(id=1, coords={ 1 })
├── Device(id=2, coords={ 2 })
└── Device(id=3, coords={ 3 })
┌ Logical coordinates:
└── data=0
└── model=0,1 -> devices=0,1
└── data=1
└── model=0,1 -> devices=2,3
```
The previous syntax is still available, and ZML will align logical axes
to physical axes like we do before this PR.
This part as unit tests, and I checked it work as I expect on CUDA but I
would need to double check the logics still works on TPU/Neuron, because
there is a few platform specific branches in that code.
## Test plan
As part of the benchmark Codex generated a test suite with hard-coded
offsets, allowing to check that Buffer.from didn't change behavior
through the commits, checking we correctly compute each individual
shard.
TODO check that LLMD perf isn't degraded.
2026-10-16 15:27:30 +02:00
|
|
|
|
|
|
|
|
|
2025-06-18 18:32:15 +02:00
|
|
|
|
|
|
|
|
|
2026-10-13 17:11:38 +02:00
|
|
|
|
2025-06-18 18:32:15 +02:00
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-01-11 22:23:25 +01:00
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-01-11 22:23:25 +01:00
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-01-11 22:23:25 +01:00
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-01-08 09:09:31 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-03-10 11:17:57 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-01-08 09:09:31 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-08-27 14:11:22 +02:00
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-06-06 11:24:21 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-01-11 09:56:47 +01:00
|
|
|
|
2025-06-06 11:24:21 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-01-11 22:23:25 +01:00
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-12-09 14:58:56 +01:00
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-06-06 19:03:09 +02:00
|
|
|
|
2026-01-11 10:22:12 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-06-06 19:03:09 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-01-11 22:23:25 +01:00
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-12-09 14:58:56 +01:00
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
2024-11-15 17:12:07 +01:00
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-12-22 15:10:41 +01:00
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
2025-06-06 11:24:21 +02:00
|
|
|
|
2024-11-15 17:12:07 +01:00
|
|
|
|
2025-09-18 15:35:25 +02:00
|
|
|
|
2024-11-15 17:12:07 +01:00
|
|
|
|
2026-01-11 09:56:47 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
2026-01-11 09:56:47 +01:00
|
|
|
|
2026-01-11 10:17:39 +01:00
|
|
|
|
2026-01-11 09:56:47 +01:00
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
2026-01-11 10:17:39 +01:00
|
|
|
|
|
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-01-11 22:23:25 +01:00
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-06-10 14:59:08 +02:00
|
|
|
|
|
|
|
|
|
2025-09-22 09:11:05 +02:00
|
|
|
|
2025-06-10 14:59:08 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-09-22 09:11:05 +02:00
|
|
|
|
2025-06-10 14:59:08 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-09-22 09:11:05 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-03-14 09:49:23 +01:00
|
|
|
|
|
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-03-14 09:49:23 +01:00
|
|
|
|
|
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
2025-12-05 14:11:24 +01:00
|
|
|
|
2025-12-09 14:58:56 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-12-09 14:58:56 +01:00
|
|
|
|
|
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-12-09 14:58:56 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-12-09 14:58:56 +01:00
|
|
|
|
|
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-12-09 14:58:56 +01:00
|
|
|
|
2025-12-05 14:11:24 +01:00
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
2026-09-16 17:57:41 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
2026-03-10 11:17:57 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-01-11 22:23:25 +01:00
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-11-01 17:38:14 +01:00
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
2025-12-08 17:46:17 +01:00
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
2024-11-01 17:38:14 +01:00
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
2026-01-11 22:23:25 +01:00
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-10-13 17:11:38 +02:00
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
2026-01-11 22:23:25 +01:00
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-09-30 17:18:42 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-01-11 22:23:25 +01:00
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-12-09 14:58:56 +01:00
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
2026-01-11 22:23:25 +01:00
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-01-11 22:23:25 +01:00
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
2026-01-11 22:23:25 +01:00
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
2026-01-11 22:23:25 +01:00
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-01-11 22:23:25 +01:00
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-01-11 22:23:25 +01:00
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-08-17 15:16:33 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
2025-02-04 11:11:02 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-01-11 10:17:39 +01:00
|
|
|
|
2025-02-04 11:11:02 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-06-06 11:24:21 +02:00
|
|
|
|
2025-02-04 11:11:02 +01:00
|
|
|
|
|
|
|
|
|
2026-06-25 14:17:46 +02:00
|
|
|
|
2025-02-04 11:11:02 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-09-16 17:57:41 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-10-01 21:12:25 +02:00
|
|
|
|
2026-01-11 09:56:47 +01:00
|
|
|
|
2025-12-09 14:58:56 +01:00
|
|
|
|
2026-01-11 09:56:47 +01:00
|
|
|
|
|
|
|
|
|
2024-10-01 21:12:25 +02:00
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
2026-01-02 22:14:00 +00:00
|
|
|
|
|
|
|
|
|
2026-09-21 19:42:46 +02:00
|
|
|
|
zml/sharding: use logical meshes (#943)
## Motivation
This is the big sharding rewrite we need to achieve complex sharding.
Currently we always resolve a mesh logical axis to a mesh physical axis
(or several of them).
But on GPU we always have 1D topology, so we can't have 2D logical
meshes.
So this PR drops the logical -> physical axis bindings, and instead
store a logical -> devices binding.
This is more flexible, and has the nice side effect of simplifying
`Buffer.from`
On stablehlo side this changes how we generate the MLIR because we now
use logical meshes, and `manualComputation` uses logical axes.
(In shardy, GSPM is not affected AFAICT because it uses device ID
directly)
## Buffer.from
This also accelerates `Buffer.from` that I rewrote to be FAST.
Version | Fixed cost | Per device
-- | -- | --
origin/master | 109.6 ns | 35.39 ns
Rewrite | 35.8 ns | 4.38 ns
Improvement | 3.1× | 8.1×
## Creating mesh
I'm adding a new syntax to create LogicalMesh, to allow the more complex
meshes we want to do, and in particular, I now allow to force a specific
number of dimensions.
You can see it in action in `mnist.zig`.
```
const dp_mp = try platform.registerMesh("DP-MP", try .init(
.{ .data, .model },
&.{ .low_bandwidth, .high_bandwidth },
&.{ 2, -1 },
));
info: topology used: Mesh(name=DP-MP, partitions=4, replicas=1)
┌ Mappings (logical coordinates from physical coordinates):
├── data (low_bandwidth) -> (bus / 2)
└── model (high_bandwidth) -> (bus % 2)
┌ PhysicalMesh(platform=cpu shardable_axes={ .bus } num_devices=4)
└── [bus] 0..4 (tree)
├── Device(id=0, coords={ 0 })
├── Device(id=1, coords={ 1 })
├── Device(id=2, coords={ 2 })
└── Device(id=3, coords={ 3 })
┌ Logical coordinates:
└── data=0
└── model=0,1 -> devices=0,1
└── data=1
└── model=0,1 -> devices=2,3
```
The previous syntax is still available, and ZML will align logical axes
to physical axes like we do before this PR.
This part as unit tests, and I checked it work as I expect on CUDA but I
would need to double check the logics still works on TPU/Neuron, because
there is a few platform specific branches in that code.
## Test plan
As part of the benchmark Codex generated a test suite with hard-coded
offsets, allowing to check that Buffer.from didn't change behavior
through the commits, checking we correctly compute each individual
shard.
TODO check that LLMD perf isn't degraded.
2026-10-16 15:27:30 +02:00
|
|
|
|
2026-01-02 22:14:00 +00:00
|
|
|
|
|
|
|
|
|
2026-01-27 23:19:10 +00:00
|
|
|
|
2026-01-02 22:14:00 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-01-27 23:19:10 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-01-02 22:14:00 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
zml/sharding: use logical meshes (#943)
## Motivation
This is the big sharding rewrite we need to achieve complex sharding.
Currently we always resolve a mesh logical axis to a mesh physical axis
(or several of them).
But on GPU we always have 1D topology, so we can't have 2D logical
meshes.
So this PR drops the logical -> physical axis bindings, and instead
store a logical -> devices binding.
This is more flexible, and has the nice side effect of simplifying
`Buffer.from`
On stablehlo side this changes how we generate the MLIR because we now
use logical meshes, and `manualComputation` uses logical axes.
(In shardy, GSPM is not affected AFAICT because it uses device ID
directly)
## Buffer.from
This also accelerates `Buffer.from` that I rewrote to be FAST.
Version | Fixed cost | Per device
-- | -- | --
origin/master | 109.6 ns | 35.39 ns
Rewrite | 35.8 ns | 4.38 ns
Improvement | 3.1× | 8.1×
## Creating mesh
I'm adding a new syntax to create LogicalMesh, to allow the more complex
meshes we want to do, and in particular, I now allow to force a specific
number of dimensions.
You can see it in action in `mnist.zig`.
```
const dp_mp = try platform.registerMesh("DP-MP", try .init(
.{ .data, .model },
&.{ .low_bandwidth, .high_bandwidth },
&.{ 2, -1 },
));
info: topology used: Mesh(name=DP-MP, partitions=4, replicas=1)
┌ Mappings (logical coordinates from physical coordinates):
├── data (low_bandwidth) -> (bus / 2)
└── model (high_bandwidth) -> (bus % 2)
┌ PhysicalMesh(platform=cpu shardable_axes={ .bus } num_devices=4)
└── [bus] 0..4 (tree)
├── Device(id=0, coords={ 0 })
├── Device(id=1, coords={ 1 })
├── Device(id=2, coords={ 2 })
└── Device(id=3, coords={ 3 })
┌ Logical coordinates:
└── data=0
└── model=0,1 -> devices=0,1
└── data=1
└── model=0,1 -> devices=2,3
```
The previous syntax is still available, and ZML will align logical axes
to physical axes like we do before this PR.
This part as unit tests, and I checked it work as I expect on CUDA but I
would need to double check the logics still works on TPU/Neuron, because
there is a few platform specific branches in that code.
## Test plan
As part of the benchmark Codex generated a test suite with hard-coded
offsets, allowing to check that Buffer.from didn't change behavior
through the commits, checking we correctly compute each individual
shard.
TODO check that LLMD perf isn't degraded.
2026-10-16 15:27:30 +02:00
|
|
|
|
2026-01-02 22:14:00 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-01-27 23:19:10 +00:00
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
2026-01-27 23:19:10 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-02-04 11:11:02 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-09-18 15:35:25 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-02-04 11:11:02 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-04-08 16:28:39 +02:00
|
|
|
|
2025-02-04 11:11:02 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-04-08 16:28:39 +02:00
|
|
|
|
2025-09-18 15:35:25 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-02-04 11:11:02 +01:00
|
|
|
|
|
|
|
|
|
2026-02-03 10:59:39 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-02-04 11:11:02 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-12-09 14:58:56 +01:00
|
|
|
|
2025-02-04 11:11:02 +01:00
|
|
|
|
2025-12-09 14:58:56 +01:00
|
|
|
|
2025-02-04 11:11:02 +01:00
|
|
|
|
|
|
|
|
|
2025-12-09 14:58:56 +01:00
|
|
|
|
2025-02-04 11:11:02 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-01-11 09:56:47 +01:00
|
|
|
|
2025-02-04 11:11:02 +01:00
|
|
|
|
|
|
|
|
|
2026-01-11 09:56:47 +01:00
|
|
|
|
2025-02-04 11:11:02 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-01-11 09:56:47 +01:00
|
|
|
|
2025-02-04 11:11:02 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-01-11 09:56:47 +01:00
|
|
|
|
2025-02-04 11:11:02 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-01-11 09:56:47 +01:00
|
|
|
|
2025-02-04 11:11:02 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-01-11 10:17:39 +01:00
|
|
|
|
2025-02-04 11:11:02 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-01-08 09:08:52 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
2026-10-13 17:11:38 +02:00
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-01-08 09:08:52 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-10-13 17:11:38 +02:00
|
|
|
|
2026-01-08 09:08:52 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-10-13 17:11:38 +02:00
|
|
|
|
2026-01-08 09:08:52 +00:00
|
|
|
|
2026-01-11 10:17:39 +01:00
|
|
|
|
2026-01-08 09:08:52 +00:00
|
|
|
|
2026-01-11 10:17:39 +01:00
|
|
|
|
2026-01-08 09:08:52 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
2025-09-18 11:36:23 +02:00
|
|
|
|
2026-01-08 09:08:52 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-09-17 11:11:18 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-11-15 11:22:01 +01:00
|
|
|
|
2025-09-04 16:57:34 +02:00
|
|
|
|
2025-06-06 11:24:21 +02:00
|
|
|
|
2024-11-15 11:22:01 +01:00
|
|
|
|
2025-06-06 11:24:21 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-11-12 11:55:48 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-06-06 11:24:21 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-09-04 16:57:34 +02:00
|
|
|
|
2025-04-08 16:28:39 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-09-04 16:57:34 +02:00
|
|
|
|
2025-04-08 16:28:39 +02:00
|
|
|
|
2026-01-11 09:56:47 +01:00
|
|
|
|
2025-04-08 16:28:39 +02:00
|
|
|
|
|
|
|
|
|
2026-01-16 10:59:12 +01:00
|
|
|
|
2025-04-08 16:28:39 +02:00
|
|
|
|
|
|
|
|
|
2025-09-04 16:57:34 +02:00
|
|
|
|
2026-01-11 09:56:47 +01:00
|
|
|
|
2026-01-15 18:20:15 +00:00
|
|
|
|
2024-11-15 11:22:01 +01:00
|
|
|
|
|
|
|
|
|
2025-06-06 11:24:21 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-11-12 11:55:48 +01:00
|
|
|
|
2026-01-11 09:56:47 +01:00
|
|
|
|
2025-06-06 11:24:21 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-11-12 11:55:48 +01:00
|
|
|
|
2026-01-11 09:56:47 +01:00
|
|
|
|
2025-11-12 11:55:48 +01:00
|
|
|
|
2025-06-06 11:24:21 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-09-04 16:57:34 +02:00
|
|
|
|
2025-06-06 11:24:21 +02:00
|
|
|
|
|
|
|
|
|
2025-09-04 16:57:34 +02:00
|
|
|
|
2026-01-11 09:56:47 +01:00
|
|
|
|
2025-06-06 11:24:21 +02:00
|
|
|
|
|
|
|
|
|
2026-01-11 09:56:47 +01:00
|
|
|
|
2025-06-06 11:24:21 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2024-11-15 11:22:01 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-03-10 11:17:57 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|