0
Fork 0
mirror of https://github.com/zml/zml.git synced 2026-11-16 19:11:09 +00:00
zml/pjrt/pjrt.zig

1864 lines
70 KiB

2024-09-17 11:11:18 +02:00
2024-09-17 11:11:18 +02:00
2024-09-17 11:11:18 +02:00
zml/sharding: use logical meshes (#943) ## Motivation This is the big sharding rewrite we need to achieve complex sharding. Currently we always resolve a mesh logical axis to a mesh physical axis (or several of them). But on GPU we always have 1D topology, so we can't have 2D logical meshes. So this PR drops the logical -> physical axis bindings, and instead store a logical -> devices binding. This is more flexible, and has the nice side effect of simplifying `Buffer.from` On stablehlo side this changes how we generate the MLIR because we now use logical meshes, and `manualComputation` uses logical axes. (In shardy, GSPM is not affected AFAICT because it uses device ID directly) ## Buffer.from This also accelerates `Buffer.from` that I rewrote to be FAST. Version | Fixed cost | Per device -- | -- | -- origin/master | 109.6 ns | 35.39 ns Rewrite | 35.8 ns | 4.38 ns Improvement | 3.1× | 8.1× ## Creating mesh I'm adding a new syntax to create LogicalMesh, to allow the more complex meshes we want to do, and in particular, I now allow to force a specific number of dimensions. You can see it in action in `mnist.zig`. ``` const dp_mp = try platform.registerMesh("DP-MP", try .init( .{ .data, .model }, &.{ .low_bandwidth, .high_bandwidth }, &.{ 2, -1 }, )); info: topology used: Mesh(name=DP-MP, partitions=4, replicas=1) ┌ Mappings (logical coordinates from physical coordinates): ├── data (low_bandwidth) -> (bus / 2) └── model (high_bandwidth) -> (bus % 2) ┌ PhysicalMesh(platform=cpu shardable_axes={ .bus } num_devices=4) └── [bus] 0..4 (tree) ├── Device(id=0, coords={ 0 }) ├── Device(id=1, coords={ 1 }) ├── Device(id=2, coords={ 2 }) └── Device(id=3, coords={ 3 }) ┌ Logical coordinates: └── data=0 └── model=0,1 -> devices=0,1 └── data=1 └── model=0,1 -> devices=2,3 ``` The previous syntax is still available, and ZML will align logical axes to physical axes like we do before this PR. This part as unit tests, and I checked it work as I expect on CUDA but I would need to double check the logics still works on TPU/Neuron, because there is a few platform specific branches in that code. ## Test plan As part of the benchmark Codex generated a test suite with hard-coded offsets, allowing to check that Buffer.from didn't change behavior through the commits, checking we correctly compute each individual shard. TODO check that LLMD perf isn't degraded.
2026-10-16 15:27:30 +02:00
2024-09-17 11:11:18 +02:00
2026-01-15 18:20:15 +00:00
2026-01-11 09:48:15 +01:00
2024-09-17 11:11:18 +02:00
2026-01-11 09:48:15 +01:00
2026-01-11 09:48:15 +01:00
2026-01-11 09:48:15 +01:00
2026-01-11 09:48:15 +01:00
2025-12-05 14:11:24 +01:00
2024-09-17 11:11:18 +02:00
zml/sharding: use logical meshes (#943) ## Motivation This is the big sharding rewrite we need to achieve complex sharding. Currently we always resolve a mesh logical axis to a mesh physical axis (or several of them). But on GPU we always have 1D topology, so we can't have 2D logical meshes. So this PR drops the logical -> physical axis bindings, and instead store a logical -> devices binding. This is more flexible, and has the nice side effect of simplifying `Buffer.from` On stablehlo side this changes how we generate the MLIR because we now use logical meshes, and `manualComputation` uses logical axes. (In shardy, GSPM is not affected AFAICT because it uses device ID directly) ## Buffer.from This also accelerates `Buffer.from` that I rewrote to be FAST. Version | Fixed cost | Per device -- | -- | -- origin/master | 109.6 ns | 35.39 ns Rewrite | 35.8 ns | 4.38 ns Improvement | 3.1× | 8.1× ## Creating mesh I'm adding a new syntax to create LogicalMesh, to allow the more complex meshes we want to do, and in particular, I now allow to force a specific number of dimensions. You can see it in action in `mnist.zig`. ``` const dp_mp = try platform.registerMesh("DP-MP", try .init( .{ .data, .model }, &.{ .low_bandwidth, .high_bandwidth }, &.{ 2, -1 }, )); info: topology used: Mesh(name=DP-MP, partitions=4, replicas=1) ┌ Mappings (logical coordinates from physical coordinates): ├── data (low_bandwidth) -> (bus / 2) └── model (high_bandwidth) -> (bus % 2) ┌ PhysicalMesh(platform=cpu shardable_axes={ .bus } num_devices=4) └── [bus] 0..4 (tree) ├── Device(id=0, coords={ 0 }) ├── Device(id=1, coords={ 1 }) ├── Device(id=2, coords={ 2 }) └── Device(id=3, coords={ 3 }) ┌ Logical coordinates: └── data=0 └── model=0,1 -> devices=0,1 └── data=1 └── model=0,1 -> devices=2,3 ``` The previous syntax is still available, and ZML will align logical axes to physical axes like we do before this PR. This part as unit tests, and I checked it work as I expect on CUDA but I would need to double check the logics still works on TPU/Neuron, because there is a few platform specific branches in that code. ## Test plan As part of the benchmark Codex generated a test suite with hard-coded offsets, allowing to check that Buffer.from didn't change behavior through the commits, checking we correctly compute each individual shard. TODO check that LLMD perf isn't degraded.
2026-10-16 15:27:30 +02:00
zml/sharding: use logical meshes (#943) ## Motivation This is the big sharding rewrite we need to achieve complex sharding. Currently we always resolve a mesh logical axis to a mesh physical axis (or several of them). But on GPU we always have 1D topology, so we can't have 2D logical meshes. So this PR drops the logical -> physical axis bindings, and instead store a logical -> devices binding. This is more flexible, and has the nice side effect of simplifying `Buffer.from` On stablehlo side this changes how we generate the MLIR because we now use logical meshes, and `manualComputation` uses logical axes. (In shardy, GSPM is not affected AFAICT because it uses device ID directly) ## Buffer.from This also accelerates `Buffer.from` that I rewrote to be FAST. Version | Fixed cost | Per device -- | -- | -- origin/master | 109.6 ns | 35.39 ns Rewrite | 35.8 ns | 4.38 ns Improvement | 3.1× | 8.1× ## Creating mesh I'm adding a new syntax to create LogicalMesh, to allow the more complex meshes we want to do, and in particular, I now allow to force a specific number of dimensions. You can see it in action in `mnist.zig`. ``` const dp_mp = try platform.registerMesh("DP-MP", try .init( .{ .data, .model }, &.{ .low_bandwidth, .high_bandwidth }, &.{ 2, -1 }, )); info: topology used: Mesh(name=DP-MP, partitions=4, replicas=1) ┌ Mappings (logical coordinates from physical coordinates): ├── data (low_bandwidth) -> (bus / 2) └── model (high_bandwidth) -> (bus % 2) ┌ PhysicalMesh(platform=cpu shardable_axes={ .bus } num_devices=4) └── [bus] 0..4 (tree) ├── Device(id=0, coords={ 0 }) ├── Device(id=1, coords={ 1 }) ├── Device(id=2, coords={ 2 }) └── Device(id=3, coords={ 3 }) ┌ Logical coordinates: └── data=0 └── model=0,1 -> devices=0,1 └── data=1 └── model=0,1 -> devices=2,3 ``` The previous syntax is still available, and ZML will align logical axes to physical axes like we do before this PR. This part as unit tests, and I checked it work as I expect on CUDA but I would need to double check the logics still works on TPU/Neuron, because there is a few platform specific branches in that code. ## Test plan As part of the benchmark Codex generated a test suite with hard-coded offsets, allowing to check that Buffer.from didn't change behavior through the commits, checking we correctly compute each individual shard. TODO check that LLMD perf isn't degraded.
2026-10-16 15:27:30 +02:00
2024-09-17 11:11:18 +02:00
2024-09-17 11:11:18 +02:00
2024-09-17 11:11:18 +02:00
2024-09-17 11:11:18 +02:00
2024-09-17 11:11:18 +02:00
2024-09-17 11:11:18 +02:00
2026-02-12 21:00:05 +01:00
pjrt: fix misleading pjrt plugin load success log (#622) Move the PJRT plugin “Loaded” log so it is only emitted after the dynamic library `.so` is opened and the PJRT API is initialized successfully. Previously, `Api.loadFrom` used a `defer` for the success log. Failed plugin load attempts could still print `Loaded: ...`. This was misleading during platform auto-detection, for example when probing TPU before falling back to CUDA. Example: ``` 2026-07-01 13:12:17 ███████╗███╗ ███╗██╗ ██╗ ██╗ ██╗ ███╗ ███╗██████╗ 2026-07-01 13:12:17 ╚══███╔╝████╗ ████║██║ ██╔╝ ██║ ██║ ████╗ ████║██╔══██╗ 2026-07-01 13:12:17 ███╔╝ ██╔████╔██║██║ ██╔╝ ██║ ██║ ██╔████╔██║██║ ██║ 2026-07-01 13:12:17 ███╔╝ ██║╚██╔╝██║██║ .ai ██╔╝ ██║ ██║ ██║╚██╔╝██║██║ ██║ 2026-07-01 13:12:17 ███████╗██║ ╚═╝ ██║███████╗ ██╔╝ ███████╗███████╗██║ ╚═╝ ██║██████╔╝ 2026-07-01 13:12:17 ╚══════╝╚═╝ ╚═╝╚══════╝ ╚═╝ ╚══════╝╚══════╝╚═╝ ╚═╝╚═════╝ 2026-07-01 13:12:17 Copyright (c) 2026 ZML 2026-07-01 13:12:17 warning(zml/io/vfs/s3): S3 initialized without full credentials (AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY). Access will be unauthenticated. Using default endpoint: https://s3.amazonaws.com (set AWS_ENDPOINT_URL or AWS_ENDPOINT_URL_S3). Using default region: us-east-1 (set AWS_REGION or AWS_DEFAULT_REGION). 2026-07-01 13:12:17 info(pjrt): Loading: libpjrt_tpu.so... 2026-07-01 13:12:17 info(llmd): Loading tokenizer 2026-07-01 13:12:17 error(pjrt): Unable to dlopen plugin: /llmd/llmd.runfiles/zml++tpu_packages+libpjrt_tpu/sandbox/lib/libpjrt_tpu.so 2026-07-01 13:12:17 info(pjrt): Loaded: libpjrt_tpu.so #################### HERE #################### 2026-07-01 13:12:17 info(llmd): Resolving model repo 2026-07-01 13:12:17 info(llmd): Loading model config 2026-07-01 13:12:17 info(llmd): Loaded model config [67.405ms] 2026-07-01 13:12:18 info(pjrt): Loading: libpjrt_cuda.so... 2026-07-01 13:12:18 warning(zml/platforms/cuda): Detected NVIDIA GPU that requires CUDA compatibility libraries. 2026-07-01 13:12:18 info(zml/platforms/cuda): Loaded CUDA compatibility libraries. 2026-07-01 13:12:18 info(pjrt): Loaded: libpjrt_cuda.so 2026-07-01 13:12:18 info(llmd): Loaded tokenizer [334.168ms] 2026-07-01 13:12:20 info(llmd): Devices: 2026-07-01 13:12:20 info(llmd): - NVIDIA RTX PRO 6000 Blackwell Server Edition (cuda:0) 2026-07-01 13:12:20 info: Using 31711 pages for the KvCache 2026-07-01 13:12:20 info: sizes: { 256 } 2026-07-01 13:12:27 info: Compiled all models [7.336s] 2026-07-01 13:12:50 info: Listening on 0.0.0.0:8000 2026-07-01 13:12:50 info(llama): Loaded weights [14.96GiB, 22.778s, 672.41MiB/s] ``` Moreover make `fromDynLib` return errors for missing `GetPjrtApi` and failed `PJRT_Plugin_Initialize` instead of panicking or using `catch unreachable`. So that `Api.loadFrom()` can catch it. After the change, the output is: ``` 2026-07-01 14:50:50 ███████╗███╗ ███╗██╗ ██╗ ██╗ ██╗ ███╗ ███╗██████╗ 2026-07-01 14:50:50 ╚══███╔╝████╗ ████║██║ ██╔╝ ██║ ██║ ████╗ ████║██╔══██╗ 2026-07-01 14:50:50 ███╔╝ ██╔████╔██║██║ ██╔╝ ██║ ██║ ██╔████╔██║██║ ██║ 2026-07-01 14:50:50 ███╔╝ ██║╚██╔╝██║██║ .ai ██╔╝ ██║ ██║ ██║╚██╔╝██║██║ ██║ 2026-07-01 14:50:50 ███████╗██║ ╚═╝ ██║███████╗ ██╔╝ ███████╗███████╗██║ ╚═╝ ██║██████╔╝ 2026-07-01 14:50:50 ╚══════╝╚═╝ ╚═╝╚══════╝ ╚═╝ ╚══════╝╚══════╝╚═╝ ╚═╝╚═════╝ 2026-07-01 14:50:50 Copyright (c) 2026 ZML 2026-07-01 14:50:51 warning(zml/io/vfs/s3): S3 initialized without full credentials (AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY). Access will be unauthenticated. Using default endpoint: https://s3.amazonaws.com (set AWS_ENDPOINT_URL or AWS_ENDPOINT_URL_S3). Using default region: us-east-1 (set AWS_REGION or AWS_DEFAULT_REGION). 2026-07-01 14:50:51 info(llmd): Resolving model repo 2026-07-01 14:50:51 info(llmd): Loading model config 2026-07-01 14:50:51 info(llmd): Loaded model config [101.024ms] 2026-07-01 14:50:51 info(llmd): Loading tokenizer 2026-07-01 14:50:51 info(pjrt): Loading: libpjrt_tpu.so... 2026-07-01 14:50:51 error(pjrt): Unable to dlopen plugin: /llmd/llmd.runfiles/zml++tpu_packages+libpjrt_tpu/sandbox/lib/libpjrt_tpu.so 2026-07-01 14:50:52 info(llmd): Loaded tokenizer [349.499ms] 2026-07-01 14:50:53 info(pjrt): Loading: libpjrt_cuda.so... 2026-07-01 14:50:53 warning(zml/platforms/cuda): Detected NVIDIA GPU that requires CUDA compatibility libraries. 2026-07-01 14:50:53 info(zml/platforms/cuda): Loaded CUDA compatibility libraries. 2026-07-01 14:51:11 info(pjrt): Loaded: libpjrt_cuda.so 2026-07-01 14:51:14 info: Using 31711 pages for the KvCache 2026-07-01 14:51:14 info: sizes: { 256 } 2026-07-01 14:51:14 info(llmd): Devices: 2026-07-01 14:51:14 info(llmd): - NVIDIA RTX PRO 6000 Blackwell Server Edition (cuda:0) 2026-07-01 14:51:22 info: Compiled all models [8.395s] 2026-07-01 14:51:46 info: Listening on 0.0.0.0:8000 2026-07-01 14:51:46 info(llama): Loaded weights [14.96GiB, 23.629s, 648.20MiB/s] ```
2026-07-02 14:01:18 +02:00
pjrt: fix misleading pjrt plugin load success log (#622) Move the PJRT plugin “Loaded” log so it is only emitted after the dynamic library `.so` is opened and the PJRT API is initialized successfully. Previously, `Api.loadFrom` used a `defer` for the success log. Failed plugin load attempts could still print `Loaded: ...`. This was misleading during platform auto-detection, for example when probing TPU before falling back to CUDA. Example: ``` 2026-07-01 13:12:17 ███████╗███╗ ███╗██╗ ██╗ ██╗ ██╗ ███╗ ███╗██████╗ 2026-07-01 13:12:17 ╚══███╔╝████╗ ████║██║ ██╔╝ ██║ ██║ ████╗ ████║██╔══██╗ 2026-07-01 13:12:17 ███╔╝ ██╔████╔██║██║ ██╔╝ ██║ ██║ ██╔████╔██║██║ ██║ 2026-07-01 13:12:17 ███╔╝ ██║╚██╔╝██║██║ .ai ██╔╝ ██║ ██║ ██║╚██╔╝██║██║ ██║ 2026-07-01 13:12:17 ███████╗██║ ╚═╝ ██║███████╗ ██╔╝ ███████╗███████╗██║ ╚═╝ ██║██████╔╝ 2026-07-01 13:12:17 ╚══════╝╚═╝ ╚═╝╚══════╝ ╚═╝ ╚══════╝╚══════╝╚═╝ ╚═╝╚═════╝ 2026-07-01 13:12:17 Copyright (c) 2026 ZML 2026-07-01 13:12:17 warning(zml/io/vfs/s3): S3 initialized without full credentials (AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY). Access will be unauthenticated. Using default endpoint: https://s3.amazonaws.com (set AWS_ENDPOINT_URL or AWS_ENDPOINT_URL_S3). Using default region: us-east-1 (set AWS_REGION or AWS_DEFAULT_REGION). 2026-07-01 13:12:17 info(pjrt): Loading: libpjrt_tpu.so... 2026-07-01 13:12:17 info(llmd): Loading tokenizer 2026-07-01 13:12:17 error(pjrt): Unable to dlopen plugin: /llmd/llmd.runfiles/zml++tpu_packages+libpjrt_tpu/sandbox/lib/libpjrt_tpu.so 2026-07-01 13:12:17 info(pjrt): Loaded: libpjrt_tpu.so #################### HERE #################### 2026-07-01 13:12:17 info(llmd): Resolving model repo 2026-07-01 13:12:17 info(llmd): Loading model config 2026-07-01 13:12:17 info(llmd): Loaded model config [67.405ms] 2026-07-01 13:12:18 info(pjrt): Loading: libpjrt_cuda.so... 2026-07-01 13:12:18 warning(zml/platforms/cuda): Detected NVIDIA GPU that requires CUDA compatibility libraries. 2026-07-01 13:12:18 info(zml/platforms/cuda): Loaded CUDA compatibility libraries. 2026-07-01 13:12:18 info(pjrt): Loaded: libpjrt_cuda.so 2026-07-01 13:12:18 info(llmd): Loaded tokenizer [334.168ms] 2026-07-01 13:12:20 info(llmd): Devices: 2026-07-01 13:12:20 info(llmd): - NVIDIA RTX PRO 6000 Blackwell Server Edition (cuda:0) 2026-07-01 13:12:20 info: Using 31711 pages for the KvCache 2026-07-01 13:12:20 info: sizes: { 256 } 2026-07-01 13:12:27 info: Compiled all models [7.336s] 2026-07-01 13:12:50 info: Listening on 0.0.0.0:8000 2026-07-01 13:12:50 info(llama): Loaded weights [14.96GiB, 22.778s, 672.41MiB/s] ``` Moreover make `fromDynLib` return errors for missing `GetPjrtApi` and failed `PJRT_Plugin_Initialize` instead of panicking or using `catch unreachable`. So that `Api.loadFrom()` can catch it. After the change, the output is: ``` 2026-07-01 14:50:50 ███████╗███╗ ███╗██╗ ██╗ ██╗ ██╗ ███╗ ███╗██████╗ 2026-07-01 14:50:50 ╚══███╔╝████╗ ████║██║ ██╔╝ ██║ ██║ ████╗ ████║██╔══██╗ 2026-07-01 14:50:50 ███╔╝ ██╔████╔██║██║ ██╔╝ ██║ ██║ ██╔████╔██║██║ ██║ 2026-07-01 14:50:50 ███╔╝ ██║╚██╔╝██║██║ .ai ██╔╝ ██║ ██║ ██║╚██╔╝██║██║ ██║ 2026-07-01 14:50:50 ███████╗██║ ╚═╝ ██║███████╗ ██╔╝ ███████╗███████╗██║ ╚═╝ ██║██████╔╝ 2026-07-01 14:50:50 ╚══════╝╚═╝ ╚═╝╚══════╝ ╚═╝ ╚══════╝╚══════╝╚═╝ ╚═╝╚═════╝ 2026-07-01 14:50:50 Copyright (c) 2026 ZML 2026-07-01 14:50:51 warning(zml/io/vfs/s3): S3 initialized without full credentials (AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY). Access will be unauthenticated. Using default endpoint: https://s3.amazonaws.com (set AWS_ENDPOINT_URL or AWS_ENDPOINT_URL_S3). Using default region: us-east-1 (set AWS_REGION or AWS_DEFAULT_REGION). 2026-07-01 14:50:51 info(llmd): Resolving model repo 2026-07-01 14:50:51 info(llmd): Loading model config 2026-07-01 14:50:51 info(llmd): Loaded model config [101.024ms] 2026-07-01 14:50:51 info(llmd): Loading tokenizer 2026-07-01 14:50:51 info(pjrt): Loading: libpjrt_tpu.so... 2026-07-01 14:50:51 error(pjrt): Unable to dlopen plugin: /llmd/llmd.runfiles/zml++tpu_packages+libpjrt_tpu/sandbox/lib/libpjrt_tpu.so 2026-07-01 14:50:52 info(llmd): Loaded tokenizer [349.499ms] 2026-07-01 14:50:53 info(pjrt): Loading: libpjrt_cuda.so... 2026-07-01 14:50:53 warning(zml/platforms/cuda): Detected NVIDIA GPU that requires CUDA compatibility libraries. 2026-07-01 14:50:53 info(zml/platforms/cuda): Loaded CUDA compatibility libraries. 2026-07-01 14:51:11 info(pjrt): Loaded: libpjrt_cuda.so 2026-07-01 14:51:14 info: Using 31711 pages for the KvCache 2026-07-01 14:51:14 info: sizes: { 256 } 2026-07-01 14:51:14 info(llmd): Devices: 2026-07-01 14:51:14 info(llmd): - NVIDIA RTX PRO 6000 Blackwell Server Edition (cuda:0) 2026-07-01 14:51:22 info: Compiled all models [8.395s] 2026-07-01 14:51:46 info: Listening on 0.0.0.0:8000 2026-07-01 14:51:46 info(llama): Loaded weights [14.96GiB, 23.629s, 648.20MiB/s] ```
2026-07-02 14:01:18 +02:00
2026-02-12 21:00:05 +01:00
pjrt: fix misleading pjrt plugin load success log (#622) Move the PJRT plugin “Loaded” log so it is only emitted after the dynamic library `.so` is opened and the PJRT API is initialized successfully. Previously, `Api.loadFrom` used a `defer` for the success log. Failed plugin load attempts could still print `Loaded: ...`. This was misleading during platform auto-detection, for example when probing TPU before falling back to CUDA. Example: ``` 2026-07-01 13:12:17 ███████╗███╗ ███╗██╗ ██╗ ██╗ ██╗ ███╗ ███╗██████╗ 2026-07-01 13:12:17 ╚══███╔╝████╗ ████║██║ ██╔╝ ██║ ██║ ████╗ ████║██╔══██╗ 2026-07-01 13:12:17 ███╔╝ ██╔████╔██║██║ ██╔╝ ██║ ██║ ██╔████╔██║██║ ██║ 2026-07-01 13:12:17 ███╔╝ ██║╚██╔╝██║██║ .ai ██╔╝ ██║ ██║ ██║╚██╔╝██║██║ ██║ 2026-07-01 13:12:17 ███████╗██║ ╚═╝ ██║███████╗ ██╔╝ ███████╗███████╗██║ ╚═╝ ██║██████╔╝ 2026-07-01 13:12:17 ╚══════╝╚═╝ ╚═╝╚══════╝ ╚═╝ ╚══════╝╚══════╝╚═╝ ╚═╝╚═════╝ 2026-07-01 13:12:17 Copyright (c) 2026 ZML 2026-07-01 13:12:17 warning(zml/io/vfs/s3): S3 initialized without full credentials (AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY). Access will be unauthenticated. Using default endpoint: https://s3.amazonaws.com (set AWS_ENDPOINT_URL or AWS_ENDPOINT_URL_S3). Using default region: us-east-1 (set AWS_REGION or AWS_DEFAULT_REGION). 2026-07-01 13:12:17 info(pjrt): Loading: libpjrt_tpu.so... 2026-07-01 13:12:17 info(llmd): Loading tokenizer 2026-07-01 13:12:17 error(pjrt): Unable to dlopen plugin: /llmd/llmd.runfiles/zml++tpu_packages+libpjrt_tpu/sandbox/lib/libpjrt_tpu.so 2026-07-01 13:12:17 info(pjrt): Loaded: libpjrt_tpu.so #################### HERE #################### 2026-07-01 13:12:17 info(llmd): Resolving model repo 2026-07-01 13:12:17 info(llmd): Loading model config 2026-07-01 13:12:17 info(llmd): Loaded model config [67.405ms] 2026-07-01 13:12:18 info(pjrt): Loading: libpjrt_cuda.so... 2026-07-01 13:12:18 warning(zml/platforms/cuda): Detected NVIDIA GPU that requires CUDA compatibility libraries. 2026-07-01 13:12:18 info(zml/platforms/cuda): Loaded CUDA compatibility libraries. 2026-07-01 13:12:18 info(pjrt): Loaded: libpjrt_cuda.so 2026-07-01 13:12:18 info(llmd): Loaded tokenizer [334.168ms] 2026-07-01 13:12:20 info(llmd): Devices: 2026-07-01 13:12:20 info(llmd): - NVIDIA RTX PRO 6000 Blackwell Server Edition (cuda:0) 2026-07-01 13:12:20 info: Using 31711 pages for the KvCache 2026-07-01 13:12:20 info: sizes: { 256 } 2026-07-01 13:12:27 info: Compiled all models [7.336s] 2026-07-01 13:12:50 info: Listening on 0.0.0.0:8000 2026-07-01 13:12:50 info(llama): Loaded weights [14.96GiB, 22.778s, 672.41MiB/s] ``` Moreover make `fromDynLib` return errors for missing `GetPjrtApi` and failed `PJRT_Plugin_Initialize` instead of panicking or using `catch unreachable`. So that `Api.loadFrom()` can catch it. After the change, the output is: ``` 2026-07-01 14:50:50 ███████╗███╗ ███╗██╗ ██╗ ██╗ ██╗ ███╗ ███╗██████╗ 2026-07-01 14:50:50 ╚══███╔╝████╗ ████║██║ ██╔╝ ██║ ██║ ████╗ ████║██╔══██╗ 2026-07-01 14:50:50 ███╔╝ ██╔████╔██║██║ ██╔╝ ██║ ██║ ██╔████╔██║██║ ██║ 2026-07-01 14:50:50 ███╔╝ ██║╚██╔╝██║██║ .ai ██╔╝ ██║ ██║ ██║╚██╔╝██║██║ ██║ 2026-07-01 14:50:50 ███████╗██║ ╚═╝ ██║███████╗ ██╔╝ ███████╗███████╗██║ ╚═╝ ██║██████╔╝ 2026-07-01 14:50:50 ╚══════╝╚═╝ ╚═╝╚══════╝ ╚═╝ ╚══════╝╚══════╝╚═╝ ╚═╝╚═════╝ 2026-07-01 14:50:50 Copyright (c) 2026 ZML 2026-07-01 14:50:51 warning(zml/io/vfs/s3): S3 initialized without full credentials (AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY). Access will be unauthenticated. Using default endpoint: https://s3.amazonaws.com (set AWS_ENDPOINT_URL or AWS_ENDPOINT_URL_S3). Using default region: us-east-1 (set AWS_REGION or AWS_DEFAULT_REGION). 2026-07-01 14:50:51 info(llmd): Resolving model repo 2026-07-01 14:50:51 info(llmd): Loading model config 2026-07-01 14:50:51 info(llmd): Loaded model config [101.024ms] 2026-07-01 14:50:51 info(llmd): Loading tokenizer 2026-07-01 14:50:51 info(pjrt): Loading: libpjrt_tpu.so... 2026-07-01 14:50:51 error(pjrt): Unable to dlopen plugin: /llmd/llmd.runfiles/zml++tpu_packages+libpjrt_tpu/sandbox/lib/libpjrt_tpu.so 2026-07-01 14:50:52 info(llmd): Loaded tokenizer [349.499ms] 2026-07-01 14:50:53 info(pjrt): Loading: libpjrt_cuda.so... 2026-07-01 14:50:53 warning(zml/platforms/cuda): Detected NVIDIA GPU that requires CUDA compatibility libraries. 2026-07-01 14:50:53 info(zml/platforms/cuda): Loaded CUDA compatibility libraries. 2026-07-01 14:51:11 info(pjrt): Loaded: libpjrt_cuda.so 2026-07-01 14:51:14 info: Using 31711 pages for the KvCache 2026-07-01 14:51:14 info: sizes: { 256 } 2026-07-01 14:51:14 info(llmd): Devices: 2026-07-01 14:51:14 info(llmd): - NVIDIA RTX PRO 6000 Blackwell Server Edition (cuda:0) 2026-07-01 14:51:22 info: Compiled all models [8.395s] 2026-07-01 14:51:46 info: Listening on 0.0.0.0:8000 2026-07-01 14:51:46 info(llama): Loaded weights [14.96GiB, 23.629s, 648.20MiB/s] ```
2026-07-02 14:01:18 +02:00
2024-09-17 11:11:18 +02:00
pjrt: fix misleading pjrt plugin load success log (#622) Move the PJRT plugin “Loaded” log so it is only emitted after the dynamic library `.so` is opened and the PJRT API is initialized successfully. Previously, `Api.loadFrom` used a `defer` for the success log. Failed plugin load attempts could still print `Loaded: ...`. This was misleading during platform auto-detection, for example when probing TPU before falling back to CUDA. Example: ``` 2026-07-01 13:12:17 ███████╗███╗ ███╗██╗ ██╗ ██╗ ██╗ ███╗ ███╗██████╗ 2026-07-01 13:12:17 ╚══███╔╝████╗ ████║██║ ██╔╝ ██║ ██║ ████╗ ████║██╔══██╗ 2026-07-01 13:12:17 ███╔╝ ██╔████╔██║██║ ██╔╝ ██║ ██║ ██╔████╔██║██║ ██║ 2026-07-01 13:12:17 ███╔╝ ██║╚██╔╝██║██║ .ai ██╔╝ ██║ ██║ ██║╚██╔╝██║██║ ██║ 2026-07-01 13:12:17 ███████╗██║ ╚═╝ ██║███████╗ ██╔╝ ███████╗███████╗██║ ╚═╝ ██║██████╔╝ 2026-07-01 13:12:17 ╚══════╝╚═╝ ╚═╝╚══════╝ ╚═╝ ╚══════╝╚══════╝╚═╝ ╚═╝╚═════╝ 2026-07-01 13:12:17 Copyright (c) 2026 ZML 2026-07-01 13:12:17 warning(zml/io/vfs/s3): S3 initialized without full credentials (AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY). Access will be unauthenticated. Using default endpoint: https://s3.amazonaws.com (set AWS_ENDPOINT_URL or AWS_ENDPOINT_URL_S3). Using default region: us-east-1 (set AWS_REGION or AWS_DEFAULT_REGION). 2026-07-01 13:12:17 info(pjrt): Loading: libpjrt_tpu.so... 2026-07-01 13:12:17 info(llmd): Loading tokenizer 2026-07-01 13:12:17 error(pjrt): Unable to dlopen plugin: /llmd/llmd.runfiles/zml++tpu_packages+libpjrt_tpu/sandbox/lib/libpjrt_tpu.so 2026-07-01 13:12:17 info(pjrt): Loaded: libpjrt_tpu.so #################### HERE #################### 2026-07-01 13:12:17 info(llmd): Resolving model repo 2026-07-01 13:12:17 info(llmd): Loading model config 2026-07-01 13:12:17 info(llmd): Loaded model config [67.405ms] 2026-07-01 13:12:18 info(pjrt): Loading: libpjrt_cuda.so... 2026-07-01 13:12:18 warning(zml/platforms/cuda): Detected NVIDIA GPU that requires CUDA compatibility libraries. 2026-07-01 13:12:18 info(zml/platforms/cuda): Loaded CUDA compatibility libraries. 2026-07-01 13:12:18 info(pjrt): Loaded: libpjrt_cuda.so 2026-07-01 13:12:18 info(llmd): Loaded tokenizer [334.168ms] 2026-07-01 13:12:20 info(llmd): Devices: 2026-07-01 13:12:20 info(llmd): - NVIDIA RTX PRO 6000 Blackwell Server Edition (cuda:0) 2026-07-01 13:12:20 info: Using 31711 pages for the KvCache 2026-07-01 13:12:20 info: sizes: { 256 } 2026-07-01 13:12:27 info: Compiled all models [7.336s] 2026-07-01 13:12:50 info: Listening on 0.0.0.0:8000 2026-07-01 13:12:50 info(llama): Loaded weights [14.96GiB, 22.778s, 672.41MiB/s] ``` Moreover make `fromDynLib` return errors for missing `GetPjrtApi` and failed `PJRT_Plugin_Initialize` instead of panicking or using `catch unreachable`. So that `Api.loadFrom()` can catch it. After the change, the output is: ``` 2026-07-01 14:50:50 ███████╗███╗ ███╗██╗ ██╗ ██╗ ██╗ ███╗ ███╗██████╗ 2026-07-01 14:50:50 ╚══███╔╝████╗ ████║██║ ██╔╝ ██║ ██║ ████╗ ████║██╔══██╗ 2026-07-01 14:50:50 ███╔╝ ██╔████╔██║██║ ██╔╝ ██║ ██║ ██╔████╔██║██║ ██║ 2026-07-01 14:50:50 ███╔╝ ██║╚██╔╝██║██║ .ai ██╔╝ ██║ ██║ ██║╚██╔╝██║██║ ██║ 2026-07-01 14:50:50 ███████╗██║ ╚═╝ ██║███████╗ ██╔╝ ███████╗███████╗██║ ╚═╝ ██║██████╔╝ 2026-07-01 14:50:50 ╚══════╝╚═╝ ╚═╝╚══════╝ ╚═╝ ╚══════╝╚══════╝╚═╝ ╚═╝╚═════╝ 2026-07-01 14:50:50 Copyright (c) 2026 ZML 2026-07-01 14:50:51 warning(zml/io/vfs/s3): S3 initialized without full credentials (AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY). Access will be unauthenticated. Using default endpoint: https://s3.amazonaws.com (set AWS_ENDPOINT_URL or AWS_ENDPOINT_URL_S3). Using default region: us-east-1 (set AWS_REGION or AWS_DEFAULT_REGION). 2026-07-01 14:50:51 info(llmd): Resolving model repo 2026-07-01 14:50:51 info(llmd): Loading model config 2026-07-01 14:50:51 info(llmd): Loaded model config [101.024ms] 2026-07-01 14:50:51 info(llmd): Loading tokenizer 2026-07-01 14:50:51 info(pjrt): Loading: libpjrt_tpu.so... 2026-07-01 14:50:51 error(pjrt): Unable to dlopen plugin: /llmd/llmd.runfiles/zml++tpu_packages+libpjrt_tpu/sandbox/lib/libpjrt_tpu.so 2026-07-01 14:50:52 info(llmd): Loaded tokenizer [349.499ms] 2026-07-01 14:50:53 info(pjrt): Loading: libpjrt_cuda.so... 2026-07-01 14:50:53 warning(zml/platforms/cuda): Detected NVIDIA GPU that requires CUDA compatibility libraries. 2026-07-01 14:50:53 info(zml/platforms/cuda): Loaded CUDA compatibility libraries. 2026-07-01 14:51:11 info(pjrt): Loaded: libpjrt_cuda.so 2026-07-01 14:51:14 info: Using 31711 pages for the KvCache 2026-07-01 14:51:14 info: sizes: { 256 } 2026-07-01 14:51:14 info(llmd): Devices: 2026-07-01 14:51:14 info(llmd): - NVIDIA RTX PRO 6000 Blackwell Server Edition (cuda:0) 2026-07-01 14:51:22 info: Compiled all models [8.395s] 2026-07-01 14:51:46 info: Listening on 0.0.0.0:8000 2026-07-01 14:51:46 info(llama): Loaded weights [14.96GiB, 23.629s, 648.20MiB/s] ```
2026-07-02 14:01:18 +02:00
2024-09-17 11:11:18 +02:00
2026-01-11 09:48:15 +01:00
2024-09-17 11:11:18 +02:00
2026-01-11 09:48:15 +01:00
2024-09-17 11:11:18 +02:00
zml/sharding: use logical meshes (#943) ## Motivation This is the big sharding rewrite we need to achieve complex sharding. Currently we always resolve a mesh logical axis to a mesh physical axis (or several of them). But on GPU we always have 1D topology, so we can't have 2D logical meshes. So this PR drops the logical -> physical axis bindings, and instead store a logical -> devices binding. This is more flexible, and has the nice side effect of simplifying `Buffer.from` On stablehlo side this changes how we generate the MLIR because we now use logical meshes, and `manualComputation` uses logical axes. (In shardy, GSPM is not affected AFAICT because it uses device ID directly) ## Buffer.from This also accelerates `Buffer.from` that I rewrote to be FAST. Version | Fixed cost | Per device -- | -- | -- origin/master | 109.6 ns | 35.39 ns Rewrite | 35.8 ns | 4.38 ns Improvement | 3.1× | 8.1× ## Creating mesh I'm adding a new syntax to create LogicalMesh, to allow the more complex meshes we want to do, and in particular, I now allow to force a specific number of dimensions. You can see it in action in `mnist.zig`. ``` const dp_mp = try platform.registerMesh("DP-MP", try .init( .{ .data, .model }, &.{ .low_bandwidth, .high_bandwidth }, &.{ 2, -1 }, )); info: topology used: Mesh(name=DP-MP, partitions=4, replicas=1) ┌ Mappings (logical coordinates from physical coordinates): ├── data (low_bandwidth) -> (bus / 2) └── model (high_bandwidth) -> (bus % 2) ┌ PhysicalMesh(platform=cpu shardable_axes={ .bus } num_devices=4) └── [bus] 0..4 (tree) ├── Device(id=0, coords={ 0 }) ├── Device(id=1, coords={ 1 }) ├── Device(id=2, coords={ 2 }) └── Device(id=3, coords={ 3 }) ┌ Logical coordinates: └── data=0 └── model=0,1 -> devices=0,1 └── data=1 └── model=0,1 -> devices=2,3 ``` The previous syntax is still available, and ZML will align logical axes to physical axes like we do before this PR. This part as unit tests, and I checked it work as I expect on CUDA but I would need to double check the logics still works on TPU/Neuron, because there is a few platform specific branches in that code. ## Test plan As part of the benchmark Codex generated a test suite with hard-coded offsets, allowing to check that Buffer.from didn't change behavior through the commits, checking we correctly compute each individual shard. TODO check that LLMD perf isn't degraded.
2026-10-16 15:27:30 +02:00
2026-01-11 09:48:15 +01:00
zml/sharding: use logical meshes (#943) ## Motivation This is the big sharding rewrite we need to achieve complex sharding. Currently we always resolve a mesh logical axis to a mesh physical axis (or several of them). But on GPU we always have 1D topology, so we can't have 2D logical meshes. So this PR drops the logical -> physical axis bindings, and instead store a logical -> devices binding. This is more flexible, and has the nice side effect of simplifying `Buffer.from` On stablehlo side this changes how we generate the MLIR because we now use logical meshes, and `manualComputation` uses logical axes. (In shardy, GSPM is not affected AFAICT because it uses device ID directly) ## Buffer.from This also accelerates `Buffer.from` that I rewrote to be FAST. Version | Fixed cost | Per device -- | -- | -- origin/master | 109.6 ns | 35.39 ns Rewrite | 35.8 ns | 4.38 ns Improvement | 3.1× | 8.1× ## Creating mesh I'm adding a new syntax to create LogicalMesh, to allow the more complex meshes we want to do, and in particular, I now allow to force a specific number of dimensions. You can see it in action in `mnist.zig`. ``` const dp_mp = try platform.registerMesh("DP-MP", try .init( .{ .data, .model }, &.{ .low_bandwidth, .high_bandwidth }, &.{ 2, -1 }, )); info: topology used: Mesh(name=DP-MP, partitions=4, replicas=1) ┌ Mappings (logical coordinates from physical coordinates): ├── data (low_bandwidth) -> (bus / 2) └── model (high_bandwidth) -> (bus % 2) ┌ PhysicalMesh(platform=cpu shardable_axes={ .bus } num_devices=4) └── [bus] 0..4 (tree) ├── Device(id=0, coords={ 0 }) ├── Device(id=1, coords={ 1 }) ├── Device(id=2, coords={ 2 }) └── Device(id=3, coords={ 3 }) ┌ Logical coordinates: └── data=0 └── model=0,1 -> devices=0,1 └── data=1 └── model=0,1 -> devices=2,3 ``` The previous syntax is still available, and ZML will align logical axes to physical axes like we do before this PR. This part as unit tests, and I checked it work as I expect on CUDA but I would need to double check the logics still works on TPU/Neuron, because there is a few platform specific branches in that code. ## Test plan As part of the benchmark Codex generated a test suite with hard-coded offsets, allowing to check that Buffer.from didn't change behavior through the commits, checking we correctly compute each individual shard. TODO check that LLMD perf isn't degraded.
2026-10-16 15:27:30 +02:00
2026-01-11 09:48:15 +01:00
2024-09-17 11:11:18 +02:00
zml/sharding: use logical meshes (#943) ## Motivation This is the big sharding rewrite we need to achieve complex sharding. Currently we always resolve a mesh logical axis to a mesh physical axis (or several of them). But on GPU we always have 1D topology, so we can't have 2D logical meshes. So this PR drops the logical -> physical axis bindings, and instead store a logical -> devices binding. This is more flexible, and has the nice side effect of simplifying `Buffer.from` On stablehlo side this changes how we generate the MLIR because we now use logical meshes, and `manualComputation` uses logical axes. (In shardy, GSPM is not affected AFAICT because it uses device ID directly) ## Buffer.from This also accelerates `Buffer.from` that I rewrote to be FAST. Version | Fixed cost | Per device -- | -- | -- origin/master | 109.6 ns | 35.39 ns Rewrite | 35.8 ns | 4.38 ns Improvement | 3.1× | 8.1× ## Creating mesh I'm adding a new syntax to create LogicalMesh, to allow the more complex meshes we want to do, and in particular, I now allow to force a specific number of dimensions. You can see it in action in `mnist.zig`. ``` const dp_mp = try platform.registerMesh("DP-MP", try .init( .{ .data, .model }, &.{ .low_bandwidth, .high_bandwidth }, &.{ 2, -1 }, )); info: topology used: Mesh(name=DP-MP, partitions=4, replicas=1) ┌ Mappings (logical coordinates from physical coordinates): ├── data (low_bandwidth) -> (bus / 2) └── model (high_bandwidth) -> (bus % 2) ┌ PhysicalMesh(platform=cpu shardable_axes={ .bus } num_devices=4) └── [bus] 0..4 (tree) ├── Device(id=0, coords={ 0 }) ├── Device(id=1, coords={ 1 }) ├── Device(id=2, coords={ 2 }) └── Device(id=3, coords={ 3 }) ┌ Logical coordinates: └── data=0 └── model=0,1 -> devices=0,1 └── data=1 └── model=0,1 -> devices=2,3 ``` The previous syntax is still available, and ZML will align logical axes to physical axes like we do before this PR. This part as unit tests, and I checked it work as I expect on CUDA but I would need to double check the logics still works on TPU/Neuron, because there is a few platform specific branches in that code. ## Test plan As part of the benchmark Codex generated a test suite with hard-coded offsets, allowing to check that Buffer.from didn't change behavior through the commits, checking we correctly compute each individual shard. TODO check that LLMD perf isn't degraded.
2026-10-16 15:27:30 +02:00
2024-09-17 11:11:18 +02:00
zml/sharding: use logical meshes (#943) ## Motivation This is the big sharding rewrite we need to achieve complex sharding. Currently we always resolve a mesh logical axis to a mesh physical axis (or several of them). But on GPU we always have 1D topology, so we can't have 2D logical meshes. So this PR drops the logical -> physical axis bindings, and instead store a logical -> devices binding. This is more flexible, and has the nice side effect of simplifying `Buffer.from` On stablehlo side this changes how we generate the MLIR because we now use logical meshes, and `manualComputation` uses logical axes. (In shardy, GSPM is not affected AFAICT because it uses device ID directly) ## Buffer.from This also accelerates `Buffer.from` that I rewrote to be FAST. Version | Fixed cost | Per device -- | -- | -- origin/master | 109.6 ns | 35.39 ns Rewrite | 35.8 ns | 4.38 ns Improvement | 3.1× | 8.1× ## Creating mesh I'm adding a new syntax to create LogicalMesh, to allow the more complex meshes we want to do, and in particular, I now allow to force a specific number of dimensions. You can see it in action in `mnist.zig`. ``` const dp_mp = try platform.registerMesh("DP-MP", try .init( .{ .data, .model }, &.{ .low_bandwidth, .high_bandwidth }, &.{ 2, -1 }, )); info: topology used: Mesh(name=DP-MP, partitions=4, replicas=1) ┌ Mappings (logical coordinates from physical coordinates): ├── data (low_bandwidth) -> (bus / 2) └── model (high_bandwidth) -> (bus % 2) ┌ PhysicalMesh(platform=cpu shardable_axes={ .bus } num_devices=4) └── [bus] 0..4 (tree) ├── Device(id=0, coords={ 0 }) ├── Device(id=1, coords={ 1 }) ├── Device(id=2, coords={ 2 }) └── Device(id=3, coords={ 3 }) ┌ Logical coordinates: └── data=0 └── model=0,1 -> devices=0,1 └── data=1 └── model=0,1 -> devices=2,3 ``` The previous syntax is still available, and ZML will align logical axes to physical axes like we do before this PR. This part as unit tests, and I checked it work as I expect on CUDA but I would need to double check the logics still works on TPU/Neuron, because there is a few platform specific branches in that code. ## Test plan As part of the benchmark Codex generated a test suite with hard-coded offsets, allowing to check that Buffer.from didn't change behavior through the commits, checking we correctly compute each individual shard. TODO check that LLMD perf isn't degraded.
2026-10-16 15:27:30 +02:00
2026-01-11 09:48:15 +01:00
2024-09-17 11:11:18 +02:00
2026-01-11 09:48:15 +01:00
zml/sharding: use logical meshes (#943) ## Motivation This is the big sharding rewrite we need to achieve complex sharding. Currently we always resolve a mesh logical axis to a mesh physical axis (or several of them). But on GPU we always have 1D topology, so we can't have 2D logical meshes. So this PR drops the logical -> physical axis bindings, and instead store a logical -> devices binding. This is more flexible, and has the nice side effect of simplifying `Buffer.from` On stablehlo side this changes how we generate the MLIR because we now use logical meshes, and `manualComputation` uses logical axes. (In shardy, GSPM is not affected AFAICT because it uses device ID directly) ## Buffer.from This also accelerates `Buffer.from` that I rewrote to be FAST. Version | Fixed cost | Per device -- | -- | -- origin/master | 109.6 ns | 35.39 ns Rewrite | 35.8 ns | 4.38 ns Improvement | 3.1× | 8.1× ## Creating mesh I'm adding a new syntax to create LogicalMesh, to allow the more complex meshes we want to do, and in particular, I now allow to force a specific number of dimensions. You can see it in action in `mnist.zig`. ``` const dp_mp = try platform.registerMesh("DP-MP", try .init( .{ .data, .model }, &.{ .low_bandwidth, .high_bandwidth }, &.{ 2, -1 }, )); info: topology used: Mesh(name=DP-MP, partitions=4, replicas=1) ┌ Mappings (logical coordinates from physical coordinates): ├── data (low_bandwidth) -> (bus / 2) └── model (high_bandwidth) -> (bus % 2) ┌ PhysicalMesh(platform=cpu shardable_axes={ .bus } num_devices=4) └── [bus] 0..4 (tree) ├── Device(id=0, coords={ 0 }) ├── Device(id=1, coords={ 1 }) ├── Device(id=2, coords={ 2 }) └── Device(id=3, coords={ 3 }) ┌ Logical coordinates: └── data=0 └── model=0,1 -> devices=0,1 └── data=1 └── model=0,1 -> devices=2,3 ``` The previous syntax is still available, and ZML will align logical axes to physical axes like we do before this PR. This part as unit tests, and I checked it work as I expect on CUDA but I would need to double check the logics still works on TPU/Neuron, because there is a few platform specific branches in that code. ## Test plan As part of the benchmark Codex generated a test suite with hard-coded offsets, allowing to check that Buffer.from didn't change behavior through the commits, checking we correctly compute each individual shard. TODO check that LLMD perf isn't degraded.
2026-10-16 15:27:30 +02:00
2024-09-17 11:11:18 +02:00
2026-01-11 10:17:39 +01:00
2026-01-11 10:17:39 +01:00
2026-01-11 10:17:39 +01:00
2026-01-11 10:31:05 +01:00
2026-01-11 10:17:39 +01:00
2024-09-17 11:11:18 +02:00
2026-01-11 10:17:39 +01:00
2026-01-11 10:17:39 +01:00
workspace: bump llvm / xla / stablehlo / pjrt plugins and restore disabled tests xla@4663f04 (#98) - workspace: new module versions added to Bazel registry: - deps/xla: https://github.com/openxla/xla/commit/4663f04d2653d686833f9c306bd0d899c3127358 (20241025.0-4663f04) This version introduces an upgraded overlay with Python repository initialization for the TSL extension. The changes were necessitated by this XLA commit: https://github.com/openxla/xla/commit/19b4ecbe98e2c2910d6ff6532cdb64636a71e6d5#diff-df5471accc0ccfdac45fb8397fb8e1488565baf112c72487912deda919680f8aR841 , which triggers a repository call. Python's hermetic version is provided via the .bazelrc file. - deps/stablehlo: https://github.com/openxla/stablehlo/commit/f7f8e4e35296deeff2e12e39421ac8d9599ba340 (20241021.0-1c0b606) - deps/llvm: https://github.com/llvm/llvm-project/commit/f7f8e4e35296deeff2e12e39421ac8d9599ba340 (20241022.0-6c4267f) This version includes an upgraded module declaration with rules_python dependency added, as LLVM prepares its overlay for Bazel v8. Relevant LLVM changes here: https://github.com/llvm/llvm-project/commit/3903cb4695012fb85a76c83b5616f2ffe6fa10f4 - deps/runtimes: use PJRT artifacts v3.0.0 (https://github.com/zml/pjrt-artifacts/releases/tag/v3.0.0) - workspace: align used versions of rules_zig in the Bazel dependency graph - pjrt: expose getPluginAttributes and getPluginAttribute private APIs and stablehloCurrentVersion public API - pjrt: updated customCallRegistry function with the new FFI provided by the PJRT API GPU extension Minimal support for this feature was added, as this is not the main focus of the current PR. See related changes in XLA here: https://github.com/openxla/xla/commit/ef85a7bcc308313492ebc50295a8a08b4e51b8f5 - dialects/mlir/stablehlo: expose getCurrentVersion API and fix broken stablehloVersionFromCompatibilityRequirement API - zml/pjrtx: serialize IR to the PJRT plugin's supported HLO version. If the plugin attribute is unavailable, it falls back to the StableHLO's compatibility requirement version. - zml/tensor: tests using batching dims for gather and scatter are now re-enabled
2024-11-25 15:53:25 +01:00
workspace: bump llvm / xla / stablehlo / pjrt plugins and restore disabled tests xla@4663f04 (#98) - workspace: new module versions added to Bazel registry: - deps/xla: https://github.com/openxla/xla/commit/4663f04d2653d686833f9c306bd0d899c3127358 (20241025.0-4663f04) This version introduces an upgraded overlay with Python repository initialization for the TSL extension. The changes were necessitated by this XLA commit: https://github.com/openxla/xla/commit/19b4ecbe98e2c2910d6ff6532cdb64636a71e6d5#diff-df5471accc0ccfdac45fb8397fb8e1488565baf112c72487912deda919680f8aR841 , which triggers a repository call. Python's hermetic version is provided via the .bazelrc file. - deps/stablehlo: https://github.com/openxla/stablehlo/commit/f7f8e4e35296deeff2e12e39421ac8d9599ba340 (20241021.0-1c0b606) - deps/llvm: https://github.com/llvm/llvm-project/commit/f7f8e4e35296deeff2e12e39421ac8d9599ba340 (20241022.0-6c4267f) This version includes an upgraded module declaration with rules_python dependency added, as LLVM prepares its overlay for Bazel v8. Relevant LLVM changes here: https://github.com/llvm/llvm-project/commit/3903cb4695012fb85a76c83b5616f2ffe6fa10f4 - deps/runtimes: use PJRT artifacts v3.0.0 (https://github.com/zml/pjrt-artifacts/releases/tag/v3.0.0) - workspace: align used versions of rules_zig in the Bazel dependency graph - pjrt: expose getPluginAttributes and getPluginAttribute private APIs and stablehloCurrentVersion public API - pjrt: updated customCallRegistry function with the new FFI provided by the PJRT API GPU extension Minimal support for this feature was added, as this is not the main focus of the current PR. See related changes in XLA here: https://github.com/openxla/xla/commit/ef85a7bcc308313492ebc50295a8a08b4e51b8f5 - dialects/mlir/stablehlo: expose getCurrentVersion API and fix broken stablehloVersionFromCompatibilityRequirement API - zml/pjrtx: serialize IR to the PJRT plugin's supported HLO version. If the plugin attribute is unavailable, it falls back to the StableHLO's compatibility requirement version. - zml/tensor: tests using batching dims for gather and scatter are now re-enabled
2024-11-25 15:53:25 +01:00
2026-01-15 18:20:15 +00:00
workspace: bump llvm / xla / stablehlo / pjrt plugins and restore disabled tests xla@4663f04 (#98) - workspace: new module versions added to Bazel registry: - deps/xla: https://github.com/openxla/xla/commit/4663f04d2653d686833f9c306bd0d899c3127358 (20241025.0-4663f04) This version introduces an upgraded overlay with Python repository initialization for the TSL extension. The changes were necessitated by this XLA commit: https://github.com/openxla/xla/commit/19b4ecbe98e2c2910d6ff6532cdb64636a71e6d5#diff-df5471accc0ccfdac45fb8397fb8e1488565baf112c72487912deda919680f8aR841 , which triggers a repository call. Python's hermetic version is provided via the .bazelrc file. - deps/stablehlo: https://github.com/openxla/stablehlo/commit/f7f8e4e35296deeff2e12e39421ac8d9599ba340 (20241021.0-1c0b606) - deps/llvm: https://github.com/llvm/llvm-project/commit/f7f8e4e35296deeff2e12e39421ac8d9599ba340 (20241022.0-6c4267f) This version includes an upgraded module declaration with rules_python dependency added, as LLVM prepares its overlay for Bazel v8. Relevant LLVM changes here: https://github.com/llvm/llvm-project/commit/3903cb4695012fb85a76c83b5616f2ffe6fa10f4 - deps/runtimes: use PJRT artifacts v3.0.0 (https://github.com/zml/pjrt-artifacts/releases/tag/v3.0.0) - workspace: align used versions of rules_zig in the Bazel dependency graph - pjrt: expose getPluginAttributes and getPluginAttribute private APIs and stablehloCurrentVersion public API - pjrt: updated customCallRegistry function with the new FFI provided by the PJRT API GPU extension Minimal support for this feature was added, as this is not the main focus of the current PR. See related changes in XLA here: https://github.com/openxla/xla/commit/ef85a7bcc308313492ebc50295a8a08b4e51b8f5 - dialects/mlir/stablehlo: expose getCurrentVersion API and fix broken stablehloVersionFromCompatibilityRequirement API - zml/pjrtx: serialize IR to the PJRT plugin's supported HLO version. If the plugin attribute is unavailable, it falls back to the StableHLO's compatibility requirement version. - zml/tensor: tests using batching dims for gather and scatter are now re-enabled
2024-11-25 15:53:25 +01:00
workspace: bump llvm / xla / stablehlo / pjrt plugins and restore disabled tests xla@4663f04 (#98) - workspace: new module versions added to Bazel registry: - deps/xla: https://github.com/openxla/xla/commit/4663f04d2653d686833f9c306bd0d899c3127358 (20241025.0-4663f04) This version introduces an upgraded overlay with Python repository initialization for the TSL extension. The changes were necessitated by this XLA commit: https://github.com/openxla/xla/commit/19b4ecbe98e2c2910d6ff6532cdb64636a71e6d5#diff-df5471accc0ccfdac45fb8397fb8e1488565baf112c72487912deda919680f8aR841 , which triggers a repository call. Python's hermetic version is provided via the .bazelrc file. - deps/stablehlo: https://github.com/openxla/stablehlo/commit/f7f8e4e35296deeff2e12e39421ac8d9599ba340 (20241021.0-1c0b606) - deps/llvm: https://github.com/llvm/llvm-project/commit/f7f8e4e35296deeff2e12e39421ac8d9599ba340 (20241022.0-6c4267f) This version includes an upgraded module declaration with rules_python dependency added, as LLVM prepares its overlay for Bazel v8. Relevant LLVM changes here: https://github.com/llvm/llvm-project/commit/3903cb4695012fb85a76c83b5616f2ffe6fa10f4 - deps/runtimes: use PJRT artifacts v3.0.0 (https://github.com/zml/pjrt-artifacts/releases/tag/v3.0.0) - workspace: align used versions of rules_zig in the Bazel dependency graph - pjrt: expose getPluginAttributes and getPluginAttribute private APIs and stablehloCurrentVersion public API - pjrt: updated customCallRegistry function with the new FFI provided by the PJRT API GPU extension Minimal support for this feature was added, as this is not the main focus of the current PR. See related changes in XLA here: https://github.com/openxla/xla/commit/ef85a7bcc308313492ebc50295a8a08b4e51b8f5 - dialects/mlir/stablehlo: expose getCurrentVersion API and fix broken stablehloVersionFromCompatibilityRequirement API - zml/pjrtx: serialize IR to the PJRT plugin's supported HLO version. If the plugin attribute is unavailable, it falls back to the StableHLO's compatibility requirement version. - zml/tensor: tests using batching dims for gather and scatter are now re-enabled
2024-11-25 15:53:25 +01:00
workspace: bump llvm / xla / stablehlo / pjrt plugins and restore disabled tests xla@4663f04 (#98) - workspace: new module versions added to Bazel registry: - deps/xla: https://github.com/openxla/xla/commit/4663f04d2653d686833f9c306bd0d899c3127358 (20241025.0-4663f04) This version introduces an upgraded overlay with Python repository initialization for the TSL extension. The changes were necessitated by this XLA commit: https://github.com/openxla/xla/commit/19b4ecbe98e2c2910d6ff6532cdb64636a71e6d5#diff-df5471accc0ccfdac45fb8397fb8e1488565baf112c72487912deda919680f8aR841 , which triggers a repository call. Python's hermetic version is provided via the .bazelrc file. - deps/stablehlo: https://github.com/openxla/stablehlo/commit/f7f8e4e35296deeff2e12e39421ac8d9599ba340 (20241021.0-1c0b606) - deps/llvm: https://github.com/llvm/llvm-project/commit/f7f8e4e35296deeff2e12e39421ac8d9599ba340 (20241022.0-6c4267f) This version includes an upgraded module declaration with rules_python dependency added, as LLVM prepares its overlay for Bazel v8. Relevant LLVM changes here: https://github.com/llvm/llvm-project/commit/3903cb4695012fb85a76c83b5616f2ffe6fa10f4 - deps/runtimes: use PJRT artifacts v3.0.0 (https://github.com/zml/pjrt-artifacts/releases/tag/v3.0.0) - workspace: align used versions of rules_zig in the Bazel dependency graph - pjrt: expose getPluginAttributes and getPluginAttribute private APIs and stablehloCurrentVersion public API - pjrt: updated customCallRegistry function with the new FFI provided by the PJRT API GPU extension Minimal support for this feature was added, as this is not the main focus of the current PR. See related changes in XLA here: https://github.com/openxla/xla/commit/ef85a7bcc308313492ebc50295a8a08b4e51b8f5 - dialects/mlir/stablehlo: expose getCurrentVersion API and fix broken stablehloVersionFromCompatibilityRequirement API - zml/pjrtx: serialize IR to the PJRT plugin's supported HLO version. If the plugin attribute is unavailable, it falls back to the StableHLO's compatibility requirement version. - zml/tensor: tests using batching dims for gather and scatter are now re-enabled
2024-11-25 15:53:25 +01:00
2024-09-17 11:11:18 +02:00
2024-09-17 11:11:18 +02:00
2024-09-17 11:11:18 +02:00
2024-09-17 11:11:18 +02:00
2024-09-17 11:11:18 +02:00
2024-09-17 11:11:18 +02:00
2024-09-17 11:11:18 +02:00
2026-01-11 09:48:15 +01:00
2025-12-15 18:55:43 +01:00
2026-01-11 09:48:15 +01:00
2026-01-11 10:17:39 +01:00
2025-12-15 18:55:43 +01:00
2026-01-11 09:48:15 +01:00
2025-12-18 16:07:17 +01:00
2024-09-17 11:11:18 +02:00
2026-01-11 10:17:39 +01:00
2024-09-17 11:11:18 +02:00
2024-09-17 11:11:18 +02:00
2024-09-17 11:11:18 +02:00
2024-09-17 11:11:18 +02:00
2024-09-17 11:11:18 +02:00
2026-01-11 09:48:15 +01:00
2024-09-17 11:11:18 +02:00
2026-01-11 09:48:15 +01:00
2024-09-17 11:11:18 +02:00
2026-01-11 09:48:15 +01:00
2024-09-17 11:11:18 +02:00
2024-09-17 11:11:18 +02:00
2024-09-17 11:11:18 +02:00
2024-09-17 11:11:18 +02:00
2024-09-17 11:11:18 +02:00
2024-09-17 11:11:18 +02:00
2026-01-11 10:17:39 +01:00
2024-09-17 11:11:18 +02:00
2026-01-11 10:17:39 +01:00
2024-09-17 11:11:18 +02:00
2026-01-11 09:56:47 +01:00
2024-09-17 11:11:18 +02:00
2024-09-17 11:11:18 +02:00
2024-09-17 11:11:18 +02:00
2024-09-17 11:11:18 +02:00
zml/buffer: add Buffer.HostAccessible (#899) `Buffer` placed in `.host_pinned` memory can be read by executables, but the host can't address it. `Buffer.HostAccessible` gives the host typed views over its allocations, to write inputs and read outputs in place between executions, without staging copies. Like `Buffer`, it is a value: executions read its `buffer` field or donate it through `&buffer`, and `await` then picks up the handles they replaced. - `init` takes a `Buffer.Donation`. On CPU, CUDA and ROCm, the devices holding a shard of an `.undonatable` buffer share one allocation: the first allocates it and the others read it through `createViewOfDeviceBuffer`, which now takes the memory the view belongs to; an external reference keeps the allocation alive. Elsewhere each device owns an allocation, and writes reach all of them. Each device owns its allocation of a `.donatable` buffer, so that no device reads another's output. - `View(T)` addresses a row-major range of the array: `slice`, `get`, `set`, `fill`, `fillIota`, `copyFrom`, `copyTo`, `copyFromView`, `indexOfScalar`, `indexOfAny`, and `blocks`/`reverseBlocks`, which visit runs contiguous both in memory and in global order. Reads use the first replica of a region, and writes reach every replica. `constSlice` and `edit` expose a range as one slice: in place when every shard holds the whole array densely, through a scratch slice otherwise. Addressing follows PJRT's layouts: row-major, strided, and tiled as XLA's LayoutUtil::LinearIndexForNestedTiling expands them, including TPU's `T(8,128)(2,1)` packing of bf16 and 8-bit arrays. The test reference is a direct port of that function, checked against both figures of https://openxla.org/xla/tiled_layout. Buffers whose shards all hold the whole array densely, the case on every platform but TPU, index elements directly. The buffer is built for llmd's decode loop, which touches it thousands of times per step: - Reading a host pointer from PJRT takes and releases a hold on the buffer and allocates an external reference, about 0.9 us. Pointers are read at creation, and by `await` only when execution replaced the handles; `view` panics before that `await` instead of writing to a released allocation. - The layout metadata and allocations take several KB. They are prepared once in a heap-allocated mapping, which views (24 B) and blocks borrow rather than copy. On Qwen3.8-27B in llmd (TP=2 on GB300, batch 256, 8k context), this takes host time between graph completion and the next Execute from 2.07 ms, when views copied the layout and donated buffers read their pointers in every view, to 0.27-0.30 ms per step (bpftrace probes). `initForTests` builds a buffer over host memory, for the tiled layouts and shardings that no available platform produces. `Buffer.from` and `Buffer.uninitialized` share the new `emptyShell` with `HostAccessible.init` to compute the PJRT shape, placement and layout.
2026-10-08 14:52:10 +02:00
2024-09-17 11:11:18 +02:00
2024-09-17 11:11:18 +02:00
2024-09-17 11:11:18 +02:00
2026-01-11 09:56:47 +01:00
2024-09-17 11:11:18 +02:00
2026-01-11 09:56:47 +01:00
zml/buffer: add Buffer.HostAccessible (#899) `Buffer` placed in `.host_pinned` memory can be read by executables, but the host can't address it. `Buffer.HostAccessible` gives the host typed views over its allocations, to write inputs and read outputs in place between executions, without staging copies. Like `Buffer`, it is a value: executions read its `buffer` field or donate it through `&buffer`, and `await` then picks up the handles they replaced. - `init` takes a `Buffer.Donation`. On CPU, CUDA and ROCm, the devices holding a shard of an `.undonatable` buffer share one allocation: the first allocates it and the others read it through `createViewOfDeviceBuffer`, which now takes the memory the view belongs to; an external reference keeps the allocation alive. Elsewhere each device owns an allocation, and writes reach all of them. Each device owns its allocation of a `.donatable` buffer, so that no device reads another's output. - `View(T)` addresses a row-major range of the array: `slice`, `get`, `set`, `fill`, `fillIota`, `copyFrom`, `copyTo`, `copyFromView`, `indexOfScalar`, `indexOfAny`, and `blocks`/`reverseBlocks`, which visit runs contiguous both in memory and in global order. Reads use the first replica of a region, and writes reach every replica. `constSlice` and `edit` expose a range as one slice: in place when every shard holds the whole array densely, through a scratch slice otherwise. Addressing follows PJRT's layouts: row-major, strided, and tiled as XLA's LayoutUtil::LinearIndexForNestedTiling expands them, including TPU's `T(8,128)(2,1)` packing of bf16 and 8-bit arrays. The test reference is a direct port of that function, checked against both figures of https://openxla.org/xla/tiled_layout. Buffers whose shards all hold the whole array densely, the case on every platform but TPU, index elements directly. The buffer is built for llmd's decode loop, which touches it thousands of times per step: - Reading a host pointer from PJRT takes and releases a hold on the buffer and allocates an external reference, about 0.9 us. Pointers are read at creation, and by `await` only when execution replaced the handles; `view` panics before that `await` instead of writing to a released allocation. - The layout metadata and allocations take several KB. They are prepared once in a heap-allocated mapping, which views (24 B) and blocks borrow rather than copy. On Qwen3.8-27B in llmd (TP=2 on GB300, batch 256, 8k context), this takes host time between graph completion and the next Execute from 2.07 ms, when views copied the layout and donated buffers read their pointers in every view, to 0.27-0.30 ms per step (bpftrace probes). `initForTests` builds a buffer over host memory, for the tiled layouts and shardings that no available platform produces. `Buffer.from` and `Buffer.uninitialized` share the new `emptyShell` with `HostAccessible.init` to compute the PJRT shape, placement and layout.
2026-10-08 14:52:10 +02:00
2024-09-17 11:11:18 +02:00
2026-01-07 12:08:31 +01:00
2026-01-11 10:17:39 +01:00
2026-01-07 12:08:31 +01:00
2026-01-11 10:17:39 +01:00
2026-01-11 10:17:39 +01:00
2026-01-11 10:17:39 +01:00
2026-01-11 09:56:47 +01:00
2026-01-11 09:56:47 +01:00
2026-01-11 09:56:47 +01:00
2024-09-17 11:11:18 +02:00
2024-09-17 11:11:18 +02:00
2024-09-17 11:11:18 +02:00
2026-01-11 10:22:12 +01:00
2026-01-11 10:22:12 +01:00
zml/sharding: use logical meshes (#943) ## Motivation This is the big sharding rewrite we need to achieve complex sharding. Currently we always resolve a mesh logical axis to a mesh physical axis (or several of them). But on GPU we always have 1D topology, so we can't have 2D logical meshes. So this PR drops the logical -> physical axis bindings, and instead store a logical -> devices binding. This is more flexible, and has the nice side effect of simplifying `Buffer.from` On stablehlo side this changes how we generate the MLIR because we now use logical meshes, and `manualComputation` uses logical axes. (In shardy, GSPM is not affected AFAICT because it uses device ID directly) ## Buffer.from This also accelerates `Buffer.from` that I rewrote to be FAST. Version | Fixed cost | Per device -- | -- | -- origin/master | 109.6 ns | 35.39 ns Rewrite | 35.8 ns | 4.38 ns Improvement | 3.1× | 8.1× ## Creating mesh I'm adding a new syntax to create LogicalMesh, to allow the more complex meshes we want to do, and in particular, I now allow to force a specific number of dimensions. You can see it in action in `mnist.zig`. ``` const dp_mp = try platform.registerMesh("DP-MP", try .init( .{ .data, .model }, &.{ .low_bandwidth, .high_bandwidth }, &.{ 2, -1 }, )); info: topology used: Mesh(name=DP-MP, partitions=4, replicas=1) ┌ Mappings (logical coordinates from physical coordinates): ├── data (low_bandwidth) -> (bus / 2) └── model (high_bandwidth) -> (bus % 2) ┌ PhysicalMesh(platform=cpu shardable_axes={ .bus } num_devices=4) └── [bus] 0..4 (tree) ├── Device(id=0, coords={ 0 }) ├── Device(id=1, coords={ 1 }) ├── Device(id=2, coords={ 2 }) └── Device(id=3, coords={ 3 }) ┌ Logical coordinates: └── data=0 └── model=0,1 -> devices=0,1 └── data=1 └── model=0,1 -> devices=2,3 ``` The previous syntax is still available, and ZML will align logical axes to physical axes like we do before this PR. This part as unit tests, and I checked it work as I expect on CUDA but I would need to double check the logics still works on TPU/Neuron, because there is a few platform specific branches in that code. ## Test plan As part of the benchmark Codex generated a test suite with hard-coded offsets, allowing to check that Buffer.from didn't change behavior through the commits, checking we correctly compute each individual shard. TODO check that LLMD perf isn't degraded.
2026-10-16 15:27:30 +02:00
2024-09-17 11:11:18 +02:00
2024-09-17 11:11:18 +02:00
2024-09-17 11:11:18 +02:00
2024-09-17 11:11:18 +02:00
2024-09-17 11:11:18 +02:00
2024-09-17 11:11:18 +02:00
2026-01-11 09:56:47 +01:00
2024-09-17 11:11:18 +02:00
2024-09-17 11:11:18 +02:00
2025-12-09 14:58:56 +01:00
2024-09-17 11:11:18 +02:00
2026-01-11 10:22:12 +01:00
2024-09-17 11:11:18 +02:00
2024-09-17 11:11:18 +02:00
2025-12-09 14:58:56 +01:00
2024-09-17 11:11:18 +02:00
2024-09-17 11:11:18 +02:00
2024-09-17 11:11:18 +02:00
2026-01-11 09:56:47 +01:00
2024-09-17 11:11:18 +02:00
2026-01-11 09:56:47 +01:00
2026-01-11 10:17:39 +01:00
2026-01-11 09:56:47 +01:00
2024-09-17 11:11:18 +02:00
2026-01-11 10:17:39 +01:00
2024-09-17 11:11:18 +02:00
2024-09-17 11:11:18 +02:00
2024-09-17 11:11:18 +02:00
2024-09-17 11:11:18 +02:00
2024-09-17 11:11:18 +02:00
2025-12-05 14:11:24 +01:00
2025-12-09 14:58:56 +01:00
2024-09-17 11:11:18 +02:00
2025-12-09 14:58:56 +01:00
2024-09-17 11:11:18 +02:00
2025-12-09 14:58:56 +01:00
2024-09-17 11:11:18 +02:00
2025-12-09 14:58:56 +01:00
2024-09-17 11:11:18 +02:00
2025-12-09 14:58:56 +01:00
2025-12-05 14:11:24 +01:00
2024-09-17 11:11:18 +02:00
2024-09-17 11:11:18 +02:00
2024-09-17 11:11:18 +02:00
2024-09-17 11:11:18 +02:00
2024-09-17 11:11:18 +02:00
2025-12-08 17:46:17 +01:00
2024-09-17 11:11:18 +02:00
2024-09-17 11:11:18 +02:00
2024-09-17 11:11:18 +02:00
2024-09-17 11:11:18 +02:00
2024-09-17 11:11:18 +02:00
2024-09-17 11:11:18 +02:00
2024-09-17 11:11:18 +02:00
2025-12-09 14:58:56 +01:00
2024-09-17 11:11:18 +02:00
2024-09-17 11:11:18 +02:00
2024-09-17 11:11:18 +02:00
2024-09-17 11:11:18 +02:00
2024-09-17 11:11:18 +02:00
2024-09-17 11:11:18 +02:00
2024-09-17 11:11:18 +02:00
2024-09-17 11:11:18 +02:00
2026-01-11 10:17:39 +01:00
2024-09-17 11:11:18 +02:00
2026-01-11 09:56:47 +01:00
2025-12-09 14:58:56 +01:00
2026-01-11 09:56:47 +01:00
2024-09-17 11:11:18 +02:00
zml/sharding: use logical meshes (#943) ## Motivation This is the big sharding rewrite we need to achieve complex sharding. Currently we always resolve a mesh logical axis to a mesh physical axis (or several of them). But on GPU we always have 1D topology, so we can't have 2D logical meshes. So this PR drops the logical -> physical axis bindings, and instead store a logical -> devices binding. This is more flexible, and has the nice side effect of simplifying `Buffer.from` On stablehlo side this changes how we generate the MLIR because we now use logical meshes, and `manualComputation` uses logical axes. (In shardy, GSPM is not affected AFAICT because it uses device ID directly) ## Buffer.from This also accelerates `Buffer.from` that I rewrote to be FAST. Version | Fixed cost | Per device -- | -- | -- origin/master | 109.6 ns | 35.39 ns Rewrite | 35.8 ns | 4.38 ns Improvement | 3.1× | 8.1× ## Creating mesh I'm adding a new syntax to create LogicalMesh, to allow the more complex meshes we want to do, and in particular, I now allow to force a specific number of dimensions. You can see it in action in `mnist.zig`. ``` const dp_mp = try platform.registerMesh("DP-MP", try .init( .{ .data, .model }, &.{ .low_bandwidth, .high_bandwidth }, &.{ 2, -1 }, )); info: topology used: Mesh(name=DP-MP, partitions=4, replicas=1) ┌ Mappings (logical coordinates from physical coordinates): ├── data (low_bandwidth) -> (bus / 2) └── model (high_bandwidth) -> (bus % 2) ┌ PhysicalMesh(platform=cpu shardable_axes={ .bus } num_devices=4) └── [bus] 0..4 (tree) ├── Device(id=0, coords={ 0 }) ├── Device(id=1, coords={ 1 }) ├── Device(id=2, coords={ 2 }) └── Device(id=3, coords={ 3 }) ┌ Logical coordinates: └── data=0 └── model=0,1 -> devices=0,1 └── data=1 └── model=0,1 -> devices=2,3 ``` The previous syntax is still available, and ZML will align logical axes to physical axes like we do before this PR. This part as unit tests, and I checked it work as I expect on CUDA but I would need to double check the logics still works on TPU/Neuron, because there is a few platform specific branches in that code. ## Test plan As part of the benchmark Codex generated a test suite with hard-coded offsets, allowing to check that Buffer.from didn't change behavior through the commits, checking we correctly compute each individual shard. TODO check that LLMD perf isn't degraded.
2026-10-16 15:27:30 +02:00
zml/sharding: use logical meshes (#943) ## Motivation This is the big sharding rewrite we need to achieve complex sharding. Currently we always resolve a mesh logical axis to a mesh physical axis (or several of them). But on GPU we always have 1D topology, so we can't have 2D logical meshes. So this PR drops the logical -> physical axis bindings, and instead store a logical -> devices binding. This is more flexible, and has the nice side effect of simplifying `Buffer.from` On stablehlo side this changes how we generate the MLIR because we now use logical meshes, and `manualComputation` uses logical axes. (In shardy, GSPM is not affected AFAICT because it uses device ID directly) ## Buffer.from This also accelerates `Buffer.from` that I rewrote to be FAST. Version | Fixed cost | Per device -- | -- | -- origin/master | 109.6 ns | 35.39 ns Rewrite | 35.8 ns | 4.38 ns Improvement | 3.1× | 8.1× ## Creating mesh I'm adding a new syntax to create LogicalMesh, to allow the more complex meshes we want to do, and in particular, I now allow to force a specific number of dimensions. You can see it in action in `mnist.zig`. ``` const dp_mp = try platform.registerMesh("DP-MP", try .init( .{ .data, .model }, &.{ .low_bandwidth, .high_bandwidth }, &.{ 2, -1 }, )); info: topology used: Mesh(name=DP-MP, partitions=4, replicas=1) ┌ Mappings (logical coordinates from physical coordinates): ├── data (low_bandwidth) -> (bus / 2) └── model (high_bandwidth) -> (bus % 2) ┌ PhysicalMesh(platform=cpu shardable_axes={ .bus } num_devices=4) └── [bus] 0..4 (tree) ├── Device(id=0, coords={ 0 }) ├── Device(id=1, coords={ 1 }) ├── Device(id=2, coords={ 2 }) └── Device(id=3, coords={ 3 }) ┌ Logical coordinates: └── data=0 └── model=0,1 -> devices=0,1 └── data=1 └── model=0,1 -> devices=2,3 ``` The previous syntax is still available, and ZML will align logical axes to physical axes like we do before this PR. This part as unit tests, and I checked it work as I expect on CUDA but I would need to double check the logics still works on TPU/Neuron, because there is a few platform specific branches in that code. ## Test plan As part of the benchmark Codex generated a test suite with hard-coded offsets, allowing to check that Buffer.from didn't change behavior through the commits, checking we correctly compute each individual shard. TODO check that LLMD perf isn't degraded.
2026-10-16 15:27:30 +02:00
2024-09-17 11:11:18 +02:00
2024-09-17 11:11:18 +02:00
2024-09-17 11:11:18 +02:00
2025-12-09 14:58:56 +01:00
2025-12-09 14:58:56 +01:00
2025-12-09 14:58:56 +01:00
2026-01-11 09:56:47 +01:00
2026-01-11 09:56:47 +01:00
2026-01-11 09:56:47 +01:00
2026-01-11 09:56:47 +01:00
2026-01-11 09:56:47 +01:00
2026-01-11 10:17:39 +01:00
2024-09-17 11:11:18 +02:00
2026-01-08 09:08:52 +00:00
2024-09-17 11:11:18 +02:00
2024-09-17 11:11:18 +02:00
2026-01-08 09:08:52 +00:00
2024-09-17 11:11:18 +02:00
2026-01-08 09:08:52 +00:00
2026-01-08 09:08:52 +00:00
2026-01-11 10:17:39 +01:00
2026-01-08 09:08:52 +00:00
2026-01-11 10:17:39 +01:00
2026-01-08 09:08:52 +00:00
2024-09-17 11:11:18 +02:00
2026-01-08 09:08:52 +00:00
2024-09-17 11:11:18 +02:00
2025-11-12 11:55:48 +01:00
2026-01-11 09:56:47 +01:00
2026-01-16 10:59:12 +01:00
2026-01-11 09:56:47 +01:00
2026-01-15 18:20:15 +00:00
2025-11-12 11:55:48 +01:00
2026-01-11 09:56:47 +01:00
2025-11-12 11:55:48 +01:00
2026-01-11 09:56:47 +01:00
2025-11-12 11:55:48 +01:00
2026-01-11 09:56:47 +01:00
2026-01-11 09:56:47 +01:00