0
Fork 0
mirror of https://github.com/zml/zml.git synced 2026-11-16 19:11:09 +00:00
Commit graph

81 commits

Author SHA1 Message Date
Guillaume Wenzek
a7a4fcd3f8
zml/sharding: use logical meshes (#943)
## Motivation

This is the big sharding rewrite we need to achieve complex sharding.

Currently we always resolve a mesh logical axis to a mesh physical axis
(or several of them).
But on GPU we always have 1D topology, so we can't have 2D logical
meshes.
So this PR drops the logical -> physical axis bindings, and instead
store a logical -> devices binding.
This is more flexible, and has the nice side effect of simplifying
`Buffer.from`

On stablehlo side this changes how we generate the MLIR because we now
use logical meshes, and `manualComputation` uses logical axes.
(In shardy, GSPM is not affected AFAICT because it uses device ID
directly)

## Buffer.from

This also accelerates `Buffer.from` that I rewrote to be FAST.
Version | Fixed cost | Per device
-- | -- | --
origin/master | 109.6 ns | 35.39 ns
Rewrite | 35.8 ns | 4.38 ns
Improvement | 3.1× | 8.1×

## Creating mesh

I'm adding a new syntax to create LogicalMesh, to allow the more complex
meshes we want to do, and in particular, I now allow to force a specific
number of dimensions.
You can see it in action in `mnist.zig`. 

```
const dp_mp = try platform.registerMesh("DP-MP", try .init(
        .{ .data, .model },
        &.{ .low_bandwidth, .high_bandwidth },
        &.{ 2, -1 },
    ));
 
info: topology used: Mesh(name=DP-MP, partitions=4, replicas=1)
┌ Mappings (logical coordinates from physical coordinates):
├── data (low_bandwidth) -> (bus / 2)
└── model (high_bandwidth) -> (bus % 2)

┌ PhysicalMesh(platform=cpu shardable_axes={ .bus } num_devices=4)
└── [bus] 0..4 (tree)
    ├── Device(id=0, coords={ 0 })
    ├── Device(id=1, coords={ 1 })
    ├── Device(id=2, coords={ 2 })
    └── Device(id=3, coords={ 3 })

┌ Logical coordinates:
└── data=0
    └── model=0,1 -> devices=0,1
└── data=1
    └── model=0,1 -> devices=2,3
```

The previous syntax is still available, and ZML will align logical axes
to physical axes like we do before this PR.
This part as unit tests, and I checked it work as I expect on CUDA but I
would need to double check the logics still works on TPU/Neuron, because
there is a few platform specific branches in that code.

## Test plan

As part of the benchmark Codex generated a test suite with hard-coded
offsets, allowing to check that Buffer.from didn't change behavior
through the commits, checking we correctly compute each individual
shard.

TODO check that LLMD perf isn't degraded.
2026-10-16 15:27:30 +02:00
Guillaume Wenzek
00a0af4a55
zig 0.17 (second attempt) (#923)
Bump for zig 0.17

Deps:
- uucode bumped to
1fb73433bb
- zigimg bumped to
734fa919fb
- libvaxis bumped to
14b7684272
- zio bumped to:
417757935e
- rules_zig upgraded to [version
0.17](https://github.com/hermeticbuild/rules_zig/releases/tag/v0.17.0)
- zls is using a commit from master, but they don't have yet a tagged
release so Codex did a bit of packaging. I confirmed it correctly
autocompletes on my machine
2026-10-13 17:11:38 +02:00
Benjamin Rabier
3972d4a011
zml/buffer: add Buffer.HostAccessible (#899)
`Buffer` placed in `.host_pinned` memory can be read by executables,
but the host can't address it. `Buffer.HostAccessible` gives the host
typed views over its allocations, to write inputs and read outputs in
place between executions, without staging copies. Like `Buffer`, it is
a value: executions read its `buffer` field or donate it through
`&buffer`, and `await` then picks up the handles they replaced.

- `init` takes a `Buffer.Donation`. On CPU, CUDA and ROCm, the devices
  holding a shard of an `.undonatable` buffer share one allocation: the
  first allocates it and the others read it through
  `createViewOfDeviceBuffer`, which now takes the memory the view
  belongs to; an external reference keeps the allocation alive.
  Elsewhere each device owns an allocation, and writes reach all of
  them. Each device owns its allocation of a `.donatable` buffer, so
  that no device reads another's output.
- `View(T)` addresses a row-major range of the array: `slice`, `get`,
  `set`, `fill`, `fillIota`, `copyFrom`, `copyTo`, `copyFromView`,
  `indexOfScalar`, `indexOfAny`, and `blocks`/`reverseBlocks`, which
  visit runs contiguous both in memory and in global order. Reads use
  the first replica of a region, and writes reach every replica.
  `constSlice` and `edit` expose a range as one slice: in place when
  every shard holds the whole array densely, through a scratch slice
  otherwise.

Addressing follows PJRT's layouts: row-major, strided, and tiled as
XLA's LayoutUtil::LinearIndexForNestedTiling expands them, including
TPU's `T(8,128)(2,1)` packing of bf16 and 8-bit arrays. The test
reference is a direct port of that function, checked against both
figures of https://openxla.org/xla/tiled_layout. Buffers whose shards
all hold the whole array densely, the case on every platform but TPU,
index elements directly.

The buffer is built for llmd's decode loop, which touches it thousands
of times per step:
- Reading a host pointer from PJRT takes and releases a hold on the
  buffer and allocates an external reference, about 0.9 us. Pointers
  are read at creation, and by `await` only when execution replaced the
  handles; `view` panics before that `await` instead of writing to a
  released allocation.
- The layout metadata and allocations take several KB. They are
  prepared once in a heap-allocated mapping, which views (24 B) and
  blocks borrow rather than copy.
On Qwen3.8-27B in llmd (TP=2 on GB300, batch 256, 8k context), this
takes host time between graph completion and the next Execute from
2.07 ms, when views copied the layout and donated buffers read their
pointers in every view, to 0.27-0.30 ms per step (bpftrace probes).

`initForTests` builds a buffer over host memory, for the tiled layouts
and shardings that no available platform produces. `Buffer.from` and
`Buffer.uninitialized` share the new `emptyShell` with
`HostAccessible.init` to compute the PJRT shape, placement and layout.
2026-10-08 14:52:10 +02:00
Hugo Mano
b148f0b919
workspace: bump XLA + LLVM + IRs (#903) 2026-10-07 16:33:44 +02:00
Guillaume Wenzek
303ff3e6a4
Revert "zig 0.17 (#693)" (#909)
This reverts commit 5a18757368.

Our internal repos has a few zig patches, I need to also upgrade them
before we ugprade ZML
2026-10-07 11:34:51 +02:00
Guillaume Wenzek
5a18757368
zig 0.17 (#693)
Bump for zig 0.17

Deps:
- uucode bumped to
1fb73433bb
- zio bumped to:
417757935e
- rules_zig upgraded to [version
0.17](https://github.com/hermeticbuild/rules_zig/releases/tag/v0.17.0)

There is one Codex generated patch:
- zigimg: minor fixes to allow zig 0.17 to compile it, until they
officially support
2026-10-07 11:08:06 +02:00
Benjamin Rabier
8c2e4c6dc9
pjrt,zml/platform: expose devices' NUMA node (#891)
Allows placing all the threads/memory on the right numa nodes. It's
faster for a device to access memory on the same numa node and overall
the it's better to have all threads on the same numa node if all devices
are on the same one.
2026-10-05 11:15:57 +02:00
Corentin Godeau
397aa43e0b
pjrt: properly interpret error when an event is ready (#844)
A ready event can still contain an error that we need to report.
2026-09-21 19:42:46 +02:00
Corentin Godeau
78a89679fa
zml/buffer: introduce simple createView (#817)
`Buffer.createView` can be used to create a view from a host pinned
buffer and use it as if it was one the device. This avoids XLA
limitations of doing operations directly on host pinned buffers.
2026-09-16 17:57:41 +02:00
Corentin Godeau
7adc862cbd
pjrt: rename ambiguous field of CreateViewOfDeviceBufferArgs (#814) 2026-09-16 17:57:40 +02:00
Benjamin Rabier
54bf33a02f
zml/Compiler: Provide accurate device memory size to XLA (#764)
By default, XLA assumes 80% of the memory to be available during
compilation. It wasn't an issue as long as we had one executable per
layer, because most of the weights weren't parts of the inputs, but for
a full forward executable we exceed that 80% threshold in llmd with the
cache being sized to use almost everything up to the BFC limit. So XLA
ends up generating an executable the least possible temporary buffers
even though it has more than that.
2026-09-03 17:17:09 +02:00
Kevin Barre
49a12235bb
zml/pjrt: propagate ready-event failures and validate opaque pointers (#703)
Fix two PJRT error-handling cases that could allow execution to continue
after a backend failure.

- Use the same error conversion and destruction path for ready-event and
callback errors.
- Reject a successful `PJRT_Buffer_OpaqueDeviceMemoryDataPointer` result
when `device_memory_ptr` is null.
2026-08-17 15:16:33 +02:00
Guillaume Wenzek
1c65836a44
pjrt: better error message on dlopen failure (#657)
I hit an issue with the plugin refusing to load on macos.
There was no information.
After adding this I saw that the the file was apparently not existing in
the bazel sandbox,
so I could reduce the bug to a Bazel issue.
2026-07-16 17:44:50 +02:00
Olivier Boulant
28e5d9ed7f
pjrt: fix misleading pjrt plugin load success log (#622)
Move the PJRT plugin “Loaded” log so it is only emitted after the
dynamic library `.so` is opened and the PJRT API is initialized
successfully.

Previously, `Api.loadFrom` used a `defer` for the success log. Failed
plugin load attempts could still print `Loaded: ...`. This was
misleading during platform auto-detection, for example when probing TPU
before falling back to CUDA. Example:

```
2026-07-01 13:12:17   ███████╗███╗   ███╗██╗             ██╗    ██╗     ██╗     ███╗   ███╗██████╗
2026-07-01 13:12:17   ╚══███╔╝████╗ ████║██║            ██╔╝    ██║     ██║     ████╗ ████║██╔══██╗
2026-07-01 13:12:17     ███╔╝ ██╔████╔██║██║           ██╔╝     ██║     ██║     ██╔████╔██║██║  ██║
2026-07-01 13:12:17    ███╔╝  ██║╚██╔╝██║██║  .ai     ██╔╝      ██║     ██║     ██║╚██╔╝██║██║  ██║
2026-07-01 13:12:17   ███████╗██║ ╚═╝ ██║███████╗    ██╔╝       ███████╗███████╗██║ ╚═╝ ██║██████╔╝
2026-07-01 13:12:17   ╚══════╝╚═╝     ╚═╝╚══════╝    ╚═╝        ╚══════╝╚══════╝╚═╝     ╚═╝╚═════╝
2026-07-01 13:12:17                                                          Copyright (c) 2026 ZML
2026-07-01 13:12:17 warning(zml/io/vfs/s3): S3 initialized without full credentials (AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY). Access will be unauthenticated. Using default endpoint: https://s3.amazonaws.com (set AWS_ENDPOINT_URL or AWS_ENDPOINT_URL_S3). Using default region: us-east-1 (set AWS_REGION or AWS_DEFAULT_REGION).
2026-07-01 13:12:17 info(pjrt): Loading: libpjrt_tpu.so...
2026-07-01 13:12:17 info(llmd): Loading tokenizer
2026-07-01 13:12:17 error(pjrt): Unable to dlopen plugin: /llmd/llmd.runfiles/zml++tpu_packages+libpjrt_tpu/sandbox/lib/libpjrt_tpu.so
2026-07-01 13:12:17 info(pjrt): Loaded: libpjrt_tpu.so           #################### HERE ####################
2026-07-01 13:12:17 info(llmd): Resolving model repo
2026-07-01 13:12:17 info(llmd): Loading model config
2026-07-01 13:12:17 info(llmd): Loaded model config [67.405ms]
2026-07-01 13:12:18 info(pjrt): Loading: libpjrt_cuda.so...
2026-07-01 13:12:18 warning(zml/platforms/cuda): Detected NVIDIA GPU that requires CUDA compatibility libraries.
2026-07-01 13:12:18 info(zml/platforms/cuda): Loaded CUDA compatibility libraries.
2026-07-01 13:12:18 info(pjrt): Loaded: libpjrt_cuda.so
2026-07-01 13:12:18 info(llmd): Loaded tokenizer [334.168ms]
2026-07-01 13:12:20 info(llmd): Devices:
2026-07-01 13:12:20 info(llmd):         - NVIDIA RTX PRO 6000 Blackwell Server Edition (cuda:0)
2026-07-01 13:12:20 info: Using 31711 pages for the KvCache
2026-07-01 13:12:20 info: sizes: { 256 }
2026-07-01 13:12:27 info: Compiled all models [7.336s]
2026-07-01 13:12:50 info: Listening on 0.0.0.0:8000
2026-07-01 13:12:50 info(llama): Loaded weights [14.96GiB, 22.778s, 672.41MiB/s]
```

Moreover make `fromDynLib` return errors for missing `GetPjrtApi` and
failed `PJRT_Plugin_Initialize` instead of panicking or using `catch
unreachable`. So that `Api.loadFrom()` can catch it.

After the change, the output is:
```
2026-07-01 14:50:50   ███████╗███╗   ███╗██╗             ██╗    ██╗     ██╗     ███╗   ███╗██████╗
2026-07-01 14:50:50   ╚══███╔╝████╗ ████║██║            ██╔╝    ██║     ██║     ████╗ ████║██╔══██╗
2026-07-01 14:50:50     ███╔╝ ██╔████╔██║██║           ██╔╝     ██║     ██║     ██╔████╔██║██║  ██║
2026-07-01 14:50:50    ███╔╝  ██║╚██╔╝██║██║  .ai     ██╔╝      ██║     ██║     ██║╚██╔╝██║██║  ██║
2026-07-01 14:50:50   ███████╗██║ ╚═╝ ██║███████╗    ██╔╝       ███████╗███████╗██║ ╚═╝ ██║██████╔╝
2026-07-01 14:50:50   ╚══════╝╚═╝     ╚═╝╚══════╝    ╚═╝        ╚══════╝╚══════╝╚═╝     ╚═╝╚═════╝
2026-07-01 14:50:50                                                          Copyright (c) 2026 ZML
2026-07-01 14:50:51 warning(zml/io/vfs/s3): S3 initialized without full credentials (AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY). Access will be unauthenticated. Using default endpoint: https://s3.amazonaws.com (set AWS_ENDPOINT_URL or AWS_ENDPOINT_URL_S3). Using default region: us-east-1 (set AWS_REGION or AWS_DEFAULT_REGION).
2026-07-01 14:50:51 info(llmd): Resolving model repo
2026-07-01 14:50:51 info(llmd): Loading model config
2026-07-01 14:50:51 info(llmd): Loaded model config [101.024ms]
2026-07-01 14:50:51 info(llmd): Loading tokenizer
2026-07-01 14:50:51 info(pjrt): Loading: libpjrt_tpu.so...
2026-07-01 14:50:51 error(pjrt): Unable to dlopen plugin: /llmd/llmd.runfiles/zml++tpu_packages+libpjrt_tpu/sandbox/lib/libpjrt_tpu.so
2026-07-01 14:50:52 info(llmd): Loaded tokenizer [349.499ms]
2026-07-01 14:50:53 info(pjrt): Loading: libpjrt_cuda.so...
2026-07-01 14:50:53 warning(zml/platforms/cuda): Detected NVIDIA GPU that requires CUDA compatibility libraries.
2026-07-01 14:50:53 info(zml/platforms/cuda): Loaded CUDA compatibility libraries.
2026-07-01 14:51:11 info(pjrt): Loaded: libpjrt_cuda.so
2026-07-01 14:51:14 info: Using 31711 pages for the KvCache
2026-07-01 14:51:14 info: sizes: { 256 }
2026-07-01 14:51:14 info(llmd): Devices:
2026-07-01 14:51:14 info(llmd):         - NVIDIA RTX PRO 6000 Blackwell Server Edition (cuda:0)
2026-07-01 14:51:22 info: Compiled all models [8.395s]
2026-07-01 14:51:46 info: Listening on 0.0.0.0:8000
2026-07-01 14:51:46 info(llama): Loaded weights [14.96GiB, 23.629s, 648.20MiB/s]
```
2026-07-02 14:01:18 +02:00
Corentin Kerisit
f9ec63fd4e
workspace: Update to XLA @ 5e83525729fd21c4ad8d55073a3d047360d13f9c (#599)
Updates to latest XLA which added support for bazel 8.x workspace and
fixed a bunch of things that made bazel 9 impossible.

3 problematic changes with this bump:
- PJRT headers were again incompatible with translate-c so a patch is
made and PR has been done in XLA
- PJRT introduced a proper type fo PJRT_Error so I used the InnerMixin
pattern for those.
- //tsl/platform/default package force evaluates Label("@cuda_nvcc")
even for CPU builds and we don't have this repo. I commented for now
until I find a solution on XLA side.

Bumped a few deps necessary for this and additional llvm deps that are
now evaluated as part of the llvm-project graph on XLA side (llvm_zlib
and llvm_zstd).

Next will be to investigate bazel 9 for real.
2026-06-25 14:17:46 +02:00
Hugo Mano
abde0a04dd
zml: sharding (#407)
Co-authored-by: Benjamin Rabier <benjamin@zml.ai>
2026-03-10 11:17:57 +01:00
Steeve Morin
dcbf5f2e27 workspace: update to 0.16.0-dev.2611+f996d2866, add remote exec
also:
- use toolchains_llvm_bootstrapped for macOS builds
2026-02-16 12:05:01 +01:00
Steeve Morin
639850347e pjrt: use dlopen on macos too
to keep symbols private
2026-02-16 12:05:01 +01:00
Steeve Morin
24354d34b3 pjrt: add from DynLib 2026-02-16 12:05:01 +01:00
Corentin Godeau
2218ec0f3f
V2: Tensor.print() (#380)
Added `Tensor.print()` back
2026-02-10 10:11:48 +01:00
Corentin Godeau
1fd90f1e45
Various improvements (#379)
This pull request includes several improvements and bug fixes across the
codebase, focusing on memory handling, correctness in tensor testing,
and API consistency. The most significant changes are grouped below:

### Tensor Testing and Memory Management Improvements

* Fixed a bug in `zml/testing.zig`'s `testlayer` by updating the
callback signature to take a mutable context and increment the index
correctly, ensuring tensors are correctly initialized.
* Improved memory management for expected output slices in `testlayer`
by switching to `zml.Slice.alloc` for allocation, using `readSliceAll`,
and freeing only the data instead of the whole slice, removing the
potential alignment problems.
* Enhanced test logging to include the tolerance value in success
messages for better diagnostics.

### API and Code Quality Enhancements

* Increased the evaluation branch quota in `MapRestrict` to support more
complex compile-time type mapping, preventing quota exceeded errors.
* Updated the `fromCStruct` function in `pjrt/pjrt.zig` to accept a
`meta.Struct`.

### StableHLO Output Operand Alias Handling

* Improved handling of output operand aliases in
`mlir/dialects/stablehlo/stablehlo.zig::custom_call` by passing
`output_tuple_indices` for multi-result functions, increasing
correctness in custom call attribute construction.
2026-02-09 12:01:22 +01:00
Steeve Morin
0f460fb636 zml: expose Memory, pass Platform as *const, add zml.io.load 2026-02-03 10:59:39 +00:00
Steeve Morin
24137cdc11 pjrt: sugarize Event.onReady
Dont't expose callconv stuff, just pass a comptimed function
2026-01-27 23:20:17 +00:00
Corentin Godeau
7bbce6c3d0
Attention Backend on ZML V2 (#373) 2026-01-16 10:59:12 +01:00
Corentin Godeau
2ea598a90f pjrt: missing fixes 2026-01-15 18:24:34 +00:00
Hugo Mano
25a5ee10f6 rework zml.io.* 2026-01-14 18:04:55 +01:00
Steeve Morin
b03b6299fc pjrt: remove get prefixes when not matching with PJRT 2026-01-11 22:23:25 +01:00
Steeve Morin
b0f35d4815 pjrt: remove dep to stdx 2026-01-11 22:13:57 +01:00
Steeve Morin
431dcd7a1c pjrt: guard against out of bounds access
when calling into an API, make sure that the method is inside the
struct_size, else we would read into undefined memory
2026-01-11 16:07:33 +01:00
Steeve Morin
a05fa3299e pjrt: rework extensions support through tagged union 2026-01-11 16:07:33 +01:00
Steeve Morin
c1c97d8dec pjrt: more cosmetics 2026-01-11 16:07:33 +01:00
Steeve Morin
89aa189d8d pjrt: add addressableMemory from pjrtx
it should be part of the binding
2026-01-11 16:07:33 +01:00
Steeve Morin
470359f39a pjrt: memorystats structs in structs 2026-01-11 16:07:33 +01:00
Steeve Morin
9eae5baaf4 pjrt: meta 2026-01-11 16:07:33 +01:00
Steeve Morin
55488f676d PJRT: meta 2026-01-11 16:07:33 +01:00
Steeve Morin
132c7cf390 pjrt:WIP 2026-01-11 16:07:33 +01:00
Steeve Morin
75903732c1 pjrt: do not deinit Event after await
PJRT_Buffer_ReadyEvent semantics say it allocates a new object
everytime, of which the caller is the owner.
2026-01-11 16:07:33 +01:00
Steeve Morin
ab62c536e2 runtimes/neuron: wip 2026-01-11 16:07:33 +01:00
Steeve Morin
d42e4f99ad pjrt: add DeviceDescription.attributes
and attribute helper for getting only one
2026-01-08 09:13:24 +00:00
Steeve Morin
1f73e85005 pjrt: rework NamedValue
it's simpler
2026-01-08 09:13:24 +00:00
Steeve Morin
89afb43e64 pjrt,zml: trim pjrtx
with the aim to remove it
2026-01-02 22:14:00 +00:00
Corentin Godeau
c0522ca6aa Events can be null when executing a LoadedExecutable 2025-12-22 15:18:16 +01:00
Corentin Godeau
3581770d26 Fix tests 2025-12-19 11:09:34 +01:00
Corentin Godeau
f680e17178 First clean version of Mnist 2025-12-15 18:55:43 +01:00
Corentin Godeau
0916e7d7de Various PJRT fixes in bindings 2025-12-09 14:58:56 +01:00
Corentin Godeau
86b5bb84df Various small fixes 2025-12-08 17:46:17 +01:00
Corentin Godeau
9cd56326e3 Basic support of zml.Buffer 2025-12-08 17:45:09 +01:00
Corentin Godeau
8e0e987968 Made it working on Zig 0.16 2025-12-05 16:42:28 +01:00
Hugo Mano
e5bc971c75
xla: bump to 9a77a88 (#357) 2025-11-12 11:55:48 +01:00
Guillaume Wenzek
21cd3ed532
mlir,pjrt,zml: expose all missing dtype (#328)
Adding the following missing data types:

- u2
- i1
- i2
- f4e2m1fn
- f8e3m4
- f8e4m3
- f8e8m0fnu

(ground truth is PJRT)

I've also realized the code in from/to Float32 wasn't correct and was
truncating to zero to often.
Hence the rewrite and bug fix.
2025-09-22 09:11:05 +02:00