0
Fork 0
mirror of https://github.com/zml/zml.git synced 2026-11-16 19:11:09 +00:00
Commit graph

262 commits

Author SHA1 Message Date
Guillaume Wenzek
3bec541823 flat Bufferized struct 2025-10-07 11:03:03 +02:00
Guillaume Wenzek
323be7bfb8 describeLayout 2025-10-06 16:28:22 +02:00
Guillaume Wenzek
b03b8f93aa pinToCore 2025-10-06 11:32:44 +02:00
Guillaume Wenzek
f5c0bf8d09 add documentation for command_buffer_compatible 2025-10-03 14:26:56 +02:00
Guillaume Wenzek
0ca0f44aa8 add patches to http_deb_archive 2025-09-29 14:46:53 +02:00
Guillaume Wenzek
25bd4d08ee
zml: cleanup some todos (#331) 2025-09-25 14:05:26 +02:00
Guillaume Wenzek
e9b42f0ca8
examples: gpt oss (#329)
This implement support for OpenAI "gpt-oss" models. 

It's mostly a take over of #292 branch, but MoE code is kept inside
examples folder cause it's still a bit warm.
In particular the routing logic can break apart when there are undefined
memory in the inputs, where it wasn't an issue for normal MLP.

I only tested with the 20B version.
2025-09-25 14:03:37 +02:00
Guillaume Wenzek
0312d6f678
zml: gather (#330)
Revamp the `gather` api to match `scatter`

(prerequesite of #329 )

## Gather new api

Now the syntax is to give a named list of indices. Each name must
correspond to one axis in the gathered tensor.
This match `scatter`, and avoid the user to have to concatenate indices
in the same order than the input axes,
which is error prone cause any screw up will compile but index at the
wrong offsets.

For 1 axis it doesn't change much: 

```
// before
self.weight.withTags(.{ .voc, .d }).gather(.{.voc}, .{ idx }, .{});
// after
self.weight.withTags(.{ .voc, .d }).gather(.{ .voc = idx }, .{});
```

But for two it helps a lot:

```
// before
self.weight.withTags(.{ .voc, .d }).gather(.{.voc, .d}, .{ .concatenate(&.{idx_voc, idx_d}, .last)}, .{});
// after
self.weight.withTags(.{ .voc, .d }).gather(.{ .voc = idx_voc, .d = idx_d }, .{});
```

Also adds a `gather_` that offers the same apis but with axis position
instead of names.
This one is more suitable inside library functions when dealing with
unknown names, making it complicated to create the "indices" struct.
I think we could generalize this pattern of offering "non-tagged" api
with an underscore suffix, simplifying the job of LSP.

## topk new API

I realized that most of the code calling topk was renaming the output
axis.
The idea is that logprob.topk(.voc) doesn't return you a tensor with voc
entry but only top_voc entries.
For the values it doesn't make much a difference but for the indices
it's embarrassing and it break usages of gather/scatter.

Now the user of `topK` can either use the unnamed api and pass a number,
of if using the name api must give a new name for the indexed axis.

## Yarn

Support Yarn rope embeddings. It's equivalent to the "llama3" embedding
except they do the interpolation in time space instead of freq space.
2025-09-25 12:29:46 +02:00
Guillaume Wenzek
21cd3ed532
mlir,pjrt,zml: expose all missing dtype (#328)
Adding the following missing data types:

- u2
- i1
- i2
- f4e2m1fn
- f8e3m4
- f8e4m3
- f8e8m0fnu

(ground truth is PJRT)

I've also realized the code in from/to Float32 wasn't correct and was
truncating to zero to often.
Hence the rewrite and bug fix.
2025-09-22 09:11:05 +02:00
Guillaume Wenzek
1480f77a55
zml: add the platform tag inside the Buffer (#326)
CPU buffers behave differently from Cuda/Rocm/TPU buffers,
and in particular they are addressable from main memory.
This was annoying and prevented us to add some safety checks that
weren't valid for CPU.

I'm adding a tag to buffers so that we know which target they refer to.

Also there is a bug in PJRT unitialized which doesn't respected the
memory. I've opened https://github.com/openxla/xla/pull/31292
but in the meantime I've a workaround in ZML that uses the regular
.fromHostBuffer.
(I've also a patched version of PJRT plugins in
https://github.com/zml/pjrt-artifacts/pull/41 but not merged yet)
2025-09-18 15:35:25 +02:00
Guillaume Wenzek
dda7f6c3f0
zml: remove old writer interface APIs (#324)
We still had a few old reader/writer left over.
I think we are good now
2025-09-18 11:36:23 +02:00
Guillaume Wenzek
df77b33d8e
async: remove asynk, asyncc, awaitt, await_ (#325) 2025-09-18 10:05:54 +02:00
Emmanuel Ferdman
03865b2f3d
Update bazel.sh source (#323)
## PR Summary
This small PR fixes the `bazel.sh` source.

Signed-off-by: Emmanuel Ferdman <emmanuelferdman@gmail.com>
2025-09-16 11:39:51 +02:00
Guillaume Wenzek
988f2711c5
zml: loadById (#322)
We already assign a unique tensor id to each tensor object created by a
BufferStore.
But we weren't using the id to load find the corresponding buffer in the
store, instead relying on the name.

This is allow to decouple the shape of the model in the struct vs the
shape of the model inside the safetensor file.
I also moved the id logic out of Tensor and into BufferStore.

I also updated llama/main.zig to use the new logic. 

Bonus:
* fix to `zml.call` tag hashing leading to different MLIR for every
recompile.
* expose  Cpu device count to control the number of CPU "devices"
2025-09-11 10:28:16 +02:00
Guillaume Wenzek
3f95bdd900
zml: callback api (#319)
Resuming work from @hugomano #244 
This is still a bit experimental.

The core API is here, but there are some caveats.

ZML should help with copying buffers to host memory (which is often
needed) but I'm not sure the current implementation using PJRT is good:
- does it allow graph capture
- what's the overhead ?
We should probably use directly cuMemcpy here instead of going through
PJRT.
Current implem should work with all plugins though.

This fix `tensor.print()`

I've also made a slightly more interesting examples/callback that
manually calls a cuda kernel. (there are other way to achieve this but
this shows you can do whatever you want with `zml.callback`

Note: 
the feature was originally named `custom_call` but it was conflicting
with existing bindings for xla builtin custom calls,
that are fully handled by the backend.
So I renamed it to `callback`

---------

Co-authored-by: Hugo Mano <hugo@zml.ai>
2025-09-04 16:57:34 +02:00
Hugo Mano
10d3da2449
xla: bump to b3fbfee (#321)
- Temporarily disable libnvptxcompiler usage since it's not available in
the PjRT CUDA plugin v13.0
- Add nvshmem in the sandbox, required by the PjRT CUDA plugin
2025-09-03 14:24:04 +02:00
Hugo Mano
5d6c04ec69
ci: revamp commit workflow (#320)
- Deduplicate workflows (introduced by
57fd9cda27)
- Fix linking issue on Linux since upb was introduced
- Fix duplicated `_start` symbol for some tests targets
- Temporarily use `-c opt` instead of `--config=debug` to bypass static
archive bug
2025-09-03 14:23:22 +02:00
Guillaume Wenzek
8994a37792
bump runtimes/* code for 0.15 (#318)
Fix #316, close #304 

- Bump platform specific Zig code that doesn't get built in CI.
- Restore pytorch loader, using `std.fs.File` instead of `async.File`,
allowing to reuse `std.zip`
- Update `zig fmt` in CI to `0.15.1` and ran it on the codebase
(third_party excluded)
- Remove `stdx.io` which was pointing to remove
`std.io.BufferededReader`.

The Neuron target is still broken though:
`bazel build //runtimes/neuron:libneuronxla`

As well as the Cuda plugin in `--config=debug` mode:
 `bazel build --config=debug //runtimes/cuda:zmlxcuda`
2025-09-02 15:16:09 +02:00
Steeve Morin
57fd9cda27
workspace: zig 0.15 (#316)
Much of the work was done by @gwenzek

Sadly we drop the torch loader in the process, to be added back later !

---------

Co-authored-by: Guillaume Wenzek <wzk@zml.ai>
2025-08-27 14:11:22 +02:00
Steeve Morin
56ac3e27dd
zml: remove gguf and torch loaders (#317)
Torch loader might be reintroduced later, if the need arises.
2025-08-27 09:43:39 +02:00
Steeve Morin
5f3a2e74c3
workspace: buildifier pass, remove rules_uv, rework tools/hf deps (#315)
Title says it all
2025-08-22 16:37:36 +02:00
Steeve Morin
fc384eb43e
runtimes/neuron: fix neuron runtime (#314)
This PR fixes the neuron runtime with the following:

Proxy the PJRT Api method to enforce the client struct sizes since the
neuron PJRT plugin doesn't use `>=` but `==` to assert them, breaking
PJRT compatibility guarantees.
Fixes https://github.com/aws-neuron/aws-neuron-sdk/issues/1095

Reimplement `libneuronxla` in Zig to control neuronx-cc sandboxing and
invocation.

Implement a python bootstrapper in Zig to create a full blown
`neuronx-cc` executable, avoiding the infamous chicken and egg problem
of python executables boostrapping when sandboxed (due to fixed path
shebangs).

---------

Co-authored-by: Corentin Kerisit <corentin.kerisit@gmail.com>
2025-08-22 16:28:23 +02:00
Steeve Morin
0c796302e2
runtimes/cuda: implement zmlxcuda in Zig (#313) 2025-08-21 18:34:09 +02:00
Steeve Morin
b0978b4b2d
runtimes/rocm: implement zmlxrocm in Zig (#312)
Also, sandbox `amdgpu.ids` and restore safetensors json parsing.
2025-08-21 18:15:30 +02:00
Steeve Morin
24575c2232
workspace: switch from zig_cc_binary to zig_binary (#311)
Following the rules_zig bump that adds the `linkmode` setting, we can
use the C linker directly.

We can now remove the `zig_cc_binary` hack.
2025-08-21 13:12:15 +02:00
Steeve Morin
cd8a65334f
zml: remove nemo, sentencepiece loaders, remove zig-yaml (#310)
Progress towards #298
2025-08-20 00:15:55 +02:00
Steeve Morin
de4a0e0beb
third_party/xla: de-modularize xla (#309)
This PR intends to remove XLA as a bzlmod and transfer it as a non
bzlmod dep.

This is because this module will never be upstreamed as is, so keep it
private.

Also, we fetch llvm-raw and stablehlo from it, which is fine.

While there, dummify the various local_config XLA symbols to please the
imports, as we don't use those parts in ZML itself.

Closes #305
2025-08-19 23:35:01 +02:00
Steeve Morin
aa87c024c1 workspace: re-add .vscode folder
removed by mistake in #308
2025-08-19 19:44:20 +02:00
Steeve Morin
3b81c5630e
zml: remove examples workspace (#308)
Closes #302
2025-08-19 19:42:50 +02:00
Steeve Morin
2044cd2de5
zml: remove zig-protobuf, switch to upb (#307)
Closes #301
2025-08-19 16:14:54 +02:00
Steeve Morin
8279e43038
pjrt,zml: remove profiler (#306)
Closes #303

Progress towards #301
2025-08-19 16:04:43 +02:00
Guillaume Wenzek
9f2ccf8a54
package HF cli so users don't have to create Python venv (#293) 2025-08-01 16:42:29 +02:00
Corentin Godeau
5263a170d6
examples/llama: switch to --hf-model-path flag (#270)
Instead of giving config, model weights and tokenizer paths, rely on
`huggingface-cli` download.
2025-08-01 09:46:07 +02:00
Corentin Kerisit
52d36e1c7e
runtimes/neuron: sandbox neuron dependencies (#290)
The difference with other sandboxing PRs is that we also define a
trampoline PJRT so file which will prove itself useful to circumvent
PJRT version mismatch because of
https://github.com/aws-neuron/aws-neuron-sdk/issues/1095

Additionally, we artificially create a empty repository so that we can
sandbox the distroless deps inside the neuron module extension.

---------

Co-authored-by: EC2 Default User <ec2-user@ip-172-31-64-93.ec2.internal>
2025-07-25 11:35:04 +02:00
Hugo Mano
4cf4ee3c14
workspace: xla 20250718.0-6319f0d with ROCm 6.4.1 (#289)
Note: CUDA plugin is not bumped due to an error with libnvptxcompiler. A
specific update will be done in another commit.
2025-07-22 23:50:38 +02:00
Guillaume Wenzek
ba8ff439ba
protobuf, libxev: remove usingnamespace (#280)
remove usingnamespace from libxev and zig-protobuf
    
relevant PRs:
* libxev:
7f803181b1
 * zig-protobuf: https://github.com/zml/zig-protobuf/pull/1/files

This means we can now build ZML with `-fincremental` (in debug mode only
and linux x86-64)

---------

Signed-off-by: Guillaume Wenzek <wzk@zml.ai>
2025-07-21 18:03:01 +02:00
Guillaume Wenzek
981b6becc3
remove usingnamespace from asyncio (#205)
Small PR to remove our last usingnamespace. 

Also usingnamespace will be out in 0.15.0:
https://github.com/ziglang/zig/pull/24362
2025-07-21 16:21:05 +02:00
Jean-Baptiste Dalido
a1752e8de9
ci: leverage disk cache and leverage runs-on snapshot feature (#288) 2025-07-18 16:30:54 +02:00
Steeve Morin
e13acae80b
runtimes/rocm: fetch libdrm from amdgpu repo, add amdgpu.ids layer (#287)
Properly fetch libdrm related artifacts from the amdgpu driver repo
instead of ubuntu.

While there, add `amdgpu.ids` file to the ROCm layers.
2025-07-17 22:16:27 +02:00
Steeve Morin
81fa2ec44c
workspace: disable glibc constraint, bump rules_rust, re-enable framepointers (#286)
Since the rework of runtimes sandboxing, explicitely compiling against a certain glibc version isn't needed anymore.
2025-07-17 18:20:07 +02:00
Corentin Kerisit
32216e8c54
runtimes/rocm: use inprocess lld instead of sandboxing it (#284)
XLA rencetly added the --xla_gpu_use_inprocess_lld bool flag which lets
use ditch the rocm-llvm//:lld from the sandbox :)

See
d01947de6f
2025-07-17 16:39:00 +02:00
Corentin Kerisit
4362fe323e
workspace: bump rules_zig to 20250714.0-b14a4f1 (#283)
This dummy fixes the presence of the static-lib in the runfiles of the
final target in case of kind != exe.
2025-07-16 18:01:09 +02:00
Hugo Mano
8066d2abdd
ci: use large disk on runs-on (#282)
With the latests developments the default disk space (40GB) is too small
for our new needs so we use large disks (80GB).
2025-07-16 15:21:37 +02:00
Corentin Kerisit
4c14652fb7
runtimes/rocm: sandbox ROCm dependencies (#276)
ROCm counterpart of #275 

The most important thing with this change is that ROCm dependencies
(along with the PJRT plugin) must be loaded in the main thread.

This is because some ROCm dependencies (amdhip64) use Thread Local
Storage in the destructor of static C++ instances.
Because all dependencies are unloaded at exit of the main program,
destructors are run on whatever thread exited (in general the main
thread), and any access to the TLS will except values from a different
threads....
2025-07-16 14:41:07 +02:00
Corentin Kerisit
d3dda29db5
runtimes/tpu: sandbox TPU PJRT Plugin (#278)
It has no dependencies so fairly straighforward
2025-07-16 14:13:22 +02:00
Hugo Mano
fc118e058f
deps/hftokenizers: bump dependency (#281) 2025-07-16 13:44:19 +02:00
Corentin Kerisit
12293da6af
runtimes/cpu: sandbox CPU PJRT Plugin (#279)
Same as TPU, simplified because there are no other NEEDED dependencies.
2025-07-16 13:08:55 +02:00
Corentin Kerisit
8274f9c2c0
runtimes/cuda: sandbox CUDA dependencies (#275)
This PR introduces CUDA sandboxing.

This essentially means:
- Removing all CUDA related dependencies off the leaf binary.
- Sandboxing the whole CUDA dependency graph.
- Mark dlopen direct dependencies of the PJRT shared library as NEEDED.
- Set the RPATH to the sandbox for all libraries that have sandbox
dependencies.
- Load the PJRT plugin from the sandbox.

And in the case of CUDA specifically for now:
- Enable usage of weak CUDA symbols without linking against CUDA
directly.
2025-07-16 11:33:48 +02:00
Corentin Kerisit
8ca8d069f9
pjrt: warn instead of panic if FFI Extension is not found (#277)
This is the case for TPU where FFI Extension doesn't not exist.
2025-07-15 14:55:15 +02:00
Corentin Kerisit
57abcb5aab
workspace: bump toolchains_llvm_bootstrapped to 0.2.4 (#274)
This update is important because it orders correctly the CRT which
contains .init and .fini family sections.
2025-07-15 14:54:57 +02:00