Use the new zigopt global setting from rules_zig to set -fllvm to all
zig copmile actions.
The Zig self-hosted backend is more fragile that we thought before. It
fails in numerous cases, in debug, but also in release sometimes. USe
LLVM backend until it is more stable.
zig 0.15.2 changed the meaning of `reader.takeDelimiterExclusive` to not
consume the separator. I rewrote the code to use
`takeDelimiterInclusive` since we want to consume the delimiter.
0.15.2 made this test fail with:
112/117 aio.torch.pickle.test.parse protocol 0...error(zml/aio): Unknow
pickle operator @enumFromInt(10), note we are only supporting pickle
protocol up to version 5.
cc @gwenzek
Zig modules that want to access the "c" module needs to have a non empty
CcCompilationContext for now.
So basically depending on an empty cc_library is not enough, it needs to
contain something (defines or headers or includes or wathever)
This PR uses the new release of rules_zig where we upstreamed all the
work from the zml rules_zig fork.
Not much changes from the POV of the public API except for `copts`
becoming `zigopts`.
Also, the global `c` module is only available to Zig modules that depend
on a non empty `cc_library`.
This is behavior that can probably be modified in `rules_zig` later on.
Finally, ZLS support had to be completely rethought to comply with the
change in shape of ZigModuleInfo providers and subsequent rules.
This PR is blocked until `rules_zig` exposes `ZigModuleInfo` for
`zig_binaries`.
This implement support for OpenAI "gpt-oss" models.
It's mostly a take over of #292 branch, but MoE code is kept inside
examples folder cause it's still a bit warm.
In particular the routing logic can break apart when there are undefined
memory in the inputs, where it wasn't an issue for normal MLP.
I only tested with the 20B version.
Revamp the `gather` api to match `scatter`
(prerequesite of #329 )
## Gather new api
Now the syntax is to give a named list of indices. Each name must
correspond to one axis in the gathered tensor.
This match `scatter`, and avoid the user to have to concatenate indices
in the same order than the input axes,
which is error prone cause any screw up will compile but index at the
wrong offsets.
For 1 axis it doesn't change much:
```
// before
self.weight.withTags(.{ .voc, .d }).gather(.{.voc}, .{ idx }, .{});
// after
self.weight.withTags(.{ .voc, .d }).gather(.{ .voc = idx }, .{});
```
But for two it helps a lot:
```
// before
self.weight.withTags(.{ .voc, .d }).gather(.{.voc, .d}, .{ .concatenate(&.{idx_voc, idx_d}, .last)}, .{});
// after
self.weight.withTags(.{ .voc, .d }).gather(.{ .voc = idx_voc, .d = idx_d }, .{});
```
Also adds a `gather_` that offers the same apis but with axis position
instead of names.
This one is more suitable inside library functions when dealing with
unknown names, making it complicated to create the "indices" struct.
I think we could generalize this pattern of offering "non-tagged" api
with an underscore suffix, simplifying the job of LSP.
## topk new API
I realized that most of the code calling topk was renaming the output
axis.
The idea is that logprob.topk(.voc) doesn't return you a tensor with voc
entry but only top_voc entries.
For the values it doesn't make much a difference but for the indices
it's embarrassing and it break usages of gather/scatter.
Now the user of `topK` can either use the unnamed api and pass a number,
of if using the name api must give a new name for the indexed axis.
## Yarn
Support Yarn rope embeddings. It's equivalent to the "llama3" embedding
except they do the interpolation in time space instead of freq space.
Adding the following missing data types:
- u2
- i1
- i2
- f4e2m1fn
- f8e3m4
- f8e4m3
- f8e8m0fnu
(ground truth is PJRT)
I've also realized the code in from/to Float32 wasn't correct and was
truncating to zero to often.
Hence the rewrite and bug fix.
CPU buffers behave differently from Cuda/Rocm/TPU buffers,
and in particular they are addressable from main memory.
This was annoying and prevented us to add some safety checks that
weren't valid for CPU.
I'm adding a tag to buffers so that we know which target they refer to.
Also there is a bug in PJRT unitialized which doesn't respected the
memory. I've opened https://github.com/openxla/xla/pull/31292
but in the meantime I've a workaround in ZML that uses the regular
.fromHostBuffer.
(I've also a patched version of PJRT plugins in
https://github.com/zml/pjrt-artifacts/pull/41 but not merged yet)
We already assign a unique tensor id to each tensor object created by a
BufferStore.
But we weren't using the id to load find the corresponding buffer in the
store, instead relying on the name.
This is allow to decouple the shape of the model in the struct vs the
shape of the model inside the safetensor file.
I also moved the id logic out of Tensor and into BufferStore.
I also updated llama/main.zig to use the new logic.
Bonus:
* fix to `zml.call` tag hashing leading to different MLIR for every
recompile.
* expose Cpu device count to control the number of CPU "devices"
Resuming work from @hugomano #244
This is still a bit experimental.
The core API is here, but there are some caveats.
ZML should help with copying buffers to host memory (which is often
needed) but I'm not sure the current implementation using PJRT is good:
- does it allow graph capture
- what's the overhead ?
We should probably use directly cuMemcpy here instead of going through
PJRT.
Current implem should work with all plugins though.
This fix `tensor.print()`
I've also made a slightly more interesting examples/callback that
manually calls a cuda kernel. (there are other way to achieve this but
this shows you can do whatever you want with `zml.callback`
Note:
the feature was originally named `custom_call` but it was conflicting
with existing bindings for xla builtin custom calls,
that are fully handled by the backend.
So I renamed it to `callback`
---------
Co-authored-by: Hugo Mano <hugo@zml.ai>
- Temporarily disable libnvptxcompiler usage since it's not available in
the PjRT CUDA plugin v13.0
- Add nvshmem in the sandbox, required by the PjRT CUDA plugin
- Deduplicate workflows (introduced by
57fd9cda27)
- Fix linking issue on Linux since upb was introduced
- Fix duplicated `_start` symbol for some tests targets
- Temporarily use `-c opt` instead of `--config=debug` to bypass static
archive bug
Fix#316, close#304
- Bump platform specific Zig code that doesn't get built in CI.
- Restore pytorch loader, using `std.fs.File` instead of `async.File`,
allowing to reuse `std.zip`
- Update `zig fmt` in CI to `0.15.1` and ran it on the codebase
(third_party excluded)
- Remove `stdx.io` which was pointing to remove
`std.io.BufferededReader`.
The Neuron target is still broken though:
`bazel build //runtimes/neuron:libneuronxla`
As well as the Cuda plugin in `--config=debug` mode:
`bazel build --config=debug //runtimes/cuda:zmlxcuda`
Much of the work was done by @gwenzek
Sadly we drop the torch loader in the process, to be added back later !
---------
Co-authored-by: Guillaume Wenzek <wzk@zml.ai>
This PR fixes the neuron runtime with the following:
Proxy the PJRT Api method to enforce the client struct sizes since the
neuron PJRT plugin doesn't use `>=` but `==` to assert them, breaking
PJRT compatibility guarantees.
Fixes https://github.com/aws-neuron/aws-neuron-sdk/issues/1095
Reimplement `libneuronxla` in Zig to control neuronx-cc sandboxing and
invocation.
Implement a python bootstrapper in Zig to create a full blown
`neuronx-cc` executable, avoiding the infamous chicken and egg problem
of python executables boostrapping when sandboxed (due to fixed path
shebangs).
---------
Co-authored-by: Corentin Kerisit <corentin.kerisit@gmail.com>
This PR intends to remove XLA as a bzlmod and transfer it as a non
bzlmod dep.
This is because this module will never be upstreamed as is, so keep it
private.
Also, we fetch llvm-raw and stablehlo from it, which is fine.
While there, dummify the various local_config XLA symbols to please the
imports, as we don't use those parts in ZML itself.
Closes#305