This implement support for OpenAI "gpt-oss" models.
It's mostly a take over of #292 branch, but MoE code is kept inside
examples folder cause it's still a bit warm.
In particular the routing logic can break apart when there are undefined
memory in the inputs, where it wasn't an issue for normal MLP.
I only tested with the 20B version.
Revamp the `gather` api to match `scatter`
(prerequesite of #329 )
## Gather new api
Now the syntax is to give a named list of indices. Each name must
correspond to one axis in the gathered tensor.
This match `scatter`, and avoid the user to have to concatenate indices
in the same order than the input axes,
which is error prone cause any screw up will compile but index at the
wrong offsets.
For 1 axis it doesn't change much:
```
// before
self.weight.withTags(.{ .voc, .d }).gather(.{.voc}, .{ idx }, .{});
// after
self.weight.withTags(.{ .voc, .d }).gather(.{ .voc = idx }, .{});
```
But for two it helps a lot:
```
// before
self.weight.withTags(.{ .voc, .d }).gather(.{.voc, .d}, .{ .concatenate(&.{idx_voc, idx_d}, .last)}, .{});
// after
self.weight.withTags(.{ .voc, .d }).gather(.{ .voc = idx_voc, .d = idx_d }, .{});
```
Also adds a `gather_` that offers the same apis but with axis position
instead of names.
This one is more suitable inside library functions when dealing with
unknown names, making it complicated to create the "indices" struct.
I think we could generalize this pattern of offering "non-tagged" api
with an underscore suffix, simplifying the job of LSP.
## topk new API
I realized that most of the code calling topk was renaming the output
axis.
The idea is that logprob.topk(.voc) doesn't return you a tensor with voc
entry but only top_voc entries.
For the values it doesn't make much a difference but for the indices
it's embarrassing and it break usages of gather/scatter.
Now the user of `topK` can either use the unnamed api and pass a number,
of if using the name api must give a new name for the indexed axis.
## Yarn
Support Yarn rope embeddings. It's equivalent to the "llama3" embedding
except they do the interpolation in time space instead of freq space.
Adding the following missing data types:
- u2
- i1
- i2
- f4e2m1fn
- f8e3m4
- f8e4m3
- f8e8m0fnu
(ground truth is PJRT)
I've also realized the code in from/to Float32 wasn't correct and was
truncating to zero to often.
Hence the rewrite and bug fix.
CPU buffers behave differently from Cuda/Rocm/TPU buffers,
and in particular they are addressable from main memory.
This was annoying and prevented us to add some safety checks that
weren't valid for CPU.
I'm adding a tag to buffers so that we know which target they refer to.
Also there is a bug in PJRT unitialized which doesn't respected the
memory. I've opened https://github.com/openxla/xla/pull/31292
but in the meantime I've a workaround in ZML that uses the regular
.fromHostBuffer.
(I've also a patched version of PJRT plugins in
https://github.com/zml/pjrt-artifacts/pull/41 but not merged yet)
We already assign a unique tensor id to each tensor object created by a
BufferStore.
But we weren't using the id to load find the corresponding buffer in the
store, instead relying on the name.
This is allow to decouple the shape of the model in the struct vs the
shape of the model inside the safetensor file.
I also moved the id logic out of Tensor and into BufferStore.
I also updated llama/main.zig to use the new logic.
Bonus:
* fix to `zml.call` tag hashing leading to different MLIR for every
recompile.
* expose Cpu device count to control the number of CPU "devices"
Resuming work from @hugomano #244
This is still a bit experimental.
The core API is here, but there are some caveats.
ZML should help with copying buffers to host memory (which is often
needed) but I'm not sure the current implementation using PJRT is good:
- does it allow graph capture
- what's the overhead ?
We should probably use directly cuMemcpy here instead of going through
PJRT.
Current implem should work with all plugins though.
This fix `tensor.print()`
I've also made a slightly more interesting examples/callback that
manually calls a cuda kernel. (there are other way to achieve this but
this shows you can do whatever you want with `zml.callback`
Note:
the feature was originally named `custom_call` but it was conflicting
with existing bindings for xla builtin custom calls,
that are fully handled by the backend.
So I renamed it to `callback`
---------
Co-authored-by: Hugo Mano <hugo@zml.ai>
- Temporarily disable libnvptxcompiler usage since it's not available in
the PjRT CUDA plugin v13.0
- Add nvshmem in the sandbox, required by the PjRT CUDA plugin
- Deduplicate workflows (introduced by
57fd9cda27)
- Fix linking issue on Linux since upb was introduced
- Fix duplicated `_start` symbol for some tests targets
- Temporarily use `-c opt` instead of `--config=debug` to bypass static
archive bug
Fix#316, close#304
- Bump platform specific Zig code that doesn't get built in CI.
- Restore pytorch loader, using `std.fs.File` instead of `async.File`,
allowing to reuse `std.zip`
- Update `zig fmt` in CI to `0.15.1` and ran it on the codebase
(third_party excluded)
- Remove `stdx.io` which was pointing to remove
`std.io.BufferededReader`.
The Neuron target is still broken though:
`bazel build //runtimes/neuron:libneuronxla`
As well as the Cuda plugin in `--config=debug` mode:
`bazel build --config=debug //runtimes/cuda:zmlxcuda`
Much of the work was done by @gwenzek
Sadly we drop the torch loader in the process, to be added back later !
---------
Co-authored-by: Guillaume Wenzek <wzk@zml.ai>
This PR fixes the neuron runtime with the following:
Proxy the PJRT Api method to enforce the client struct sizes since the
neuron PJRT plugin doesn't use `>=` but `==` to assert them, breaking
PJRT compatibility guarantees.
Fixes https://github.com/aws-neuron/aws-neuron-sdk/issues/1095
Reimplement `libneuronxla` in Zig to control neuronx-cc sandboxing and
invocation.
Implement a python bootstrapper in Zig to create a full blown
`neuronx-cc` executable, avoiding the infamous chicken and egg problem
of python executables boostrapping when sandboxed (due to fixed path
shebangs).
---------
Co-authored-by: Corentin Kerisit <corentin.kerisit@gmail.com>
This PR intends to remove XLA as a bzlmod and transfer it as a non
bzlmod dep.
This is because this module will never be upstreamed as is, so keep it
private.
Also, we fetch llvm-raw and stablehlo from it, which is fine.
While there, dummify the various local_config XLA symbols to please the
imports, as we don't use those parts in ZML itself.
Closes#305
The difference with other sandboxing PRs is that we also define a
trampoline PJRT so file which will prove itself useful to circumvent
PJRT version mismatch because of
https://github.com/aws-neuron/aws-neuron-sdk/issues/1095
Additionally, we artificially create a empty repository so that we can
sandbox the distroless deps inside the neuron module extension.
---------
Co-authored-by: EC2 Default User <ec2-user@ip-172-31-64-93.ec2.internal>
remove usingnamespace from libxev and zig-protobuf
relevant PRs:
* libxev:
7f803181b1
* zig-protobuf: https://github.com/zml/zig-protobuf/pull/1/files
This means we can now build ZML with `-fincremental` (in debug mode only
and linux x86-64)
---------
Signed-off-by: Guillaume Wenzek <wzk@zml.ai>
ROCm counterpart of #275
The most important thing with this change is that ROCm dependencies
(along with the PJRT plugin) must be loaded in the main thread.
This is because some ROCm dependencies (amdhip64) use Thread Local
Storage in the destructor of static C++ instances.
Because all dependencies are unloaded at exit of the main program,
destructors are run on whatever thread exited (in general the main
thread), and any access to the TLS will except values from a different
threads....
This PR introduces CUDA sandboxing.
This essentially means:
- Removing all CUDA related dependencies off the leaf binary.
- Sandboxing the whole CUDA dependency graph.
- Mark dlopen direct dependencies of the PJRT shared library as NEEDED.
- Set the RPATH to the sandbox for all libraries that have sandbox
dependencies.
- Load the PJRT plugin from the sandbox.
And in the case of CUDA specifically for now:
- Enable usage of weak CUDA symbols without linking against CUDA
directly.