Adding the following missing data types:
- u2
- i1
- i2
- f4e2m1fn
- f8e3m4
- f8e4m3
- f8e8m0fnu
(ground truth is PJRT)
I've also realized the code in from/to Float32 wasn't correct and was
truncating to zero to often.
Hence the rewrite and bug fix.
CPU buffers behave differently from Cuda/Rocm/TPU buffers,
and in particular they are addressable from main memory.
This was annoying and prevented us to add some safety checks that
weren't valid for CPU.
I'm adding a tag to buffers so that we know which target they refer to.
Also there is a bug in PJRT unitialized which doesn't respected the
memory. I've opened https://github.com/openxla/xla/pull/31292
but in the meantime I've a workaround in ZML that uses the regular
.fromHostBuffer.
(I've also a patched version of PJRT plugins in
https://github.com/zml/pjrt-artifacts/pull/41 but not merged yet)
Resuming work from @hugomano #244
This is still a bit experimental.
The core API is here, but there are some caveats.
ZML should help with copying buffers to host memory (which is often
needed) but I'm not sure the current implementation using PJRT is good:
- does it allow graph capture
- what's the overhead ?
We should probably use directly cuMemcpy here instead of going through
PJRT.
Current implem should work with all plugins though.
This fix `tensor.print()`
I've also made a slightly more interesting examples/callback that
manually calls a cuda kernel. (there are other way to achieve this but
this shows you can do whatever you want with `zml.callback`
Note:
the feature was originally named `custom_call` but it was conflicting
with existing bindings for xla builtin custom calls,
that are fully handled by the backend.
So I renamed it to `callback`
---------
Co-authored-by: Hugo Mano <hugo@zml.ai>
- Temporarily disable libnvptxcompiler usage since it's not available in
the PjRT CUDA plugin v13.0
- Add nvshmem in the sandbox, required by the PjRT CUDA plugin
Fix#316, close#304
- Bump platform specific Zig code that doesn't get built in CI.
- Restore pytorch loader, using `std.fs.File` instead of `async.File`,
allowing to reuse `std.zip`
- Update `zig fmt` in CI to `0.15.1` and ran it on the codebase
(third_party excluded)
- Remove `stdx.io` which was pointing to remove
`std.io.BufferededReader`.
The Neuron target is still broken though:
`bazel build //runtimes/neuron:libneuronxla`
As well as the Cuda plugin in `--config=debug` mode:
`bazel build --config=debug //runtimes/cuda:zmlxcuda`
Much of the work was done by @gwenzek
Sadly we drop the torch loader in the process, to be added back later !
---------
Co-authored-by: Guillaume Wenzek <wzk@zml.ai>
This PR introduces CUDA sandboxing.
This essentially means:
- Removing all CUDA related dependencies off the leaf binary.
- Sandboxing the whole CUDA dependency graph.
- Mark dlopen direct dependencies of the PJRT shared library as NEEDED.
- Set the RPATH to the sandbox for all libraries that have sandbox
dependencies.
- Load the PJRT plugin from the sandbox.
And in the case of CUDA specifically for now:
- Enable usage of weak CUDA symbols without linking against CUDA
directly.
This is the non-controversial part of #262
It contains:
* the new `zig_srcs` helper,
* bug fix related to sources in rules_zig,
* cleanups of zml/BUILD.bazel
* test fixes in async/coro.zig (those tests weren't run in CI but now
are)
* disabled nemo and yaml models loader (don't compile anymore since
0.14.0 bump)
* buffer/hostbuffer utils
* more precise f32 -> bf16 conversion
* visit should not read irrelevant fields
* HostBuffer.data -> HostBuffer.bytes()
* zml.aio loadBuffers: type inference
* store expected input shapes in the executable
* zml.meta.visit support std.BoundedArray
* stdx.json.TaggedUnion
* protection against undefined float hyperparams
* avoid meta.TagPayload which can be quite slow
* improve logging
Ensure that our Bazel setup for the framework and examples work through
queries, builds and tests.
Also fix broken code and deps in the codebase.
As of today only Neuron is broken (cf:
https://github.com/zml/zml/pull/111)
Rewrite trivial transpose into reshapes.
Transpose over a 1-d axis are effectively no-op.
Normally the backend can detect this, but we observe that some backend
can get confused. So avoiding those extra transpose can help.
I also changed the default profiler size to 1_000_000 events because
above that the perfetto ui can't load the generated file anyway
* create one folder per target, then one sub-folder per mlir module.
* pass this folder to `xla_dump_to` so that all the xla files arrive
there.
* explicitly save the mlir and the serialized executable there
I'm also exposing James profile conversion tool into ZML as
`profiler.dumpDataAsJson(file)`
The Pjrt api allows to passe string flags to the plugin on client
creation.
Since we don't like working with strings I've converted the strings
supported by the cuda plugin into a struct.
For now I've focused on the strings corresponding to the allocator
settings following #125
More settings are not exposed ATM, will follow up with #1323d31c48c71/xla/pjrt/c/pjrt_c_api_gpu_internal.cc (L76-L86)
AFAICT the cpu plugin doesn't read flags at all.
Neuron doesn't have known flags, TPU requires more investigation.
**Breaking change**: `context.autoPlatform()` ->
`context.autoPlatform(.{})`
- workspace: new module versions added to Bazel registry:
- deps/xla: 4663f04d26 (20241025.0-4663f04)
This version introduces an upgraded overlay with Python repository initialization for the TSL extension. The changes were necessitated by this XLA commit: 19b4ecbe98 (diff-df5471accc) , which triggers a repository call. Python's hermetic version is provided via the .bazelrc file.
- deps/stablehlo: f7f8e4e352 (20241021.0-1c0b606)
- deps/llvm: f7f8e4e352 (20241022.0-6c4267f)
This version includes an upgraded module declaration with rules_python dependency added, as LLVM prepares its overlay for Bazel v8. Relevant LLVM changes here: 3903cb4695
- deps/runtimes: use PJRT artifacts v3.0.0 (https://github.com/zml/pjrt-artifacts/releases/tag/v3.0.0)
- workspace: align used versions of rules_zig in the Bazel dependency graph
- pjrt: expose getPluginAttributes and getPluginAttribute private APIs and stablehloCurrentVersion public API
- pjrt: updated customCallRegistry function with the new FFI provided by the PJRT API GPU extension
Minimal support for this feature was added, as this is not the main focus of the current PR. See related changes in XLA here: ef85a7bcc3
- dialects/mlir/stablehlo: expose getCurrentVersion API and fix broken stablehloVersionFromCompatibilityRequirement API
- zml/pjrtx: serialize IR to the PJRT plugin's supported HLO version. If the plugin attribute is unavailable, it falls back to the StableHLO's compatibility requirement version.
- zml/tensor: tests using batching dims for gather and scatter are now re-enabled
This one is a chonky.
This commit adds a new `stdx` module with a few utility functions. It is
very far reaching. Forgive the size.
Also, clean the logs in the executables.
* restore the integration with PJRT custom call extension
* add a generic `zmlHostBufferCallback` custom call that copies the
received tensor to the host, and call a user supplied callback on it.
* add a default "print" method on `Tensor` that print the runtime value
of a given tensor
* print itself has very little code, it's easy for a user to define
their own print if they don't like ours.
limitations:
* only supported on Cuda atm
* pre allocates a buffer on the host to copy the content of the device
buffer,
this buffer won't be freed. There is exactly one buffer per "print" call
in the IR.
* it uses cuda api, maybe I should be using `createViewOfDeviceBuffer`
to make it less cuda specific
Add a tiny abstraction layer between ZML and PJRT.
Most notably, to implement custom logic to decide wether or not we
should/can load a PJRT plugin.
For instance, `libtpu` will block the program trying to reach the GCP
metadata server, which doesn't exist outside GCP.
This also allows for comptime detection of wether a runtime is linked
with the program, and its dependencies.
Some runtimes do not implement `PJRT_Event`. For those, we must dispatch
on our own thread pool for buffer copies and wait on events only if the
runtime returns them.
If there are no events, assume all operations are synchronous.
First rough support for in-process sharding.
The goal of sharding is to allow leveraging several GPUs to collaborate
to run one big model.
Current implementation:
* added `platform.sharding()` describing how the set of devices managed
by the current process should be used.
* added a `_sharding_info` to shape.
* A tensor / shape axis can be marked as needing sharding using
`.withSharding(.{ .a })` or `.withSharding(.{ 1 })`.
* When using `populateModelWeights` tensors with sharded axes, will be
populated by sharded buffers.
* when generating the MLIR bytecode for a ZML function the **input**
arguments, will see their `_sharding_info` converted into
`mhlo.sharding` attributes
some dead code or commented out code we should have removed before
release.
copyslice is also broken as per the commented out tests.
it will need to be reimplemented if we need it again.