This PR introduces CUDA sandboxing.
This essentially means:
- Removing all CUDA related dependencies off the leaf binary.
- Sandboxing the whole CUDA dependency graph.
- Mark dlopen direct dependencies of the PJRT shared library as NEEDED.
- Set the RPATH to the sandbox for all libraries that have sandbox
dependencies.
- Load the PJRT plugin from the sandbox.
And in the case of CUDA specifically for now:
- Enable usage of weak CUDA symbols without linking against CUDA
directly.
This is the non-controversial part of #262
It contains:
* the new `zig_srcs` helper,
* bug fix related to sources in rules_zig,
* cleanups of zml/BUILD.bazel
* test fixes in async/coro.zig (those tests weren't run in CI but now
are)
* disabled nemo and yaml models loader (don't compile anymore since
0.14.0 bump)
* buffer/hostbuffer utils
* more precise f32 -> bf16 conversion
* visit should not read irrelevant fields
* HostBuffer.data -> HostBuffer.bytes()
* zml.aio loadBuffers: type inference
* store expected input shapes in the executable
* zml.meta.visit support std.BoundedArray
* stdx.json.TaggedUnion
* protection against undefined float hyperparams
* avoid meta.TagPayload which can be quite slow
* improve logging
Ensure that our Bazel setup for the framework and examples work through
queries, builds and tests.
Also fix broken code and deps in the codebase.
As of today only Neuron is broken (cf:
https://github.com/zml/zml/pull/111)
Rewrite trivial transpose into reshapes.
Transpose over a 1-d axis are effectively no-op.
Normally the backend can detect this, but we observe that some backend
can get confused. So avoiding those extra transpose can help.
I also changed the default profiler size to 1_000_000 events because
above that the perfetto ui can't load the generated file anyway
* create one folder per target, then one sub-folder per mlir module.
* pass this folder to `xla_dump_to` so that all the xla files arrive
there.
* explicitly save the mlir and the serialized executable there
I'm also exposing James profile conversion tool into ZML as
`profiler.dumpDataAsJson(file)`
The Pjrt api allows to passe string flags to the plugin on client
creation.
Since we don't like working with strings I've converted the strings
supported by the cuda plugin into a struct.
For now I've focused on the strings corresponding to the allocator
settings following #125
More settings are not exposed ATM, will follow up with #1323d31c48c71/xla/pjrt/c/pjrt_c_api_gpu_internal.cc (L76-L86)
AFAICT the cpu plugin doesn't read flags at all.
Neuron doesn't have known flags, TPU requires more investigation.
**Breaking change**: `context.autoPlatform()` ->
`context.autoPlatform(.{})`
- workspace: new module versions added to Bazel registry:
- deps/xla: 4663f04d26 (20241025.0-4663f04)
This version introduces an upgraded overlay with Python repository initialization for the TSL extension. The changes were necessitated by this XLA commit: 19b4ecbe98 (diff-df5471accc) , which triggers a repository call. Python's hermetic version is provided via the .bazelrc file.
- deps/stablehlo: f7f8e4e352 (20241021.0-1c0b606)
- deps/llvm: f7f8e4e352 (20241022.0-6c4267f)
This version includes an upgraded module declaration with rules_python dependency added, as LLVM prepares its overlay for Bazel v8. Relevant LLVM changes here: 3903cb4695
- deps/runtimes: use PJRT artifacts v3.0.0 (https://github.com/zml/pjrt-artifacts/releases/tag/v3.0.0)
- workspace: align used versions of rules_zig in the Bazel dependency graph
- pjrt: expose getPluginAttributes and getPluginAttribute private APIs and stablehloCurrentVersion public API
- pjrt: updated customCallRegistry function with the new FFI provided by the PJRT API GPU extension
Minimal support for this feature was added, as this is not the main focus of the current PR. See related changes in XLA here: ef85a7bcc3
- dialects/mlir/stablehlo: expose getCurrentVersion API and fix broken stablehloVersionFromCompatibilityRequirement API
- zml/pjrtx: serialize IR to the PJRT plugin's supported HLO version. If the plugin attribute is unavailable, it falls back to the StableHLO's compatibility requirement version.
- zml/tensor: tests using batching dims for gather and scatter are now re-enabled
This one is a chonky.
This commit adds a new `stdx` module with a few utility functions. It is
very far reaching. Forgive the size.
Also, clean the logs in the executables.
* restore the integration with PJRT custom call extension
* add a generic `zmlHostBufferCallback` custom call that copies the
received tensor to the host, and call a user supplied callback on it.
* add a default "print" method on `Tensor` that print the runtime value
of a given tensor
* print itself has very little code, it's easy for a user to define
their own print if they don't like ours.
limitations:
* only supported on Cuda atm
* pre allocates a buffer on the host to copy the content of the device
buffer,
this buffer won't be freed. There is exactly one buffer per "print" call
in the IR.
* it uses cuda api, maybe I should be using `createViewOfDeviceBuffer`
to make it less cuda specific
Add a tiny abstraction layer between ZML and PJRT.
Most notably, to implement custom logic to decide wether or not we
should/can load a PJRT plugin.
For instance, `libtpu` will block the program trying to reach the GCP
metadata server, which doesn't exist outside GCP.
This also allows for comptime detection of wether a runtime is linked
with the program, and its dependencies.
Some runtimes do not implement `PJRT_Event`. For those, we must dispatch
on our own thread pool for buffer copies and wait on events only if the
runtime returns them.
If there are no events, assume all operations are synchronous.
First rough support for in-process sharding.
The goal of sharding is to allow leveraging several GPUs to collaborate
to run one big model.
Current implementation:
* added `platform.sharding()` describing how the set of devices managed
by the current process should be used.
* added a `_sharding_info` to shape.
* A tensor / shape axis can be marked as needing sharding using
`.withSharding(.{ .a })` or `.withSharding(.{ 1 })`.
* When using `populateModelWeights` tensors with sharded axes, will be
populated by sharded buffers.
* when generating the MLIR bytecode for a ZML function the **input**
arguments, will see their `_sharding_info` converted into
`mhlo.sharding` attributes
some dead code or commented out code we should have removed before
release.
copyslice is also broken as per the commented out tests.
it will need to be reimplemented if we need it again.