This PR add the support of nvfp4 x nvfp4 moe backend on sm120 / sm100
(natively supported nvfp4 hardware). It :
- retrieves the released artifact cutlass moe flashinfer :
https://github.com/zml/flashinfer/releases/tag/cutlass-moe-v0.1.0
- loads the .so
- implements the nvfp4 x nvfp4 backend (added to existing triton and
mosaic tpu).
Resuming work from @hugomano #244
This is still a bit experimental.
The core API is here, but there are some caveats.
ZML should help with copying buffers to host memory (which is often
needed) but I'm not sure the current implementation using PJRT is good:
- does it allow graph capture
- what's the overhead ?
We should probably use directly cuMemcpy here instead of going through
PJRT.
Current implem should work with all plugins though.
This fix `tensor.print()`
I've also made a slightly more interesting examples/callback that
manually calls a cuda kernel. (there are other way to achieve this but
this shows you can do whatever you want with `zml.callback`
Note:
the feature was originally named `custom_call` but it was conflicting
with existing bindings for xla builtin custom calls,
that are fully handled by the backend.
So I renamed it to `callback`
---------
Co-authored-by: Hugo Mano <hugo@zml.ai>
Fix#316, close#304
- Bump platform specific Zig code that doesn't get built in CI.
- Restore pytorch loader, using `std.fs.File` instead of `async.File`,
allowing to reuse `std.zip`
- Update `zig fmt` in CI to `0.15.1` and ran it on the codebase
(third_party excluded)
- Remove `stdx.io` which was pointing to remove
`std.io.BufferededReader`.
The Neuron target is still broken though:
`bazel build //runtimes/neuron:libneuronxla`
As well as the Cuda plugin in `--config=debug` mode:
`bazel build --config=debug //runtimes/cuda:zmlxcuda`
Much of the work was done by @gwenzek
Sadly we drop the torch loader in the process, to be added back later !
---------
Co-authored-by: Guillaume Wenzek <wzk@zml.ai>