0
Fork 0
mirror of https://github.com/zml/zml.git synced 2026-11-16 19:11:09 +00:00
Commit graph

99 commits

Author SHA1 Message Date
Steeve Morin
032f2d7d59 WIP 2024-11-29 16:12:14 +01:00
Steeve Morin
5a2803c21a WIP 2024-11-29 14:58:59 +01:00
Steeve Morin
8399c3f756 WIP 2024-11-29 14:58:59 +01:00
Guillaume Wenzek
c41e161a7d remove unused test 2024-11-28 15:06:29 +01:00
Guillaume Wenzek
c03448363c llama: update to new rope 2024-11-28 15:02:13 +01:00
Guillaume Wenzek
31e5070848 fixup! make rope accept token_idx directly 2024-11-28 15:00:25 +01:00
Guillaume Wenzek
eb500ace79 make rope accept token_idx directly 2024-11-28 15:00:20 +01:00
Guillaume Wenzek
db3675242a bazelrc: --config=debug 2024-11-28 14:50:17 +01:00
Guillaume Wenzek
ddfc9497c2 f8: add comment 2024-11-28 14:48:42 +01:00
Guillaume Wenzek
e984f3d2fc fix float conversions 2024-11-28 14:48:09 +01:00
Guillaume Wenzek
71f134b9b3 fix small inconsistencies between different versions of sdpa 2024-11-28 14:48:07 +01:00
Guillaume Wenzek
0e092f4caf fix Tensor.constant for types which aren't valid MLIR attributes 2024-11-28 14:48:01 +01:00
Guillaume Wenzek
6412aa1613 better error message for broadcast 2024-11-28 14:47:47 +01:00
Steeve Morin
02527de9b9
runtimes/cuda: fetch nccl from the pip package (#110)
Exactly as XLA does
2024-11-28 11:14:05 +01:00
Steeve Morin
6f6dfe9fe5
runtimes/cuda: rework sandboxing and dynamic symbol renaming (#109)
This change reworks the sandboxing and dynamic symbol renaming for the
cuda runtime.

Switch to the jax-cuda-pjrt pre-built PJRT plugin until we fix the pjrt
artifacts build. Most notably because our build of the plugin didn't
include the right compute capabilities.

Take the opportunity to bump the `cuda=12.6.2` `cudnn=9.5.1`.

Fixes #19
2024-11-27 19:39:17 +01:00
Hugo Mano
85685174c9
zml/pjrtx: use our own stablehlo version if plugin provided version is newer (#108)
Discovered use case:

- PJRT plugin attribute `stablehlo_current_version` is `1.7.8`
- `stablehloVersionFromCompatibilityRequirement(WEEK_12)` is `1.1.0`
-  `stablehloGetCurrentVersion` is `1.7.6`

The target version for artifact serialization is `1.7.8` but should be
`1.7.6`.

XLA ref:
2f99455cdf (diff-7f335a7892)
2024-11-27 18:29:04 +01:00
Hugo Mano
2f70fa4a00
workspace: log message before return NotFound (#107)
This allow us to find which NotFound is raised
2024-11-27 17:21:34 +01:00
Hugo Mano
db55475b17
examples/loader: missing stdx dep (#106) 2024-11-27 17:09:25 +01:00
Corentin Godeau
6a1183f8ad
zml/pjrtx: call bufferFromHostBuffer on the thread pool (#103)
This was causing very bad performance for weight loading on Neuron
because the calls are blocking
2024-11-26 18:57:59 +01:00
Corentin Godeau
58f5895d50
zml/platform: increase max device limit (#104)
Some platform (such as Neuron) can have up to 32 devices, one per core.
2024-11-26 18:57:21 +01:00
Steeve Morin
43c4348de9
runtimes/neuron: add support for AWS neuron / trainium / inferentia 2 (#102)
This PR adds support for AWS Neuron / Trainium 1 / Inferentia 2.

It can be enabled via the `--@zml//runtimes:neuron=true` flag.

Everything should work as expected.

---------

Co-authored-by: Corentin Godeau <corentin@zml.ai>
2024-11-26 14:20:47 +01:00
Guillaume Wenzek
09da897fe8
zml: fix memory usage of sdpaMemEfficient (#90)
use `zml.ops.while` instead of `zml.ops.for` the problem with the latter is that it concatenate all intermediary results, and thus using lot of memory unless the concat was optimized away which wasn't always the case
2024-11-26 13:57:27 +01:00
Hugo Mano
d366f00aed
workspace: bump llvm / xla / stablehlo / pjrt plugins and restore disabled tests xla@4663f04 (#98)
- workspace: new module versions added to Bazel registry:
  - deps/xla: 4663f04d26 (20241025.0-4663f04) 
This version introduces an upgraded overlay with Python repository initialization for the TSL extension. The changes were necessitated by this XLA commit: 19b4ecbe98 (diff-df5471accc) , which triggers a repository call. Python's hermetic version is provided via the .bazelrc file.
  - deps/stablehlo: f7f8e4e352 (20241021.0-1c0b606)  
  - deps/llvm: f7f8e4e352 (20241022.0-6c4267f)
 This version includes an upgraded module declaration with rules_python dependency added, as LLVM prepares its overlay for Bazel v8. Relevant LLVM changes here: 3903cb4695
  - deps/runtimes: use PJRT artifacts v3.0.0 (https://github.com/zml/pjrt-artifacts/releases/tag/v3.0.0)
- workspace: align used versions of rules_zig in the Bazel dependency graph
- pjrt: expose getPluginAttributes and getPluginAttribute private APIs and stablehloCurrentVersion public API
- pjrt: updated customCallRegistry function with the new FFI provided by the PJRT API GPU extension 
Minimal support for this feature was added, as this is not the main focus of the current PR. See related changes in XLA here: ef85a7bcc3
- dialects/mlir/stablehlo: expose getCurrentVersion API and fix broken stablehloVersionFromCompatibilityRequirement API
- zml/pjrtx: serialize IR to the PJRT plugin's supported HLO version. If the plugin attribute is unavailable, it falls back to the StableHLO's compatibility requirement version.
- zml/tensor: tests using batching dims for gather and scatter are now re-enabled
2024-11-25 15:53:25 +01:00
Steeve Morin
db88158518
async: rework async runtime to use coroutines and cross thread signaling (#100)
This mostly reworks the runtime to enable proper cross thread signaling
for `callBlocking`.

API changes include:
- `async_ -> asyncc`
- `await_ -> awaitt`
- proper type inference for non generic arguments even for generic
functions
- bump `libxev`
- use `epoll` on linux by default (can be comptime overridden through
`xev_options.linux_backend` in the root module)

Mostly, also, it's pretty fast.
2024-11-22 17:56:22 +01:00
Guillaume Wenzek
15b0cdc9ad
Fix crash in for_ (#99)
Contains two bug fixes

1. values should be pushed to their block before we open a new block,
otherwise this lead to us assigning values to the wrong block.
* to detect future instance of this bug, and misusage of ZML apis I've
added a few more asserts.
* now every mlir block we create is either "open" or "hermetic", and we
check that hermetic block don't reference variables from outside the
block.

2. in `for_` make sure `first_step` is always used. This is a workaround
for #97. For some reason you need two nested loops to trigger the bug.
Note that this used to work, so I'm not sure what changed in how we
generate MLIR or on MLIR side.

to enable this I had to use the `.pad(.{ ._0 = ... })` syntax. I feel
this could solve a lot of usability usage we have, without having to
introduce too many `.operation1d` operators.
2024-11-22 17:46:47 +01:00
Corentin Godeau
4d1d03cfaa
examples: fixes related to recent async changes (#96)
Recent changes to the `async` module reverted `asyncGeneric` to `asyncc`
but examples were not updated
2024-11-20 15:34:38 +01:00
Guillaume Wenzek
5c60687cde
zml: makeBlocks returns the inner tensor to propagate tags (#92)
Now zml.ops.makeBlock returns both the mlir.Block created and also the
Tensors returned by `func`.

The returned tensors should not be returned to the user,
because their `mlir.Value` must not escape the block that created them.
But their shapes/tags can be safely propagated further.

This makes `zml.ops.makeBlock` a bit more footgunny, but also it's a
utility function that most users shouldn't use. And it allows to use
tags created inside the block and apply them to the return value. In
particular tags and sharding info should now be propagated more
consistently for while loops, summation, reduceWindow, ...

I've also enabled tests to run on non-cpu platform.
this is useful to figure out if a specific platform has limitation in
it's pjrt support
2024-11-20 13:36:01 +01:00
Guillaume Wenzek
7f380a9882
fix a few compileError issues (#93)
fix a few `compileError` calls introduced by #89 . Zig compiler don't
catch those until you trigger a comptime error.
2024-11-20 13:26:44 +01:00
Rene Schallner
e99837a59f
Module-cache: increase max_pjrt_executable_size to 400MB (#95)
Some PJRT executables get really large (e.g. ca. 300MB). We want to be
able to load them.
2024-11-20 10:02:29 +01:00
Corentin Godeau
56c77707ac
third_party/modules/zig-protobuf: revert indentation changes (#94)
This was breaking old branches that didn't have these indentation
changes
2024-11-19 15:09:22 +01:00
James Deal
0ea2e65419
PJRT: Profiling Output to JSON (#79)
Implements conversion of PJRT protobuf profiling output to JSON format.
2024-11-18 11:32:42 -06:00
Steeve Morin
c65f07a123
zml: hotfixes for #89 (#91)
Due to miscommunication on my part, the tests were _not_ supposed to be
failing!
2024-11-15 18:03:00 +01:00
Steeve Morin
783a3cdeb8
zml: add stdx, rework async signature inference (#89)
This one is a chonky.

This commit adds a new `stdx` module with a few utility functions. It is
very far reaching. Forgive the size.

Also, clean the logs in the executables.
2024-11-15 17:12:07 +01:00
Guillaume Wenzek
28afd46728
zml: small cleanup (#88)
* add more scatterSlices testcases     
* zml: remove helpers.mapTensors in favor of zml.meta.map
* fix shape when for loop is fully unrolled
* zml.Tensor.pad should accept i64 to be compatible with dims
* allow to have array of tensor inside model struct
* split Buffer.asViewOf into asViewOfHostBuffer, asViewOfDeviceBuffer
2024-11-15 14:37:14 +01:00
Guillaume Wenzek
0be024db54
zml.nn: add dynamic sampling, supporting top-k, top-p, min-p settings (#87)
Given the output of the last layer of a LM with a `.voc` axis,
Compute indices for the next tokens, following the given sampling
strategy.
The dynamic sampling strategy is more expressive but top_p requires
computing the softmax.

Options are:

* top_k: only sample among the k top scoring tokens,
* max_top_k: limit a compilation time what is the max possible runtime
value for top_k, saving memory and compute by not having to fully sort
the tokens.
* top_p: only sample among top scoring tokens whose probabilities sum up
to top_p
* min_p: drop tokens whose probabilities are lower than a ratio of the
most likely token
2024-11-15 14:26:10 +01:00
Guillaume Wenzek
282cf58b3c
zml: iota default to .i32 (#86)
It's midly annoying to have to specify iota dtype everywhere to .i32.

In practice if people want to work with floating point range of number
they generally use arange.
Also `iota(x.shape(), 0).convert(.f32)` also work and isn't much work
than saying `iota(x.shape(), .f32, 0)`

For good measure I've added a fallback to .i64 when the given axis has
too many elements. I don't think this would cause much trouble, just
because the tensor that big aren't very common in the first place.
2024-11-15 14:05:36 +01:00
Guillaume Wenzek
ecef330aa1
stablehlo: add DotAlgorithm support to stablehlo.dot_general (#85)
In the API I merged the "precision" and "algorithm" attributes into a
union because according to the spec,
precision must be default when algorithm is set, and precision=default
means "fast" when algorithm is not set.

https://github.com/openxla/stablehlo/blob/main/docs/spec.md#dot_general

For now it's not exposed to users because the API is a bit limited. 
In particular only a subset of algorithm are supported by openxla, and
only a subset of that is supported by chips.

I was able to successfully change the accumulation type for dot on Cuda,
but want to experiment more before deciding what to expose.

---------

Co-authored-by: Corentin Godeau <corentin@zml.ai>
2024-11-15 14:00:18 +01:00
Guillaume Wenzek
d33208a10e
Add custom call support, and tensor.print() (#82)
* restore the integration with PJRT custom call extension
* add a generic `zmlHostBufferCallback` custom call that copies the
received tensor to the host, and call a user supplied callback on it.
* add a default "print" method on `Tensor` that print the runtime value
of a given tensor
* print itself has very little code, it's easy for a user to define
their own print if they don't like ours.

limitations:
* only supported on Cuda atm
* pre allocates a buffer on the host to copy the content of the device
buffer,
this buffer won't be freed. There is exactly one buffer per "print" call
in the IR.
* it uses cuda api, maybe I should be using `createViewOfDeviceBuffer`
to make it less cuda specific
2024-11-15 11:22:01 +01:00
Corentin Godeau
03feafa85a
examples: small cleanup (#84)
Changes were made to `asynk` and not applied everywhere
2024-11-15 10:23:11 +01:00
Guillaume Wenzek
c32c0da46c
pjrtx: error instead of panic (#81)
We currently have an issue with our version of openxla not being able to
serialize the new "batching_dim" attribute of gather/scatter.
This is a version mismatch we should fix, but also it's bound to happen
again, so since the compilation can fail, I'd rather return an error in
that case.
2024-11-12 22:23:49 +01:00
Steeve Morin
fad7cc0876
zml: reintroduce pjrtx (#78)
In an async scenario, it is very easy to footgun yourself and block the
reactor, especially with Events.

Reintroduce pjrtx that deals with the specifics.
2024-11-05 19:25:50 +01:00
Steeve Morin
44d1b24beb
async: add threaded implementation (#77)
This implements the `async` module using a thread pool. This is now the
default.

Implementations can be swapped using the
`@zml//async:impl=[threaded,zigcoro]` flag.

Because io_uring is not enabled in the default seccomp profile, some
container implementations will block on io_uring calls. Use the threaded
implementation to work around this, for now.
2024-11-05 16:47:41 +01:00
Guillaume Wenzek
f661e1a45d
zml/aio: allow metadata to be read from index.json (#76)
For sharded safetensor files, we already have a separate json file we
need to read.
Let's allow writing metadata there, to avoid having a separate json file
with the model config.
2024-11-05 16:45:45 +01:00
Guillaume Wenzek
2febb10baa
zml.tensor: add triangular and toDiagonal (#75)
- triangular: zeroes out the upper right part of a matrix. Options
control how much is actually zeroes with offset from the diagonal.
- toDiagonal: equivalent to pytorch "diag_embed". The base case is
converting a vector into a diagonal matrix. We already have a `diag`
operator doing this, but the name was wrong cause in general `diag` does
the reverse operation of extracting the diagonal from a matrix into a
vector. Also it was using old style ZML, and was a bit too restrictive
on input shapes.
- elu: one more activation function. it's getting a bit out of hand, I've put it under zml.nn.Activation
2024-11-04 15:30:55 +01:00
Guillaume Wenzek
cb0bcf2f7b
zml.tensor: add cumulativeSum, small refactor of maxPoolND (#64)
This PR adds the "cumulative sum" operator.

This is based on what jax does, which is leveraging reduce window
operator,
where for each n in [0..N] you compute the some of elements [0..n] 
I'm not sure how this get optimized in complex programs, to be
investigated.

**BREAKING CHANGES**:
* `reduceWindow` is still pretty hard to used but I merged the
`padding_shape` and `padding_value` into one operator and made the
signature more explicit.
* this also lead me to change a bit how `maxPool1D` and `maxPool2D` take
arguments, using proper tuples instead of slices that need to be of
specific len.
* `pad` now take options using 
     * "tagged syntax":`x.pad(0, .{ .a = .{ .low = 1, .high = -1 }});` 
* or the "AOS syntax": `x.pad(0, &.{.{}, { .low = 1, .high = -1 }});`
* the "SOA syntax" is **REMOVED**: `x.pad(0, .{ .low = &.{1}, .high =
&.{-1} });`
2024-11-04 09:45:32 +01:00
Steeve Morin
09f8506763
runtimes: add intermediate abstraction layer (#74)
Add a tiny abstraction layer between ZML and PJRT.

Most notably, to implement custom logic to decide wether or not we
should/can load a PJRT plugin.

For instance, `libtpu` will block the program trying to reach the GCP
metadata server, which doesn't exist outside GCP.

This also allows for comptime detection of wether a runtime is linked
with the program, and its dependencies.
2024-11-01 18:07:33 +01:00
Steeve Morin
ee6aaea850
zml: configure runfiles env at context start (#73)
If we start bazel built binaries inside a ZML process, the environment
has to be configured properly for the sub processes to find their
runfiles properly.
Do it process wide at context start.
2024-11-01 17:44:47 +01:00
Steeve Morin
a3a5a6336c
zml,pjrt: account for runtimes that not implementing Events (#72)
Some runtimes do not implement `PJRT_Event`. For those, we must dispatch
on our own thread pool for buffer copies and wait on events only if the
runtime returns them.

If there are no events, assume all operations are synchronous.
2024-11-01 17:38:14 +01:00
Steeve Morin
ccd925309b
async: rename async_ to asyncc and add Generic slugs (#71)
`asyncc` is arguably nicer to write than `async_`, so use that.

Also, add an explicit `Generic` version that can work with `anytype`
functions (but on which type inference will be disabled).
2024-11-01 17:36:08 +01:00
Steeve Morin
be510cbb6b
bazel,runtimes/rocm: use dynamic symbol import renaming (#70)
Currently, in ROCm, we use
`__attribute__((ifunc("_zml_rocm_resolve_dlopen")))` to make sure
`dlsym(RTLD_NEXT, "dlopen")` will resolve to the real `dlopen` and not
the always failing stub.

This is a bit brittle.

Instead, leverage `patchelf`'s `rename_dynamic_symbols` feature to
import a custom `dlopen` implementation, and forward to the real one
normally.
2024-11-01 17:33:57 +01:00