0
Fork 0
mirror of https://github.com/zml/zml.git synced 2026-11-16 19:11:09 +00:00
Commit graph

42 commits

Author SHA1 Message Date
Rene Schallner
46b5383bc3
WIP cleanups 2025-01-20 15:06:02 +01:00
Rene Schallner
788b72fafa
WIP rene: it generates 2025-01-20 13:57:27 +01:00
Rene Schallner
fc0ec52d5e
WIP rene : rework llama to work more like llmd thx @corendos 2025-01-17 12:43:38 +01:00
Rene Schallner
f327be0091
WIP rene 2025-01-17 12:24:58 +01:00
Steeve Morin
d34fc356fc
WIP 2025-01-17 12:23:03 +01:00
Steeve Morin
30325919bd
WIP 2025-01-17 12:23:03 +01:00
Steeve Morin
e91d6374c1
WIP 2025-01-17 12:23:03 +01:00
Steeve Morin
c4a569cc6f
WIP 2025-01-17 12:23:03 +01:00
Guillaume Wenzek
424bbc5dcb
zml: improve source locations (#148)
Put the name of the operator in the source location.
This seemed redundant at first because it's not needed when looking at
the stablehlo,
but it's nice to have it to understand the optimization passes.

I'm also including a few QoL:

* sdpa don't need a bias argument, attn_mask already play this role.
* gatherSlices should accept a shape literal `.{ .a = 5, .b = 128 }`
* Shape.outer to combine two shapes into the superset of their axes
* Tensor.all (since we already have Tensor.any)
* argMax doesn't need to specify the dtype, it can be inferred from the
size of the input tensors. If the user care about the dtype they can
manually convert.

This PR is stacked on top of #147
2025-01-09 17:18:20 +01:00
Guillaume Wenzek
b360a92f34
zml: rewrite simple transpose (#142)
Rewrite trivial transpose into reshapes.
Transpose over a 1-d axis are effectively no-op.
Normally the backend can detect this, but we observe that some backend
can get confused. So avoiding those extra transpose can help.

I also changed the default profiler size to 1_000_000 events because
above that the perfetto ui can't load the generated file anyway
2025-01-07 10:56:58 +01:00
Hugo Mano
80ec28f3c7
workspace: bump llvm / xla / stablehlo / pjrt plugins xla@20250103.0-5f1fe6a (#141) 2025-01-06 15:06:52 +01:00
Guillaume Wenzek
cf135bd428
zml: Reorganize the compilation output folders (#135)
* create one folder per target, then one sub-folder per mlir module.
* pass this folder to `xla_dump_to` so that all the xla files arrive
there.
* explicitly save the mlir and the serialized executable there

I'm also exposing James profile conversion tool into ZML as
`profiler.dumpDataAsJson(file)`
2024-12-17 16:55:11 +01:00
Steeve Morin
e0041974d2
workspace: rework code completion using a custom build runner (#131)
Following the new release of the VSCode Zig plugin, rework the code
completion logic to leverage a build runner. This is much lighter,
faster and more reliable.
2024-12-16 13:20:27 +01:00
Guillaume Wenzek
e8fa410537
pjrt/zml: expose client creation flags (#129)
The Pjrt api allows to passe string flags to the plugin on client
creation.
Since we don't like working with strings I've converted the strings
supported by the cuda plugin into a struct.
For now I've focused on the strings corresponding to the allocator
settings following #125 

More settings are not exposed ATM, will follow up with #132 

3d31c48c71/xla/pjrt/c/pjrt_c_api_gpu_internal.cc (L76-L86)

AFAICT the cpu plugin doesn't read flags at all.
Neuron doesn't have known flags, TPU requires more investigation.

**Breaking change**: `context.autoPlatform()` ->
`context.autoPlatform(.{})`
2024-12-11 13:38:04 +01:00
Allison Durham
88e0cc7da9
zml: Llama 3.2 (text only models) support (#117)
llama 3.2 1B and 3B do not have lm_head. 

I've added transposing over the embed_tokens as a replacement and set
lm_head as optional for backward compatibility with the other models.

I also added both repositories+executions to bazel and updated the
README.
2024-12-10 16:30:21 +01:00
Guillaume Wenzek
6578a3d11a
zml: emit func.call operations (#116)
Restore the machinery to emit individual functions and calls to them.

this should unblock composites, and generates cleaner MLIR in general

* functions are cached based on the input shapes including tags.
* donations/tags are propagated through function calls

---------

Co-authored-by: Corentin Godeau <corentin@zml.ai>
2024-12-05 17:45:07 +01:00
Guillaume Wenzek
d669bdc6bf
zml: cleanup module.zig (#115)
* compileInternal now works with function not with "module"
* BaseExe now allocates all the memory needed to call `pjrtExecute`
* `Exe` is the type variant of `BaseExe` and enforce the right Zig arg
are passed to `call`. `exe.prepare(model)` don't need to allocate
anymore, the allocation happened during compilation.
* public entry points like `compile`, `compileModel`, `compileFn` are
now at the top of the file
* the caching logic has been mostly inlined inside `compileInternal`
* `compile`, `compileFn`, `Exe`, `BaseExe` which are the user visible
API have been moved to `exe.zig`
* `CompilationContext` and all the compilation logic stays in
`module.zig`
2024-12-05 17:33:19 +01:00
Corentin Godeau
bed7a57ae4
examples/llama+neuron: enable buffer donation (#112)
Donate verything except the `token_index` buffer as it messes up the
output
2024-12-04 10:51:43 +01:00
Guillaume Wenzek
a6ea0bd905
zml: revisit rope implementation (#114)
The current implementation first compute the full cos/sin matrix for the
full sentence,
then slice into it to extract the positions for the current token.
This is generally optimized by the compiler so it's not a performance
issue, but it's a readability issue,
because it means calling `rope` requires first a call to `ropeCosSin`.

I rewrote this to directly produce the optimized IR where we only
compute rope for given offsets.
2024-12-02 14:51:36 +01:00
Guillaume Wenzek
8de3d19bb3
zml: fix float8 <-> float32 conversions, support for Tensor.constant(.{}, .{ .f8 = 1.0}) (#113)
Mostly:
* fix float8 <-> float32 conversions
* support for `Tensor.constant(.{}, .{ .f8 = 1.0})` 

Misc:
* fix small inconsistencies between different versions of sdpa
* better error message for broadcast
* bazelrc: --config=debug
2024-11-29 16:14:41 +01:00
Steeve Morin
6f6dfe9fe5
runtimes/cuda: rework sandboxing and dynamic symbol renaming (#109)
This change reworks the sandboxing and dynamic symbol renaming for the
cuda runtime.

Switch to the jax-cuda-pjrt pre-built PJRT plugin until we fix the pjrt
artifacts build. Most notably because our build of the plugin didn't
include the right compute capabilities.

Take the opportunity to bump the `cuda=12.6.2` `cudnn=9.5.1`.

Fixes #19
2024-11-27 19:39:17 +01:00
Hugo Mano
db55475b17
examples/loader: missing stdx dep (#106) 2024-11-27 17:09:25 +01:00
Steeve Morin
43c4348de9
runtimes/neuron: add support for AWS neuron / trainium / inferentia 2 (#102)
This PR adds support for AWS Neuron / Trainium 1 / Inferentia 2.

It can be enabled via the `--@zml//runtimes:neuron=true` flag.

Everything should work as expected.

---------

Co-authored-by: Corentin Godeau <corentin@zml.ai>
2024-11-26 14:20:47 +01:00
Guillaume Wenzek
09da897fe8
zml: fix memory usage of sdpaMemEfficient (#90)
use `zml.ops.while` instead of `zml.ops.for` the problem with the latter is that it concatenate all intermediary results, and thus using lot of memory unless the concat was optimized away which wasn't always the case
2024-11-26 13:57:27 +01:00
Hugo Mano
d366f00aed
workspace: bump llvm / xla / stablehlo / pjrt plugins and restore disabled tests xla@4663f04 (#98)
- workspace: new module versions added to Bazel registry:
  - deps/xla: 4663f04d26 (20241025.0-4663f04) 
This version introduces an upgraded overlay with Python repository initialization for the TSL extension. The changes were necessitated by this XLA commit: 19b4ecbe98 (diff-df5471accc) , which triggers a repository call. Python's hermetic version is provided via the .bazelrc file.
  - deps/stablehlo: f7f8e4e352 (20241021.0-1c0b606)  
  - deps/llvm: f7f8e4e352 (20241022.0-6c4267f)
 This version includes an upgraded module declaration with rules_python dependency added, as LLVM prepares its overlay for Bazel v8. Relevant LLVM changes here: 3903cb4695
  - deps/runtimes: use PJRT artifacts v3.0.0 (https://github.com/zml/pjrt-artifacts/releases/tag/v3.0.0)
- workspace: align used versions of rules_zig in the Bazel dependency graph
- pjrt: expose getPluginAttributes and getPluginAttribute private APIs and stablehloCurrentVersion public API
- pjrt: updated customCallRegistry function with the new FFI provided by the PJRT API GPU extension 
Minimal support for this feature was added, as this is not the main focus of the current PR. See related changes in XLA here: ef85a7bcc3
- dialects/mlir/stablehlo: expose getCurrentVersion API and fix broken stablehloVersionFromCompatibilityRequirement API
- zml/pjrtx: serialize IR to the PJRT plugin's supported HLO version. If the plugin attribute is unavailable, it falls back to the StableHLO's compatibility requirement version.
- zml/tensor: tests using batching dims for gather and scatter are now re-enabled
2024-11-25 15:53:25 +01:00
Steeve Morin
db88158518
async: rework async runtime to use coroutines and cross thread signaling (#100)
This mostly reworks the runtime to enable proper cross thread signaling
for `callBlocking`.

API changes include:
- `async_ -> asyncc`
- `await_ -> awaitt`
- proper type inference for non generic arguments even for generic
functions
- bump `libxev`
- use `epoll` on linux by default (can be comptime overridden through
`xev_options.linux_backend` in the root module)

Mostly, also, it's pretty fast.
2024-11-22 17:56:22 +01:00
Corentin Godeau
4d1d03cfaa
examples: fixes related to recent async changes (#96)
Recent changes to the `async` module reverted `asyncGeneric` to `asyncc`
but examples were not updated
2024-11-20 15:34:38 +01:00
Steeve Morin
c65f07a123
zml: hotfixes for #89 (#91)
Due to miscommunication on my part, the tests were _not_ supposed to be
failing!
2024-11-15 18:03:00 +01:00
Steeve Morin
783a3cdeb8
zml: add stdx, rework async signature inference (#89)
This one is a chonky.

This commit adds a new `stdx` module with a few utility functions. It is
very far reaching. Forgive the size.

Also, clean the logs in the executables.
2024-11-15 17:12:07 +01:00
Corentin Godeau
03feafa85a
examples: small cleanup (#84)
Changes were made to `asynk` and not applied everywhere
2024-11-15 10:23:11 +01:00
Steeve Morin
ccd925309b
async: rename async_ to asyncc and add Generic slugs (#71)
`asyncc` is arguably nicer to write than `async_`, so use that.

Also, add an explicit `Generic` version that can work with `anytype`
functions (but on which type inference will be disabled).
2024-11-01 17:36:08 +01:00
Steeve Morin
be510cbb6b
bazel,runtimes/rocm: use dynamic symbol import renaming (#70)
Currently, in ROCm, we use
`__attribute__((ifunc("_zml_rocm_resolve_dlopen")))` to make sure
`dlsym(RTLD_NEXT, "dlopen")` will resolve to the real `dlopen` and not
the always failing stub.

This is a bit brittle.

Instead, leverage `patchelf`'s `rename_dynamic_symbols` feature to
import a custom `dlopen` implementation, and forward to the real one
normally.
2024-11-01 17:33:57 +01:00
Corentin Godeau
c61ef6bd1d
Fix LSP in Neovim (#65)
`zig_exe_path` needs to be set for the LSP to work properly
2024-10-23 14:07:48 +02:00
Corentin Godeau
7a35d53ea3
Add Llama 3.1 70B Instruct model (#56)
This can be handy to test beefier models on powerful accelerators
2024-10-17 17:05:24 +02:00
Corentin Godeau
cb31e4a749
Llama weights sharding (#58)
This PR introduces sharding attributes to Llama weights to take
advantage of Tensor Parallelism
2024-10-17 17:05:10 +02:00
Guillaume Wenzek
895492dee4
zml.aio.torch: rename zml.aio.Value -> zml.aio.Metadata (#55)
Part of ongoing improvements to the torch loader

* rename zml.aio.Value -> zml.aio.Metadata
* simplify the types zml.aio.Metadata can have
* pickle `parse` function moved to `pickle.zig`
* add dedicated test case
* fix small bug in parse where `short_binstring` was converted to
`bin_string`
* pickle `evaluate` function now returns `[]const zml.aio.torch.Value`

note: I didn't modified the core logic.
2024-10-17 16:16:05 +02:00
Guillaume Wenzek
164a2f2c70
zml.aio.json: fix loading empty array (#50)
safetensor loader was segfaulting when loading metadata with an empty
json array.

I also rewrote some of the torch loader code to improve the readability.
Notably every slice in `ops.zig` is now const. I've also realized that
our strategy to load Pytorch tensor is not future proof see the warning
at: https://pytorch.org/docs/stable/storage.html
2024-10-11 13:26:04 +02:00
Hugo Mano
f26a022aa2
zml/sharding: user sharding constraints api and hlo integration (#46)
- Introduce sharding hints from the user code to specify how operations
should be split across user's devices mesh leveraging the HLO sharding
annotations
- Remove options arguments in `zml.module` including
`opts.add_donations_attributes`: it's for the function caching because inner function can't do donation, so we don't want to put the donation attributes there but for now we it's removed and we will do it differently next time around
2024-10-10 13:56:49 +02:00
Guillaume Wenzek
2251b1d815
zml: sharding (#26)
First rough support for in-process sharding.
The goal of sharding is to allow leveraging several GPUs to collaborate
to run one big model.

Current implementation:
* added `platform.sharding()` describing how the set of devices managed
by the current process should be used.
* added a `_sharding_info` to shape.
* A tensor / shape axis can be marked as needing sharding using
`.withSharding(.{ .a })` or `.withSharding(.{ 1 })`.
* When using `populateModelWeights` tensors with sharded axes, will be
populated by sharded buffers.
* when generating the MLIR bytecode for a ZML function the **input**
arguments, will see their `_sharding_info` converted into
`mhlo.sharding` attributes
2024-10-01 21:12:25 +02:00
Guillaume Wenzek
1c7d7dacef
workspace: bump xla to enabled batching in gather (#21)
Since we were using an "old" version of XLA we where missing the
batching dims stablehlo attribute.
This was preventing the usage of batching in the gather operator.

Since I had already wrote the batching test before hand and was just
ignoring the compilation error,
this PR just bump xla and remove the error suppression code in tests.
2024-09-24 09:40:55 +02:00
Guillaume Wenzek
493f61d31b
zml/tensor: gatherValues (#12)
gather1d was quite limited by design.
gatherValues is a generalization of gather1d, that handles tuple of
indices and batching.

For each coordinate in `indices`,
`gatherValues` extracts a single value of the given tensor.

* axes_ is a single axis, or a tuple of axis: .b, or .{ .b, .c }
* indices is an integer tensor
* result is a tensor whose shape is similar to the input shape
where the gathered axes have been replaced by axes from 'indices'.

Some example input for the base case where we work on one axis:
- gatherValues(f:[a]->float, .a, ind:[n]->int)[n] == f[ind[n]]
- gatherValues(f:[a, b], .a, ind:[n])[n, b] == f[ind[n], b]
- gatherValues(f: [a,b,c], .{.b}, ind: [n,m])[a, n, m, c] == f[a, ind[n,
m], c]

If an axis in common between `self` and `indices`,
it is treated as a "batching" axis, meaning that semantically
the operator is doing a gatherValues one time per dimension of this
axis:
- gatherValues(f: [a,b,c], .{.b}, ind: [a,n])[a, n] == f[a, ind[a, n]]

It is an error to have an axis present in `self`, `axes_` and `indices`.

If several axes are passed, then the last axis of indices is treated as
coordinates:
- gatherValues(f: [a,b,c], .{.b, .c}, ind: [n,2])[a, n] == f[a,
ind[n][0], ind[n][1]]
- gatherValues(f: [a,b,c,d], .{.b, .c}, ind: [a, n,2])[a, n, d] == f[a,
ind[a, n][0], ind[a, n][1], d]

It is possible to use gatherValues without tags, but batching won't be
available.


> [!WARNING]
> The semantic of inserting the indices axis where we are dropping the
input axis is very useful in practice:
> `roll`, `nearest`, `resize` benefit greatly from it.
> But it doesn't always play nice with batching dimensions.
> In particular the following currently isn't accepted:
> `gatherValues(.{ .{ .a = 10, .b = 20 }, .b, .{ .n = 8, .a = 10 })`
> because MLIR will produce `.{ .n = 8, .a = 10 }` following the indices
shape,
> while we produce `.{.a = 10, n=8}` following the semantic of
"replacing .b with axes from indices".
> If you do `gatherValues(.{ .{ .a = 10, .b = 20 }, .b, .{ .a = 10, .n =
8 })` then both convention agrees and it's a non issue.
> I think this inconsistency is ok for now because it only happens when
you have both batch axes and extra-indices axes which act like second
set of batching dimensions.
> the issue is avoidable from the user side by eg broadcasting the
values along `.n`, or transposing one of the operand.

---------

Signed-off-by: Guillaume Wenzek <wzk@zml.ai>
2024-09-19 16:17:07 +02:00
Steeve Morin
13d84615bb zml: initial release 2024-09-17 11:11:18 +02:00