0
Fork 0
mirror of https://github.com/zml/zml.git synced 2026-11-16 19:11:09 +00:00
Commit graph

40 commits

Author SHA1 Message Date
Guillaume Wenzek
876ca22ff2
add patches to http_deb_archive (#338)
add patches to http_deb_archive
add documentation for command_buffer_compatible
pinToCore
Optional head dim for Llama 3.1
2025-10-15 15:06:58 +02:00
Guillaume Wenzek
1480f77a55
zml: add the platform tag inside the Buffer (#326)
CPU buffers behave differently from Cuda/Rocm/TPU buffers,
and in particular they are addressable from main memory.
This was annoying and prevented us to add some safety checks that
weren't valid for CPU.

I'm adding a tag to buffers so that we know which target they refer to.

Also there is a bug in PJRT unitialized which doesn't respected the
memory. I've opened https://github.com/openxla/xla/pull/31292
but in the meantime I've a workaround in ZML that uses the regular
.fromHostBuffer.
(I've also a patched version of PJRT plugins in
https://github.com/zml/pjrt-artifacts/pull/41 but not merged yet)
2025-09-18 15:35:25 +02:00
Guillaume Wenzek
df77b33d8e
async: remove asynk, asyncc, awaitt, await_ (#325) 2025-09-18 10:05:54 +02:00
Guillaume Wenzek
988f2711c5
zml: loadById (#322)
We already assign a unique tensor id to each tensor object created by a
BufferStore.
But we weren't using the id to load find the corresponding buffer in the
store, instead relying on the name.

This is allow to decouple the shape of the model in the struct vs the
shape of the model inside the safetensor file.
I also moved the id logic out of Tensor and into BufferStore.

I also updated llama/main.zig to use the new logic. 

Bonus:
* fix to `zml.call` tag hashing leading to different MLIR for every
recompile.
* expose  Cpu device count to control the number of CPU "devices"
2025-09-11 10:28:16 +02:00
Steeve Morin
57fd9cda27
workspace: zig 0.15 (#316)
Much of the work was done by @gwenzek

Sadly we drop the torch loader in the process, to be added back later !

---------

Co-authored-by: Guillaume Wenzek <wzk@zml.ai>
2025-08-27 14:11:22 +02:00
Steeve Morin
24575c2232
workspace: switch from zig_cc_binary to zig_binary (#311)
Following the rules_zig bump that adds the `linkmode` setting, we can
use the C linker directly.

We can now remove the `zig_cc_binary` hack.
2025-08-21 13:12:15 +02:00
Steeve Morin
3b81c5630e
zml: remove examples workspace (#308)
Closes #302
2025-08-19 19:42:50 +02:00
Corentin Godeau
5263a170d6
examples/llama: switch to --hf-model-path flag (#270)
Instead of giving config, model weights and tokenizer paths, rely on
`huggingface-cli` download.
2025-08-01 09:46:07 +02:00
Hugo Mano
b4716dba03
ci: revamp commit workflow (#246)
- Use runs-on s3 cache
- Use mlugg/setup-zig@v2 with ZML mirror instead of deprecated and unreachable goto-bus-stop/setup-zig@v2
- Setup Bazel with bazel-contrib/setup-bazel@0.15.0, leverage disk cache and bazelrc flags 
- Leverage Bazel `manual` tag to filter out unneeded targets for `//...`

Co-authored-by: Jean-Baptiste Dalido <jb@zml.ai>
2025-06-11 13:30:34 +02:00
Corentin Godeau
7ac7a7f210
examples/llama: set model.norm.eps correctly (#256) 2025-06-06 18:12:25 +02:00
Guillaume Wenzek
bf79263919
Fix llama3 rope scaling (#230) 2025-05-09 14:32:15 +02:00
Guillaume Wenzek
a85b53fba0
zig 0.14.0 (#189)
Co-authored-by: Hugo Mano <hugo@zml.ai>
2025-03-14 09:49:23 +01:00
Hugo Mano
2fd9516773
workspace: query, build and test (#187)
Ensure that our Bazel setup for the framework and examples work through
queries, builds and tests.
Also fix broken code and deps in the codebase.

As of today only Neuron is broken (cf:
https://github.com/zml/zml/pull/111)
2025-02-12 10:40:20 +01:00
Guillaume Wenzek
5de2adcec3
Fix llama generations (#181)
Current implementation of llama is discarding the first outputted token,
and feeding the last prompt token a second time in the model.

Due to how self attention works this wasn't too disturbing for attention
but it was technically wrong.
Also it was doing a bit extra compute for nothing.

---------

Co-authored-by: Corentin Godeau <corentin@zml.ai>
2025-02-06 14:10:02 +01:00
Rene Schallner
a9057737f3
llama: clean up llama3 prompt template encoding. (#172)
There was an unnecessary re-encoding of the newline string, and we added
a newline at the end.
2025-02-03 12:10:19 +01:00
Guillaume Wenzek
f1d0c64db3
Restore tinyllama models (#173)
This is a follow up on #162 

* restore homemade tokenizer, integrate it with the new zml/tokenizer
library. This will only be used for working with tinyllama, I could also
make it work to support GGUF if needed (couple lines of code).
* this allows us to use the original tinyllama files from karpathy repo
which generates actual sentences. This is important when working on
modeling because subtle bugs can degrade the output quality, which is
not detectable if the output is crap to begin with.
* fix casing of zml/tokenizer api: fromFile, tokenToId. Let's stick to
the Zig style guide. Function written in Rust follows rust convention,
but the Zig bindings should follow Zig convention.
* make tokenizer/main.zig a proper tokenizer test tool reusing
llama/test_tokenizer.zig

~~use native_binary instead of cc_binary~~
Reverted because apparently native_binary break the runfiles and
therefore the cuda sandbox
2025-02-03 12:06:33 +01:00
Rene Schallner
edf5d9a93d
llama: fix dims & dtypes -> fix #170 (#171)
This PR fixes issue #170 , making Llama run again on CUDA and RoCM.

- Latest Llama changes had introduced an unnecessary batching dimension:
fixed
- Ops returning indices return i32 yet we now use u32 as token indices
-> dtype conversion requred: fixed

Tested on CUDA.
2025-01-27 17:06:03 +01:00
Steeve Morin
d76e5b9cd4
examples: fix logFn usages (#167)
Ooops! Forgot to update the examples.
2025-01-24 21:07:36 +01:00
Rene Schallner
f41e023e22
zml/tokenizer: use HuggingFace Tokenizers and SentencePiece & Llama improvements (#162)
# HF tokenizers & Llama improvements

1. Added HuggingFace tokenizer and SentencePiece bindings, removed old
tokenizers
2. Improved Llama example

## 1. zml/tokenizer: add HuggingFace tokenizers binding, remove old
tokenizers

`zml.Tokenizer` now uses the new HuggingFace tokenizer bindings when
loading a `tokenizer.json`.

To move fast, we dropped the `tinyllama`, `gguf`, and `sentencepiece`
tokenizers.

## 2. examples/llama

Changes to the llama example:

- config.json support -> less cmdline flags
- switched to [zig-clap](https://github.com/Hejsil/zig-clap) for cmdline
args parsing
- using HuggingFace tokenizer
- re-worked and simplified the code
    - single token prediction
- removed esoteric models from BUILD.bazel
- default container now uses Llama-3.2-1B-Instruct

---------

Co-authored-by: Steeve Morin <steeve@zml.ai>
Co-authored-by: Corentin Godeau <corentin.godeau@zml.ai>
Co-authored-by: Corentin Kerisit <corentin.kerisit@gmail.com>
2025-01-24 18:25:57 +01:00
Guillaume Wenzek
b517170674
Llama3.2: fix tokenizer (#157)
Fix #134 , superseed #133

Basically the llama3 tokenizer.json has the following config: 

```
  "model": {
    "type": "BPE",
    "dropout": null,
    "unk_token": null,
    "continuing_subword_prefix": null,
    "end_of_word_suffix": null,
    "fuse_unk": false,
    "byte_fallback": false,
    ...
}
```

this it technically invalid because it says the model has no byte
fallback and no unknown token either.
This means that unknown emoji can't be converted to bytes nor to the
special unknown token.
This ended up in tokenizer.zig using the value of MAX_INT for the UNK
token then panicking for out of bound.

So I've added some logic to recognize that in the case of GPT2 encoded
vocab (like llama3) that actually contains individual bytes, we should
use the "byte_fallback" algorithm even if disabled.

I've also added a Llama3 "well known" normalizer because HF tokenizers
doesn't add a dummy prefix for llama3 tokenizer.

Technically this is still not a correct implementation of llama3
tokenizer because we don't do the regex based splitting they do. But
even for quite long snippets I tested it now produces exact same tokens.
2025-01-21 13:41:57 +01:00
Steeve Morin
f228ec2edc
async: fix async hangs (#154)
Rework the libxev epoll backend to play nice with zigcoro based
scheduler.

In the process, load the PJRT plugins using `callBlocking` because it
can a bit slow sometimes.

Should have a slight performance increase, too.
2025-01-18 00:50:51 +01:00
Guillaume Wenzek
b360a92f34
zml: rewrite simple transpose (#142)
Rewrite trivial transpose into reshapes.
Transpose over a 1-d axis are effectively no-op.
Normally the backend can detect this, but we observe that some backend
can get confused. So avoiding those extra transpose can help.

I also changed the default profiler size to 1_000_000 events because
above that the perfetto ui can't load the generated file anyway
2025-01-07 10:56:58 +01:00
Guillaume Wenzek
cf135bd428
zml: Reorganize the compilation output folders (#135)
* create one folder per target, then one sub-folder per mlir module.
* pass this folder to `xla_dump_to` so that all the xla files arrive
there.
* explicitly save the mlir and the serialized executable there

I'm also exposing James profile conversion tool into ZML as
`profiler.dumpDataAsJson(file)`
2024-12-17 16:55:11 +01:00
Guillaume Wenzek
e8fa410537
pjrt/zml: expose client creation flags (#129)
The Pjrt api allows to passe string flags to the plugin on client
creation.
Since we don't like working with strings I've converted the strings
supported by the cuda plugin into a struct.
For now I've focused on the strings corresponding to the allocator
settings following #125 

More settings are not exposed ATM, will follow up with #132 

3d31c48c71/xla/pjrt/c/pjrt_c_api_gpu_internal.cc (L76-L86)

AFAICT the cpu plugin doesn't read flags at all.
Neuron doesn't have known flags, TPU requires more investigation.

**Breaking change**: `context.autoPlatform()` ->
`context.autoPlatform(.{})`
2024-12-11 13:38:04 +01:00
Allison Durham
88e0cc7da9
zml: Llama 3.2 (text only models) support (#117)
llama 3.2 1B and 3B do not have lm_head. 

I've added transposing over the embed_tokens as a replacement and set
lm_head as optional for backward compatibility with the other models.

I also added both repositories+executions to bazel and updated the
README.
2024-12-10 16:30:21 +01:00
Guillaume Wenzek
6578a3d11a
zml: emit func.call operations (#116)
Restore the machinery to emit individual functions and calls to them.

this should unblock composites, and generates cleaner MLIR in general

* functions are cached based on the input shapes including tags.
* donations/tags are propagated through function calls

---------

Co-authored-by: Corentin Godeau <corentin@zml.ai>
2024-12-05 17:45:07 +01:00
Guillaume Wenzek
d669bdc6bf
zml: cleanup module.zig (#115)
* compileInternal now works with function not with "module"
* BaseExe now allocates all the memory needed to call `pjrtExecute`
* `Exe` is the type variant of `BaseExe` and enforce the right Zig arg
are passed to `call`. `exe.prepare(model)` don't need to allocate
anymore, the allocation happened during compilation.
* public entry points like `compile`, `compileModel`, `compileFn` are
now at the top of the file
* the caching logic has been mostly inlined inside `compileInternal`
* `compile`, `compileFn`, `Exe`, `BaseExe` which are the user visible
API have been moved to `exe.zig`
* `CompilationContext` and all the compilation logic stays in
`module.zig`
2024-12-05 17:33:19 +01:00
Corentin Godeau
bed7a57ae4
examples/llama+neuron: enable buffer donation (#112)
Donate verything except the `token_index` buffer as it messes up the
output
2024-12-04 10:51:43 +01:00
Guillaume Wenzek
a6ea0bd905
zml: revisit rope implementation (#114)
The current implementation first compute the full cos/sin matrix for the
full sentence,
then slice into it to extract the positions for the current token.
This is generally optimized by the compiler so it's not a performance
issue, but it's a readability issue,
because it means calling `rope` requires first a call to `ropeCosSin`.

I rewrote this to directly produce the optimized IR where we only
compute rope for given offsets.
2024-12-02 14:51:36 +01:00
Steeve Morin
6f6dfe9fe5
runtimes/cuda: rework sandboxing and dynamic symbol renaming (#109)
This change reworks the sandboxing and dynamic symbol renaming for the
cuda runtime.

Switch to the jax-cuda-pjrt pre-built PJRT plugin until we fix the pjrt
artifacts build. Most notably because our build of the plugin didn't
include the right compute capabilities.

Take the opportunity to bump the `cuda=12.6.2` `cudnn=9.5.1`.

Fixes #19
2024-11-27 19:39:17 +01:00
Steeve Morin
43c4348de9
runtimes/neuron: add support for AWS neuron / trainium / inferentia 2 (#102)
This PR adds support for AWS Neuron / Trainium 1 / Inferentia 2.

It can be enabled via the `--@zml//runtimes:neuron=true` flag.

Everything should work as expected.

---------

Co-authored-by: Corentin Godeau <corentin@zml.ai>
2024-11-26 14:20:47 +01:00
Steeve Morin
db88158518
async: rework async runtime to use coroutines and cross thread signaling (#100)
This mostly reworks the runtime to enable proper cross thread signaling
for `callBlocking`.

API changes include:
- `async_ -> asyncc`
- `await_ -> awaitt`
- proper type inference for non generic arguments even for generic
functions
- bump `libxev`
- use `epoll` on linux by default (can be comptime overridden through
`xev_options.linux_backend` in the root module)

Mostly, also, it's pretty fast.
2024-11-22 17:56:22 +01:00
Steeve Morin
c65f07a123
zml: hotfixes for #89 (#91)
Due to miscommunication on my part, the tests were _not_ supposed to be
failing!
2024-11-15 18:03:00 +01:00
Steeve Morin
783a3cdeb8
zml: add stdx, rework async signature inference (#89)
This one is a chonky.

This commit adds a new `stdx` module with a few utility functions. It is
very far reaching. Forgive the size.

Also, clean the logs in the executables.
2024-11-15 17:12:07 +01:00
Steeve Morin
ccd925309b
async: rename async_ to asyncc and add Generic slugs (#71)
`asyncc` is arguably nicer to write than `async_`, so use that.

Also, add an explicit `Generic` version that can work with `anytype`
functions (but on which type inference will be disabled).
2024-11-01 17:36:08 +01:00
Corentin Godeau
7a35d53ea3
Add Llama 3.1 70B Instruct model (#56)
This can be handy to test beefier models on powerful accelerators
2024-10-17 17:05:24 +02:00
Corentin Godeau
cb31e4a749
Llama weights sharding (#58)
This PR introduces sharding attributes to Llama weights to take
advantage of Tensor Parallelism
2024-10-17 17:05:10 +02:00
Guillaume Wenzek
895492dee4
zml.aio.torch: rename zml.aio.Value -> zml.aio.Metadata (#55)
Part of ongoing improvements to the torch loader

* rename zml.aio.Value -> zml.aio.Metadata
* simplify the types zml.aio.Metadata can have
* pickle `parse` function moved to `pickle.zig`
* add dedicated test case
* fix small bug in parse where `short_binstring` was converted to
`bin_string`
* pickle `evaluate` function now returns `[]const zml.aio.torch.Value`

note: I didn't modified the core logic.
2024-10-17 16:16:05 +02:00
Guillaume Wenzek
493f61d31b
zml/tensor: gatherValues (#12)
gather1d was quite limited by design.
gatherValues is a generalization of gather1d, that handles tuple of
indices and batching.

For each coordinate in `indices`,
`gatherValues` extracts a single value of the given tensor.

* axes_ is a single axis, or a tuple of axis: .b, or .{ .b, .c }
* indices is an integer tensor
* result is a tensor whose shape is similar to the input shape
where the gathered axes have been replaced by axes from 'indices'.

Some example input for the base case where we work on one axis:
- gatherValues(f:[a]->float, .a, ind:[n]->int)[n] == f[ind[n]]
- gatherValues(f:[a, b], .a, ind:[n])[n, b] == f[ind[n], b]
- gatherValues(f: [a,b,c], .{.b}, ind: [n,m])[a, n, m, c] == f[a, ind[n,
m], c]

If an axis in common between `self` and `indices`,
it is treated as a "batching" axis, meaning that semantically
the operator is doing a gatherValues one time per dimension of this
axis:
- gatherValues(f: [a,b,c], .{.b}, ind: [a,n])[a, n] == f[a, ind[a, n]]

It is an error to have an axis present in `self`, `axes_` and `indices`.

If several axes are passed, then the last axis of indices is treated as
coordinates:
- gatherValues(f: [a,b,c], .{.b, .c}, ind: [n,2])[a, n] == f[a,
ind[n][0], ind[n][1]]
- gatherValues(f: [a,b,c,d], .{.b, .c}, ind: [a, n,2])[a, n, d] == f[a,
ind[a, n][0], ind[a, n][1], d]

It is possible to use gatherValues without tags, but batching won't be
available.


> [!WARNING]
> The semantic of inserting the indices axis where we are dropping the
input axis is very useful in practice:
> `roll`, `nearest`, `resize` benefit greatly from it.
> But it doesn't always play nice with batching dimensions.
> In particular the following currently isn't accepted:
> `gatherValues(.{ .{ .a = 10, .b = 20 }, .b, .{ .n = 8, .a = 10 })`
> because MLIR will produce `.{ .n = 8, .a = 10 }` following the indices
shape,
> while we produce `.{.a = 10, n=8}` following the semantic of
"replacing .b with axes from indices".
> If you do `gatherValues(.{ .{ .a = 10, .b = 20 }, .b, .{ .a = 10, .n =
8 })` then both convention agrees and it's a non issue.
> I think this inconsistency is ok for now because it only happens when
you have both batch axes and extra-indices axes which act like second
set of batching dimensions.
> the issue is avoidable from the user side by eg broadcasting the
values along `.n`, or transposing one of the operand.

---------

Signed-off-by: Guillaume Wenzek <wzk@zml.ai>
2024-09-19 16:17:07 +02:00
Steeve Morin
13d84615bb zml: initial release 2024-09-17 11:11:18 +02:00