0
Fork 0
mirror of https://github.com/zml/zml.git synced 2026-11-16 19:11:09 +00:00
Commit graph

159 commits

Author SHA1 Message Date
Corentin Godeau
f79fb574ae zml/ops.zig: Added zml.ops.case operation
This can be used to select which branch will be run at runtime.

It wraps the `stablehlo.case` operation.
2025-02-11 16:51:30 +00:00
Corentin Godeau
a336507857 pjrt+zml: rework profiler imports
Also expose TraceContainer as it can be useful.
2025-02-05 15:24:27 +00:00
Corentin Godeau
c06ba80e7c zml/buffer.zig: added Buffer.fromBytes 2025-02-05 13:31:03 +00:00
Guillaume Wenzek
b9dd101f88 Merge branch 'gw/fix_llama' into gw/dev2
# Conflicts:
#	examples/llama/main.zig
#	zml/aio.zig
#	zml/pjrtx.zig
2025-02-04 17:21:08 +01:00
Guillaume Wenzek
d2aa8c8398 aio.awaitAll 2025-02-04 17:17:05 +01:00
Guillaume Wenzek
44874b3b7a --xla_dump_hlo_as_proto 2025-02-04 16:45:46 +01:00
Corentin Godeau
576130b750 awaitAll 2025-02-04 16:40:36 +01:00
Guillaume Wenzek
7baf092501 examples/llama: read the token generated in prefill 2025-02-04 16:40:36 +01:00
Guillaume Wenzek
6db444bf9a fix partialSoftmax 2025-02-04 15:06:41 +01:00
Guillaume Wenzek
1c9ec996ee fixup! bazel: use prebuilt protoc binaries (#178) 2025-02-04 15:06:37 +01:00
Guillaume Wenzek
90903b7816 fix fromArray 2025-02-04 09:45:16 +01:00
Guillaume Wenzek
87b5722261 zml.buffer.awaitAll 2025-02-03 14:24:35 +01:00
Rene Schallner
a9057737f3
llama: clean up llama3 prompt template encoding. (#172)
There was an unnecessary re-encoding of the newline string, and we added
a newline at the end.
2025-02-03 12:10:19 +01:00
Guillaume Wenzek
f1d0c64db3
Restore tinyllama models (#173)
This is a follow up on #162 

* restore homemade tokenizer, integrate it with the new zml/tokenizer
library. This will only be used for working with tinyllama, I could also
make it work to support GGUF if needed (couple lines of code).
* this allows us to use the original tinyllama files from karpathy repo
which generates actual sentences. This is important when working on
modeling because subtle bugs can degrade the output quality, which is
not detectable if the output is crap to begin with.
* fix casing of zml/tokenizer api: fromFile, tokenToId. Let's stick to
the Zig style guide. Function written in Rust follows rust convention,
but the Zig bindings should follow Zig convention.
* make tokenizer/main.zig a proper tokenizer test tool reusing
llama/test_tokenizer.zig

~~use native_binary instead of cc_binary~~
Reverted because apparently native_binary break the runfiles and
therefore the cuda sandbox
2025-02-03 12:06:33 +01:00
Corentin Kerisit
71fa3764d1
bazel: use prebuilt protoc binaries (#178)
This removes ~1300 build steps by depending on prebuilt binaries.

Note that this PR is blocked by
https://github.com/aspect-build/toolchains_protoc/pull/44 since
`rules_proto` depends on `protobuf@29.1` and
https://github.com/aspect-build/toolchains_protoc doesn't expose
`protoc@29.1` and we can't pin module versions unless adding an entry in
the local BCR which I think is overkill for this.
2025-01-31 23:12:04 +01:00
Steeve Morin
fd4310f418
bazel: fix random huggingface redownloads by using sha256 (#179)
If a file is in Git LFS, use the oid as the sha256 for the download.
2025-01-31 20:41:55 +01:00
Guillaume Wenzek
bc514c191d revert zml: exe should collect events 2025-01-30 18:14:34 +01:00
Guillaume Wenzek
6050d4f032 zml: remove api from Shard 2025-01-29 10:28:55 +01:00
Guillaume Wenzek
01c54e0e72 zml: exe should collect events 2025-01-29 10:28:49 +01:00
Corentin Godeau
d9f2fa8d19 zml/tensor.zig: add maximum as scatter operation 2025-01-28 11:09:01 +00:00
Rene Schallner
edf5d9a93d
llama: fix dims & dtypes -> fix #170 (#171)
This PR fixes issue #170 , making Llama run again on CUDA and RoCM.

- Latest Llama changes had introduced an unnecessary batching dimension:
fixed
- Ops returning indices return i32 yet we now use u32 as token indices
-> dtype conversion requred: fixed

Tested on CUDA.
2025-01-27 17:06:03 +01:00
Steeve Morin
f4eb7594b4 zml/module: disable CUDA libnvptx for older plugin 2025-01-26 12:16:32 +00:00
Steeve Morin
6e3879546b zml/hostbuffer: is buffer is ready, mark it ready 2025-01-26 12:16:04 +00:00
Steeve Morin
39a64d1041 zml/buffer: await constant buffer creation 2025-01-26 12:15:35 +00:00
Steeve Morin
b3fdf197fe zml: add asynchronous HostBuffer event 2025-01-24 21:15:50 +01:00
Steeve Morin
1141fa7f9d zml,mnist: fix async buffers handling 2025-01-24 21:11:03 +01:00
Rene Schallner
9a683a87b2 zml(buffer): fix const bug in toHostAlloc 2025-01-24 21:10:40 +01:00
Steeve Morin
85698b515e zml/exe: don't await for shard after execution 2025-01-24 21:10:36 +01:00
Steeve Morin
2612952a44 zml: add Buffer.Shard type 2025-01-24 21:10:25 +01:00
Steeve Morin
9517504d02 zml/aio: await weights transfers 2025-01-24 21:09:50 +01:00
Rene Schallner
9ca43972d8 llama: fix async toHost usage when fetching token from device. 2025-01-24 21:09:43 +01:00
Steeve Morin
8a5b73e70b zml/aio: wait for buffer readiness in loadBuffers 2025-01-24 21:09:40 +01:00
Steeve Morin
7980fb51ff zml: make buffers and execute asynchronous
This change makes buffer and execution completely asynchronous. This
enable easy pipelining of multiple operations. It doesn't map well to
`async` because the PJRT API is already asynchronous, so might as well
use it as is.
2025-01-24 21:09:12 +01:00
Steeve Morin
d76e5b9cd4
examples: fix logFn usages (#167)
Ooops! Forgot to update the examples.
2025-01-24 21:07:36 +01:00
Steeve Morin
f0c2be0280
async: async mutex on log and fixes on ResetCondition (#166)
Stop using a real mutex for the `logFn` and use an `Mutex` and allow for
a fallback logger if outside a coroutine.
2025-01-24 20:58:14 +01:00
Steeve Morin
8394632b19
runtimes: properly set sonames (#165)
Sometimes we require prebuilt PJRT plugins from outside (neuron,
TPU...), and their `SONAME` are wrong.

Let's ensure all the plugins have their soname proper.
2025-01-24 20:44:41 +01:00
Steeve Morin
a9e17d35fd
runtimes/cuda: downgrade the PJRT plugin back to 0.4.38 (#164)
There is a performance regression on XLA master and 0.5.0.

Get back to 0.4.38 for now.
2025-01-24 19:42:22 +01:00
Rene Schallner
f41e023e22
zml/tokenizer: use HuggingFace Tokenizers and SentencePiece & Llama improvements (#162)
# HF tokenizers & Llama improvements

1. Added HuggingFace tokenizer and SentencePiece bindings, removed old
tokenizers
2. Improved Llama example

## 1. zml/tokenizer: add HuggingFace tokenizers binding, remove old
tokenizers

`zml.Tokenizer` now uses the new HuggingFace tokenizer bindings when
loading a `tokenizer.json`.

To move fast, we dropped the `tinyllama`, `gguf`, and `sentencepiece`
tokenizers.

## 2. examples/llama

Changes to the llama example:

- config.json support -> less cmdline flags
- switched to [zig-clap](https://github.com/Hejsil/zig-clap) for cmdline
args parsing
- using HuggingFace tokenizer
- re-worked and simplified the code
    - single token prediction
- removed esoteric models from BUILD.bazel
- default container now uses Llama-3.2-1B-Instruct

---------

Co-authored-by: Steeve Morin <steeve@zml.ai>
Co-authored-by: Corentin Godeau <corentin.godeau@zml.ai>
Co-authored-by: Corentin Kerisit <corentin.kerisit@gmail.com>
2025-01-24 18:25:57 +01:00
Corentin Kerisit
52c5d4fe02
deps: Upgrade rules_distroless to 0.4.2 (#161)
The reason for the upgrade is because the //:lock target of the apt
repository checks for bazel marker files to write the lockfile at the
right relative path from the root module.

In 0.3.8, only WORKSPACE and WORKSPACE.bazel files were checked as
marker files, and since we are full bzlmod with a single MODULE.bazel
marker file, the run target would fail like so:

```bash
> bazel run @neuron_bookworm//:lock
Writing lockfile to runtimes/neuron/packages.lock.json
Error: neither WORKSPACE nor WORKSPACE.bazel file was found
```

0.4.2 brings that support.

# Tests

I only tested `cd examples; bazel run //mnist:image` to validate that it
still works. Let me know what else is needed.
2025-01-24 13:58:23 +01:00
Guillaume Wenzek
68d0b5b713
zml: misc oneliner fixes (#160)
* tinyllama tokenizer: rewrite "<0x0A>" token to "\n"
* HostBuffer.isContiguous: don't get triggerd by 1 dim axes
* HostBuffer.slice1d: improve error messages
* module.zig: just print where the .mlir file is instead of cluttering
the screen. most of the time the full IR is not very useful anyway
* setFlags: correctly handle comptime int/float
* tokenizer.zig: don't panic when detokenizing high number just use
`<oob>` instead.
* Buffer.constant -> speedup creation up to 2.5G/s on cuda.
2025-01-24 10:20:19 +01:00
Jean-Baptiste Dalido
3682e16da9
workspace: upgrade to Bazel 8 (#143)
This PR covers the migration to Bazel 8.

---------

Co-authored-by: Hugo Mano <hugo@zml.ai>
2025-01-23 18:36:57 +01:00
Hugo Mano
acce597ae9
workspace: bump xla to version 20250122.0-cc075be (#159)
* Bump XLA to 20250122.0-cc075be
* Use nvptx compiler and nvlink with nvjitlink support
* Warn user if cuda path detected in LD_LIBRARY_PATH
* Revert CUDA sandbox fix added here: 6d67fcf and fixed by openxla/xla@a04b262
2025-01-23 16:37:39 +01:00
Guillaume Wenzek
b517170674
Llama3.2: fix tokenizer (#157)
Fix #134 , superseed #133

Basically the llama3 tokenizer.json has the following config: 

```
  "model": {
    "type": "BPE",
    "dropout": null,
    "unk_token": null,
    "continuing_subword_prefix": null,
    "end_of_word_suffix": null,
    "fuse_unk": false,
    "byte_fallback": false,
    ...
}
```

this it technically invalid because it says the model has no byte
fallback and no unknown token either.
This means that unknown emoji can't be converted to bytes nor to the
special unknown token.
This ended up in tokenizer.zig using the value of MAX_INT for the UNK
token then panicking for out of bound.

So I've added some logic to recognize that in the case of GPT2 encoded
vocab (like llama3) that actually contains individual bytes, we should
use the "byte_fallback" algorithm even if disabled.

I've also added a Llama3 "well known" normalizer because HF tokenizers
doesn't add a dummy prefix for llama3 tokenizer.

Technically this is still not a correct implementation of llama3
tokenizer because we don't do the regex based splitting they do. But
even for quite long snippets I tested it now produces exact same tokens.
2025-01-21 13:41:57 +01:00
Steeve Morin
9cc0f96a7a
third_party: bump libxev epoll patch (#156)
This should fix crash and hangs in the epoll and kqueue implementations.
2025-01-20 14:48:29 +01:00
Steeve Morin
bc7fec1261
runtimes/rocm: fix sandboxing and zmlxrocm hooks (#155)
Only the PJRT plugin needs hooking.

Make sure that hipblastlt bytecodes are optional, too.
2025-01-18 00:59:17 +01:00
Steeve Morin
f228ec2edc
async: fix async hangs (#154)
Rework the libxev epoll backend to play nice with zigcoro based
scheduler.

In the process, load the PJRT plugins using `callBlocking` because it
can a bit slow sometimes.

Should have a slight performance increase, too.
2025-01-18 00:50:51 +01:00
Steeve Morin
6d67fcf399
async,runtimes: bump rocm, fixe cuda and rocm sandboxes, bump libxev (#153)
This PR is a collection of fixes related to CUDA and ROCm sandbox
discovery.

In the process, update the epoll libxev patch which caused 100% CPU on
linux.

While fixing CUDA add `xla_gpu_enable_latency_hiding_scheduler` as per
NVIDIA's recommendations.

Finally, bump cuDNN to 9.6.0 while we're here.

---------

Co-authored-by: Corentin Godeau <corentin@zml.ai>
2025-01-16 01:11:36 +01:00
Steeve Morin
5309a25420
async: vendor zigcoro and unify APIs (#152)
This PR merges zigcoro inside the repo and reworks its internals to play
well with `stdx.meta`.

It also adds `Channel.try_{send,recv}` methods as well as dynamically
sized channels (with a `comptime` capacity).

Although not used, it also introduces a `PoolStackAllocator` for
allocating coroutine stacks from a pool
2025-01-14 19:33:16 +01:00
Guillaume Wenzek
ec48976ce6
zml: revamp scatterSlices (#145)
Main issue with current `scatter` implementation is that it uses
broadcasting dims of `stablehlo.scatter`.
While nice in theory, the optimizer doesn't handle them well and they
often are unrolled into while loop.
Here I convert the batching dim to extra iotas indices.
2025-01-14 16:49:39 +01:00
Guillaume Wenzek
52f40144ce
fix #97 (#150)
Before we where using `module.op().writeBytecode(writer)` to compute the
hash of a model
but it crashes on some inputs, notably for unused variables.

So I used the text representation of the mlir.
2025-01-09 17:19:32 +01:00