This PR fixes issue #170 , making Llama run again on CUDA and RoCM.
- Latest Llama changes had introduced an unnecessary batching dimension:
fixed
- Ops returning indices return i32 yet we now use u32 as token indices
-> dtype conversion requred: fixed
Tested on CUDA.
Sometimes we require prebuilt PJRT plugins from outside (neuron,
TPU...), and their `SONAME` are wrong.
Let's ensure all the plugins have their soname proper.
# HF tokenizers & Llama improvements
1. Added HuggingFace tokenizer and SentencePiece bindings, removed old
tokenizers
2. Improved Llama example
## 1. zml/tokenizer: add HuggingFace tokenizers binding, remove old
tokenizers
`zml.Tokenizer` now uses the new HuggingFace tokenizer bindings when
loading a `tokenizer.json`.
To move fast, we dropped the `tinyllama`, `gguf`, and `sentencepiece`
tokenizers.
## 2. examples/llama
Changes to the llama example:
- config.json support -> less cmdline flags
- switched to [zig-clap](https://github.com/Hejsil/zig-clap) for cmdline
args parsing
- using HuggingFace tokenizer
- re-worked and simplified the code
- single token prediction
- removed esoteric models from BUILD.bazel
- default container now uses Llama-3.2-1B-Instruct
---------
Co-authored-by: Steeve Morin <steeve@zml.ai>
Co-authored-by: Corentin Godeau <corentin.godeau@zml.ai>
Co-authored-by: Corentin Kerisit <corentin.kerisit@gmail.com>
The reason for the upgrade is because the //:lock target of the apt
repository checks for bazel marker files to write the lockfile at the
right relative path from the root module.
In 0.3.8, only WORKSPACE and WORKSPACE.bazel files were checked as
marker files, and since we are full bzlmod with a single MODULE.bazel
marker file, the run target would fail like so:
```bash
> bazel run @neuron_bookworm//:lock
Writing lockfile to runtimes/neuron/packages.lock.json
Error: neither WORKSPACE nor WORKSPACE.bazel file was found
```
0.4.2 brings that support.
# Tests
I only tested `cd examples; bazel run //mnist:image` to validate that it
still works. Let me know what else is needed.
* tinyllama tokenizer: rewrite "<0x0A>" token to "\n"
* HostBuffer.isContiguous: don't get triggerd by 1 dim axes
* HostBuffer.slice1d: improve error messages
* module.zig: just print where the .mlir file is instead of cluttering
the screen. most of the time the full IR is not very useful anyway
* setFlags: correctly handle comptime int/float
* tokenizer.zig: don't panic when detokenizing high number just use
`<oob>` instead.
* Buffer.constant -> speedup creation up to 2.5G/s on cuda.
* Bump XLA to 20250122.0-cc075be
* Use nvptx compiler and nvlink with nvjitlink support
* Warn user if cuda path detected in LD_LIBRARY_PATH
* Revert CUDA sandbox fix added here: 6d67fcf and fixed by openxla/xla@a04b262
Fix#134 , superseed #133
Basically the llama3 tokenizer.json has the following config:
```
"model": {
"type": "BPE",
"dropout": null,
"unk_token": null,
"continuing_subword_prefix": null,
"end_of_word_suffix": null,
"fuse_unk": false,
"byte_fallback": false,
...
}
```
this it technically invalid because it says the model has no byte
fallback and no unknown token either.
This means that unknown emoji can't be converted to bytes nor to the
special unknown token.
This ended up in tokenizer.zig using the value of MAX_INT for the UNK
token then panicking for out of bound.
So I've added some logic to recognize that in the case of GPT2 encoded
vocab (like llama3) that actually contains individual bytes, we should
use the "byte_fallback" algorithm even if disabled.
I've also added a Llama3 "well known" normalizer because HF tokenizers
doesn't add a dummy prefix for llama3 tokenizer.
Technically this is still not a correct implementation of llama3
tokenizer because we don't do the regex based splitting they do. But
even for quite long snippets I tested it now produces exact same tokens.
Rework the libxev epoll backend to play nice with zigcoro based
scheduler.
In the process, load the PJRT plugins using `callBlocking` because it
can a bit slow sometimes.
Should have a slight performance increase, too.
This PR is a collection of fixes related to CUDA and ROCm sandbox
discovery.
In the process, update the epoll libxev patch which caused 100% CPU on
linux.
While fixing CUDA add `xla_gpu_enable_latency_hiding_scheduler` as per
NVIDIA's recommendations.
Finally, bump cuDNN to 9.6.0 while we're here.
---------
Co-authored-by: Corentin Godeau <corentin@zml.ai>
This PR merges zigcoro inside the repo and reworks its internals to play
well with `stdx.meta`.
It also adds `Channel.try_{send,recv}` methods as well as dynamically
sized channels (with a `comptime` capacity).
Although not used, it also introduces a `PoolStackAllocator` for
allocating coroutine stacks from a pool
Main issue with current `scatter` implementation is that it uses
broadcasting dims of `stablehlo.scatter`.
While nice in theory, the optimizer doesn't handle them well and they
often are unrolled into while loop.
Here I convert the batching dim to extra iotas indices.
Before we where using `module.op().writeBytecode(writer)` to compute the
hash of a model
but it crashes on some inputs, notably for unused variables.
So I used the text representation of the mlir.
Put the name of the operator in the source location.
This seemed redundant at first because it's not needed when looking at
the stablehlo,
but it's nice to have it to understand the optimization passes.
I'm also including a few QoL:
* sdpa don't need a bias argument, attn_mask already play this role.
* gatherSlices should accept a shape literal `.{ .a = 5, .b = 128 }`
* Shape.outer to combine two shapes into the superset of their axes
* Tensor.all (since we already have Tensor.any)
* argMax doesn't need to specify the dtype, it can be inferred from the
size of the input tensors. If the user care about the dtype they can
manually convert.
This PR is stacked on top of #147
Add pretty printing of HostBuffer.
This will be leverage by the debug helper `x.print()`
It can also be used like this: `std.log.info("my buffer: {}",
.{host_buffer.pretty()})`
Rewrite trivial transpose into reshapes.
Transpose over a 1-d axis are effectively no-op.
Normally the backend can detect this, but we observe that some backend
can get confused. So avoiding those extra transpose can help.
I also changed the default profiler size to 1_000_000 events because
above that the perfetto ui can't load the generated file anyway
* create one folder per target, then one sub-folder per mlir module.
* pass this folder to `xla_dump_to` so that all the xla files arrive
there.
* explicitly save the mlir and the serialized executable there
I'm also exposing James profile conversion tool into ZML as
`profiler.dumpDataAsJson(file)`
Following the new release of the VSCode Zig plugin, rework the code
completion logic to leverage a build runner. This is much lighter,
faster and more reliable.
* add an arena to the CompilationContext
There was already an allocator in the CompilationContext, this PR makes
it into an arena and expose the arena allocator.
This allows ops that need to allocate to do so without abusing
`std.BoundedArray` with arbitrary bounds, and in a safe way.
For now little ops use it, which I believe should stay the case, because
most of the time you want to know how many tensor you're working with.
It'll also simplify the life of Zig new-comers
this include some **breaking changes** to `chunkAllowTrailing` and
`split` that are allocating
* upgrade a few `axis_: i64` into `axis_: anytype` to handle tags
* add a few TODO for the next version of the Tensor API
---------
Signed-off-by: Guillaume Wenzek <wzk@zml.ai>
The Pjrt api allows to passe string flags to the plugin on client
creation.
Since we don't like working with strings I've converted the strings
supported by the cuda plugin into a struct.
For now I've focused on the strings corresponding to the allocator
settings following #125
More settings are not exposed ATM, will follow up with #1323d31c48c71/xla/pjrt/c/pjrt_c_api_gpu_internal.cc (L76-L86)
AFAICT the cpu plugin doesn't read flags at all.
Neuron doesn't have known flags, TPU requires more investigation.
**Breaking change**: `context.autoPlatform()` ->
`context.autoPlatform(.{})`
* testLayer was silently broken by #115 : test + fix so we detect this
in the future.
* mapAlloc had a regression in #115 wrt struct with 0 size fields: test
+ fix
* add Python `venv` to gitignore
Run `bazel build //llama` in CI to make sure we don't accidentally break
it.
I'm also building all the examples target.
For reference it took 5 minutes to run this CI.
llama 3.2 1B and 3B do not have lm_head.
I've added transposing over the embed_tokens as a replacement and set
lm_head as optional for backward compatibility with the other models.
I also added both repositories+executions to bazel and updated the
README.
all code contributed by @steeve
* add intrusive queue
* change the constructor of Channel with default AsyncThread executor
---------
Co-authored-by: Steeve Morin <steeve@zml.ai>
Restore the machinery to emit individual functions and calls to them.
this should unblock composites, and generates cleaner MLIR in general
* functions are cached based on the input shapes including tags.
* donations/tags are propagated through function calls
---------
Co-authored-by: Corentin Godeau <corentin@zml.ai>
* compileInternal now works with function not with "module"
* BaseExe now allocates all the memory needed to call `pjrtExecute`
* `Exe` is the type variant of `BaseExe` and enforce the right Zig arg
are passed to `call`. `exe.prepare(model)` don't need to allocate
anymore, the allocation happened during compilation.
* public entry points like `compile`, `compileModel`, `compileFn` are
now at the top of the file
* the caching logic has been mostly inlined inside `compileInternal`
* `compile`, `compileFn`, `Exe`, `BaseExe` which are the user visible
API have been moved to `exe.zig`
* `CompilationContext` and all the compilation logic stays in
`module.zig`
The current implementation first compute the full cos/sin matrix for the
full sentence,
then slice into it to extract the positions for the current token.
This is generally optimized by the compiler so it's not a performance
issue, but it's a readability issue,
because it means calling `rope` requires first a call to `ropeCosSin`.
I rewrote this to directly produce the optimized IR where we only
compute rope for given offsets.
Mostly:
* fix float8 <-> float32 conversions
* support for `Tensor.constant(.{}, .{ .f8 = 1.0})`
Misc:
* fix small inconsistencies between different versions of sdpa
* better error message for broadcast
* bazelrc: --config=debug
This change reworks the sandboxing and dynamic symbol renaming for the
cuda runtime.
Switch to the jax-cuda-pjrt pre-built PJRT plugin until we fix the pjrt
artifacts build. Most notably because our build of the plugin didn't
include the right compute capabilities.
Take the opportunity to bump the `cuda=12.6.2` `cudnn=9.5.1`.
Fixes#19
Discovered use case:
- PJRT plugin attribute `stablehlo_current_version` is `1.7.8`
- `stablehloVersionFromCompatibilityRequirement(WEEK_12)` is `1.1.0`
- `stablehloGetCurrentVersion` is `1.7.6`
The target version for artifact serialization is `1.7.8` but should be
`1.7.6`.
XLA ref:
2f99455cdf (diff-7f335a7892)
This PR adds support for AWS Neuron / Trainium 1 / Inferentia 2.
It can be enabled via the `--@zml//runtimes:neuron=true` flag.
Everything should work as expected.
---------
Co-authored-by: Corentin Godeau <corentin@zml.ai>