`testLayer` was broken because it wasn't directly used by `llama.zig`.
Remove the uneeded "compile_options" argument,
update the testing logic to the new sharded output.
We had issues with the CodeLlama tokenizer.
https://huggingface.co/codellama/CodeLlama-7b-hf
Looking at the config, it's actually a "sentencepiece" tokenizer, that
uses the special "▁" token to represent spaces.
```
"normalizer": {
"type": "Sequence",
"normalizers": [
{
"type": "Prepend",
"prepend": "▁"
},
{
"type": "Replace",
"pattern": {
"String": " "
},
"content": "▁"
}
]
},
```
I've added a `Normalizer.fromHfJson` that reads this json and tries to
convert this style to our options.
This config can express more things than we can, so it's a best effort.
I've also made the magic space token configurable, requiring adding a
small buffer in Normalizer to store it.
There is a small change that can have importance, is that now in most
cases the setting "remove_extra_whitespaces" will not be set, meaning
that user input will be tokenized differently in face of double spaces.
I generally don't think this is a good practice, but it respect HF
defaults.
I've also added a `debug` boolean on the tokenizer struct that shows the
individual merges of tokens.
It helped me previously when debugging tokenizer issues.
Before you needed to explicitly uncomment some code to use it, now it's
behind a runtime boolean.
---------
Co-authored-by: Rene Schallner <rene@zml.ai>
safetensor loader was segfaulting when loading metadata with an empty
json array.
I also rewrote some of the torch loader code to improve the readability.
Notably every slice in `ops.zig` is now const. I've also realized that
our strategy to load Pytorch tensor is not future proof see the warning
at: https://pytorch.org/docs/stable/storage.html
- Introduce sharding hints from the user code to specify how operations
should be split across user's devices mesh leveraging the HLO sharding
annotations
- Remove options arguments in `zml.module` including
`opts.add_donations_attributes`: it's for the function caching because inner function can't do donation, so we don't want to put the donation attributes there but for now we it's removed and we will do it differently next time around
The end goal is to allow zml users to use our async test runner in their
project.
This approach works but is a bit hacky because it relies on the fact
that `test_runner.zig` is part of the `zml` library
otherwise `zig_test` doesn't correctly put the test_runner into the
build sandbox.
I made this dependency explicit, but it was already the case because of
the glob pattern used in zml library.
And I think `test_runner` should be `zig_library` in the first place,
and not a filegroup.
See https://github.com/zml/rules_zig/issues/2 for follow up.
Indicate that it's also the standard location used by the CLI, and add
the environment variable, too.
---------
Co-authored-by: Rene Schallner <rene@renerocks.ai>
pjrtx.zig was mean to be a ZML specific wrapper over pjrt.zig.
But pjrt.zig in its current form is pretty usable without wrapper,
the extra layer of indirection is IMHO confusing because it mostly
transfer the calls directly to pjrt.zig but make some call suspending.
I moved the few helpers left in pjrtx.zig where they are used, ie
`pjrtx.Buffer.bufferFromHostBuffer`-> in `buffer.Buffer.from`
`pjrtx.Client.compile` -> in `module.compileModuleToPjrtExecutable`
`pjrtx.Event.await` -> `zml.Platform.awaitEvent()`
This also fix a bug in `pjrtx.Event.await` which was blocking for the
event before suspending for the event.
Some tokenizers have bytes encoded in hex like this: "<0x40>".
This break the tokenization algorithm because the input text will
contain "@" not "<0x40>",
and if the input contains "<0x40>" it needs to not be treated as a
single byte.
So we replace byte fallbacks strings, by their corresponding character.
This enables the normal tokenization algorithm to work.
This PR also implement the main byte fallback logic, that split unknow
utf8 codepoint into bytes.
---------
Co-authored-by: cryptodeal <james@zml.ai>
First rough support for in-process sharding.
The goal of sharding is to allow leveraging several GPUs to collaborate
to run one big model.
Current implementation:
* added `platform.sharding()` describing how the set of devices managed
by the current process should be used.
* added a `_sharding_info` to shape.
* A tensor / shape axis can be marked as needing sharding using
`.withSharding(.{ .a })` or `.withSharding(.{ 1 })`.
* When using `populateModelWeights` tensors with sharded axes, will be
populated by sharded buffers.
* when generating the MLIR bytecode for a ZML function the **input**
arguments, will see their `_sharding_info` converted into
`mhlo.sharding` attributes
When doing a `max` over a dimension, you have to start with the minimum
value possible, otherwise you might miss the maximum.
Example:
Let's imagine you have a Tensor with the following values: `[-5, -4, -3,
-2, -1]`
If you start with a value of `0`, the `max` in reduce will always return
0. Thus in the end, you get a maximum of `0` whereas `0` is not in the
set of values.
Same for `min`
some dead code or commented out code we should have removed before
release.
copyslice is also broken as per the commented out tests.
it will need to be reimplemented if we need it again.
User should use either `chunkExact` or `chunkAllowTrailing` to clarify
what they want.
previous `chunk` was crashing on `Tensor(.{.a=10}).chunk(.a, 3)`
`chunkAllowTrailing` will return **4** tensors: `.{.a=3}, .{.a=3},
.{.a=3}, .{.a=1}`
Signed-off-by: Guillaume Wenzek <wzk@zml.ai>
Since we were using an "old" version of XLA we where missing the
batching dims stablehlo attribute.
This was preventing the usage of batching in the gather operator.
Since I had already wrote the batching test before hand and was just
ignoring the compilation error,
this PR just bump xla and remove the error suppression code in tests.
This PR restore skipped tests.
It was skipped because we managed to have a CI infrastructure for the
public release. There was a problem regarding computation precision
on ubuntu/cpu platform so it was skipped. It's now fixed using `f32`
instead of `f16` precision so we have lower diffs.
I also added `std.testing.refAllDecls` on each main namespace to ensure
we run every declared tests in `zml/`.
We have to rework about how we want to reference all tests but for now
it does the job.
**Motivation**
In some contexts we want to run only a subset of our test suite.
This allow us to debug and trace more easily what's going on with
failing tests.
Since we maintain our own test runner mainly to run in a async stack,
we can easily filter the list filled by `builtin.test_functions`
**Implementation**
I purpose a simple filtering without flags management, based on the
first process argument.
- `bazel run -c opt //zml:test` will ran the full test suite as before
this pull request.
- `bazel run -c opt //zml:test -- sdpa` or `bazel run -c opt //zml:test
-- "sdpaMemEfficient without mask"` will run
only tests with these words in the function name.
For later, it will gives us the opportunity to have a testing strategy.
Eg: `test "sdpaMemEfficient without mask - [tag1, tag2]" { ...} `
**Caveats**
- While we are able to call a subset of test functions, the tests target
need to compile
every necessary deps and sources, so your code should compile without
failure.
- Maybe we will have to add cumulative arguments to create matrix tests
gather1d was quite limited by design.
gatherValues is a generalization of gather1d, that handles tuple of
indices and batching.
For each coordinate in `indices`,
`gatherValues` extracts a single value of the given tensor.
* axes_ is a single axis, or a tuple of axis: .b, or .{ .b, .c }
* indices is an integer tensor
* result is a tensor whose shape is similar to the input shape
where the gathered axes have been replaced by axes from 'indices'.
Some example input for the base case where we work on one axis:
- gatherValues(f:[a]->float, .a, ind:[n]->int)[n] == f[ind[n]]
- gatherValues(f:[a, b], .a, ind:[n])[n, b] == f[ind[n], b]
- gatherValues(f: [a,b,c], .{.b}, ind: [n,m])[a, n, m, c] == f[a, ind[n,
m], c]
If an axis in common between `self` and `indices`,
it is treated as a "batching" axis, meaning that semantically
the operator is doing a gatherValues one time per dimension of this
axis:
- gatherValues(f: [a,b,c], .{.b}, ind: [a,n])[a, n] == f[a, ind[a, n]]
It is an error to have an axis present in `self`, `axes_` and `indices`.
If several axes are passed, then the last axis of indices is treated as
coordinates:
- gatherValues(f: [a,b,c], .{.b, .c}, ind: [n,2])[a, n] == f[a,
ind[n][0], ind[n][1]]
- gatherValues(f: [a,b,c,d], .{.b, .c}, ind: [a, n,2])[a, n, d] == f[a,
ind[a, n][0], ind[a, n][1], d]
It is possible to use gatherValues without tags, but batching won't be
available.
> [!WARNING]
> The semantic of inserting the indices axis where we are dropping the
input axis is very useful in practice:
> `roll`, `nearest`, `resize` benefit greatly from it.
> But it doesn't always play nice with batching dimensions.
> In particular the following currently isn't accepted:
> `gatherValues(.{ .{ .a = 10, .b = 20 }, .b, .{ .n = 8, .a = 10 })`
> because MLIR will produce `.{ .n = 8, .a = 10 }` following the indices
shape,
> while we produce `.{.a = 10, n=8}` following the semantic of
"replacing .b with axes from indices".
> If you do `gatherValues(.{ .{ .a = 10, .b = 20 }, .b, .{ .a = 10, .n =
8 })` then both convention agrees and it's a non issue.
> I think this inconsistency is ok for now because it only happens when
you have both batch axes and extra-indices axes which act like second
set of batching dimensions.
> the issue is avoidable from the user side by eg broadcasting the
values along `.n`, or transposing one of the operand.
---------
Signed-off-by: Guillaume Wenzek <wzk@zml.ai>
**Changes**
- zml_utils: fix `AttributeError: type object 'ActivationCollector' has
no attribute 'CollectionOver'`
- zml_utils: remove dead code and comments
- docs(howto_torch2zml): add explanation on why `output` variable can be
`None`