for now it's not exposed to users because the API is a bit limited.
in particular only a subset of algorithm are supported by openxla,
and only a subset of that is supported by chips.
I was able to successfully change the accumulation type for dot on Cuda,
but want to experiment more before deciding what to expose.
Some runtimes require additional layers to be built. This change adds
a generic `layers` filegroup to each runtime. It should be embedded into
OCI images.
Notably:
- `--enable-fast-loading-neuron-binaries
- `--optlevel=1`: 30% speed up in compilation, no performance penalty on llama
- NEURON_RT_STOCHASTIC_ROUNDING_EN=1 for more precision
In an async scenario, it is very easy to footgun yourself and block the
reactor, especially with Events.
Reintroduce pjrtx that deals with the specifics.
This implements the `async` module using a thread pool. This is now the
default.
Implementations can be swapped using the
`@zml//async:impl=[threaded,zigcoro]` flag.
Because io_uring is not enabled in the default seccomp profile, some
container implementations will block on io_uring calls. Use the threaded
implementation to work around this, for now.
For sharded safetensor files, we already have a separate json file we
need to read.
Let's allow writing metadata there, to avoid having a separate json file
with the model config.
- triangular: zeroes out the upper right part of a matrix. Options
control how much is actually zeroes with offset from the diagonal.
- toDiagonal: equivalent to pytorch "diag_embed". The base case is
converting a vector into a diagonal matrix. We already have a `diag`
operator doing this, but the name was wrong cause in general `diag` does
the reverse operation of extracting the diagonal from a matrix into a
vector. Also it was using old style ZML, and was a bit too restrictive
on input shapes.
- elu: one more activation function. it's getting a bit out of hand, I've put it under zml.nn.Activation
This PR adds the "cumulative sum" operator.
This is based on what jax does, which is leveraging reduce window
operator,
where for each n in [0..N] you compute the some of elements [0..n]
I'm not sure how this get optimized in complex programs, to be
investigated.
**BREAKING CHANGES**:
* `reduceWindow` is still pretty hard to used but I merged the
`padding_shape` and `padding_value` into one operator and made the
signature more explicit.
* this also lead me to change a bit how `maxPool1D` and `maxPool2D` take
arguments, using proper tuples instead of slices that need to be of
specific len.
* `pad` now take options using
* "tagged syntax":`x.pad(0, .{ .a = .{ .low = 1, .high = -1 }});`
* or the "AOS syntax": `x.pad(0, &.{.{}, { .low = 1, .high = -1 }});`
* the "SOA syntax" is **REMOVED**: `x.pad(0, .{ .low = &.{1}, .high =
&.{-1} });`
Add a tiny abstraction layer between ZML and PJRT.
Most notably, to implement custom logic to decide wether or not we
should/can load a PJRT plugin.
For instance, `libtpu` will block the program trying to reach the GCP
metadata server, which doesn't exist outside GCP.
This also allows for comptime detection of wether a runtime is linked
with the program, and its dependencies.
If we start bazel built binaries inside a ZML process, the environment
has to be configured properly for the sub processes to find their
runfiles properly.
Do it process wide at context start.
Some runtimes do not implement `PJRT_Event`. For those, we must dispatch
on our own thread pool for buffer copies and wait on events only if the
runtime returns them.
If there are no events, assume all operations are synchronous.
`asyncc` is arguably nicer to write than `async_`, so use that.
Also, add an explicit `Generic` version that can work with `anytype`
functions (but on which type inference will be disabled).
Currently, in ROCm, we use
`__attribute__((ifunc("_zml_rocm_resolve_dlopen")))` to make sure
`dlsym(RTLD_NEXT, "dlopen")` will resolve to the real `dlopen` and not
the always failing stub.
This is a bit brittle.
Instead, leverage `patchelf`'s `rename_dynamic_symbols` feature to
import a custom `dlopen` implementation, and forward to the real one
normally.
For `silu`, `logistic` is more precise than `sigmoid` since the first
one is bound to StableHLO while the second one is a custom
implementation. Implemented `logistic` implementation in `sigmoid`
and aliased `logistic` to `sigmoid`.
* add support for `Tensor.concatenate(&.{x, y}, .d)`
* start removing some `broadcastLeft` usage
* use f32 to compute head scaling constant in transformer. We should
always compute constant with max precision.
* use more complicated test cases with long integers
* add tests for protocol 0. Pickle as evolve a lot since protocol 0 was
there,
and it's unlikely we receive such pickle, I thought it was justified
since a big part of the code is related to protocol 0 anyway.
* add tests to the "zipped pickle"
* simplify structs hierarchy -> `PickleData` has been merged with
`Parser` into `torch.File`.
* rename `torch/value.zig` -> `torch/py_object.zig`
* intermediary results: []pickle.Op, []py_object.AnyPy, should not be
stored inside objects but passed around function calls.
* we should use buffered reader to parse the pickle and the zip headers
* python Dict should not be represented by a singleton seq of seq of seq
of kv, just a single seq with keys and values following each other is
enough (like in pickle protocol)
* kwargs should not be mixed with args
* added harder pytorch test with a raw tensor and a `torch.nn.Conv2d`
module
* simplified a bit lifecylcle of `BufferStore` by passing the files
during init.