This PR adds support for models downloaded via `hf download` without the
`--local-dir` flag.
This is done by keeping the given path instead of getting it from the
`std.Io.File`. The problem with `File.realpath` is that in the case of
models downloaded from HF, the files points to blobs whose name are hex
strings, thus losing the required file extension. Unfortunately, we rely
on the file extension to detect the kind of repository the path
represents.
I also added `//examples/io:playground` to the `//:completion` target so
that we have completion when working on it.
This PR bumps rules_zig back to upstream/main where we now have all the
necessary
features to enable zig local actions while the reste on remote.
It also adds the necessary configurable to actually enable this.
Additionally, since rules_zig now ships ZLS support, we remove zls
entirely from zml.
### How the Bazel resolution works
Bazel resolves execution platforms and toolchains during analysis for
each
target that requires toolchains. If a rule defines exe, resolution
happens separately for the default target execution group and for each
`exec_group`;
actions assigned to an exec group use that group’s selected execution
platform
and toolchains.
Execution platforms come from an ordered set. In this repository’s
remote
configuration, the relevant order is:
1. `//:rbe_platform`, supplied by
`--extra_execution_platforms=@zml//:rbe_platform`
2. `//bazel/zig_toolchain_config:local_execution_platform`, registered
by this PR
3. the host platform, included automatically by Bazel
Earlier platforms are selected when they can satisfy the requested
toolchains.
For each target or exec group, Bazel takes the required toolchain types,
the
target platform, the ordered execution platforms, and the ordered
registered
toolchains. It filters execution platforms by execution constraints,
then tries
to find a matching toolchain for every required toolchain type. A
toolchain
matches when:
- its `exec_compatible_with` constraints are present on the candidate
execution
platform
- its `target_compatible_with` constraints are present on the target
platform
- its `target_settings` match the current configuration
Any execution platform missing a required toolchain is rejected. The
first
remaining execution platform wins, and the first matching toolchain for
each type
is selected.
### How this makes Zig local-only
This PR defines a custom execution constraint:
- `//bazel/zig_toolchain_config:local_zig_execution`
It then defines a host-derived execution platform:
- `//bazel/zig_toolchain_config:local_execution_platform`
That platform inherits the host OS/CPU constraints from
`@platforms//host`, adds
`local_zig_execution`, and sets:
```starlark
exec_properties = {
"no-remote-exec": "1",
}
```
The zig.toolchain(...) module extension call now passes:
extra_exec_compatible_with =
["//bazel/zig_toolchain_config:local_zig_execution"]
rules_zig appends that extra execution constraint to every generated Zig
SDK
toolchain. As a result, when rules_zig resolves
@rules_zig//zig:toolchain_type
for its zig execution group:
- the remote execution platform is considered first, but it does not
have
local_zig_execution, so it cannot match the Zig SDK toolchain
- the custom local execution platform has both the host OS/CPU
constraints and
local_zig_execution, so it matches
- the selected platform carries no-remote-exec, so Zig actions execute
locally
even in remote builds
Other toolchains are not constrained by local_zig_execution, so they can
continue
to resolve to the remote execution platform first. This keeps the build
remote by
default while carving out Zig actions as local-only.
---------
Co-authored-by: Steeve Morin <steeve@zml.ai>
Adding the [IREE
tokenizer](https://github.com/iree-org/iree/blob/main/runtime/src/iree/tokenizer/README.md).Being
made from the ground up with streaming (de-)tokenization in mind, I've
extended the APIs to expose a `std.Io.Writer` for the tokenization.
I've quickly benchmarked IREE's tokenizer vs HuggingFace's one with a
request I retrieved from a coding agent (opencode) in the middle of a
session. The request body was 360KB encoded in 95K tokens. I also did
another test with a similarly sized request with the same results as
shown hereafter.
I've tested two cases:
- oneshot: the template is rendered all at once and only then we encode
the whole string.
- stream: the template rendering and encoding are pipelined to avoid the
intermediate allocation.
There is a slight difference in the generated tokens between HuggingFace
and IREE. Some tokens like `introdu` were encoded as `i`, `n`, `t`, etc.
Decoding them returns the same text. It's very close in terms of total
token count:
- hf: 95779
- iree/stream: 95705
- iree/oneshot: 95587
| Model | IREE avg (oneshot / stream) ms | HuggingFace avg (ms) | Uplift
vs HF (oneshot / stream) |
|---|---:|---:|---|
| liquidai/LFM2.5-1.2B-Instruct | 8.5 / 7.3 | 40.5 | 4.8x / 5.5x |
| Qwen/Qwen3.5-9B | 6.0 / 8.2 | 59.4 | 9.9x / 7.2x |
| mistralai/Ministral-3-3B-Reasoning-2512 | 7.4 / 6.9 | 59.2 | 8x / 8.6x
|
It's unclear why the Qwen's chat template is somehow slower in the
stream case, maybe because there are more writes. Adding non-empty
buffer in the interface helps a bit, but tends to slow down mistral
case. So well
---------
Co-authored-by: Corentin Kerisit <corentin.kerisit@gmail.com>
This allows the ZLS server to work.
Before this fix, `bazel run //:completion.print_build_config` would fail
into
```bash
ERROR: no such package 'examples/transfer-benchmark': BUILD file not found in any of the following directories. Add a BUILD file to a directory to mark it as a package.
- examples/transfer-benchmark
```
This PR uses the new release of rules_zig where we upstreamed all the
work from the zml rules_zig fork.
Not much changes from the POV of the public API except for `copts`
becoming `zigopts`.
Also, the global `c` module is only available to Zig modules that depend
on a non empty `cc_library`.
This is behavior that can probably be modified in `rules_zig` later on.
Finally, ZLS support had to be completely rethought to comply with the
change in shape of ZigModuleInfo providers and subsequent rules.
This PR is blocked until `rules_zig` exposes `ZigModuleInfo` for
`zig_binaries`.
Much of the work was done by @gwenzek
Sadly we drop the torch loader in the process, to be added back later !
---------
Co-authored-by: Guillaume Wenzek <wzk@zml.ai>
Following the new release of the VSCode Zig plugin, rework the code
completion logic to leverage a build runner. This is much lighter,
faster and more reliable.
This mostly reworks the runtime to enable proper cross thread signaling
for `callBlocking`.
API changes include:
- `async_ -> asyncc`
- `await_ -> awaitt`
- proper type inference for non generic arguments even for generic
functions
- bump `libxev`
- use `epoll` on linux by default (can be comptime overridden through
`xev_options.linux_backend` in the root module)
Mostly, also, it's pretty fast.