0
Fork 0
mirror of https://github.com/zml/zml.git synced 2026-11-16 19:11:09 +00:00
Commit graph

1,078 commits

Author SHA1 Message Date
Steeve Morin
adee932efa
zml/io/vfs: add GCS VFS (#453)
This pull request adds support for Google Cloud Storage (GCS) to the
virtual file system (VFS) module, alongside some build configuration
updates. The main changes introduce the new `GCS` backend, make it
available in the VFS API, and register it in the example application.

Authentication is handled like other GCS libraries via the
`GOOGLE_APPLICATION_CREDENTIALS` env or application default account
files.

**IMPORTANT NOTE**: Only Authorized User and Metadata Server auth are
implemented. Service Account auth isn't implemented yet.
2026-04-07 19:05:43 +02:00
elogir
57546912f1
bin/zml-smi: Add CSV export and driver version reporting (#452)
- Restructured `bin/zml-smi/`
- Added CSV export
- Added driver version reporting
- Minor fixes
2026-04-07 16:44:52 +02:00
Benjamin Rabier
448084908a
zml/tokenizer: add IREE tokenizer (#416)
Adding the [IREE
tokenizer](https://github.com/iree-org/iree/blob/main/runtime/src/iree/tokenizer/README.md).Being
made from the ground up with streaming (de-)tokenization in mind, I've
extended the APIs to expose a `std.Io.Writer` for the tokenization.

I've quickly benchmarked IREE's tokenizer vs HuggingFace's one with a
request I retrieved from a coding agent (opencode) in the middle of a
session. The request body was 360KB encoded in 95K tokens. I also did
another test with a similarly sized request with the same results as
shown hereafter.

I've tested two cases:

- oneshot: the template is rendered all at once and only then we encode
the whole string.
- stream: the template rendering and encoding are pipelined to avoid the
intermediate allocation.

There is a slight difference in the generated tokens between HuggingFace
and IREE. Some tokens like `introdu` were encoded as `i`, `n`, `t`, etc.
Decoding them returns the same text. It's very close in terms of total
token count:

- hf: 95779
- iree/stream: 95705
- iree/oneshot: 95587

| Model | IREE avg (oneshot / stream) ms | HuggingFace avg (ms) | Uplift
vs HF (oneshot / stream) |
|---|---:|---:|---|
| liquidai/LFM2.5-1.2B-Instruct | 8.5 / 7.3 | 40.5 | 4.8x / 5.5x |
| Qwen/Qwen3.5-9B | 6.0 / 8.2 | 59.4 | 9.9x / 7.2x |
| mistralai/Ministral-3-3B-Reasoning-2512 | 7.4 / 6.9 | 59.2 | 8x / 8.6x
|


It's unclear why the Qwen's chat template is somehow slower in the
stream case, maybe because there are more writes. Adding non-empty
buffer in the interface helps a bit, but tends to slow down mistral
case. So well

---------

Co-authored-by: Corentin Kerisit <corentin.kerisit@gmail.com>
2026-04-07 16:42:55 +02:00
Benjamin Rabier
70c5d1c588
zml/ops: Stronger typing of custom calls and more code re-use beyond flashattn (#454)
Today, creating a custom call is a bit tedious and error prone. The
primary goal of this PR is to only accept structs as
input/output/attributes and rely more strongly on the names everywhere
when creating as custom call. I've migrated the `fa2` custom to provide
an example. Can migrate the rest.
- Custom call relies on the position of the input/outputs/attributes to
be in the correct order to work. The `output_operand_aliases` also maps
the outputs to the input field position.
- If the custom call is "sharding aware", as in it knows it works on a
single partition on multi devices setup, one must use
`zml.ops.manualComputation` which is a bit tedious to set up.
- `flashattn.zig` has a bunch of logic to register said custom call but
also to map inputs into something more usable which can be shared.
- both `zml.ops.customCall` and `zml.ops.manualComputation` tend to
accept to many different types IMHO which makes it a bit hard to follow.
But well, to be fair not sure how much simpler my code is. I do
constrain myself to only accepting simple structs for the generic types.

I've re-used the existing `manualComputationInternal` and
`customCallInteral` for this new API without touching them. The only
exception is the `output_operand_alises` where I needed a bit more
flexibility.
2026-04-07 15:17:33 +02:00
Steeve Morin
1a9e66c1c3 bazel/stamp: generate .c file and replace genrule by run_binary+tool
This makes it remote build compatible and cross platform compatible.
2026-04-03 15:53:54 +02:00
Steeve Morin
96414d29c4 workspace: add stamping and helpers 2026-04-03 12:55:13 +00:00
Steeve Morin
e69508f364 zml/io/vfs/s3: don't treat 404 as Unexpected
It's completely normal to stat a missing file.
2026-04-03 12:06:18 +00:00
Tristan Gautié
333ec93c47
Fix slice lifetime issue (#451)
We fix a small slice lifetime issue in the prompt template of the Qwen3.5 model in LLM CLI
2026-04-03 13:11:45 +02:00
Steeve Morin
c6931ba2be bazel/patchelf: use --force-rpath in rpath ops
This allows setting DT_RPATH instead of DT_RUNPATH. DT_RPATH takes
precendence to LD_LIBRARY_PATH.

Sometimes LD_LIBRARY_PATH is useful, sometimes it's a pain. Using this
enables us at least to precede it.
2026-04-03 09:18:40 +00:00
Benjamin Rabier
c3c0a69689
ci: update actions (#447)
- Update setup-bazel & checkout to latest
- Add a bunch of `tags = ["manual"]` for docker image related targets
2026-04-01 10:48:10 +02:00
Benjamin Rabier
c817fe94f8
Revert "flashattn: update flash-attention to v2026.3.23" (#446)
This reverts commit 7f71fa58c2.
2026-04-01 10:13:12 +02:00
Hugo Mano
65e9b4487e
examples/llm: replicate Qwen 3.5 KV heads when TP exceeds KV head count (#444)
Handle Qwen3.5 full-attention K/V heads like vLLM when
num_key_value_heads < tp: replicate K/V heads across the model mesh
instead of sharding the KV head axis.

**Motivation**

Qwen3.5 9B has 4 KV heads. On 8-way TP, sharding the KV head axis over
`.model` is invalid because the dimension is not divisible by the
tensor-parallel degree.

This is a placement issue, not a model-shape issue. The model still has
4 logical KV heads, so the right fix is to replicate those heads across
ranks rather than materialize fake extra heads.

This also matches cost modeling better: replication represents
duplicated placement of the same logical data, while repeating would
incorrectly make the tensor look larger than the model actually defines.

**Changes**

- Add a policy helper to detect when KV heads should be replicated
instead of model-sharded.
- Apply that policy to projected K/V tensors in Qwen3.5 self-attention.
- Apply the same policy to KV cache shape partitioning.
- Thread model partition count through the Qwen3.5 inference compilation
path so cache placement follows the same rule.

**Why replicate instead of repeat**

Replicating:

- preserves the logical model shape
- matches grouped-query attention semantics
- keeps cost modeling honest by representing placement, not model
expansion

Repeating:

- would change the graph from 4 KV heads to 8 KV heads
- would increase KV cache size and bandwidth for no new information
- would distort cost modeling by turning a placement choice into fake
tensor growth

**Result**

Qwen3.5 now supports TP setups such as 8 GPUs where KV head count is
smaller than the tensor-parallel degree.
2026-03-31 18:07:02 +02:00
elogir
67b4fbdbb8
bin: Introduce zml-smi (#440)
This PR introduces **zml-smi**, a tool to monitor metrics of any
platform supported by ZML, featuring but not limited to:

- real time `--top` mode
- per process tracking
- responsive layout

---------

Co-authored-by: Steeve Morin <steeve@zml.ai>
2026-03-31 12:47:07 +02:00
Steeve Morin
b675cfbbe3 workspace: bump @llvm=0.7.0
For macOS SDK fix
2026-03-31 10:28:15 +02:00
elogir
4ecdd92cb0
platforms/rocm: merge rocm and mesa amdgpu.ids (#441)
This PR introduces a script to merge `amdgpu.ids` from two sources: Mesa
and ROCm.

Both provide GPU ID-to-name mappings, but each may contain entries the
other lacks. For example, Mesa includes `AMD Radeon 8060S Graphics`
while ROCm does not, and ROCm lists newer GPUs absent from Mesa.

The script merges the two sources together, with ROCm entries taking
priority in case of conflicts.
2026-03-27 14:45:35 +01:00
Hugo Mano
6e7cb90728
zml/platforms/rocm: update to ROCm 7.2.1 with full hermeticity (#439)
Fix ROCm runtime hermeticity in the packaged LLM sandbox by patching transitive ROCm libraries to use zmlxrocm_dlopen and by adding missing libdw/liblzma/libbz2 runtime deps for distroless execution. Also update the ROCm packaging to 7.2.1 and refresh the pinned libpjrt_rocm
2026-03-26 17:52:48 +01:00
Benjamin Rabier
9fa7768808
examples/llm: better single prompt UX + --topk (#438)
- Same UX as the chat
- Add `--topk` argument
2026-03-26 15:58:03 +01:00
Corentin Kerisit
127b195a52
workspace: minor README edits (#432) 2026-03-26 15:47:47 +01:00
Steeve Morin
3828a571bc platforms/neuron: minor neuron fixes 2026-03-25 18:05:23 +00:00
Benjamin Rabier
268ef6e058
examples/llm: fix --help & relative/home paths (#437) 2026-03-25 15:14:22 +01:00
Benjamin Rabier
bba9230f0b
ci: Increase CPU core count on runners (#434)
```
4cpus: 
    total: 2m56s
    ci-cpu: 2m14
    ci-neuron: 2m07s
    ci-tpu: 1m58s
8cpus:
    total: 2m26s
    ci-cpu: 1m37s
    ci-neuron: 1m47s
    ci-tpu: 1m36s
16cpus (mi8-flex):
    total: 2m07s
    ci-cpu: 1m24s
    ci-neuron: 1m26s
    ci-tpu: 1m19s
16cpus (c8i-flex):
    total: 1m58s
    ci-cpu: 1m17s
    ci-neuron: 1m15s
    ci-tpu: 1m17s
32cpus:
    total: 1m48s
    ci-cpu: 1m05s
    ci-neuron: 1m06s
    ci-tpu: 1m02s
```

RunsOn tends to prefer m8i instance (memory-optimzed) rather than c8i
ones for some reason, so forced the family with `c8+c7`, maybe we could
just use `c8`. But not sure if it's worth being that selective. RunsOn
selects the best there is.
2026-03-25 14:01:03 +01:00
Hugo Mano
22db5548ec
zml/examples/llm: add sharding to Qwen 3.5 (#435)
Using tensor parallelism strategy. From 86 toks/s to 119 tok/s for Qwen
3.5 on 2x5090
2026-03-25 13:55:06 +01:00
Benjamin Rabier
0d0d5020cf
example/llm: Further simplifications (#433) 2026-03-25 10:16:26 +01:00
Corentin Kerisit
f17291e7a4
workspace: update readme for zml/v2 (#431) 2026-03-24 18:18:33 +01:00
Benjamin Rabier
8de2c42fe6
examples/llm: Simplifications (#430) 2026-03-24 18:16:25 +01:00
Corentin Godeau
2eb8423e8a
Update doc for V2 (#428) 2026-03-24 17:00:10 +01:00
Benjamin Rabier
81299c67c8
examples/llm: Common model example (#425) 2026-03-24 16:55:29 +01:00
Steeve Morin
140af6716e platforms/cuda: bump PJRT plugin 2026-03-24 15:21:47 +00:00
Steeve Morin
942d6a1f0a Revert "Set xla flags (#429)"
This reverts commit 47c0bdd94d.
2026-03-24 15:11:03 +00:00
Tristan Gautié
47c0bdd94d
Set xla flags (#429)
Fix XLA flags to avoid autotune crash on Nvidia 4090:
- Enable CublasLT on Cuda 
- Disable autotune and triton gemm
2026-03-24 15:57:14 +01:00
Hugo Mano
463ca4a762
zml: add zml.ops.while and zml.nn.GatedDeltaNet (#427) 2026-03-24 14:28:34 +01:00
Corentin Kerisit
7f71fa58c2 flashattn: update flash-attention to v2026.3.23 2026-03-23 19:03:01 +01:00
Hugo Mano
a68f0f9c85
workspace/zls: add non regression test (#426) 2026-03-23 18:33:02 +01:00
Hugo Mano
2f175540bf
ci: rework pipeline + add remote with buildbuddy
Co-authored-by: Benjamin Rabier <benjamin@zml.ai>
2026-03-23 16:07:40 +01:00
Benjamin Rabier
3b76703430
workspace: fixes for build and test //... green
Fixes:
- Added `target_compatible_with = ["@platforms//os:linux"],` for neuron
& rocm
- Updated the zig in neuron
- Updated neuron to use Python 3.12, it's at best a mess to handle two
different python version with `build //...`
- renamed `third_party/flashattn/BUILD.bazel` to
`third_party/flashattn/flashattention.BUILD.bazel` as it's not part of
this module.
- Used HTTPS & fallback URL for swig, sometimes we would end up with an
incorrect archive (wrong sha256)
- Using `--@rules_python//python/config_settings:bootstrap_impl=script`
for hermetic Python build, it relied on the system Python otherwise.
- Used a separate pip extension for neuron to avoid the extra pip args
to leak to other installations.
- Fix `zml/testing` `fn autoCall` function

---------

Co-authored-by: Hugo Mano <hugo@zml.ai>
2026-03-23 14:55:48 +01:00
Hugo Mano
a96b8bd84f platforms/cuda: fix CUDNN redist sha 2026-03-23 13:23:22 +00:00
Corentin Godeau
90fccf5f5a
Proper sharding for paged .cuda_fa3 (#423) 2026-03-23 13:07:42 +01:00
Steeve Morin
5e2bf7fc16 workspace: bump buildifier prebuilt 2026-03-21 17:33:46 +01:00
Steeve Morin
b065d976d3 third_party/zls: fix completion on single dep targets
take the opportunity to bump ZLS to a version compatible with our
current nighly.
2026-03-21 17:33:46 +01:00
Steeve Morin
78546350af stdx: remove meta.Signature
has proven unstable on Zig nightlies and mostly needed since std.Io
anyway.
2026-03-21 17:33:45 +01:00
Steeve Morin
c8e8db75e6 third_party/translate-c: bump translate-c and arocc 2026-03-21 17:33:45 +01:00
Steeve Morin
58b2a4b552 examples/llama: fix bazel_lib and tar imports 2026-03-21 16:21:57 +00:00
Steeve Morin
13c7c4bb33 platforms/rocm: bump PJRT plugin, switch to hipBLASLt instead of triton
faster compile times!

but also +10tok/s on Llama 3.1 8B (125tok/s -> 136tok/s)
2026-03-21 15:30:23 +00:00
Steeve Morin
cb5443e5da platforms/cuda: bump PJRT plugin 2026-03-21 12:15:14 +00:00
Steeve Morin
06f13f9a22 platforms: uncompress http_archives using http_bsdtar_archive
Roughly a 2x speedup according to @cerisier. So take it for what it is.
2026-03-20 16:22:26 +00:00
Benjamin Rabier
a011fb0e52
Green CI (#419)
Co-authored-by: Corentin Godeau <corentin.godeau@zml.ai>
2026-03-20 10:31:57 +01:00
Corentin Godeau
b56e86a8e5 zml/attention/flashattn: enable sharding for Fa3 backend 2026-03-19 18:10:07 +01:00
Kevin Barre
5ce4825cee fix: Correct divCeil usage in fromBackend function for proper type handling 2026-03-19 16:15:40 +00:00
Corentin Godeau
a0ef362b65 zml/attention/paged_attention: replace custom divCeil with std.math.divCeil 2026-03-18 14:59:33 +01:00
Steeve Morin
ac971b4bc6 workspace: fix upb rule loading 2026-03-18 13:03:04 +01:00