0
Fork 0
mirror of https://github.com/zml/zml.git synced 2026-11-16 19:11:09 +00:00
Commit graph

59 commits

Author SHA1 Message Date
Corentin Kerisit
439ba6c0a6
bazel: register runfiles initialization from C (#1005)
ZML initializes runfiles before `main` so its loaders can locate
packaged libraries and data.

When using `-fno-llvm -fincremental` and linking with the incremental
ELF linker, declaring the registration pointer in Zig produces incorrect
`.init_array` data.

See https://codeberg.org/ziglang/zig/issues/37453

Use a C constructor on Linux and macOS so the C compiler emits the
platform-specific startup registration, while Zig exports the existing
runfiles initialization function directly.
We can revert when https://codeberg.org/ziglang/zig/issues/37453 is
fixed.

With this fix, we bypass the wrong behavior in incremental ELF linker
and get closer to ZML compliance with zig incremental using their
incremental linker.

The registration is shared by the `use_cc_common_link=true` and `false`
paths. Keep the explicit linker retention so the constructor is included
even when its object is in a static archive.
2026-11-03 14:45:23 +01:00
Kevin Barre
e9685a3662
zml/bazel: simplify patchelf execution and fix RPATH removal (#916)
The `@patchelf` version 0.18.0 is more capable and can now construct
something like

```bash
patchelf \
  --add-needed somelib.so \
  --force-rpath \
  --set-rpath '/home/kevin/sdk/lib' \
  --rename-dynamic-symbols patched_tcc.symbols \
  --output patched_tcc/furiosa-tcc \
  somebinary
```

So now we do not need to spawn a shell and do all the cp, chmod, and the
successive patchelf command

Before 

```bash
remove_rpath = ["some/directory"]
```

`--remove-rpath` is a flag that `Removes the entire embedded search
path` and not an individual directory

will contruct something like `patchelf --remove-rpath some/directory
patched.so` but `--remove-rpath` do not take arguments

Switch to `remove_rpath = True` to ensure something like `patchelf
--remove-rpath --output patched.so original.so` the `--remove-rpath` is
now used just after the patchelf to ensure correct usage flag placement.
2026-10-20 11:49:41 +02:00
Guillaume Wenzek
00a0af4a55
zig 0.17 (second attempt) (#923)
Bump for zig 0.17

Deps:
- uucode bumped to
1fb73433bb
- zigimg bumped to
734fa919fb
- libvaxis bumped to
14b7684272
- zio bumped to:
417757935e
- rules_zig upgraded to [version
0.17](https://github.com/hermeticbuild/rules_zig/releases/tag/v0.17.0)
- zls is using a commit from master, but they don't have yet a tagged
release so Codex did a bit of packaging. I confirmed it correctly
autocompletes on my machine
2026-10-13 17:11:38 +02:00
Guillaume Wenzek
303ff3e6a4
Revert "zig 0.17 (#693)" (#909)
This reverts commit 5a18757368.

Our internal repos has a few zig patches, I need to also upgrade them
before we ugprade ZML
2026-10-07 11:34:51 +02:00
Guillaume Wenzek
5a18757368
zig 0.17 (#693)
Bump for zig 0.17

Deps:
- uucode bumped to
1fb73433bb
- zio bumped to:
417757935e
- rules_zig upgraded to [version
0.17](https://github.com/hermeticbuild/rules_zig/releases/tag/v0.17.0)

There is one Codex generated patch:
- zigimg: minor fixes to allow zig 0.17 to compile it, until they
officially support
2026-10-07 11:08:06 +02:00
Steeve Morin
3831492aa0 third_party/zio: bump zio=v0.19.0 2026-10-05 18:11:42 +02:00
Olivier Boulant
8cbd3bd38c
platform/metal: support toolchain extraction on Linux x86_64 (#880)
Even if `exec_compatible_with = ["@platforms//os:macos"],` is removed
from `platforms/metal/BUILD.bazel`, fetching the pinned Metal toolchain
on Linux currently fails:
```
ERROR: /home/oboulant/.cache/bazel/_bazel_oboulant/4ea653214b23f6728bdb9f5b40de2f50/external/zml+/bazel/http_aea_archive.bzl:20:13: An error occurred during the fetch of repository 'zml++http_aea_archive+metal_toolchain':
   Traceback (most recent call last):
        File "/home/oboulant/.cache/bazel/_bazel_oboulant/4ea653214b23f6728bdb9f5b40de2f50/external/zml+/bazel/http_aea_archive.bzl", line 43, column 19, in _http_aea_archive_impl
                _require_macos(rctx)
        File "/home/oboulant/.cache/bazel/_bazel_oboulant/4ea653214b23f6728bdb9f5b40de2f50/external/zml+/bazel/http_aea_archive.bzl", line 20, column 13, in _require_macos
                fail("http_aea_archive is macOS-only; current repository OS is %r" % rctx.os.name)
Error in fail: http_aea_archive is macOS-only; current repository OS is "linux"
```

Add checksum-pinned `ipsw` and `7-Zip` binaries for Linux x86_64.
`bazel/http_aea_archive.bzl` now selects the right binaries according to
running host. It preserves the existing macOS extraction path.
Linux ARM64 remains uncovered.

Validation:

```bash
bazel  build \
  # selects compiler binaries from the extracted external repository \
  # filegroup defined in platforms/metal/metal_toolchain.BUILD.bazel \
  @metal_toolchain//:metal_compile_tools \
  # selects the compiler’s runtime files \
  # filegroup also defined in platforms/metal/metal_toolchain.BUILD.bazel \
  @metal_toolchain//:metal_compile_runtime \
  # copies compiler binaries and runtime files into the layout consumed by ZML \
  //platforms/metal:metal_toolchain \
  # selects macOS ARM64 as the build target \
  --platforms=//platforms:macos_arm64 \
  --lockfile_mode=off
```

---------

Co-authored-by: Corentin Kerisit <corentin.kerisit@gmail.com>
2026-10-01 18:41:45 +02:00
Steeve Morin
9b7d8fb58c bazel: rework zig_options to be config_settting compatible
zig_option can now take a `flag` dependency to expose existing flags or
`bool`, `int or `string` for config_setting flags
2026-09-24 22:21:44 +02:00
Olivier Boulant
1f0c465f9c
bazel: fix runfiles initializer retention on macOS (#829) 2026-09-18 16:27:00 +02:00
Steeve Morin
b31dca5b12
bazel: make lto work again with bazel runfiles init (#822)
in LTO, the linker would garbage collect that exported symbol since it
wouldn't be used inside the final binary

forcing it undefined via a link flag prevents it from being removed.
2026-09-17 23:52:43 +02:00
Steeve Morin
df3a43d29f
bazel: add zig_options rule (#820)
Introduce `zig_option` and `zig_options` rules to generate the bazel
equivalent of option modules plugged on bazel flags.

On `string` flags, if the flag has a `allowed_values`, the `string` is
converted into a `enum` in zig code.

Also, port platforms to this and remove the ugly `defines` hack to
comptime test for platforms enablement.

In the process, port `arocc` to the `zig_options`.
2026-09-16 21:50:22 +02:00
elogir
a4993854da
platforms/metal: sandbox Metal toolchain (#598)
Fetch Apple's encrypted Metal toolchain and only exposes the minimal
compile tools + runtime needed
2026-06-26 14:37:56 +02:00
Corentin Kerisit
aa90f92832
workspace: bump to bazel 9.1.1 (#601)
A few missing loads
Remove all flipped flags or deperacted flags from .bazelrc

Important:
- Keep autoload rules_cc and rules_python because XLA still has
transitive deps that are missing loads, PR in progress.
2026-06-25 19:37:51 +02:00
Corentin Kerisit
31887689a9
workspace: add several bazel 9 required changed (#530)
Just a few stuff to get us closer to Bazel 9
XLA is the main blocker and i'll tackle this as soon as other
dependencies merge my PRs.
2026-05-12 10:27:18 +02:00
Olivier Boulant
df4bb4c3dd
bazel: suppress SWIG c-target warning due to intended behavior (#504)
This suppresses SWIG Warning 524 in `bazel/swig.bzl`. 

The custom SWIG rule intentionally parses C++ interfaces while
generating a C ABI wrapper for Zig consumption (in
[sentencepiece.zig](zml/tokenizer/sentencepiece/sentencepiece.zig)), so
this warning is expected noise from an intentional -c backend selection
rather than a build problem.

**Reproduce**
```
$ bazel build //examples/llm
[...]
SWIG:1: Warning 524: Experimental target language. Target language C specified by -c is an experimental language. See the 'Target Languages' section in the Introduction chapter of the SWIG documentation.
[...]
```

**Validation**
* Built `//examples/llm:llm` after a clean
* Confirmed Warning 524 no longer appears
* Build still succeeds without changing generated binding behavior
2026-04-29 18:24:43 +02:00
Steeve Morin
185e75f884 workspace: bump zig=0.16.0
also bump translate-c and remove a genrule in arocc
2026-04-15 21:42:47 +02:00
KB
8edf02d0c3
zml/chore: bump Zig to 0.16.0-dev.3133 and adapt codebase (#462)
Update Zig compiler to 0.16.0-dev.3133+5ec8e45f3 and adjust code
for breaking standard library changes:

- Replace ArrayListUnmanaged/ArrayList init with .empty
- Switch to unmanaged AutoArrayHashMap/StringArrayHashMap
- Use debug.GeneralPurposeAllocator (renamed upstream)
- Replace std.Io.File.ReadStreamingError with error.EndOfStream
- Refactor meta.visit to use structured visitor pattern
- Fix alignment allocation in mem.bufferizeInner

Also update third-party dependencies: flashattn 0.0.6-rc8, arocc,
uucode, libvaxis, and zigimg patches.

Co-authored-by: Kevin Barre <kevin@zml.ai>
2026-04-13 11:56:32 +02:00
Steeve Morin
1a9e66c1c3 bazel/stamp: generate .c file and replace genrule by run_binary+tool
This makes it remote build compatible and cross platform compatible.
2026-04-03 15:53:54 +02:00
Steeve Morin
96414d29c4 workspace: add stamping and helpers 2026-04-03 12:55:13 +00:00
Steeve Morin
c6931ba2be bazel/patchelf: use --force-rpath in rpath ops
This allows setting DT_RPATH instead of DT_RUNPATH. DT_RPATH takes
precendence to LD_LIBRARY_PATH.

Sometimes LD_LIBRARY_PATH is useful, sometimes it's a pain. Using this
enables us at least to precede it.
2026-04-03 09:18:40 +00:00
Hugo Mano
2f175540bf
ci: rework pipeline + add remote with buildbuddy
Co-authored-by: Benjamin Rabier <benjamin@zml.ai>
2026-03-23 16:07:40 +01:00
Steeve Morin
06f13f9a22 platforms: uncompress http_archives using http_bsdtar_archive
Roughly a 2x speedup according to @cerisier. So take it for what it is.
2026-03-20 16:22:26 +00:00
Benjamin Rabier
a011fb0e52
Green CI (#419)
Co-authored-by: Corentin Godeau <corentin.godeau@zml.ai>
2026-03-20 10:31:57 +01:00
Corentin Godeau
70e4c9953d
Bump zig to 0.16.0-dev.2722+f16eb18ce (#409)
Co-authored-by: Hugo Mano <hugo@zml.ai>
2026-03-10 15:58:48 +01:00
Hugo Mano
abde0a04dd
zml: sharding (#407)
Co-authored-by: Benjamin Rabier <benjamin@zml.ai>
2026-03-10 11:17:57 +01:00
Steeve Morin
dcbf5f2e27 workspace: update to 0.16.0-dev.2611+f996d2866, add remote exec
also:
- use toolchains_llvm_bootstrapped for macOS builds
2026-02-16 12:05:01 +01:00
Steeve Morin
b48c122756 workspace: import cc_* rules from rules_cc
This becomes mandatory starting bazel 9

Also fixed in patchelf=0.18.0.bcr.1
2026-01-27 22:49:06 +00:00
Steeve Morin
1b644887b7 bazel,platforms: centralize runfiles loading
runfiles are process-wide in space and time.
so it makes sense to keep them around

because they are bound to the process, we use the smp_allocator and
except the memory to be kept around during the lifetime of the process.
2026-01-19 18:14:06 +01:00
Steeve Morin
3a94593467 workspace: fix and bump ZLS 2026-01-06 09:05:56 +00:00
Steeve Morin
df09615256 zig bump 2026-01-05 09:09:04 +00:00
Steeve Morin
f3245c5968 MNIST running again 2026-01-01 17:56:29 +00:00
Steeve Morin
db3ded4f80 WIP 2026-01-01 08:49:18 +00:00
Steeve Morin
3fe06706e9 WORKING !!! 2025-12-01 19:17:17 +01:00
Steeve Morin
20747c4dcb WIP 2025-11-26 17:47:50 +01:00
Steeve Morin
42e9f801e6 WIP 2025-11-17 13:51:35 -08:00
Corentin Kerisit
711ed46f5b
workspace: update zig version to 0.15.2 (#349) 2025-10-17 19:05:47 +02:00
Guillaume Wenzek
876ca22ff2
add patches to http_deb_archive (#338)
add patches to http_deb_archive
add documentation for command_buffer_compatible
pinToCore
Optional head dim for Llama 3.1
2025-10-15 15:06:58 +02:00
Steeve Morin
57fd9cda27
workspace: zig 0.15 (#316)
Much of the work was done by @gwenzek

Sadly we drop the torch loader in the process, to be added back later !

---------

Co-authored-by: Guillaume Wenzek <wzk@zml.ai>
2025-08-27 14:11:22 +02:00
Steeve Morin
b0978b4b2d
runtimes/rocm: implement zmlxrocm in Zig (#312)
Also, sandbox `amdgpu.ids` and restore safetensors json parsing.
2025-08-21 18:15:30 +02:00
Steeve Morin
24575c2232
workspace: switch from zig_cc_binary to zig_binary (#311)
Following the rules_zig bump that adds the `linkmode` setting, we can
use the C linker directly.

We can now remove the `zig_cc_binary` hack.
2025-08-21 13:12:15 +02:00
Steeve Morin
2044cd2de5
zml: remove zig-protobuf, switch to upb (#307)
Closes #301
2025-08-19 16:14:54 +02:00
Corentin Kerisit
52d36e1c7e
runtimes/neuron: sandbox neuron dependencies (#290)
The difference with other sandboxing PRs is that we also define a
trampoline PJRT so file which will prove itself useful to circumvent
PJRT version mismatch because of
https://github.com/aws-neuron/aws-neuron-sdk/issues/1095

Additionally, we artificially create a empty repository so that we can
sandbox the distroless deps inside the neuron module extension.

---------

Co-authored-by: EC2 Default User <ec2-user@ip-172-31-64-93.ec2.internal>
2025-07-25 11:35:04 +02:00
Guillaume Wenzek
ee1163a113
Bump rules_zig, add zig_srcs targets, cleanup build (#267)
This is the non-controversial part of #262 

It contains:
* the new `zig_srcs` helper,
* bug fix related to sources in rules_zig,
* cleanups of zml/BUILD.bazel
* test fixes in async/coro.zig (those tests weren't run in CI but now
are)
* disabled nemo and yaml models loader (don't compile anymore since
0.14.0 bump)
2025-07-15 14:18:15 +02:00
Corentin Kerisit
8d0e6e6348
bazel: add rpath manipulation to patchelf (#272) 2025-07-12 20:02:46 +02:00
Hugo Mano
b0168d1c63
workspace: bump xla to version 20250710.0-22ea002 (#271)
Also:
- Bump XLA deps : `com_github_grpc_grpc` and `com_google_protobuf`
- Inject `rules_ml_toolchain`
- Fix `zig_proto_library` rule
2025-07-11 17:45:57 +02:00
Steeve Morin
2502c44c0f
workspace: switch to toolchains_llvm_bootstrapped (#245)
Notably, this will enable using custom linkers or enable LTO.

---------

Co-authored-by: Corentin Kerisit <corentin.kerisit@gmail.com>
2025-06-05 21:45:15 +02:00
Steeve Morin
23c7335592
workspace: bump to zig=0.14.1 (#242)
In the process, add a case where an empty tuple syntax (`&.{}`) was
detected as `*const @TypeOf(.{})`
2025-05-30 16:51:55 +02:00
Steeve Morin
42cf435f48
runtimes/neuron: bump runtime (#241)
In the process, expose the `nrt.h` header to zig.
2025-05-29 23:40:51 +02:00
Steeve Morin
a566e396d2
workspace: add debug/release configurations and various stdx (#234)
This PR bumps rules_zig to add command line to the zig compilation in
order to enable debug and release configs from the bazel command line
(and bazelrc).

Most notably, this is useful for passing `-mcpu` flags to Zig
(especially for SIMD support like AVX512).

While we're there, add `stdx.time.Duration.hz()` and
`stdx.math.roundeven()`.
2025-05-19 21:26:28 +02:00
Steeve Morin
e0336dc9b3
workspace: fetch llvm, stablehlo and triton from xla, bump XLA (#206)
This enables making sure versions are the same across the board.

Also, import `triton` repo for later use.

---------

Co-authored-by: Hugo Mano <hugo@zml.ai>
2025-03-19 10:00:27 +01:00