0
Fork 0
mirror of https://github.com/zml/zml.git synced 2026-11-16 19:11:09 +00:00
zml/bazel
Guillaume Wenzek 8ebed54029
zml: refactor safetensors (#1048)
Refactor `zml/safetensors.zig`

* Drop metadata parsing
* refactor code to be more "object oriented", code is owned by the
`TensorRegistry`
* implement a new format `model.safetensors.full_index.json` that is
similar to `index.json` but actually has data inside it:

```
{
"metadata": {"num_tensors":96085,"total_size":510286023000},
"weight_map": {
  "aligner.w1.bias": {"filename":"model-00001-of-00048.safetensors","shape":[5120],"data_offsets":[27384,37624],"dtype":"BF16"},
  "aligner.w1.weight": {"filename":"model-00001-of-00048.safetensors","shape":[5120,9216],"data_offsets":[37624,94409464],"dtype":"BF16"},
  "aligner.w2.bias": {"filename":"model-00001-of-00048.safetensors","shape":[5120],"data_offsets":[94409464,94419704],"dtype":"BF16"},
  "aligner.w2.weight": {"filename":"model-00001-of-00048.safetensors","shape":[5120,5120],"data_offsets":[94419704,146848504],"dtype":"BF16"},
  "vision.blocks.0.attn.wo.bias": {"filename":"model-00001-of-00048.safetensors","shape":[1024],"data_offsets":[146848504,146850552],"dtype":"BF16"},
  "vision.blocks.0.attn.wo.weight": {"filename":"model-00001-of-00048.safetensors","shape":[1024,1024],"data_offsets":[146850552,148947704],"dtype":"BF16"},
  "vision.blocks.0.attn.wqkv.bias": {"filename":"model-00001-of-00048.safetensors","shape":[3072],"data_offsets":[148947704,148953848],"dtype":"BF16"},
 ...
 ```
 
 I followed conventions of `model.saftensors.index.json`
History 2026-11-12 13:12:25 +01:00
..
stamp bazel/stamp: generate .c file and replace genrule by run_binary+tool 2026-04-03 15:53:54 +02:00
bazel.zig zml: refactor safetensors (#1048) 2026-11-12 13:12:25 +01:00
BUILD.bazel bazel: register runfiles initialization from C (#1005) 2026-11-03 14:45:23 +01:00
cc_import.bzl workspace: bump to bazel 9.1.1 (#601) 2026-06-25 19:37:51 +02:00
dpkg.bzl bazel,runtimes/rocm: use dynamic symbol import renaming (#70) 2024-11-01 17:33:57 +01:00
http_aea_archive.bzl platform/metal: support toolchain extraction on Linux x86_64 (#880) 2026-10-01 18:41:45 +02:00
http_deb_archive.bzl platforms: uncompress http_archives using http_bsdtar_archive 2026-03-20 16:22:26 +00:00
huggingface.bzl bazel: fix random huggingface redownloads by using sha256 (#179) 2025-01-31 20:41:55 +01:00
patchelf.bzl zml/bazel: simplify patchelf execution and fix RPATH removal (#916) 2026-10-20 11:49:41 +02:00
runfiles.bzl runtimes/neuron: add support for AWS neuron / trainium / inferentia 2 (#102) 2024-11-26 14:20:47 +01:00
runfiles_init.c bazel: register runfiles initialization from C (#1005) 2026-11-03 14:45:23 +01:00
simple_repository.bzl runtimes/neuron: sandbox neuron dependencies (#290) 2025-07-25 11:35:04 +02:00
swig.bzl workspace: add several bazel 9 required changed (#530) 2026-05-12 10:27:18 +02:00
zig_index.json zig 0.17 (second attempt) (#923) 2026-10-13 17:11:38 +02:00
zig_options.bzl third_party/zio: bump zio=v0.19.0 2026-10-05 18:11:42 +02:00
zig_proto_library.bzl zml: remove zig-protobuf, switch to upb (#307) 2025-08-19 16:14:54 +02:00
zig_srcs.bzl platforms: uncompress http_archives using http_bsdtar_archive 2026-03-20 16:22:26 +00:00