0
Fork 0
mirror of https://github.com/zml/zml.git synced 2026-11-16 19:11:09 +00:00
zml/tools
Guillaume Wenzek 8ebed54029
zml: refactor safetensors (#1048)
Refactor `zml/safetensors.zig`

* Drop metadata parsing
* refactor code to be more "object oriented", code is owned by the
`TensorRegistry`
* implement a new format `model.safetensors.full_index.json` that is
similar to `index.json` but actually has data inside it:

```
{
"metadata": {"num_tensors":96085,"total_size":510286023000},
"weight_map": {
  "aligner.w1.bias": {"filename":"model-00001-of-00048.safetensors","shape":[5120],"data_offsets":[27384,37624],"dtype":"BF16"},
  "aligner.w1.weight": {"filename":"model-00001-of-00048.safetensors","shape":[5120,9216],"data_offsets":[37624,94409464],"dtype":"BF16"},
  "aligner.w2.bias": {"filename":"model-00001-of-00048.safetensors","shape":[5120],"data_offsets":[94409464,94419704],"dtype":"BF16"},
  "aligner.w2.weight": {"filename":"model-00001-of-00048.safetensors","shape":[5120,5120],"data_offsets":[94419704,146848504],"dtype":"BF16"},
  "vision.blocks.0.attn.wo.bias": {"filename":"model-00001-of-00048.safetensors","shape":[1024],"data_offsets":[146848504,146850552],"dtype":"BF16"},
  "vision.blocks.0.attn.wo.weight": {"filename":"model-00001-of-00048.safetensors","shape":[1024,1024],"data_offsets":[146850552,148947704],"dtype":"BF16"},
  "vision.blocks.0.attn.wqkv.bias": {"filename":"model-00001-of-00048.safetensors","shape":[3072],"data_offsets":[148947704,148953848],"dtype":"BF16"},
 ...
 ```
 
 I followed conventions of `model.saftensors.index.json`
History 2026-11-12 13:12:25 +01:00
..
hf tools/hf: add hf_transfer implicit dependency (#344) 2025-10-20 12:28:45 +02:00
neuron platforms: bump plugins for CPU, ROCm, CUDA and Metal (#788) 2026-09-10 14:34:40 +00:00
safetensors zml: refactor safetensors (#1048) 2026-11-12 13:12:25 +01:00
xprof //tools/xprof: bump to xprof 2.32.2 (#823) 2026-09-17 10:02:57 +02:00
xspace_to_perfetto platforms: bump cpu + cuda plugins and align xla (#626) 2026-07-03 14:22:23 +02:00
BUILD.bazel zml/Tensor: explicit sharding in Tensor code (#868) 2026-10-06 15:37:22 +02:00
buildifier.sh workspace: rework code completion using a custom build runner (#131) 2024-12-16 13:20:27 +01:00
nsys.sh zml/profiling: unify profiling flow and tracing (#443) 2026-05-11 09:52:11 +02:00
rocprofv3.sh platforms: bump plugins for CPU, ROCm, CUDA and Metal (#788) 2026-09-10 14:34:40 +00:00
workspace_status.sh bin/zml-smi: Add CSV export and driver version reporting (#452) 2026-04-07 16:44:52 +02:00
zls.sh stdx: remove meta.Signature 2026-03-21 17:33:45 +01:00
zml_utils.py examples: gpt oss (#329) 2025-09-25 14:03:37 +02:00