0
Fork 0
mirror of https://github.com/zml/zml.git synced 2026-11-16 19:11:09 +00:00
Any model. Any hardware. Zero compromise. Built with @ziglang / @openxla / MLIR / @bazelbuild https://docs.zml.ai
  • Zig 83.3%
  • C++ 10.5%
  • Starlark 2.8%
  • C 2.5%
  • Python 0.6%
  • Other 0.3%
Find a file
Guillaume Wenzek b2f0e03043
zml/Tensor: explicit sharding in Tensor code (#868)
In Buffer code we already have explicit sharding, but not for tensor.

I'm adding a `zml.Sharding` to `.withPartitioning` to make it explicit
which mesh we are working with.
To simplify the migration, `.withPartitioning` in Tensor code can use
`.model` syntax to lookup a sharding by name.
This replace the previous logic of `selectSharding(shape)` which would
look up a sharding that matches the axes of the given partitioning which
could yield surprising behavior.

This also modifies how sharding are stored in zml.Shape.
Before it was storing a list of axis names + a tag (2 * 8 * 64bits), now
I'm storing a zml.Sharding (64 bits) and a compact PartitionArray (32
bits) that references the logical axis of the sharding by offset instead
of by name.

There is an impact on the `TensorRegistry` because it also needs to be
aware of the possible shardings, instead of delegating the matching
partition spec -> sharding to loading. This means error will be detected
sooner.

`store.createTensor` accepts `@enumLiteral()` to lookup the relevant
sharding.

This also simplifies `zml.Buffer` because the full sharding is now
stored in the shape, so it doesn't need to be passed independently. It
also means that it will propagate from the tensor who are graph inputs,
to the buffers that are eg inputs.

`Placement` logic also got a bit simpler.
History 2026-10-06 15:37:22 +02:00
.github ci: use bazel --experimental_disk_cache_gc_max_size (#824) 2026-09-24 10:31:19 +02:00
.vscode zml: remove hftokenizers (#518) 2026-04-29 18:15:26 +02:00
.zed zml: sharding (#407) 2026-03-10 11:17:57 +01:00
bazel third_party/zio: bump zio=v0.19.0 2026-10-05 18:11:42 +02:00
bin/zml-smi zml/Tensor: explicit sharding in Tensor code (#868) 2026-10-06 15:37:22 +02:00
docs zml/Tensor: explicit sharding in Tensor code (#868) 2026-10-06 15:37:22 +02:00
examples zml/Tensor: explicit sharding in Tensor code (#868) 2026-10-06 15:37:22 +02:00
ffi zml/profiler: post-process saved xplane profiles and clean up profiler FFI (#522) 2026-05-05 14:52:15 +02:00
kernels mlir/dialects: keep pointerType general in LLVM dialect (#889) 2026-10-02 15:45:47 +02:00
mlir mlir/dialects: keep pointerType general in LLVM dialect (#889) 2026-10-02 15:45:47 +02:00
pjrt pjrt,zml/platform: expose devices' NUMA node (#891) 2026-10-05 11:15:57 +02:00
platforms zml/Tensor: explicit sharding in Tensor code (#868) 2026-10-06 15:37:22 +02:00
stdx zml/Tensor: explicit sharding in Tensor code (#868) 2026-10-06 15:37:22 +02:00
third_party third_party/zio: bump zio=v0.19.0 2026-10-05 18:11:42 +02:00
tools zml/Tensor: explicit sharding in Tensor code (#868) 2026-10-06 15:37:22 +02:00
upb ci: rework pipeline + add remote with buildbuddy 2026-03-23 16:07:40 +01:00
vfs zml/vfs: handle invalid GCS json 2026-09-01 11:23:43 +02:00
zml zml/Tensor: explicit sharding in Tensor code (#868) 2026-10-06 15:37:22 +02:00
.bazelignore workspace: remove custom registry (#531) 2026-05-12 10:32:51 +02:00
.bazelrc build: disable default Xcode discovery (#630) 2026-08-03 09:35:09 +09:00
.bazelversion workspace: bump to bazel 9.1.1 (#601) 2026-06-25 19:37:51 +02:00
.editorconfig runtimes/rocm: implement zmlxrocm in Zig (#312) 2025-08-21 18:15:30 +02:00
.gitignore Add devenv files 2026-03-03 17:41:40 +01:00
.nvim.lua nvim: merge configurations table instead of overwriting (#611) 2026-07-19 11:12:00 +02:00
AGENTS.md workspace: update AGENTS.md (#548) 2026-05-25 17:14:28 +02:00
bazel.sh zml: remove examples workspace (#308) 2025-08-19 19:42:50 +02:00
BUILD.bazel zml/safetensors: support hf snapshots (#872) 2026-09-29 18:23:08 +02:00
build.zig zml: remove examples workspace (#308) 2025-08-19 19:42:50 +02:00
CONTRIBUTING.md zml: initial release 2024-09-17 11:11:18 +02:00
devenv.lock zml/Sharding: Rename Placement.Shard to Placement (#549) 2026-05-26 09:29:49 +02:00
devenv.nix examples/llm: Common model example (#425) 2026-03-24 16:55:29 +01:00
devenv.yaml Add devenv files 2026-03-03 17:41:40 +01:00
LICENSE zml: initial release 2024-09-17 11:11:18 +02:00
MODULE.bazel zml/workspace: bump llvm to 0.8.24 (#901) 2026-10-05 17:51:01 +02:00
MODULE.bazel.lock zml/workspace: bump llvm to 0.8.24 (#901) 2026-10-05 17:51:01 +02:00
README.md zml/Tensor: explicit sharding in Tensor code (#868) 2026-10-06 15:37:22 +02:00

About

ZML is a production inference stack, purpose-built to decouple AI workloads from proprietary hardware.

Any model, many hardwares, one codebase, peak performance.

Compiled directly to NVIDIA, AMD, INTEL, TPU, Trainium for peak hardware performance on any accelerator. No rewriting.

It is built using the Zig language, MLIR, and Bazel.

Getting Started

Prerequisites

We use bazel to build ZML and its dependencies. The only prerequisite is bazel, which we recommend installing through bazelisk.

macOS

brew install bazelisk

Linux

curl -L -o /usr/local/bin/bazel 'https://github.com/bazelbuild/bazelisk/releases/download/v1.28.0/bazelisk-linux-amd64'
chmod +x /usr/local/bin/bazel

30-Second Smoke Test

Run the MNIST example:

bazel run //examples/mnist

This downloads a small pretrained MNIST model, compiles it, loads the weights, and classifies a random handwritten digit.

LLM Quickstart

The main LLM example is //examples/llm. It currently supports:

  • Llama 3.1 / 3.2
  • Qwen 3.5
  • LFM 2.5

Authenticate with Hugging Face if you want to load gated repos such as Meta Llama:

bazel run //tools/hf -- auth login

Alternatively, set the HF_TOKEN environment variable.

Then run a prompt directly:

bazel run //examples/llm -- --model=hf://meta-llama/Llama-3.2-1B-Instruct --prompt="What is the capital of France?"

Open the interactive chat loop by omitting --prompt:

bazel run //examples/llm -- --model=hf://meta-llama/Llama-3.2-1B-Instruct

You can also load from:

  • a local directory: --model=/var/models/meta-llama/Llama-3.2-1B-Instruct
  • S3: --model=s3://bucket/path/to/model

Running Models on GPU / TPU

Append one or more platform flags when compiling or running:

  • NVIDIA CUDA: --@zml//platforms:cuda=true
  • AMD RoCM: --@zml//platforms:rocm=true
  • Intel OneAPI: --@zml//platforms:oneapi=true
  • Google TPU: --@zml//platforms:tpu=true
  • AWS Trainium / Inferentia 2: --@zml//platforms:neuron=true
  • Disable CPU compilation: --@zml//platforms:cpu=false

Example on CUDA:

bazel run //examples/llm --@zml//platforms:cuda=true -- --model=hf://meta-llama/Llama-3.2-1B-Instruct --prompt="Write a haiku about Zig"

Example on ROCm:

bazel run //examples/llm --@zml//platforms:rocm=true -- --model=hf://meta-llama/Llama-3.2-1B-Instruct --prompt="Write a haiku about Zig"

Example on Intel OneAPI:

bazel run //examples/llm --@zml//platforms:cpu=false --@zml//platforms:oneapi=true -- --model=hf://meta-llama/Llama-3.2-1B-Instruct --prompt="Write a haiku about Zig"

Run Tests

bazel test //zml:test

Examples

A Taste Of ZML

const Mnist = struct {
    fc1: Layer,
    fc2: Layer,

    const Layer = struct {
        weight: zml.Tensor,
        bias: zml.Tensor,

        pub fn init(store: zml.io.TensorStore.View) Layer {
            return .{
                .weight = store.createReplicatedTensor("weight", .{ .d_out, .d }),
                .bias = store.createReplicatedTensor("bias", .{.d_out}),
            };
        }

        pub fn forward(self: Layer, input: zml.Tensor) zml.Tensor {
            return self.weight.dot(input, .d).add(self.bias).relu().withTags(.{.d});
        }
    };

    pub fn init(store: zml.io.TensorStore.View) Mnist {
        return .{
            .fc1 = .init(store.withPrefix("fc1")),
            .fc2 = .init(store.withPrefix("fc2")),
        };
    }

    pub fn load(
        self: *const Mnist,
        allocator: std.mem.Allocator,
        io: std.Io,
        platform: *const zml.Platform,
        store: *const zml.io.TensorStore,
        meshes: []const *const zml.Mesh,
    ) !zml.Bufferized(Mnist) {
        return zml.io.load(Mnist, self, allocator, io, platform, store, .{
            .meshes = meshes,
            .parallelism = 1,
            .dma_chunks = 1,
            .dma_chunk_size = 16 * 1024 * 1024,
        });
    }

    pub fn unloadBuffers(self: *zml.Bufferized(Mnist)) void {
        self.fc1.weight.deinit();
        self.fc1.bias.deinit();
        self.fc2.weight.deinit();
        self.fc2.bias.deinit();
    }

    /// just two linear layers + relu activation
    pub fn forward(self: Mnist, input: zml.Tensor) zml.Tensor {
        var x = input.flatten().convert(.f32).withTags(.{.d});
        const layers: []const Layer = &.{ self.fc1, self.fc2 };
        for (layers) |layer| {
            x = layer.forward(x);
        }
        return x.argMax(0).indices.convert(.u8);
    }
};

For a full walkthrough, see:

Where To Go Next

Contributing

See here.

License

ZML is licensed under the Apache 2.0 license.

Thanks To Our Contributors