0
Fork 0
mirror of https://github.com/zml/zml.git synced 2026-11-16 19:11:09 +00:00
Any model. Any hardware. Zero compromise. Built with @ziglang / @openxla / MLIR / @bazelbuild https://docs.zml.ai
  • Zig 83.3%
  • C++ 10.5%
  • Starlark 2.8%
  • C 2.5%
  • Python 0.6%
  • Other 0.3%
Find a file
Benjamin Rabier c81078daf1 zml/profiling: preserve binary profiles and validate provider sentinels
PJRT CollectData returns serialized binary data with an explicit byte count.
Unconditionally trimming a trailing NUL can truncate a valid XSpace field,
such as an explicitly encoded zero-valued plane ID, before trace export.
However, XLA's plugin implementation also returns a buffer with one extra
NUL byte, so simply preserving the buffer requires parser compatibility.

Copy the exact returned bytes in both the direct-buffer and two-pass PJRT
paths. At the XSpace boundary, parse the complete buffer first and retry
without one trailing NUL only when the complete buffer fails to parse and
the shortened message is valid. Reject oversized and nonempty null inputs
before calling protobuf, and stop any active local host trace session on
parse failure so that subsequent profiling can start normally.
History 2026-11-11 13:22:52 +01:00
.github ci: add supported platforms with bumped and audited deps (#967) 2026-10-20 14:06:49 +02:00
.helix workspace: add Helix editor configuration for Zig (#1029) 2026-11-06 10:29:31 +01:00
.vscode zml: remove hftokenizers (#518) 2026-04-29 18:15:26 +02:00
.zed zml: sharding (#407) 2026-03-10 11:17:57 +01:00
bazel bazel: register runfiles initialization from C (#1005) 2026-11-03 14:45:23 +01:00
bin/zml-smi zig 0.17 (second attempt) (#923) 2026-10-13 17:11:38 +02:00
docs mlir/dialects: split into one package per dialect and unify them (#982) 2026-10-26 11:41:05 +01:00
dsl zml/kernels: Enable MegaMoe support in llmd (#1032) 2026-11-10 14:23:27 +01:00
examples zml/kernels/moe: make mesh and partitioning explicit (#1028) 2026-11-09 15:26:03 +01:00
ffi zml/profiler: post-process saved xplane profiles and clean up profiler FFI (#522) 2026-05-05 14:52:15 +02:00
mlir dsl/cute: add SM100 primitives and CTA-group support (#1007) 2026-11-03 10:26:31 +01:00
pjrt zml/profiling: preserve binary profiles and validate provider sentinels 2026-11-11 13:22:52 +01:00
platforms zml/kernels/flash_attention/metal: port the kernels to the Metal DSL (#1054) 2026-11-10 18:51:08 +01:00
stdx zml/sharding: use logical meshes (#943) 2026-10-16 15:27:30 +02:00
third_party third_party/zls: allow all 0.17 zig versions (#1036) 2026-11-06 15:14:38 +01:00
tools zml/Tensor: explicit sharding in Tensor code (#868) 2026-10-06 15:37:22 +02:00
upb ci: rework pipeline + add remote with buildbuddy 2026-03-23 16:07:40 +01:00
vfs zig 0.17 (second attempt) (#923) 2026-10-13 17:11:38 +02:00
zml zml/profiling: preserve binary profiles and validate provider sentinels 2026-11-11 13:22:52 +01:00
.bazelignore workspace: remove custom registry (#531) 2026-05-12 10:32:51 +02:00
.bazelrc ci: remote build (#924) 2026-10-20 09:31:03 +02:00
.bazelversion bazel: update to 9.3.0 (#964) 2026-10-20 09:30:12 +02:00
.editorconfig runtimes/rocm: implement zmlxrocm in Zig (#312) 2025-08-21 18:15:30 +02:00
.gitignore Add devenv files 2026-03-03 17:41:40 +01:00
.nvim.lua workspace: fix typo in nvim config (#927) 2026-10-12 17:55:27 +02:00
AGENTS.md zml/Compiler: remove conditionals from command buffer (#913) 2026-10-08 12:33:34 +02:00
bazel.sh zml: remove examples workspace (#308) 2025-08-19 19:42:50 +02:00
BUILD.bazel rbe: use GCR image registry instead of Docker Hub (#962) 2026-10-19 12:45:42 +02:00
build.zig zml: remove examples workspace (#308) 2025-08-19 19:42:50 +02:00
CONTRIBUTING.md zml: initial release 2024-09-17 11:11:18 +02:00
devenv.lock dsl/cute: add SM100 primitives and CTA-group support (#1007) 2026-11-03 10:26:31 +01:00
devenv.nix examples/llm: Common model example (#425) 2026-03-24 16:55:29 +01:00
devenv.yaml Add devenv files 2026-03-03 17:41:40 +01:00
LICENSE zml: initial release 2024-09-17 11:11:18 +02:00
MODULE.bazel workspace: bump ZML plugins and CUDA/ROCm runtimes (#942) 2026-10-16 10:31:34 +02:00
MODULE.bazel.lock third_party/zls: bump to 0.17.0 (#966) 2026-10-20 11:36:07 +02:00
README.md zml/Tensor: explicit sharding in Tensor code (#868) 2026-10-06 15:37:22 +02:00

About

ZML is a production inference stack, purpose-built to decouple AI workloads from proprietary hardware.

Any model, many hardwares, one codebase, peak performance.

Compiled directly to NVIDIA, AMD, INTEL, TPU, Trainium for peak hardware performance on any accelerator. No rewriting.

It is built using the Zig language, MLIR, and Bazel.

Getting Started

Prerequisites

We use bazel to build ZML and its dependencies. The only prerequisite is bazel, which we recommend installing through bazelisk.

macOS

brew install bazelisk

Linux

curl -L -o /usr/local/bin/bazel 'https://github.com/bazelbuild/bazelisk/releases/download/v1.28.0/bazelisk-linux-amd64'
chmod +x /usr/local/bin/bazel

30-Second Smoke Test

Run the MNIST example:

bazel run //examples/mnist

This downloads a small pretrained MNIST model, compiles it, loads the weights, and classifies a random handwritten digit.

LLM Quickstart

The main LLM example is //examples/llm. It currently supports:

  • Llama 3.1 / 3.2
  • Qwen 3.5
  • LFM 2.5

Authenticate with Hugging Face if you want to load gated repos such as Meta Llama:

bazel run //tools/hf -- auth login

Alternatively, set the HF_TOKEN environment variable.

Then run a prompt directly:

bazel run //examples/llm -- --model=hf://meta-llama/Llama-3.2-1B-Instruct --prompt="What is the capital of France?"

Open the interactive chat loop by omitting --prompt:

bazel run //examples/llm -- --model=hf://meta-llama/Llama-3.2-1B-Instruct

You can also load from:

  • a local directory: --model=/var/models/meta-llama/Llama-3.2-1B-Instruct
  • S3: --model=s3://bucket/path/to/model

Running Models on GPU / TPU

Append one or more platform flags when compiling or running:

  • NVIDIA CUDA: --@zml//platforms:cuda=true
  • AMD RoCM: --@zml//platforms:rocm=true
  • Intel OneAPI: --@zml//platforms:oneapi=true
  • Google TPU: --@zml//platforms:tpu=true
  • AWS Trainium / Inferentia 2: --@zml//platforms:neuron=true
  • Disable CPU compilation: --@zml//platforms:cpu=false

Example on CUDA:

bazel run //examples/llm --@zml//platforms:cuda=true -- --model=hf://meta-llama/Llama-3.2-1B-Instruct --prompt="Write a haiku about Zig"

Example on ROCm:

bazel run //examples/llm --@zml//platforms:rocm=true -- --model=hf://meta-llama/Llama-3.2-1B-Instruct --prompt="Write a haiku about Zig"

Example on Intel OneAPI:

bazel run //examples/llm --@zml//platforms:cpu=false --@zml//platforms:oneapi=true -- --model=hf://meta-llama/Llama-3.2-1B-Instruct --prompt="Write a haiku about Zig"

Run Tests

bazel test //zml:test

Examples

A Taste Of ZML

const Mnist = struct {
    fc1: Layer,
    fc2: Layer,

    const Layer = struct {
        weight: zml.Tensor,
        bias: zml.Tensor,

        pub fn init(store: zml.io.TensorStore.View) Layer {
            return .{
                .weight = store.createReplicatedTensor("weight", .{ .d_out, .d }),
                .bias = store.createReplicatedTensor("bias", .{.d_out}),
            };
        }

        pub fn forward(self: Layer, input: zml.Tensor) zml.Tensor {
            return self.weight.dot(input, .d).add(self.bias).relu().withTags(.{.d});
        }
    };

    pub fn init(store: zml.io.TensorStore.View) Mnist {
        return .{
            .fc1 = .init(store.withPrefix("fc1")),
            .fc2 = .init(store.withPrefix("fc2")),
        };
    }

    pub fn load(
        self: *const Mnist,
        allocator: std.mem.Allocator,
        io: std.Io,
        platform: *const zml.Platform,
        store: *const zml.io.TensorStore,
        meshes: []const *const zml.Mesh,
    ) !zml.Bufferized(Mnist) {
        return zml.io.load(Mnist, self, allocator, io, platform, store, .{
            .meshes = meshes,
            .parallelism = 1,
            .dma_chunks = 1,
            .dma_chunk_size = 16 * 1024 * 1024,
        });
    }

    pub fn unloadBuffers(self: *zml.Bufferized(Mnist)) void {
        self.fc1.weight.deinit();
        self.fc1.bias.deinit();
        self.fc2.weight.deinit();
        self.fc2.bias.deinit();
    }

    /// just two linear layers + relu activation
    pub fn forward(self: Mnist, input: zml.Tensor) zml.Tensor {
        var x = input.flatten().convert(.f32).withTags(.{.d});
        const layers: []const Layer = &.{ self.fc1, self.fc2 };
        for (layers) |layer| {
            x = layer.forward(x);
        }
        return x.argMax(0).indices.convert(.u8);
    }
};

For a full walkthrough, see:

Where To Go Next

Contributing

See here.

License

ZML is licensed under the Apache 2.0 license.

Thanks To Our Contributors