0
Fork 0
mirror of https://github.com/zml/zml.git synced 2026-11-16 19:11:09 +00:00
Any model. Any hardware. Zero compromise. Built with @ziglang / @openxla / MLIR / @bazelbuild https://docs.zml.ai
  • Zig 83.3%
  • C++ 10.5%
  • Starlark 2.8%
  • C 2.5%
  • Python 0.6%
  • Other 0.3%
Find a file
History 2025-07-17 11:45:04 +02:00
.github strip always on ci 2025-07-17 11:45:04 +02:00
.zed workspace: add Zed settings (#239) 2025-05-29 16:46:44 +02:00
async asyncio: remove usingnamespace 2025-07-15 17:31:54 +02:00
bazel Merge branch 'gw/no_usingnamespace' into gw/build.zig 2025-07-15 17:38:20 +02:00
docs workspace: add debug/release configurations and various stdx (#234) 2025-05-19 21:26:28 +02:00
examples examples/mnist/build.zig 2025-06-20 12:11:38 +02:00
ffi zml/tokenizer: use HuggingFace Tokenizers and SentencePiece & Llama improvements (#162) 2025-01-24 18:25:57 +01:00
mlir Bump rules_zig, add zig_srcs targets, cleanup build (#267) 2025-07-15 14:18:15 +02:00
pjrt Bump rules_zig, add zig_srcs targets, cleanup build (#267) 2025-07-15 14:18:15 +02:00
platforms workspace: switch to toolchains_llvm_bootstrapped (#245) 2025-06-05 21:45:15 +02:00
runtimes Bump rules_zig, add zig_srcs targets, cleanup build (#267) 2025-07-15 14:18:15 +02:00
stdx Bump rules_zig, add zig_srcs targets, cleanup build (#267) 2025-07-15 14:18:15 +02:00
third_party zig-protobuf: remove meta.FieldType 2025-07-17 11:43:16 +02:00
tools workspace: make zls runner target explicit (#240) 2025-05-29 23:04:20 +02:00
zml remove usingnamespace from libxev and zig-protobuf 2025-07-16 11:33:35 +02:00
.bazelignore zml: initial release 2024-09-17 11:11:18 +02:00
.bazelrc zml: initial release 2024-09-17 11:11:18 +02:00
.bazelrc-examples workspace: rework code completion using a custom build runner (#131) 2024-12-16 13:20:27 +01:00
.bazelrc-zml zml: initial release 2024-09-17 11:11:18 +02:00
.bazelversion zml: initial release 2024-09-17 11:11:18 +02:00
.gitignore Fix test layer (#124) 2024-12-11 11:04:38 +01:00
.nvim.lua zml: initial release 2024-09-17 11:11:18 +02:00
.vscode zml: initial release 2024-09-17 11:11:18 +02:00
BUILD.bazel workspace: make zls runner target explicit (#240) 2025-05-29 23:04:20 +02:00
build.zig -Dzml_deps_tar 2025-07-17 10:23:39 +02:00
build.zig.zon publish bazel files 2025-06-17 17:43:04 +02:00
CONTRIBUTING.md zml: initial release 2024-09-17 11:11:18 +02:00
LICENSE zml: initial release 2024-09-17 11:11:18 +02:00
MODULE.bazel zig-protobuf: remove meta.FieldType 2025-07-17 11:43:16 +02:00
MODULE.bazel.lock workspace: switch to toolchains_llvm_bootstrapped (#245) 2025-06-05 21:45:15 +02:00
platform_mappings zml: initial release 2024-09-17 11:11:18 +02:00
README.md workspace: add debug/release configurations and various stdx (#234) 2025-05-19 21:26:28 +02:00

Bonjour 👋

At ZML, we are creating exciting AI products on top of our high-performance AI inference stack. Our stack is built for production, using the amazing Zig language, MLIR, and the power of Bazel.

Take me straight to getting started or give me a taste 🥐!

 

We're happy to share!

We're very happy to share our inference stack with the World and hope it allows you, too, to build cool and exciting AI projects.

To give you a glimpse of what you can do with ZML, here is an early demo:

It shows a prototype running a LLaMA2 model sharded on 1 NVIDIA RTX 4090, 1 AMD 6800XT, and 1 Google Cloud TPU v2. All accelerators were hosted in different locations, with activations being passed over a VPN.

All processes used the same model code, cross-compiled on a Mac, and copied onto the servers.

For more inspiration, see also the examples below or check out the examples folder.

Getting started

Prerequisites

We use bazel to build ZML and its dependencies. The only prerequisite is bazel, which we recommend to download through bazelisk, a version manager for bazel.

Please note: If you do not wish to install bazel system-wide, we provide examples/bazel.sh which downloads it to your home folder and runs it.

Install Bazel (recommended):

macOS

brew install bazelisk

Linux

curl -L -o /usr/local/bin/bazel 'https://github.com/bazelbuild/bazelisk/releases/download/v1.25.0/bazelisk-linux-amd64'
chmod +x /usr/local/bin/bazel

Run a pre-packaged model

We have implemented a variety of example models in ZML. See our reference implementations in the examples folder.

MNIST

The classic handwritten digits recognition task. The model is tasked to recognize a handwritten digit, which has been converted to a 28x28 pixel monochrome image. Bazel will download a pre-trained model, and the test dataset. The program will load the model, compile it, and classify a randomly picked example from the test dataset.

On the command line:

cd examples
bazel run --config=release //mnist

# or
./bazel.sh run --config=release //mnist

Meta Llama 3.1 8B

This model has restrictions, see here. It requires approval from Meta on Huggingface, which can take a few hours to get granted.

While waiting, you can already generate an access token to log into HuggingFace from bazel; see here.

Once you've been granted access, you're ready to download a gated model like Meta-Llama-3.1-8B-Instruct!

# requires token in $HOME/.cache/huggingface/token, as created by the
# `huggingface-cli login` command, or the `HUGGINGFACE_TOKEN` environment variable.
cd examples
bazel run --config=release //llama:Llama-3.1-8B-Instruct
bazel run --config=release //llama:Llama-3.1-8B-Instruct -- --prompt="What is the capital of France?"

You can also try Llama-3.1-70B-Instruct if you have enough memory.

Meta Llama 3.2 1B

Like the 8B model above, this model also requires approval. See here for access requirements.

cd examples
bazel run --config=release //llama:Llama-3.2-1B-Instruct
bazel run --config=release //llama:Llama-3.2-1B-Instruct -- --prompt="What is the capital of France?"

For a larger 3.2 model, you can also try Llama-3.2-3B-Instruct.

Running Models on GPU / TPU

You can compile models for accelerator runtimes by appending one or more of the following arguments to the command line when compiling / running a model:

  • NVIDIA CUDA: --@zml//runtimes:cuda=true
  • AMD RoCM: --@zml//runtimes:rocm=true
  • Google TPU: --@zml//runtimes:tpu=true
  • AWS Trainium/Inferentia 2: --@zml//runtimes:neuron=true
  • AVOID CPU: --@zml//runtimes:cpu=false

The latter, avoiding compilation for CPU, cuts down compilation time.

So, to run the OpenLLama model from above on your host sporting an NVIDIA GPU, run the following:

cd examples
bazel run --config=release //llama:Llama-3.2-1B-Instruct             \
          --@zml//runtimes:cuda=true                       \
          -- --prompt="What is the capital of France?"

Run Tests

bazel test //zml:test

A taste of ZML

MNIST

const std = @import("std");
const zml = @import("zml");

/// Model definition
const Mnist = struct {
    fc1: Layer,
    fc2: Layer,

    const Layer = struct {
        weight: zml.Tensor,
        bias: zml.Tensor,

        pub fn forward(self: Layer, input: zml.Tensor) zml.Tensor {
            return self.weight.matmul(input).add(self.bias).relu();
        }
    };

    /// just two linear layers + relu activation
    pub fn forward(self: Mnist, input: zml.Tensor) zml.Tensor {
        std.log.info("Compiling for target: {s}", .{@tagName(input.getContext().target())});
        var x = input.flattenAll().convert(.f32);
        const layers: []const Layer = &.{ self.fc1, self.fc2 };
        for (layers) |layer| {
            x = zml.call(layer, .forward, .{x});
        }
        return x.argMax(0, .u8).indices;
    }
};

Tagged Tensors

const Sdpa = struct {
    pub fn forward(_: Sdpa, ctx: *zml.Context, q_: zml.Tensor, k_: zml.Tensor, v_: zml.Tensor) zml.Tensor {
        const q = q_.withTags(.{ .b, .h, .q, .hd });
        const k = k_.withTags(.{ .b, .h, .k, .hd });
        const v = v_.withTags(.{ .b, .h, .k, .hd });
        const attn_mask = zml.nn.causalAttnMask(ctx, .{ .q = q.dim(.q), .k = k.dim(.k) }, q.dtype(), null);
        return zml.nn.sdpa(ctx, q, k, v, .{ .attn_mask = attn_mask });
    }
};

Where to go next:

You might want to check out more examples, read through the documentation directly on GitHub, or, for the full rendering experience, browse the online documentation with included API reference.

Contributing

See here.

License

ZML is licensed under the Apache 2.0 license.

Thanks to our contributors