0
Fork 0
mirror of https://github.com/zml/zml.git synced 2026-11-16 19:11:09 +00:00
Any model. Any hardware. Zero compromise. Built with @ziglang / @openxla / MLIR / @bazelbuild https://docs.zml.ai
  • Zig 83.3%
  • C++ 10.5%
  • Starlark 2.8%
  • C 2.5%
  • Python 0.6%
  • Other 0.3%
Find a file
Corentin Godeau 82a23317ad zml/module.zig: increase SerializedExecutable size limit
On Neuron, some artifacts can reach more than 100MB.
History 2024-11-13 15:01:15 +00:00
.github zml: initial release 2024-09-17 11:11:18 +02:00
async zml: reintroduce pjrtx (#78) 2024-11-05 19:25:50 +01:00
bazel runtimes/neuron: rework neuronx-cc sandboxing 2024-11-08 14:35:34 +01:00
docs async: rename async_ to asyncc and add Generic slugs (#71) 2024-11-01 17:36:08 +01:00
examples examples,runtimes: add layers for each runtime 2024-11-08 17:49:55 +01:00
mlir add DotAlgorim support to dot_general 2024-11-13 11:57:36 +00:00
pjrt runtimes: add intermediate abstraction layer (#74) 2024-11-01 18:07:33 +01:00
platforms zml: initial release 2024-09-17 11:11:18 +02:00
runtimes examples,runtimes: add layers for each runtime 2024-11-08 17:49:55 +01:00
third_party workspace: bump xla to enabled batching in gather (#21) 2024-09-24 09:40:55 +02:00
tools zml_utils: fix CollectionOver scope (#13) 2024-09-19 13:54:21 +02:00
zml zml/module.zig: increase SerializedExecutable size limit 2024-11-13 15:01:15 +00:00
.bazelignore zml: initial release 2024-09-17 11:11:18 +02:00
.bazelrc zml: initial release 2024-09-17 11:11:18 +02:00
.bazelrc-examples zml: initial release 2024-09-17 11:11:18 +02:00
.bazelrc-zml zml: initial release 2024-09-17 11:11:18 +02:00
.bazelversion zml: initial release 2024-09-17 11:11:18 +02:00
.gitignore zml: initial release 2024-09-17 11:11:18 +02:00
.nvim.lua zml: initial release 2024-09-17 11:11:18 +02:00
.vscode zml: initial release 2024-09-17 11:11:18 +02:00
BUILD.bazel zml: initial release 2024-09-17 11:11:18 +02:00
build.zig zml: initial release 2024-09-17 11:11:18 +02:00
CONTRIBUTING.md zml: initial release 2024-09-17 11:11:18 +02:00
LICENSE zml: initial release 2024-09-17 11:11:18 +02:00
MODULE.bazel examples,runtimes: add layers for each runtime 2024-11-08 17:49:55 +01:00
MODULE.bazel.lock examples,runtimes: add layers for each runtime 2024-11-08 17:49:55 +01:00
platform_mappings zml: initial release 2024-09-17 11:11:18 +02:00
README.md workspace: be a little more precise regarding the huggingface token (#45) 2024-10-06 14:34:38 -07:00
zls.build.json zml: initial release 2024-09-17 11:11:18 +02:00

Bonjour 👋

At ZML, we are creating exciting AI products on top of our high-performance AI inference stack. Our stack is built for production, using the amazing Zig language, MLIR, and the power of Bazel.

Take me straight to getting started or give me a taste 🥐!

 

We're happy to share!

We're very happy to share our inference stack with the World and hope it allows you, too, to build cool and exciting AI projects.

To give you a glimpse of what you can do with ZML, here is an early demo:

It shows a prototype running a LLaMA2 model sharded on 1 NVIDIA RTX 4090, 1 AMD 6800XT, and 1 Google Cloud TPU v2. All accelerators were hosted in different locations, with activations being passed over a VPN.

All processes used the same model code, cross-compiled on a Mac, and copied onto the servers.

For more inspiration, see also the examples below or check out the examples folder.

Getting started

Prerequisites

We use bazel to build ZML and its dependencies. The only prerequisite is bazel, which we recommend to download through bazelisk, a version manager for bazel.

Please note: If you do not wish to install bazel system-wide, we provide examples/bazel.sh which downloads it to your home folder and runs it.

Install Bazel (recommended):

macOS

brew install bazelisk

Linux

curl -L -o /usr/local/bin/bazel 'https://github.com/bazelbuild/bazelisk/releases/download/v1.20.0/bazelisk-linux-amd64'
chmod +x /usr/local/bin/bazel

Run a pre-packaged model

We have implemented a variety of example models in ZML. See our reference implementations in the examples folder.

MNIST

The classic handwritten digits recognition task. The model is tasked to recognize a handwritten digit, which has been converted to a 28x28 pixel monochrome image. Bazel will download a pre-trained model, and the test dataset. The program will load the model, compile it, and classify a randomly picked example from the test dataset.

On the command line:

cd examples
bazel run -c opt //mnist

# or
./bazel.sh run -c opt //mnist

TinyLlama, Stories 15M

Our LLM examples start with a small model trained specifically on children's history books. This model has been trained by Andrej Karpathy; you can read more about it on his GitHub.

cd examples
bazel run -c opt //llama:TinyLlama-Stories-15M
bazel run -c opt //llama:TinyLlama-Stories-15M -- --prompt="Once upon a time, there was a cute little dragon"

OpenLLama 3B

cd examples
bazel run -c opt //llama:OpenLLaMA-3B
bazel run -c opt //llama:OpenLLaMA-3B -- --prompt="Once upon a time,"

Meta Llama 3 8B

This model has restrictions, see here. It requires approval from Meta on Huggingface, which can take a few hours to get granted.

While waiting, you can already generate an access token to log into HuggingFace from bazel; see here.

Once you've been granted access, you're ready to download a gated model like Meta-Llama-3-8b!

# requires token in $HOME/.cache/huggingface/token, as created by the
# `huggingface-cli login` command, or the `HUGGINGFACE_TOKEN` environment variable.
cd examples
bazel run -c opt //llama:Llama-3.1-8B-Instruct
bazel run -c opt //llama:Llama-3.1-8B-Instruct -- --prompt="Once upon a time,"

Running Models on GPU / TPU

You can compile models for accelerator runtimes by appending one or more of the following arguments to the command line when compiling / running a model:

  • NVIDIA CUDA: --@zml//runtimes:cuda=true
  • AMD RoCM: --@zml//runtimes:rocm=true
  • Google TPU: --@zml//runtimes:tpu=true
  • AVOID CPU: --@zml//runtimes:cpu=false

The latter, avoiding compilation for CPU, cuts down compilation time.

So, to run the OpenLLama model from above on your host sporting an NVIDIA GPU, run the following:

cd examples
bazel run -c opt //llama:OpenLLaMA-3B        \
          --@zml//runtimes:cuda=true         \
          -- --prompt="Once upon a time,"

Run Tests

bazel test //zml:test

A taste of ZML

MNIST

const std = @import("std");
const zml = @import("zml");

/// Model definition
const Mnist = struct {
    fc1: Layer,
    fc2: Layer,

    const Layer = struct {
        weight: zml.Tensor,
        bias: zml.Tensor,

        pub fn forward(self: Layer, input: zml.Tensor) zml.Tensor {
            return self.weight.matmul(input).add(self.bias).relu();
        }
    };

    /// just two linear layers + relu activation
    pub fn forward(self: Mnist, input: zml.Tensor) zml.Tensor {
        std.log.info("Compiling for target: {s}", .{@tagName(input.getContext().target())});
        var x = input.flattenAll().convert(.f32);
        const layers: []const Layer = &.{ self.fc1, self.fc2 };
        for (layers) |layer| {
            x = zml.call(layer, .forward, .{x});
        }
        return x.argMax(0, .u8).indices;
    }
};

Tagged Tensors

const Sdpa = struct {
    pub fn forward(_: Sdpa, ctx: *zml.Context, q_: zml.Tensor, k_: zml.Tensor, v_: zml.Tensor) zml.Tensor {
        const q = q_.withTags(.{ .b, .h, .q, .hd });
        const k = k_.withTags(.{ .b, .h, .k, .hd });
        const v = v_.withTags(.{ .b, .h, .k, .hd });
        const attn_mask = zml.nn.causalAttnMask(ctx, .{ .q = q.dim(.q), .k = k.dim(.k) }, q.dtype(), null);
        return zml.nn.sdpa(ctx, q, k, v, .{ .attn_mask = attn_mask });
    }
};

Where to go next:

You might want to check out more examples, read through the documentation directly on GitHub, or, for the full rendering experience, browse the online documentation with included API reference.

Contributing

See here.

License

ZML is licensed under the Apache 2.0 license.

Thanks to our contributors