- Zig 83.3%
- C++ 10.5%
- Starlark 2.8%
- C 2.5%
- Python 0.6%
- Other 0.3%
docs/learn/loader.md is the guide to zml.io.Loader: the recipe over a
VFS with Remotes, placing tensors on a mesh or in host memory,
composite tensors built from Source's shape ops, programs bound with
bindProgram, and what each handle borrows and how a load fails. The
errors get a short summary that points at Loader's doc comment, which
holds the contract. The docs index, the README's taste of ZML,
Buffer's doc and the headers of zml/io.zig and zml/io/loader.zig link
it.
write_first_model gains a last step, 'Loading weights': a few lines of
Python write the layer's weight and bias to a .safetensors file, the
layer binds them through a TensorStore view, and main loads them over
a VFS with Remotes through the loader before calling the executable.
The TensorStore paragraph now says the store holds the shapes and the
loader fills the buffers.
zml.io.next goes: nothing in zml or llmd names it any more.
The tutorial's program and every snippet of the guide were built and run
on 4 CPU devices from a scratch target that is not committed: the
tutorial prints {6,11,16} from the Python-written checkpoint, and the
placement (host_mapped included) and composite snippets load the values
they describe. The program snippet stands alone: it defines its sources
and its forward dequantizes by block and places its result as the output
is, without which the load fails with InvalidBinding; on an 8x256 weight
with one scale per row and 128 columns it loads every value as expected,
row-sharded.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
History
|
|
|---|---|---|
| .github | ||
| .vscode | ||
| .zed | ||
| bazel | ||
| bin/zml-smi | ||
| docs | ||
| dsl | ||
| examples | ||
| ffi | ||
| mlir | ||
| pjrt | ||
| platforms | ||
| stdx | ||
| third_party | ||
| tools | ||
| upb | ||
| vfs | ||
| zml | ||
| .bazelignore | ||
| .bazelrc | ||
| .bazelversion | ||
| .editorconfig | ||
| .gitignore | ||
| .nvim.lua | ||
| AGENTS.md | ||
| bazel.sh | ||
| BUILD.bazel | ||
| build.zig | ||
| CONTRIBUTING.md | ||
| devenv.lock | ||
| devenv.nix | ||
| devenv.yaml | ||
| LICENSE | ||
| MODULE.bazel | ||
| MODULE.bazel.lock | ||
| README.md | ||
About
ZML is a production inference stack, purpose-built to decouple AI workloads from proprietary hardware.
Any model, many hardwares, one codebase, peak performance.
Compiled directly to NVIDIA, AMD, INTEL, TPU, Trainium for peak hardware performance on any accelerator. No rewriting.
It is built using the Zig language, MLIR, and Bazel.
Getting Started
Prerequisites
We use bazel to build ZML and its dependencies. The only prerequisite is
bazel, which we recommend installing through bazelisk.
macOS
brew install bazelisk
Linux
curl -L -o /usr/local/bin/bazel 'https://github.com/bazelbuild/bazelisk/releases/download/v1.28.0/bazelisk-linux-amd64'
chmod +x /usr/local/bin/bazel
30-Second Smoke Test
Run the MNIST example:
bazel run //examples/mnist
This downloads a small pretrained MNIST model, compiles it, loads the weights, and classifies a random handwritten digit.
LLM Quickstart
The main LLM example is //examples/llm. It currently supports:
- Llama 3.1 / 3.2
- Qwen 3.5
- LFM 2.5
Authenticate with Hugging Face if you want to load gated repos such as Meta Llama:
bazel run //tools/hf -- auth login
Alternatively, set the HF_TOKEN environment variable.
Then run a prompt directly:
bazel run //examples/llm -- --model=hf://meta-llama/Llama-3.2-1B-Instruct --prompt="What is the capital of France?"
Open the interactive chat loop by omitting --prompt:
bazel run //examples/llm -- --model=hf://meta-llama/Llama-3.2-1B-Instruct
You can also load from:
- a local directory:
--model=/var/models/meta-llama/Llama-3.2-1B-Instruct - S3:
--model=s3://bucket/path/to/model
Running Models on GPU / TPU
Append one or more platform flags when compiling or running:
- NVIDIA CUDA:
--@zml//platforms:cuda=true - AMD RoCM:
--@zml//platforms:rocm=true - Intel OneAPI:
--@zml//platforms:oneapi=true - Google TPU:
--@zml//platforms:tpu=true - AWS Trainium / Inferentia 2:
--@zml//platforms:neuron=true - Disable CPU compilation:
--@zml//platforms:cpu=false
Example on CUDA:
bazel run //examples/llm --@zml//platforms:cuda=true -- --model=hf://meta-llama/Llama-3.2-1B-Instruct --prompt="Write a haiku about Zig"
Example on ROCm:
bazel run //examples/llm --@zml//platforms:rocm=true -- --model=hf://meta-llama/Llama-3.2-1B-Instruct --prompt="Write a haiku about Zig"
Example on Intel OneAPI:
bazel run //examples/llm --@zml//platforms:cpu=false --@zml//platforms:oneapi=true -- --model=hf://meta-llama/Llama-3.2-1B-Instruct --prompt="Write a haiku about Zig"
Run Tests
bazel test //zml:test
Examples
examples/llm: unified LLM CLI for Llama, Qwen, and LFMexamples/mnist: smallest end-to-end model runexamples/sharding: logical mesh, partitioners, shard-local execution, profiler outputexamples/io: inspect and load local,hf://,https://, ands3://repositories through the VFS layerexamples/benchmark: measure loading and execution performance
A Taste Of ZML
const Mnist = struct {
fc1: Layer,
fc2: Layer,
const Layer = struct {
weight: zml.Tensor,
bias: zml.Tensor,
pub fn init(store: zml.io.TensorStore.View) Layer {
return .{
.weight = store.createReplicatedTensor("weight", .{ .d_out, .d }),
.bias = store.createReplicatedTensor("bias", .{.d_out}),
};
}
pub fn forward(self: Layer, input: zml.Tensor) zml.Tensor {
return self.weight.dot(input, .d).add(self.bias).relu().withTags(.{.d});
}
};
pub fn init(store: zml.io.TensorStore.View) Mnist {
return .{
.fc1 = .init(store.withPrefix("fc1")),
.fc2 = .init(store.withPrefix("fc2")),
};
}
/// just two linear layers + relu activation
pub fn forward(self: Mnist, input: zml.Tensor) zml.Tensor {
var x = input.flatten().convert(.f32).withTags(.{.d});
const layers: []const Layer = &.{ self.fc1, self.fc2 };
for (layers) |layer| {
x = layer.forward(x);
}
return x.argMax(0).indices.convert(.u8);
}
};
pub fn main(init: std.process.Init) !void {
const allocator = init.gpa;
const path = ...;
// Local paths. `zml.io.VFS.Remotes` adds hf://, s3://, gs:// and https://.
var vfs: zml.io.VFS = try .init(allocator, init.io);
defer vfs.deinit();
const io = vfs.io();
const platform: *zml.Platform = try .auto(allocator, io, .{});
defer platform.deinit(allocator, io);
// Returns at once: the loader calibrates while the model is compiled.
var loader: zml.io.Loader = try .init(allocator, io, platform, .{ .checkpoint = path });
defer loader.deinit();
var registry: zml.safetensors.TensorRegistry = try .fromPath(allocator, io, path);
defer registry.deinit();
var store: zml.io.TensorStore = .init(allocator, ®istry, .{});
defer store.deinit();
const model: Mnist = .init(store.view());
const input: zml.Tensor = .init(.{ 28, 28 }, .u8);
var exe = try platform.compile(allocator, io, model, .forward, .{input}, .{});
defer exe.deinit();
var weights = try zml.mem.bufferize(allocator, &model);
defer zml.mem.deinitBufferized(allocator, &weights);
try loader.start(&model, &weights, &store, .none);
try loader.wait(); // logs "Loaded weights"
// ... call `exe` on the weights and an image
}
The first start waits for whatever calibration is left.
Loading checkpoints gives the full recipe, with
remote checkpoints, placement, lifetimes and errors.
For a full walkthrough, see:
Where To Go Next
- Run more examples in
./examples - Read the example-specific notes in
examples/llm/README.md - Learn tagged dimensions in
working_with_tensors.md - Start building a model with
write_first_model.md - Explore deployment in
deploy_on_server.md
Contributing
See here.
License
ZML is licensed under the Apache 2.0 license.