bitCast reinterprets bits like Tensor.bitCast. To a narrower type, each
element becomes the elements of a new last axis .bitcast. To a wider
type, the last axis must hold one element of the new type, and its bits
must be one contiguous run in the checkpoint, which panics otherwise.
Sub-byte types that land inside a byte fail at lowering with
SubBytePosition. FP8 and NVFP4 block scales, including the 128x4
swizzle, are laid out as sources this way: the swizzle case pads, splits,
transposes and merges, checked against Tensor.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
broad is Tensor.broad: the elements repeat along the axes the target shape
adds or grows from 1, as dimensions whose checkpoint stride is zero. The
loader reads a few large repetitions once each, and copies many small
ones from one read of the source. Shared scales are bound this way.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
pad is Tensor.pad with zero as the only padding value: low, high and
interior padding are nonnegative, so it never crops. Each piece's origin
and steps move by the padding along every axis, and the elements no piece
covers are the zeros. Rows appended after the last one move no element,
so that case only grows the shape. It replaces padding a table by hand
over the old store's writers.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
transpose is Tensor.transpose, and no axes reverses them. When the axes
that move all hold one element and the others keep their order, the
row-major order is unchanged and the transpose is a reshape that leaves
the pieces alone. Otherwise each piece's origin and dimensions are
permuted, and a transposed tensor lowers to strided parts of single
elements. Placing a transposed source with as keeps its element order.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
slice is Tensor.slice for a static, contiguous slice: it clips each
piece's lattice to the slice's range along the axis, keeping the steps
whose whole sub-lattice is inside as one piece and recursing into the
partial ones, and shifts the result's origin to the slice's start. A
singleton slice drops the axis, and .len takes precedence over the
default end as Tensor.Slice resolves it. split and chunkExact are slices;
split's result lives in the store's arena. They split fused checkpoint
tensors such as gate_up and qkv, and expert slices.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Ops that move elements rewrite a source's pieces. A piece is kept flat;
an op that addresses axes decomposes it into pieces in the coordinates of
the input shape, each dimension moving one axis, cutting it where a
dimension would carry from one axis into the next (at row boundaries when
the lattice steps regularly, else one piece per step). The op then moves
those coordinates and emits each piece flattened in the result's shape,
its dimensions outermost first and merged where one continues another.
Pieces are allocated in the store's arena, the scratch freed per op.
The first ops on it are concatenate and stack, like Tensor's. Along the
outermost axis the pieces only shift their origin. Fused QKV, gate_up and
expert stacks are interleaves built from splitAxis, concatenate and
merge, which keep one piece per checkpoint tensor at checkpoint sizes.
Sources of different stores panic.
Each op case is checked against Tensor: the same shape-op expression is
compiled and run as a program over zml.Tensor, and its bytes and tags are
compared with the source's binding lowered and replayed on the host,
which also fails on any byte written twice.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The new TensorStore now binds model tensors. A Binding is what the loader
will read of a tensor: its load arm holds the placed source the tensor is
filled from and the memory it is delivered to, default device memory.
bindingOf looks a tensor's binding up by id; the store owns its bindings
and every source in its arena, and frees them in deinit.
View gains source and maybeSource, which give a checkpoint tensor as a
Source with optional tags, and createTensor, maybeCreateTensor and
createReplicatedTensor, which are source(...).withPartitioning(...).bind().
A replicated tensor has no mesh and takes only .{} as its partitioning.
A missing key panics with the key, so binding cannot fail.
io.zig's test block reaches Source.zig, whose tests now run.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zml/io/Source.zig describes a tensor the new loader assembles by moving
checkpoint bytes, never by computing them. A source is a shape and a list
of pieces: each piece reads the bits of one checkpoint tensor through an
affine lattice, mapping the source's row-major elements to bit offsets in
the tensor, and elements no piece covers are zero. Pieces live in the
store's arena.
This change adds the source of one checkpoint tensor (init through a
View), its accessors, and the ops that only relabel or reshape it
(withTags, rename, renameTag, splitAxis, merge, reshape, flatten,
insertAxes, appendAxes, squeeze), which keep the row-major order and so
never touch a piece. Placement comes last: withPartitioning and as return
a Placed, the only thing that binds. Placed.bind records the model tensor
through a private bindNew, the one writer of bindings, so pieces stay
disjoint by construction. Placed.lower turns each piece into one
backend.LoadPart by one rule: dimensions outermost first, the innermost
run contiguous on both sides as the width, and SubBytePosition for a run
that starts or ends inside a byte. withPartitioning(.replicated, ...)
takes only .{}, checked at compile time.
The tests replay a binding's lowered parts on the host and fail on any
byte written twice. Nothing imports the file yet: the store wiring that
binds through it follows.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zml/io/backend.zig holds the contract between the new loader's front end
and its backends, which later changes build on from both sides. Nothing
calls it yet; io.zig's test block reaches it.
A LoadSpec is one destination: its shape, the memory it is delivered to,
the caller's output buffer and the parts that fill it. A LoadPart is the
bytes of one checkpoint tensor that a Lattice moves: a run of bytes,
repeated over up to 2 * MAX_RANK dimensions with a source and a
destination stride each, a zero source stride repeating the same bytes.
Lattice.Cursor walks a range of runs without a division per run, and
copyRuns copies them between two addresses. checkParts is the check the
front end runs before a backend sees a spec: at least one part, each
non-empty and inside its source and the destination, in ascending
destination order, strided runs landing in index order, interleaving
strided parts covering no more than the destination, and every file end
far enough below maxInt(u64) that a read widened to the direct-I/O
alignment does not overflow. Spans are checked for overflow before the
order, so a part with huge strides is refused instead of overflowing.
requestSize gives the source request for a DMA block: the read chunk
rounded up to whole blocks, at least one, at most 32 MiB
(max_read_request_size). A block that is not a nonzero multiple of the
direct-I/O alignment, or a request too short to hold a read widened at
both ends, has none. readAlignment is that alignment when the io is a VFS
and direct I/O is not off, else 0. backend.Options carries the checked
loader options; its dmaBlockSizes is the pinned block, else one fixed
16 MiB block.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Buffer.from, Buffer.uninitialized and HostAccessible.init created one
shard for every device of the platform, and toSlice and toSliceAlloc read
as many, whatever mesh the shape was placed on. A buffer on a mesh over
fewer devices then got shards on the wrong devices, or read past its
shards. They now walk the placement's mesh in its canonical device order,
and the readbacks walk the buffer's own shards. buffer.deviceMemory finds
a mesh device's memory of a kind, and panics naming the device when it has
none, where only device 0 was checked. placementOrPanic places a shape
without a mesh through Platform.placedShape. On a mesh over every device
nothing changes.
The test puts a buffer on two of four devices, out of the platform's
order, in default and pinned host memory. It needs test infrastructure:
- testing.Devices gives N devices, the accelerator's when it has as many,
else a CPU platform of its own, with a one-axis physical mesh over
exactly them;
- testing.expectShards reads every shard back in mesh order and checks its
memory space and its box of the tensor's row-major bytes, for a
delivery kind of the new loader's store;
- Buffer.Shard.toHost copies one shard to the host and waits.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zml/io/TensorStore.zig is the new loader's store. zml does not export it:
zml.io.TensorStore in io.zig stays the store models and the current loader
use, and keeps borrowing its registry. The new store takes the registry's
entries instead: init leaves the caller's registry empty, so its deferred
deinit stays correct, and locks the map's pointers until deinit, so an
entry's address identifies its checkpoint tensor and a later insert
panics in safe builds.
It has the old store's caller API: init with Options{meshes,
quantization}, deinit, view and mesh, and View's root, withPrefix,
withLayer, prefix, get, count, mesh and quantization. Memory names where
the loader delivers a tensor (default, host_pinned or host_mapped), with
isHost and the memory space of each device's buffer.
zml/io/log.zig adds the new loader's log scopes, load, io and mem. Their
err logs at debug in test builds, where the test runner counts a logged
error as a failure, and failure appends the error's name and logs
cancellation and OutOfMemory at debug.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ShardGrid is the first file of the new loader under zml/io/, which zml
does not export yet: io.zig's test block is the only thing that reaches
it. It places a destination's logical bytes on its devices by arithmetic,
never by listing rows: init takes the shape as PJRT allocates it, placed
on the platform and with sub-byte storage packed, and computes the even
box grid; locate maps a logical byte to its box and the offset inside it,
rankIn counts a box's bytes before a logical offset, runEnd and runCount
cut a range into the runs that stay in one box, and replicas gives the
devices per box.
Platform.placedShape names the default a buffer already applies to a shape
without a mesh: replicated over every device.
The tests walk every fixed-step byte range of replicated, packed sub-byte,
1D mirrored and folded, 2D, 3D and scalar placements on 4- and 8-device
CPU platforms and compare each byte with Placement.slices. They skip where
no CPU plugin is available.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A throttle was a charged retry: each reader backed off on its own and
retried, so a rate-limited backend kept seeing about as many requests and
a long limit exhausted every reader's retries. A throttle (429 everywhere,
503 on a backend that names it so) now arms a hold over every request of
the backend. Each request waits out the hold before it goes, waking at the
deadline plus jitter of its own, and the wait is cancelable. A throttle
arms the hold for the delay the server named or for a full-jitter backoff
on the throttles of the episode, whichever is longer, clamped between the
initial delay and max_hold (2 minutes); a later answer can extend a running
hold but never shorten it. A clean window as long as the hold ends the
episode, and an episode longer than throttle_budget (5 minutes) fails the
backend's requests with RateLimited until one does. Throttles are no longer
charged to max_retries. The governor counts holds and the time requests
spent waiting them out. Requests carry their URI's authority as the hold's
key, though one hold covers the whole backend.
Tested against MockServer only; no real throttling source was available.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A peer that stops sending without closing its connection holds the reader
waiting on it, and the load waiting on that reader, forever. Each exchange
now runs on a task of its own while the attempt watches it: range_read
counts the body bytes it receives in Attempt.progress, the attempt checks
that count twelve times per Retry.stall_timeout (60 s by default), and
after a whole timeout without progress it cancels the exchange and retries
it as a timeout. An exchange that had already finished when it was
canceled keeps its value. Where the io cannot run the exchange
concurrently, it runs unwatched. A request that a hook sends before its
exchange, such as a re-resolve, is watched by its own exchange, so waiting
on it is no stall.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
HTTP's reads and lookups now run through request.zig. HTTP.init takes
Options{protocol, retry} instead of a bare protocol, and its governor,
named http, treats a 503 as a server failure. fetchSize walks redirects
with HeadChain, which follows the relative Location that Hugging Face
answers a resolve URL with, where HTTP's own walk failed. A positional
read is one range_read GET that follows redirects and fills every buffer;
it filled only the first. Opening, stat-ing and probing a path follow the
open error contract, and access sends a HEAD where it succeeded for any
path. A failed positional read is Canceled or Unexpected and a failed
streaming read InputOutput, each logged once. A seek saturates at 0
instead of overflowing, and deinit logs the read totals at debug. Remotes
passes .{ .protocol = .https }. VFSBase's io is aligned for the i96 fields
of the governor HTTP now embeds, so recovering a backend from its base
needs no @alignCast.
HTTP has no parallel_read pool, so master's loader still reads https://
with one request per read. This breaks HTTP.init callers; llmd has none
once it uses VFS.Remotes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
std.http.Client does not follow a redirect for a HEAD, so a backend that
sizes an object with a HEAD walks the chain itself. HTTP's walk stopped at
a relative Location (Hugging Face answers a resolve URL with one), and HF
carries a walk of its own. HeadChain follows absolute and relative
Locations through three rotating 8 KiB buffers, up to a hop limit the
caller sets, and returns the URL that answered a 2xx with its
Content-Length. Each hop is one governed HEAD with open semantics, so a
missing or denied object fails the walk with FileNotFound or
PermissionDenied, and a callback chooses the credentials sent to each URL.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
performRangeRead turns one positional read into one GET with a Range
header for what the caller's buffers hold of the object, run through the
governed loop, and scatters the response into every buffer. Nothing is
sent at the end of the object. A 206 must carry a Content-Range that
covers the request, parsed strictly; a 200 that ignored the Range is
positioned by discarding up to the offset. The backend supplies the
request through a prepare hook, called once per attempt, since an S3
signature covers the Range header and a timestamp and a token or a signed
URL may expire between attempts; static serves a plain GET. It replaces
parallel_read's chunk reader and HTTP's own Content-Range parsing once the
backends move to it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each remote backend sends its requests with code of its own: HTTP retries
nothing, and HF, S3 and GCS each carry a copy of the same retry loop that
differs in which failures it retries and how it reports the rest.
request.zig holds one loop for all of them. A backend owns a Governor,
with its name, its retry knobs and its request statistics, and runs each
request through perform, which retries a retryable failure after a
full-jitter backoff, or after the delay the server named in Retry-After or
HF's RateLimit header, up to max_delay, and gives up after max_retries.
exchange owns the std.http plumbing of one attempt: it forces identity
encoding, keeps its own redirect buffer, closes a connection after a
partial send, and tells a canceled read from a failed one. A 408 is a
timeout, a 429 a throttle, a 5xx a server failure, and a 503 either a
throttle or a server failure as the backend says; any other status fails
the request, logged once at err.
Open, stat and access requests share one error contract: 404 and 410 are
FileNotFound, 401 and 403 PermissionDenied, a cancel stays Canceled, and
anything else is Unexpected, logged once by the governor. A failed read is
Canceled or Unexpected. No backend uses the loop yet; its tests drive it,
one of them against MockServer.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The remote backends' failure paths (retries, refusals, redirects, rate
limiting) cannot be tested against a real origin. MockServer serves an
object per path over a real loopback socket, with Range GETs and HEADs, and
can be told to fail the first GETs, hold GETs at a barrier until enough
arrive, throttle (the first requests, every GET, or past a rate window,
with a status and a Retry-After), refuse a path with a status, or redirect
it. It counts HEADs, GETs and the peak of concurrent GETs, and keeps the
first error it met through stdx.FirstError. It is exported as
VFS.MockServer so tests outside vfs can use it too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Tasks that run concurrently and may each fail often need to keep the first
failure and let the others see it. FirstError(E) keeps it in one atomic
word: set stores an error unless one is already kept and says whether it
kept this one, and get reads it back.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Neither read mode wins everywhere: a direct read comes from the disk even
when the page cache holds the file, and a buffered one pays a copy that
competes with DMA for host memory bandwidth. DirectIo.auto reads a file
directly when at most half of 32 pages sampled across it are cached, and
buffered otherwise. A sample is a preadv2 with RWF_NOWAIT, which answers
EAGAIN for an uncached page, needs no write permission and costs the same
whatever the file's size; unknown residency counts as cached. A sample
that misses queues the read of its page, so each probe shifts its samples
by a random offset, and a cold file read directly is still found cold by
the next probe unless its samples land in what those misses read.
predictDirectIo gives the answer useDirectIo would give for a local path,
and the file's size, on a descriptor of its own: it decides no handle and
logs nothing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A reader that reads a local file in large aligned requests can ask the VFS
to read it past the page cache: useDirectIo(file, policy) sets O_DIRECT on
the descriptor under .on, once per handle, and from then on the VFS issues
that handle's positional reads itself, with preadv in batches of at most
IOV_MAX buffers. The kernel answers a direct read it cannot serve with
EINVAL or EFAULT, which the threaded io treats as a programmer error; here
such a refusal clears the flag and the read, and every later one, goes
through the inner io buffered. It is logged at warn when the read was
aligned and at err when it was not. Streaming reads and file copies take
the handle off direct reads first. The flag goes on and off through
fcntl, retried on EINTR, and only on a descriptor opened read-only.
vfs/direct_io.zig holds the mechanism; the VFS exports DirectIo and
direct_io_alignment (4096, or 0 where O_DIRECT does not exist).
Nothing calls useDirectIo yet, so every handle stays undecided and reads
as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Creating or deleting through the VFS went through VFSBase's forwarders,
which handed the inner io a VFS handle index as a descriptor and a URI as a
local path: creating a directory from a directory opened through the VFS
used the wrong descriptor, and createFile on s3://bucket/key wrote
bucket/key on local disk. Creates and deletes (directories, plain and
atomic files) now resolve their directory through the VFS's handle table
and strip file://, and the directory dirCreateDirPathOpen opens gets a
handle of its own. A remote path, given whole or relative to a directory
opened on a remote, is refused with ReadOnlyFileSystem before any request,
and so is opening a remote file for anything but reading. The refusal is
logged through base.scopedLog, which logs err at debug in test builds, so
the tests of these failure paths pass under the test runner.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every program that reads remote checkpoints wired the same remote schemes
by hand: one std.http.Client, then the HTTP, HF, S3 and GCS backends and
their registration. The copies had drifted: examples/llm had no https and
ignored the proxy variables. VFS.Remotes builds one client on the VFS's base
io, with proxies from the environment, sets up https, hf, s3 and gs through
each backend's auto constructor and registers all four, or none on error.
It is initialized in place, since the backends point at its client.
examples/io and examples/llm use it, so examples/llm gains https and
proxies.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Local files are the VFS's own: a file:// URI now resolves to the inner io,
like a bare path, and registerBackend asserts that nobody registers the
file scheme. vfs/file.zig goes. It was a forwarding backend with a second
handle table that, for file:// URIs, also opened a second O_DIRECT
descriptor and read through it transparently whenever a read happened to
be aligned. Bare paths, which llmd and the examples use, never had that,
so only file:// URIs change: they now read buffered. The examples stop
creating and registering a VFS.File.
This breaks llmd's use of VFS.File in main.zig and zoo.zig.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A backend is now registered as a Backend: its io together with ReadHints,
the smallest request worth sending and whether the backend's latency calls
for many requests in flight. Local files read best in 8 MiB requests, the
https, s3 and gs backends in 16 MiB and hf in 32 MiB, the remote ones all
high-latency. registerBackend replaces register and unregister, and VFSBase
is no longer exported. readHints and isLocal answer for a path without
opening it, and fromIo finds the VFS behind an Io, so a reader that holds
only an Io can size its reads. examples/io and examples/llm register
through registerBackend.
This breaks llmd's register calls in main.zig and zoo.zig.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
TensorStore.fromRegistry(allocator, ®istry, meshes) becomes
init(allocator, ®istry, Options{ meshes, quantization }). The store
still borrows the registry: entries registered after init, such as
aliases of a fused tensor, are found, and each bound tensor holds a copy
of its entry.
Options.quantization is the platform's Quantization.Target, which code
binding block-FP8 weights reads through View.quantization; that panics
when the store has none, so weights cannot be laid out for a different
FP8 encoding than the activations.
View.get(subkey) returns the checkpoint tensor, or null, with one hash
lookup. It replaces getShape, getShapeOpts and hasKey; hasKey scanned
every key for one starting with the subkey, so it also matched a
tensor's prefix. View.prefix() returns "" at the root instead of null.
The examples, the torch2zml howto and testing.testLayer move to init,
and the lfm2 and llama layer tests read tensors with get and
Tensor.read. llmd's fromRegistry, hasKey and getShape calls follow in
its next zml bump.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Opening a checkpoint had several faults. fromPath opened a file path
relative to the file's own directory, so a relative path such as
models/x/model.safetensors looked for models/x/models/x/model.safetensors.
resolveModelRepo accepted a file and opened its directory, and mapped
every error but FileNotFound to InvalidPath, including a denied directory
and a canceled open. resolveModelEntrypoint ignored why each probe failed,
and fetchRegistry leaked the registry when a file failed to parse.
- fromPath opens a file by its base name in its own directory.
- resolveModelRepo takes directories only; anything else is a logged
InvalidPath, and PermissionDenied and Canceled pass through.
ModelPathResolutionError no longer lists OutOfMemory.
- resolveModelEntrypoint becomes private. It still tries
model.safetensors after the index, and reports PermissionDenied when
neither opens and one was denied: S3 answers 403 for a missing key to
a principal that may not list the bucket.
- fetchRegistry becomes the private addCheckpoint and the registry is
freed when parsing fails. FileType and resolveFiletype are deleted.
TensorRegistry.addRepo(io, repo, name) adds a second checkpoint's
tensors under name and a dot, so a speculative drafter can share its
model's registry; it refuses a name a tensor is already under before
reading anything. parseSafetensors takes that prefix and allocates from
the registry's allocator.
checkpointFiles(arena, io, path) names the safetensors files fromPath
reads for a path, in their directory, which it returns open. It reads
no header and logs nothing, for a reader that prepares a load before
the registry exists. fromPath and it share how a path locates its
checkpoint (locateCheckpoint), the directory's entrypoint
(resolveModelEntrypoint, which also names it) and the files an index
lists (indexFiles).
safetensors.Tensor gains read, which reads a whole tensor from its file,
and readRange, which reads a tensor-relative range of a file the caller
opened with positional, bounds-checked reads. TensorReader stays for
now. testing.writeSafetensors writes a fixture file and registers it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
registerTensor put each tensor in the registry, so when two files of a
checkpoint named the same tensor, the last one read silently won and
which bytes were loaded depended on the index's file order. It now
refuses a name already registered with DuplicateTensor, logging the
tensor and both files. A checkpoint that repeats a name across its
shards no longer loads.
llmd checks the registry before registering a tensor of its own, so it
is unaffected.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A checkpoint's header is untrusted input, but parseSafetensors read it
as raw JSON values and unwrapped fields with .?, so a malformed header
panicked, or was undefined behavior in release builds. It also cast
negative offsets to u64, allocated a header length read from the file
without checking it against the file, and never checked that a tensor's
data_offsets span its shape.
The header is now parsed as typed entries that ignore unknown fields.
Its length must fit the file, and each tensor's data_offsets must span
exactly the bytes of its shape within the file's data, the byte count
checked for overflow. Anything else, including a file shorter than the
8-byte length field, is InvalidSafetensorsHeader, logged with the file
and tensor. stringToDtype returns null on an unknown dtype and its
caller logs the file and tensor, and the TooManyDimensions log names
the file.
The index is parsed as a typed {weight_map}. Parsing it as raw values
with alloc_if_needed left strings pointing into the reader's buffer,
which the next refill overwrote, so an index longer than 64 KiB could
name the wrong files. An index without a weight_map is refused rather
than read as an empty checkpoint, and a read error returns its cause.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The registry parsed every header's __metadata__ and every index's
metadata into a map of typed values, about 300 lines, and nothing in zml
or llmd reads it. Metadata, Metadatas, TensorRegistry.metadata,
initWithMetadata, mergeMetadata and the parsing helpers are deleted,
along with the registry's unused mutex. A header's __metadata__ is
skipped, and the torch2zml howto's sample output loses the debug line
that metadata parsing logged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Tests that read a checkpoint need a safetensors file with known tensors.
writeSafetensorsFile writes one from (name, dtype, dims, bytes) entries,
laying the tensors out in order after the header, and pattern fills a
tensor with distinct, nonzero bytes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A partitioning written as a struct of tags, such as .{ .out = .model },
could only be parsed at comptime: Partitioning.parse captured it in a
comptime Logical{src, tgt}. Code that picks a partitioning at runtime,
such as a weight binder that chooses row or column parallelism, had no
value to pass.
Logical is now a runtime value, {entries, default}, keyed by tensor axis
tags. Each entry holds a Spec: a logical mesh axis, .replicated or
.open. Logical.init takes a struct of specs (null optional specs are
skipped), a Logical or .replicated; get(tag) returns an axis's spec, and
resolve(tags, mesh, default) turns it into a positional Partitioning for
one tensor on one mesh, so one value applies to tensors whose axes are
ordered differently and to different meshes. Shape.withPartitioning,
Tensor.withPartitioning and the old TensorStore's maybeCreateTensor
resolve through it, and maybeCreateHostPinnedTensor no longer needs a
comptime partitioning. Partitioning.parse, Logical.parse, apply and
applyUseAll are deleted.
Shape.eqlWithTagsAndPartitioning also compares the mesh and each axis's
partition spec.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Block-FP8 weights must be bound in the FP8 encoding and scale layout the
platform's GEMMs read, and activations quantized to the same encoding.
quantizeInput chose the encoding with its own platform switch, and
weights had nothing to follow. Quantization.Target{fp8, swizzled_scales}
and Target.of(platform) are now the one switch: FN on CUDA with e8m0
scales swizzled on sm_10x, FNUZ on CDNA3, FN on CDNA4 and RDNA4, and no
FP8 elsewhere or when a ROCm device's architecture is unknown.
quantizeInput reads Target.of(platform).fp8, so its result is unchanged;
on CUDA it now queries the compute capability on each call. The block128
scaled dot test takes its dtype from Target as well.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bufferize(allocator, T, &model) repeated the model's type next to a
pointer that already carries it. It becomes bufferize(allocator, &model)
and returns Bufferized of the pointed-to type. The callers in
examples/io, examples/llm and both mnist examples move with it.
Buffer.deinitAll keeps its type argument. llmd's bufferize calls follow
in its next zml bump.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
lfm2, llama, qwen3_5 and qwen3_5_moe each freed their weights with a
tree of hand-written unloadBuffers functions, one per layer struct, and
each LoadedModel carried its own copy of the same loadBuffers. A walker
that missed a field leaked its device memory.
models.LoadedModel.loadBuffers now bufferizes and loads any model arm
with one body, and main, lfm2's session and the lfm2 and llama layer
tests free buffers with mem.deinitBufferized, which follows the Bufferized
type. The per-model loaders and every unloadBuffers walker are deleted,
so no example calls nn.Linear.unloadBuffers anymore.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Exe.call panicked when PJRT failed to launch an executable or when a
waited execution event reported an error, so a caller could not recover
from, say, a device running out of memory. It now returns
pjrt.ApiError. A failed launch writes no outputs. With wait, every
partition's event is awaited and destroyed, and a failed execution
destroys the outputs it wrote, so the results hold no buffer after any
error. The neuron and non-neuron event loops become one; their behavior
is unchanged.
Runner.run and FnExe's Runner.run keep their panic. Direct callers in
the examples, the tutorial and zml.testing add try, and the old
Loader.loadExecute reports a failed call as LoadFailed; PJRT has already
logged its cause. llmd's one direct call follows in its next zml bump.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each model frees its buffers with a hand-written walker that has to
follow every field of its Bufferized struct, and leaks when a field is
added without updating it. mem.deinitBufferized frees any tree bufferize
returns: it deinits every Buffer and frees the slices bufferize
allocated, through structs, unions, optionals, arrays and nested
slices. Passing it the model instead of its buffers does not compile.
When an allocation failed partway through a model, bufferize returned
the error and leaked the slices it had already allocated. It now frees
what it built before returning the error. It also fills the slice it
allocates before storing it, so models with []const Tensor fields
bufferize too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Bufferized maps a [N]Tensor field to [N]Buffer, but bufferize had no
case for arrays and reached `unreachable` at runtime on any model with
one. It now bufferizes arrays element by element.
bufferize also walked Bufferized's field names and looked each one up
in the model. Bufferized drops the fields that hold no tensor and, for
a tuple, renumbers the ones it keeps, so bufferizing a tuple with a
tensor-free field before a tensor did not compile. bufferize now walks
the model's fields and maps each to the field Bufferized keeps for it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
pjrtx.Client.compile went through pjrt.Client.compile, which started
compileRaw with io.async and awaited it at once. Under zio that ran
XLA's compile on an 8 MiB coroutine stack, even though Compiler.compile
had already left the coroutine with zio.blockInPlace; this is the
likely cause of the stack overflow seen compiling Laguna.
pjrtx.Client.compile now calls compileRaw directly and no longer takes
an io, Compiler stops passing one, and pjrt.Client.compile, which had no
other caller, is deleted. Under threaded io the compile now runs on the
caller's thread instead of a pool thread.
Under zio, Compiler.compile now leaves the coroutine with
zio.blockInPlaceReserved rather than zio.blockInPlace. blockInPlace
queues the compile behind the blocking pool's other jobs, and the pool
adds a worker only when none is idle and at least twice as many jobs as
workers are queued, so a burst of compiles shared one to three threads:
llmd's six model compiles took 6 to 15 s under zio on the B70 against
4.5 s under threaded io, and DeepSeek's two on a GB300 ran one after the
other (73-77 s against 50 s). A compile holds its thread for its whole
run, so each now takes a pool worker of its own, idle or new, and
concurrent compiles run in parallel as they do under threaded io.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
gcsUri set an object's path to the VFS path, bucket/key, without the
leading slash, so every GCS request targeted `bucket/key` instead of
`/bucket/key`. storage.googleapis.com accepts that, but it is not a
valid origin-form target and other endpoints need not. The path is now
rooted, with a test, and gcs.zig's tests run under //vfs:test.
A streaming read whose request failed returned EndOfStream on http, hf,
s3 and gs, so a reader saw a short file instead of an error. It now
returns InputOutput; the error is still logged where it happens.
The GCS backend also loses code nothing reaches: the JWT assertion and
its commented-out signing, getTimestamp, the commented-out authHeaders,
EndpointUrl, and the `region` option, which no caller sets.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The VFS seeded its handle table with the cwd placeholder at index 0,
then stdin, stdout and stderr, so std.Io.File.stdout() (handle 1)
resolved to descriptor 0. Writing to stdout through the VFS went to
stdin instead, which only works when both are the same terminal. The
standard streams now take indices 0 to 2 and the cwd placeholder
follows them.
A local name with a colon, such as notes:old.txt, was parsed as a URI:
from the cwd it failed as an unregistered scheme, and stripScheme cut
the name after the colon. A path is now remote only when it contains
"://", stripScheme removes only up to "://", and dirRealPathFile passes
the stripped path to the backend, so a remote real path no longer
carries its scheme twice.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Docs that no longer match the code:
- huggingface_access_token.md names HF_TOKEN, the variable vfs/hf.zig
reads, instead of HUGGINGFACE_TOKEN.
- write_first_model.md calls Buffer.fromSlice with its three arguments
and explains that compile() reads each tensor's sharding from its
shape, instead of a `sharding` variable the tutorial never defines.
- howto_torch2zml.md calls Platform.deinit with its io, and replaces
maybeCreateReplicatedTensor, which does not exist, with
maybeCreateTensor(..., .replicated, .{}).
In examples/llm, lfm2, llama, qwen3_5 and qwen3_5_moe each kept a second
loader on their inner Model (loadBuffers or load) that nothing calls and
that would not compile if called; they are deleted. LoadedModel.loadBuffers
stays the only loader.
qwen3_5_moe binds its .replicated tensors with an empty partitioning,
which the store ignores for the replicated mesh anyway, and examples/io
iterates the registry's tensors directly. Neither changes behavior.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
expectClose's doc names zml.HostBuffer, which no longer exists; it
accepts a zml.Slice. BufferizedWithArgs has no user and is deleted.
Buffer's doc likewise pointed to zml.aio.loadBuffers and
HostBuffer.toDevice, neither of which exists; it now names fromSlice and
fromBytes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
quantizeMxfp8 has no caller in zml or llmd. It also kept its own copy of
the platform's FP8 encoding switch, which quantizeInput already holds.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Platform.init warned when a plugin reported more than MAX_NUM_DEVICES
devices, then used them all, although per-device state such as Exe's
events lives in arrays of MAX_NUM_DEVICES entries. validateDeviceCount
now logs and returns InvalidDeviceCount instead.
Platform.init also leaked the PJRT client, and on CUDA the FlashInfer
state, when it failed after creating them. Both are now released on
error.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`visit` walks only a bounded array's first `len` elements, but the type
MapRestrict maps it to holds the whole backing buffer. Pairing a model's
tensors with its buffers by visit order would then miss or misplace
elements, so mapping a bounded array that contains the source type is now
a compile error that points to slices and arrays instead.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Slice.absolute tested `.end == to_the_end` before `.len`. Since `.end`
defaults to `to_the_end`, a slice given as `.{ .start, .len }` took the
rest of the axis instead of `len` elements, contrary to the documented
examples. `.len` is now checked first, and the slice test covers it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Duration.hz divides the i96 nanosecond count with `/`, which does not
compile for signed integers once the function is called; it now uses
@divTrunc, with a test that calls it.
AllocatingLimitedConcurrentGroup has no user in zml or llmd and is
deleted. LimitedGroup stays: zml/io.zig's Loader uses it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>