docs/learn/loader.md is the guide to zml.io.Loader: the recipe over a
VFS with Remotes, placing tensors on a mesh or in host memory,
composite tensors built from Source's shape ops, programs bound with
bindProgram, and what each handle borrows and how a load fails. The
errors get a short summary that points at Loader's doc comment, which
holds the contract. The docs index, the README's taste of ZML,
Buffer's doc and the headers of zml/io.zig and zml/io/loader.zig link
it.
write_first_model gains a last step, 'Loading weights': a few lines of
Python write the layer's weight and bias to a .safetensors file, the
layer binds them through a TensorStore view, and main loads them over
a VFS with Remotes through the loader before calling the executable.
The TensorStore paragraph now says the store holds the shapes and the
loader fills the buffers.
zml.io.next goes: nothing in zml or llmd names it any more.
The tutorial's program and every snippet of the guide were built and run
on 4 CPU devices from a scratch target that is not committed: the
tutorial prints {6,11,16} from the Python-written checkpoint, and the
placement (host_mapped included) and composite snippets load the values
they describe. The program snippet stands alone: it defines its sources
and its forward dequantizes by block and places its result as the output
is, without which the load fails with InvalidBinding; on an 8x256 weight
with one scale per row and 128 columns it loads every value as expected,
row-sharded.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
HTTP, HF, S3 and GCS now all run their requests through request.zig, so
vfs/parallel_read.zig has no user left and goes with its BUILD entry. A
remote read's width is its caller's: a backend-local pool would multiply
it and defeat the reader's memory bounds.
vfs.zig's header named only HTTP as following the open and read error
contract; it now states it for all four backends: a missing object is
FileNotFound (404, 410, or a path absent from an HF repository's
listing), a denied one PermissionDenied (401, 403), and S3 answers a
principal without s3:ListBucket 403 for a missing key too, so a probe of
an optional object treats both as absent. The header also says that a
Loader may keep its source files open until its deinit, so the VFS and
its backends must outlive it, and that it is the loader that asks
fromIo. Remotes' doc points at the contract, and S3's HEAD doc now
refers to the header instead of restating the 403 rule.
No behaviour change; //vfs:test, //zml:test, //stdx:test and the
examples build on CPU.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
HF's repository listing, download-URL lookups and reads now run through
request.zig, under a governor named hf for which a 503 is a server
failure. HF.init takes Options{token, retry} in place of a token and
InitOpts; auto is unchanged. A file's download URL is resolved once at
open with HeadChain (at most 8 hops) and kept on its handle, so a read
is one Range GET to where the Hub's redirect chain ends, not a redirect
chain per 32 MiB chunk. The 32-worker parallel_read pool goes: a read's
width is its caller's. The token is sent to https://huggingface.co only:
never to the CDN a resolve URL redirects to, a look-alike host or plain
http, as a unit test checks. A 401 or 403 on a data GET, as a signed URL
that expired during a long hold answers, re-resolves the URL once and
stores it on the handle for later reads. The listing is fetched under
its own mutex, so reads of open files go on while it waits; the cache
was read without a lock and the fetch held the handle mutex.
Also: stat of a path absent from the listing is FileNotFound, not IsDir,
and opening a directory of the listing is IsDir, as its stat is, without
a request; a listing without Content-Length fails instead of panicking,
and one that fails partway frees the nodes it built. Opening, stat-ing
and probing follow the open error contract (a gated repository without a
token opens as PermissionDenied); a failed positional read is Canceled
or Unexpected and a failed streaming read InputOutput, each logged once.
A seek saturates, and deinit logs the read totals at debug.
Validated on CPU against the pool, both on the new loader, interleaved,
sharded on 4 devices. load_check of hf://LiquidAI/LFM2.5-1.2B-Thinking
loads 2.18 GiB in 2.71 s against 2.68 s with the pool (medians of 5,
about 830 MiB/s, this host's network), with 76 requests and no retries;
with every tensor checked back, the run takes 35 s instead of 50 s,
since a check read no longer walks the redirect chain. Of
hf://Qwen/Qwen3.5-9B it loads 17.98 GiB in 19.41-19.52 s against
19.36-19.37 s with the pool (about 950 MiB/s), with 603 requests and no
retries, and a run checking every 8th tensor and the largest matched.
With the VFS's debug log on, examples/io cat of
hf://Qwen/Qwen3.5-9B/config.json takes 4 requests and safetensors 18,
listing the same tensors as before; a missing file is FileNotFound. Not
run: a gated repository with a token (none here), a re-resolve after a
real expiry (the mock server cannot stand in for huggingface.co), and
the B70 runs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
GCS's lookups, listing, reads and token refreshes now run through
request.zig. GCS.init takes Options{credentials, endpoint_url, retry}
instead of InitOpts, and its governor, named gcs, treats a 503 as rate
limiting, since GCS answers it beside 429. A positional read is one
range_read GET; the 32-worker, 16 MiB parallel_read pool goes, and
InitError loses ConcurrentError with it.
A token refresh, from the metadata server or the ADC token endpoint, is
a governed request run under a mutex of the token's own, not the handle
mutex, since it may wait out a hold or retries for minutes. Each attempt
copies the bearer header into storage of its own: the slice handed out
before could be rewritten by a concurrent refresh, and a token that
expired during a hold is refreshed before the next try. A refresh that
fails, other than by a cancel, fails the readers queued behind it,
unlogged, instead of each sending it again with a retry budget of its
own, which kept a load waiting out one failed refresh per reader. A
refused token exchange is logged with up to 1 KiB of its body, where
the endpoint says why (invalid_grant, invalid_client), as the client
logged it before: request.zig's exchange gains an opt-in option for it,
so the other backends log as they did.
Opening, stat-ing and probing an object follow the open error contract,
and access sends a HEAD where it succeeded for any path. The listing
passes a cancel through and logs any other failure once. A failed
positional read is Canceled or Unexpected and a failed streaming read
InputOutput, each logged once. A seek saturates at 0 instead of
overflowing on minInt(i64), and deinit logs the read totals at debug.
The acceptance tests run the open contract, the streaming read and seek
case, and the 503 hold against gs:// as well.
GCS's internal Config gains the metadata server and token endpoint
URLs, which the Options leave unchanged, so that unit tests can serve
tokens and an object from a local server: 16 readers after an expiry
share one refresh and each sends the new token; a refresh refused with
401, or failing with 500 through 2 retries, is sent once (2 and 4 token
requests with the open's) and the 16 readers queued behind it fail; the
ADC refresh resends the same form after a 429, which holds the backend,
and after a 500, which takes one retry.
Checked on real GCS anonymously, through a public bucket
(vertex-model-garden-public-us): load_check of Llama-3.2-1B-Instruct
(2.30 GiB, sharded on 4 CPU devices) matches every tensor's bytes, and
four interleaved release runs load it in a median of 5.96 s against 5.98
s with the pool before this change, about 395 MiB/s, here in 152
requests and no retries. ls, stat, cat and safetensors work, a missing
object is FileNotFound and a denied one PermissionDenied. No GCS
credentials were available, so neither the metadata server nor the ADC
refresh ever got a token; with a bogus ADC file, opening
model.safetensors fails once, logging the token endpoint's 401 with its
body (invalid_client), and a local server answering the token POST with
400 and Google's JSON error gets it logged on one line. Someone with a
bucket and credentials should run load_check against it before merge.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
S3's lookups, listing and reads now run through request.zig. S3.Options
holds the keys, endpoint and region of the old Config plus a retry
policy, and replaces Config and InitOpts; requests are always signed for
the s3 service. The governor, named s3, treats a 503 as a throttle,
since S3 answers SlowDown with it, so it holds the backend's requests
instead of charging retries. Each attempt signs anew, because the SigV4
signature covers x-amz-date and the Range header: the HEAD that sizes an
object, the bucket listing, and the one Range GET per positional read.
The 32-worker pool that split each read into 16 MiB GETs goes: the
reader's width is the backend's concurrency.
Opening, stat-ing and probing a key follow the open contract, so a 403
is PermissionDenied, which is also what S3 answers a principal without
s3:ListBucket for a missing key. access sends a HEAD where it succeeded
for any path, a canceled listing or read stays Canceled, and any other
failed listing is logged once. A seek back saturates at 0 instead of
overflowing on minInt(i64), and deinit logs the read totals at debug.
The acceptance tests run the open contract, the refused streaming read
and seek, and the 503 hold on S3 beside HTTP, and the VFS's test of
refused writes registers S3 on the mock server and checks that no
request reaches it and nothing lands on local disk, where S3's base
layer would write. A unit test pins the SigV4 Authorization of the
HEAD, the listing and a Range GET, at a fixed date, to botocore's.
Checked against a local MinIO: signed HEAD, listing and Range GETs
succeed, a wrong secret or a bucket the principal may not read opens as
PermissionDenied (Unexpected before), and a missing key as FileNotFound.
load_check of LiquidAI/LFM2.5-1.2B-Thinking over s3:// on CPU, 8
interleaved runs against the pooled backend: 2.18 GiB in 310.6 ms
median (302.7-316.4) against 306.9 (294.4-324.8), with the same 3 HEADs
and 141 GETs and no retries; bytes match on every run. With the byte
check, whose reads are one tensor at a time, the pool's split made the
check faster: 1.58 s against 1.32 s median, 289 GETs against 362.
MinIO answers 404 for a missing key whatever the principal's
s3:ListBucket, so the 403 for a missing key was checked on the mock
only; real AWS was not reachable from here.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
These served only the deleted loader or its callers:
- safetensors.TensorReader, Tensor.reader and the registry's reader,
iterator and totalBytes; Tensor.read reads a whole tensor, and a
store's views look tensors up;
- mem's DmaAllocator, UninitializedBufferAllocator, DmaMapAllocator
and DynamicBufferPool: the direct backend pins its own host memory,
so mem.zig keeps only the buffer-tree helpers;
- stdx.Io.LimitedGroup;
- nn.Linear.unloadBuffers and
Buffer.freeDeviceMemoryButKeepHostMetadataMemory:
mem.deinitBufferized frees a whole tree, without a hand-written
walker that misses a field.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Nothing in zml uses the previous store and loader any more: the
borrowing TensorStore with its getReader and maybeCreateBinding, the
Loader with load, loadBuffer, loadExecute and await, ProgressWriter,
MemoryWriter, BufferedMemoryWriter, DirectShardWriter, DispatchSpans
and DirectMemoryWriter, their tests, and the imports only they used.
io.zig keeps the VFS and the io/ tree's exports.
The mesh validation comment that pointed at DirectMemoryWriter's tests
goes with them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
testLayer counted its references by scanning the registry, which the
store now owns, and loaded its inputs with zml.io.load, which does not
exist, so it compiled only while nothing called it. It now counts the
inputs and outputs under their view's prefix, loads the inputs with
testing.load, frees them with deinitBufferized, and has a test of its
own over a scale layer. Its signature is unchanged. storeOf, load and
expectShards name zml.io.
Buffer's doc says zml.io.Loader fills buffers from a checkpoint, the
extra device memory option names zml.io's hostMappedBytes, and the MoE
experts layout doc no longer names a packer.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The README's taste of ZML called zml.io.load, which does not exist,
and a hand-written unloadBuffers. It now ends with a main that
creates the loader before compiling, binds the model through a
TensorStore view, and loads it with start and wait into buffers freed
by deinitBufferized.
The torch2zml howto loads its layer with zml.testing.load, the
convenience for tests, and frees the weights with deinitBufferized.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The examples load through zml.io.Loader: bufferize the model, start,
wait, and free the buffers with deinitBufferized. The loader logs the
bytes, time and rate itself, so the hand-written timers go.
examples/llm creates its loader in a block that initializes, compiles
and loads the model: the loader sets its backend up while the model
compiles, and the block frees it, with its pinned host memory, before
the chat starts. The weights still load after compilation, to leave
room for autotuning. The lfm2 and llama tests take a loader with
default options and pass their live progress node. mnist and mnist_dp
read their checkpoint through a VFS, create the loader before
compiling, and load in main, without a Mnist.load or unloadBuffers.
examples/io's load is the same recipe over every checkpoint tensor,
and load_check names zml.io.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zml.io.Loader and zml.io.TensorStore are now the loader and store of
the io/ tree, with dma_calibration beside them, and io.zig opens with
a map of that tree's files. zml.io.next stays as a second name for the
module, so code written against it builds unchanged. The io tree's
docs and messages name zml.io.
CPU now loads through the direct backend, as CUDA, ROCm and oneAPI do.
On 4 CPU devices, Llama-3.1-8B-Instruct sharded by rows loads in about
1.60 s, against 1.78 s through the buffered backend and 9.6 s through
the previous loader.
The previous store and loader stay in io.zig under private names, with
their tests, until the following commits move their callers and delete
them; the examples build again once they move.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A loader test serves four tensors from a MockServer that lets two GETs
through per 100 ms and answers the others with 429, and loads them over
the VFS's HTTP backend with four readers. Every tensor must land intact,
the governor must have throttled and held the readers without spending
a retry, and the physical bytes must equal the tensors'. The direct
backend must size its requests from the HTTP backend's hints rather
than the local ones and read the remote file exact, without direct I/O.
Its destroy, which deinit runs, asserts that every lifecycle credit came
back once the workers stopped; the test does not read the credits
before, while a worker may still hold one for a claim that finds
nothing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
dma_calibration.measure runs the direct backend's DMA calibration
without a loader: it warms the device allocators, screens the blocks in
a workspace of its own, released before it returns, and returns the
block it selects. Where the loader only picks the candidate blocks,
measure takes every knob of Options and checks them first: an empty
block list, a block past the largest source request, a parallelism
outside 1..32, a window that is not positive or a tolerance outside
[0, 1) is InvalidOptions, logged. A platform that loads through the
buffered backend has no DMA to measure and is UnsupportedPlatform.
zml.io.next now exports dma_calibration.
`load_check dma-bench` is measure's caller. It takes the screen's knobs
from ZML_DMA_BENCH_* (candidate blocks in MiB, parallelism, window
length, transfer target, tolerance) and the platform's from the same
variables as a load, and logs the block selected. load_check also
enables the zml/io scope at debug, where the screen logs its pick and
the rate it measured on one device.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A load that reads at least four fifths of its checkpoint's bytes past
the page cache (ZML_DIRECT_IO=on, or a cold checkpoint under auto)
interleaves its page-backed arenas over every allowed node instead of
the devices' nodes: the drives write into them, and interleaving splits
that traffic, and the copies out of them, between the nodes' memory.
Before calibration maps the first arena, the direct backend predicts
how the files the registry reads for the checkpoint
(safetensors.checkpointFiles) will read, each with the VFS's own rule
(VFS.predictDirectIo), and a debug line reports the prediction.
Workspace.init takes the prediction as direct_reads.
Each of those files then reads as predicted: the planner keeps the
answers by real path, as the registry names the files, and under auto
decides a file predicted direct under on and one predicted buffered
under off, rather than sampling its residency again; under on, which
samples nothing, every file keeps on. The prediction's sample queues
the read of each page it misses and of the kernel's readahead, and a
second sample of those pages once the reads have landed finds the file
cached: on the GB300 host below (64 KiB pages), sampling again when
planning read 3 of the 330 files of five cold loads buffered, at 75 to
97% cached. A file whose residency changes before it is planned costs
speed, never bytes, and under on a filesystem that refuses the flag
still reads it buffered and logs why.
Loader.init copies the checkpoint's path for that set-up when it is
local and the VFS may read it direct, so it may also return
OutOfMemory; the set-up task frees the copy, canceled or not.
Under zio the prediction takes a pool worker of its own
(zio.blockInPlaceReserved), as Compiler.compile does. zio hands the file
operations its event loop does not run (a real path under io_uring,
every one under epoll) to its blocking pool, which adds a worker for
such a job only when none is idle and its queue holds twice as many
jobs as it has workers, and the compiles hold theirs for their whole
run. Queued there, llmd's set-up on the GB300 host below waited for the
first compile to end: it placed its arenas 49.8 and 51.9 s after llmd
started, against 1.6 and 1.9 s on a worker of its own, and with two
compiles of 28 s the load of one process in two waited about 0.3 s for
it.
On two GB300 of one Grace node, over an md RAID0 of four NVMe drives two
on each node, Qwen3.8-27B-FP8 forced direct loaded in 1.752 s with the
arenas interleaved over both nodes, against 1.873 s with them on the
devices' node, in the same session: the drives read as fast either way
(1.90 against 1.87 ms per 16 MiB read), and staging a read from the
arenas took 0.52 against 0.64 ms. Cold under auto, from a private copy
on the same RAID evicted before each load, it loaded in 1.658 against
1.772 s, and in 1.690 s on the original branch, in another session: a
16 MiB read took 1.87 ms either way, and staging it 0.49 against
0.69 ms. Each time is the median of five loads. That is about 4.2 and
4.0 ms per GiB saved on direct reads, against 10 to 16 ms per GiB that
spreading costs reads through the page cache, so spreading pays from
about 0.71 to 0.80 of the bytes read direct, and four fifths, at the top
of that range, leans to the devices' nodes. Page-cached, the checkpoint
keeps loading from the devices' node.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The page-backed DMA arenas (CUDA, oneAPI and CPU) are now interleaved
with mbind over the host's memory-bearing NUMA nodes, the ones
/sys/devices/system/node/has_memory lists, which leaves out
device-coherent HBM nodes, narrowed to the nodes of the platform's
devices. The set is masked by the task's cpuset and memory policy:
arenas stay within the nodes both allow; among several, they go to the
devices' nodes. When a device reports no node (the CPU plugin reports
none), or none of theirs is allowed, they interleave over every allowed
node. With a single allowed memory-bearing node there is nothing to
choose and the kernel places them. If the kernel refuses the policy, it
is logged once at warn and every later arena stays unplaced. ROCm and
Furiosa's pjrt_host arenas keep their per-node allocation paths.
On four GB300 over two nodes, warm replicated DeepSeek-V4-Flash loaded
in 4.35 s from one pool interleaved over both against 4.92 s from
node-local pools. On two GB300 of one Grace node, Qwen3.8-27B-FP8 loaded
in 1.265 s with a task policy holding the arenas to that node against
1.64 s interleaved over both nodes. The arena debug line now reports the
placement and its node mask.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
wait published a program group only once no other one was pending, so
one group was in flight at a time whatever the devices held. It now
admits a group as soon as it fits the room every device reports: the
allocator's limit, minus the bytes in use, minus the bytes the direct
backend was submitted but has not allocated yet, minus a 64 MiB
reserve. A group costs its inputs' placements, its outputs' and the
largest temporaries of its programs, from the executables' compiled
memory stats (zero, logged once at debug, where the plugin gives none),
beside the outputs and temporaries of the groups still pending. Pending
submissions retire, oldest first, until the group fits; once no program
group is pending, the room is read again after the retired groups'
memory is given back, and the group goes even if it is still short,
since the room is an estimate and can read low. A group that does not
fit a device fails with the device's error, which wait returns; a
smaller program_group_bytes makes smaller groups.
Freed device memory stays counted in bytes_in_use until it is given back
(Platform.releaseFreedDeviceMemory): XLA's VMM allocator, the default
for CUDA and ROCm platforms, counts a freed allocation until it is
reused or reclaimed. start gives back what was freed before the load,
the compiles' scratch and the set-up's calibration buffers, so that
neither the room nor the load's peak counts it. Admission gives back the
retired groups' inputs only for a group that would otherwise go beyond
the room, so the allocator still reuses them for the next group of the
same shapes in the common case. Through llmd on two GB300, the one
DeepSeek-V4.1-Flash group per load that goes beyond the room without the
drain (3.4 to 4.3 GiB of room for 4.7 GiB) fits in 15.5 to 22.6 GiB, and
MiMo-V2.6-Flash's peak device memory in use is 178-184 GB per device
against 184-188 GB without these drains, for at most 49 ms per call on
two devices.
The direct backend counts each sealed batch's device bytes, a box per
device of every destination outside host memory, and the bytes it
allocates for outputs; their difference, saturating, is what it still
owes the devices. Admission measures only when the backend is direct
and every device reports a limit at the first start, so CPU and the
buffered backend keep one group in flight. A debug line for each
admission that retired or went beyond the room names the device the fit
is decided on, and one at deinit reports the retires, the groups
admitted beyond the room and the least room seen.
Tests cover the room and fit arithmetic and the device the fit is
decided on, check that the direct backend allocated exactly what was
sealed for bulk, program and composite loads, and that host bindings
count on neither side.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A tensor bound to host_mapped memory now loads: its bytes stay in one
pinned host allocation per box, and each device's shard is a view of it
in the device's default memory space, so kernels read the host memory
in place. Its shards are always views, replicated or not, and the
allocation goes with the last of them through the views' PJRT deletion
callbacks; the loader keeps no owner. supportsMemory accepts both host
kinds where the platform loads directly and every device has a pinned
host space, and the front end refuses a host kind elsewhere.
The front end's tests load a host_pinned and a host_mapped binding next
to a device one, or check the refusal where the platform has no pinned
host space, and gather rows of a padded table delivered host_mapped,
sharded and replicated, on every device while it is freed behind queued
matrix products, checking that its allocations outlive the gather and
go once it is done and, where the plugin counts allocations, that the
table took no device memory. The host-resident, failure and memory-kind
loops run host_mapped too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A replicated host_pinned destination kept a buffer per device, and its
workers wrote every byte once per device. The devices holding one box
now share one allocation: the first of them creates an owner buffer in
its pinned host space, and every device's shard is a view of it. Each
view holds a reference that its PJRT deletion callback drops, and the
last drop destroys the owner, so the allocation outlives every reader
whatever order the shards are freed in, the loader included. A host
transfer is written once, through its first writer's shard.
Writing through one view shows on every device, so a shard of such a
destination must not be donated to an execution that writes it.
The host-resident test checks that the replicated outputs share one
pointer and one live allocation each, and that freeing the shards in
reverse order releases them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The direct backend now takes tensors bound to host_pinned memory, and
no DMA goes to them: the worker that reads a host destination's bytes
writes them into its PJRT host buffers. A host job's read goes straight
into the destination ranges when it has no alignment padding and its
transfers address each byte once, in order; a read widened for direct
I/O, or one that overlapping parts read twice, lands in the request's
blocks and is copied from there. The bytes no part covers are zeroed in
place by a host fill job. Host and DMA parts never share a job, and host
jobs take no DMA events. Each device of a replicated host destination
holds a buffer of its own, written from the first device's.
A first pinned allocation is slow on some drivers, so a task of the
worker group creates a batch's host destinations ahead of their reads,
in claim order, holding one completion unit of the batch; a worker that
finds its destination missing still creates it. The batch planning and
summary debug lines count host jobs and the time spent ahead.
supportsMemory accepts host_pinned where the platform loads directly
and every device has a pinned host space; host_mapped stays refused.
The composite replays, the failure loop and the loader's memory-kind
test run host_pinned on CPU, and a new test loads rows, columns,
replicas, an overlap and a device tensor from one file, read exact and
widened. load_check binds every tensor to ZML_LOAD_MEMORY.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A source whose strided parts do not expand into contiguous ones (a row
interleave, a transpose, a broadcast, a block-scale swizzle) was
refused; it now goes to the backend as strided parts, and the direct
backend plans its destination whole. The destination's logical bytes are
cut into ranges of at most a request, less two units of the widest
alignment a file may take. Each range is one landing job: for every part
with runs in it, the job reads the source range of those runs, or the
part's whole source range when its runs are out of order, from whichever
file holds it, copies the runs into its blocks box by box, zeroes the
bytes no part covers, and sends each box as one piece per block. The
copy loops over whole rows of the two innermost dimensions, and moves
runs of 1, 2, 4, 8 or 16 bytes as one word.
Since such a job may read a part's whole source range, the front end
asks the backend what each strided part reads and refuses at start, as
InvalidBinding logged with both numbers, a part that needs more than one
read: the request of the smallest block calibration may pick, so a
refusal does not depend on the pick. It also refuses strided parts into
host memory. The buffered backend reads any part.
The tests load row interleaves across files, broadcasts, transposes and
a block-scale swizzle over one, two and four devices, with reads exact
and widened, check the read bounds of local and remote parts, and fuzz
copyRuns at word widths.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A read job of a tensor sharded along any axis but its first went to the
devices in file order, one DMA piece per row and device: Llama-3.1-8B
with o_proj and down_proj column-sharded over four CPU devices planned
1.06M submissions. The planner now lists a job's segments before it
places them, and stages the job where that saves at least one device's
in-flight pieces (64) and the segments tile its bytes once: the bytes
land in the job's blocks grouped by destination box, so a box goes out
as one piece per block. The same load plans about 10,500 submissions.
A staged job reads one extent. An exact read scatters straight into the
blocks, at most one vectored read's ranges per call; a read widened for
direct I/O lands in the worker's page-aligned bounce buffer and is
copied from there. The batch planning debug line counts staged jobs.
The composite replays also run staged, with reads exact and widened,
and a new test loads column-sharded tensors over four devices in under
a hundredth of their row pieces.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A source lowered into several parts, some of them strided, now loads
as one destination. The front end expands a strided part of at most 64
runs, or of runs of 1 MiB or more, into one contiguous part per run,
sorts the parts by destination offset and checks them; a spec with any
other strided part is still refused. Padding is no longer refused on
the platforms that load through the direct backend.
The direct backend fills every device's placement exactly: the bytes
no part covers go out with the plan of the destination's first part,
as a last fill job that reads nothing and transfers from the zero
block, cut by blocks and merged per writer set like read bytes. The
batch planning debug line counts fill jobs apart from source jobs.
The new tests replay planned composites over four and eight CPU
devices, sharded by rows, by columns or replicated, and check that
every device byte is written once, from its source or from zeros; the
front end's tests load concatenated, padded, interleaved, stacked and
split sources, and the buffered backend loads a padded one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zml.io.next.Loader.Options gains the checkpoint, a DMA block and a
direct-I/O policy, and its read width becomes optional. When io is a
VFS, init asks the VFS backend serving the checkpoint for its read
hints, which size the source requests, and takes a default width of 16
for a local checkpoint and 32 for a high-latency one. It checks the
width, the hints, and every block calibration may select against the
request size and the direct-I/O alignment, and refuses the rest as
InvalidOptions, logged.
The direct backend asks the VFS once per local file whether it reads
the file directly. Such a file is read in whole alignment units: each
job's read widens to aligned bounds around its range and its transfers
address the widened read; a job covers at most a request less two
units, so the widened read still fits its slot. A remote file is read
exact without being opened by the planner, and so is every file of a
plain io or of the off policy.
load_check takes the read width, the DMA block and the direct-I/O
policy from ZML_LOAD_READ_PARALLELISM, ZML_DMA_BENCH_BLOCK_MIB and
ZML_DIRECT_IO, and passes the checkpoint to the loader.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zml.io.next exports the new Loader and its TensorStore next to the
zml.io.Loader and zml.io.TensorStore that models still use. It is
unstable, and goes away once the new loader replaces the old one.
zml.testing's loader helpers name it instead of importing its files by
path, and the docs that pointed at the unexported store now name it.
//examples/io:load_check is the tool the new loader's backends are
validated with on each host. It binds every tensor of a checkpoint,
local or remote, replicated, split along its first axis, or as tensor
parallelism splits o_proj and down_proj, loads them through
zml.io.next, and with ZML_LOAD_CHECK reads them back and compares every
replica with the checkpoint's bytes. The loader logs its rate and its
stats on the zml/io/load scope. The knobs, listed in the file's header,
also set a GPU memory pool fraction and the CPU device count. The
examples/io demo keeps the old loader.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The direct backend now counts what an A/B run needs to explain a load
time, in atomic counters the hot paths bump: reads and their bytes and
time, the time workers wait for a lifecycle credit, how long a request
holds its credit in the DMA stage, the time to create a tensor's
buffers and transfer managers, DMA submissions with their submit and
completion latency, and why each device pump stopped (empty queue or
full budget).
At debug level on the zml/io/load scope, each awaited batch logs a
'batch completed' line (when it was published, sealed and done, from the
loader's creation) and a 'batch planning' line (plans, planning time,
jobs, transfers, planned DMA submissions, coalescing ratio, average read
size), which is only computed when it would be logged. destroy logs the
'loader summary', 'loader waits' and 'loader DMA' lines, with the credits'
high water and the pinned bytes mapped. Nothing reads the counters but
these logs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The direct backend now takes its DMA block from dma_calibration.calibrate
instead of one fixed 16 MiB block. create calibrates in its own
workspace before it sizes anything else, so the ring the screen measured
in stays mapped and the pre-growth maps only what it leaves; the debug
line reports both. Each device's DMA budget is dma_calibration.depth
blocks, which moves there from direct_loader.zig, and backend.Options
.dmaBlockSizes returns the default candidates, 2 to 32 MiB, unless the
options pin one block. CPU takes the smallest candidate, 2 MiB, which
maps a quarter of the pinned set the fixed block did. A failed or
canceled calibration fails the set-up, logged once.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A short screen window is noisy, and a candidate near the selection floor
could be picked or dropped by chance. A candidate is now borderline when
its rounds disagree on whether it is near the peak, or when its median
ratio lies within 0.02 of the tolerance's floor. Each borderline
candidate runs three paired 25 ms, 256-transfer windows against the
peak, alternating which goes first, and its median ratio from those
windows replaces its screen ratio before pick. A confirmation can rate a
larger block above the screen's peak, so pick takes the best ratio from
either source.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Adds zml/io/dma_calibration.zig, which picks the direct backend's DMA
block once per loader. calibrate maps one ring for the largest
candidate in the loader's workspace, where it stays as the load's first
pinned capacity, and opens one asynchronous host-to-device transfer
manager on device 0, sliced per candidate. Each candidate (2 to 32 MiB
by default) is warmed, then measured over three rounds of fixed windows
(at least 2 ms and 32 transfers, depth transfers in flight, rotated
order), and pick returns the smallest block whose median rate is within
8% of the best. The windows' workers wait behind a start gate and are
joined on every exit, a canceled sleep included. CPU returns the
smallest candidate without measuring: its transfers are memcpys, and a
load takes the same time at every block size.
The direct backend does not call it yet.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Backend and Submission unions gain the direct arm, and
backend.loadsDirect decides per target: CUDA, ROCm, oneAPI, CPU and
Furiosa load through the direct backend; TPU, neuron and Metal stay on
the buffered one until their transfer path is checked on hardware. ROCm
and Furiosa stage in PJRT pinned_host arenas. Furiosa has never run this
backend. supportsMemory still accepts device memory only.
CPU goes direct as well, so the front end's tests on CPU now exercise
the direct backend. The direct backend writes only the bytes its parts
cover, so on a platform that loads through it the front end refuses a
padded source with InvalidBinding instead of handing it a destination it
would leave partly unwritten. A new test drives the front end over the
buffered backend on CPU, so both arms stay covered.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
submit already fails the direct loader when planning a submission
fails, and awaits what it published. A failed loader now also refuses a
submit with its first error before it builds a batch: every later
submission sees the failure that stopped the loader, and none creates
destinations or plans it cannot run.
The tests drive the failure paths a submission can take on CPU. With the
planner's allocations failing one after another, planning fails before
any plan is published, then after the first file's plan was, and past
the last the read of a truncated source fails. Each failure fails the
loader, leaves nothing queued and, once the workers stopped, every credit
and read permit free, counts the batch once, and makes later submits
return the same error. A loader that fails between a submit's open
check and its first publish, as a worker of an older batch can make it,
returns that failure rather than the scheduler's Canceled.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The file header describes the direct backend as the code now has it: how
create sizes the block, the request, the read width, the DMA reserve and
the pinned set once; how a submission is planned per file, claimed in
order, read into a credit's blocks and handed to per-device pumps, with
the last-transfer flag set by bytes; how a destination's bytes are
placed on its devices; and the order in which the runtime's locks nest.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
create now sizes the pinned set for the whole load before the first
read. The DMA block is the options' one, 16 MiB unless they pin
another, and a source request is the read chunk rounded up to whole
blocks. Beside the workers' requests the workspace keeps a DMA reserve,
eight blocks per device and at least one maximal request, which become
lifecycle credits beyond the workers' own, so transfers still holding
blocks keep the DMA engines fed: one credit beyond the read width moved
24.3 GiB/s on a GB300 loading DeepSeek-V4, against 43.8 GiB/s with eight
requests queued. The read width is clipped to what the 16 GiB mapped
ceiling leaves beside the reserve; when not even one request fits,
create logs the sizes and fails with OutOfMemory.
The workspace maps the reserve as one arena first, then the rest, since
ROCm balances host memory per allocation, and cuts everything into
blocks once, so nothing maps during a load. The first block is a block
of zeros that no credit owns.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Loader is the direct backend that loads end to end: whole tensors into
device memory, with exact reads. create carves the workspace into the
lifecycle credits' slots and starts width + 1 workers, which share width
read permits, so a full width of reads is in flight while one worker
queues what it read. Each worker takes a credit before it claims a job,
so a claimed job never waits for its blocks.
submit creates the batch's destinations and parts in its arena, then
plans and publishes one file at a time and drops the publish sentinel. A
planning failure fails the loader, since a partial submission can never
complete. The workers may already hold the plans it published, so the
batch cannot just be destroyed: submit seals it, awaits what was
published and returns the first error. awaitBatch waits for the batch,
checks that a clean batch submitted every byte of every target, retires
its events and destroys it with its transfer managers, whose destructor
errors every buffer a failure left unfinished, so such an output never
reports success. abort fails the loader; destroy stops the workers and
frees everything.
Each source path gets one slot holding its own copy of the path, opened
by the first read and closed by destroy. Every read takes a permit and
stops once the loader failed. recordError keeps the first error and logs
it once, retires every unclaimed job and releases every queued piece;
any thread may call it, a DMA callback included. Each destination
completes one progress item when its last byte is read.
The tests cover request, credit and batch lifecycles on a loader without
PJRT, and whole loads on CPU: the read width in flight, a read failing
mid-load, a failed load's unfinished outputs on two devices, source
slots, and tensors sharded by rows or columns, replicated or joined from
two files on four devices.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A request's transfer pieces go to per-device pumps. Each device has one
queue and runs at most one pump at a time, on whichever thread finds
work: a worker after it queued a job, or a completion callback. One
submitter per device is what lets a target's last-transfer flag go by
bytes. A pump submits queued pieces in arrival order while the device
has room under its budget: eight blocks of bytes in flight, and at most
64 pieces, so a model of small tensors cannot flood a device with
events. The piece that crosses the byte budget is admitted, so a device
with room always has one in flight. Per-device pumps loaded replicated
DeepSeek on four GB300 in 4.68 s, against 5.41 s with one pump for
every device.
enqueue counts a job's pieces per device, locks every pump in device
order, checks the loader's sticky error, reserves every queue and only
then queues the whole job, so a request is fully queued or not at all;
queuing piece by piece cost tens of thousands of lock and pump trips per
DeepSeek load. Each submission takes a preallocated event context from
its plan. A completion frees its device's room, which may pump more,
then hands its fired event to the device's next pump to destroy, never
inside the event's own callback, then releases its request last, since
that release may complete and free the batch. retireBatch destroys what
is left with every pump locked. Once the loader failed, abortReady
releases every queued piece, so no request waits on a pump that stopped.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A plan now allocates every context its jobs can need before it is
published: one ReadRequest per job, idle until a worker claims the job,
and one event context per planned DMA submission, a writer of a
transfer each. The batch's arena keeps them until the batch retires, so
callbacks can hold their addresses.
A ReadRequest is a claimed job with the worker's lifecycle credit. Its
run creates the device state of the job's destinations if no worker has,
reads the job into its slot's blocks in file order, counts the bytes
read toward each destination and queues the job's transfer pieces for
DMA. It holds a scheduling reference and one per queued piece; the last
release gives the credit back, then releases the batch's completion
unit, which may free the batch, so nothing is touched after it.
This and the next commit refer to the DMA stage and the loader that the
runtime commit adds: the three are mutually recursive and build together.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A Batch is one submission to the direct backend: its plans, one per
source file, and the destinations and parts they write, all in the
batch's arena. It counts completion units, one per published job plus a
publish sentinel held until the submission is sealed. Each job's unit is
released exactly once, by its request's last reference or by a failure
that retires it unclaimed; the last one sets the batch's done event, and
releasing a unit is the releaser's last access to the batch.
The Scheduler is a strict FIFO of published plans. A plan, its
completion units and its queue entry appear in one critical section, so
a queued plan's batch always holds a unit and is never freed under the
scheduler, and workers need no rendezvous with the awaiting task. Jobs
are claimed in plan order, a plan leaving the queue with its last claim.
fail stops claims and publishes, wakes waiting workers and retires every
queued plan's unclaimed units, so a failed batch still completes through
the requests already claimed.
Planner.publishFiles plans and publishes one file at a time, so the
first file can be read while the rest is planned: planning the whole
model first delayed DeepSeek's first read by 0.32 s on one MI300X.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A destination's device state is one PJRT host-to-device transfer
manager and buffer per device of its grid, in canonical order, created
once through LazyOnce by whichever worker first reads for it. Creating
them also writes the caller's output buffer, once. On failure, the
managers created so far are destroyed with their buffers.
A Target tracks one device buffer's submitted bytes against its total.
PJRT readies a buffer once every transfer to it has completed and one
carried the last-transfer flag, whatever their order, so the flag goes
on the submission that completes the buffer's bytes, never on the
highest-offset piece: flagging by offset makes tails wait for their
prefixes, which can deadlock a sharded load whose tails hold every
credit while their prefixes stay unclaimed. This follows XLA's
CommonAsyncHostToDeviceTransferManager.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Credits are the direct backend's lifecycle credits: each is a fixed
slot of pinned blocks, one source request's worth. A worker will take a
credit before it claims a job and its request gives it back after its
last DMA callback, so only the holder reads into and transfers from a
slot, and the credits are the one bound on the pinned memory in use.
acquire waits for a free slot and release wakes one waiter, not all of
them. They also count the most slots held at once.
There can be more credits than reads in flight, so transfers still
holding their blocks keep the DMA engines fed: on a GB300 loading
DeepSeek-V4, one credit beyond the read width moved 24.3 GiB/s against
43.8 GiB/s with eight requests queued.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
readFilePositionalAllV reads a file range into a list of buffers in
order, consuming the buffers as it goes. The io takes as many buffers
per call as it can, and a short read resumes where it stopped. Reading
stops once a minimum is in, so bytes a caller added past the data to
align a read may stay unread where the file ends, and a read that ends
before the minimum fails with UnexpectedEndOfFile. Every source read of
the direct backend will go through it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
LazyOnce(T) is a value the first task to touch it initializes, with the
initializer that task passes. Concurrent callers wait for it, and a
failed initialization keeps its error code and returns the same error
to every later caller. The direct backend will open each source file
and create each destination's device buffers through it, from whichever
worker reaches them first.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zml/io/direct_loader.zig starts the direct backend, which will read a
checkpoint's bytes into pinned host blocks and copy them to the devices
with PJRT host-to-device transfer managers, as a pure planner. It plans
whole tensors into device memory with exact reads. Nothing calls it yet;
io.zig's test block reaches it.
The backend works per source file. Parts, each one source range of a
destination, are sorted by file, offset and size, and each file becomes
one plan whose jobs are claimed in file order. Touching or overlapping
ranges merge into one run, never across a gap or a file, and each run is
cut into the fewest jobs of at most one request, at the latest boundary
where every earlier range has ended that still leaves room for the rest:
no part is split where a whole cut fits. A job's transfers place its
bytes on the devices through the destination's ShardGrid, one per run
and DMA block, merged where the block, the writers and both offsets
continue, so parts adjacent in the file and in the destination go out
as one transfer. Every transfer stays inside one block.
Coalescing this way took DeepSeek-V4-Flash from about 69,445 source
reads to 9,524, and cutting at tensor-safe boundaries kept 69,572 DMA
pieces where a rigid grid made 78,665.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On ROCm and Furiosa, the Workspace's arenas are PJRT pinned_host buffers:
an uninitialized u8 buffer in a device's host_pinned memory, retained
through an external reference until the workspace unmaps it, whose host
pointer the blocks use. Registering ordinary pages instead was far
slower on ROCm: with eight MI300X visible, hipHostRegister took about
6.3 s per GiB against 0.6-0.7 s for hipHostMalloc. The pointer belongs
to the PJRT buffer and is never DmaMapped, since unregistering
allocation-owned memory breaks the eventual hipHostFree. Furiosa backs
host_pinned with its SDK's dma-buf memory, which it recognizes by
address.
The workspace keeps one allocation path per host NUMA node its devices
report (Device.numaNode), or a single one when none does, and maps each
arena on the node with the fewest bytes so far: arenas differ in size,
and balancing bytes rather than arenas or devices kept replicated loads
on eight MI300X in their fast mode. Workspace.init can now fail with
OutOfMemory.
Transfers avoid staging only when the ROCm plugin recognizes these
ranges as pinned (IsHostMemoryPinned); no CPU test reaches this arm.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zml/io/host_memory.zig adds the host memory the direct backend will read
into and transfer from: a Workspace of loader-owned arenas, mapped one at
a time and then cut into blocks once. Nothing creates one yet outside its
tests; io.zig's test block reaches it.
CUDA and oneAPI arenas are pages registered with the PJRT client
(DmaMap); CPU arenas are plain pageable pages. Arenas that span a
transparent huge page are aligned to it and advised MADV_HUGEPAGE. A
failed allocation or registration is logged with its cause and returns
OutOfMemory; pages whose unregistration fails are leaked with a warning
rather than reused. The other targets panic, since only the direct
backend creates a workspace.
allocate retains one arena and refuses a growth past a fixed 16 GiB
mapped ceiling, a safety guard rather than a target. capacity counts the
whole blocks the arenas hold and the ceiling still allows. carve maps
the blocks missing below a first count as one arena, then those missing
below the total as another, and cuts every arena into blocks: nothing
maps after it, so nothing maps inside a load. The workspace owns the
arenas and the block list.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The new loader now computes the tensors a program binds
(Source.bindProgram) instead of refusing them. start checks a computed
tensor's placement like any other and queues it; the program tensors are
cut, in model order, into groups of at most Options.program_group_bytes
of outputs (8 GiB), so the inputs of one layer's experts, adjacent in
the checkpoint, are read together. Each input lands in an empty shell
through its own spec, checked as start checks every spec.
Compiles start at start, on io.concurrent tasks, or in place when io
cannot run one, so the bulk reads do not wait behind them. Tensors whose
programs are equal over the same placed shapes share one compile
(Compute.sameExecutable). Once compiled, an executable must return one
tensor of its binding's shape, placed alike on every device, or wait
fails with InvalidBinding naming the program.
wait publishes the groups oldest first. Before each, it waits for the
group's compiles, then retires pending submissions until no other
program group is pending, so one group is in flight at a time. Retiring
a group runs its programs on the caller's task through Exe.call, whose
errors fail the loader, then frees every input shell exactly once,
donated or not. A failed or canceled loader drops the queued groups and
retires the pending ones without running them. The info line wait logs
adds the tensors computed, the programs, the groups and the time spent
waiting for compiles.
A load gives back the device memory it freed. XLA's VMM allocator, the
default for CUDA and ROCm platforms, keeps a freed allocation mapped and
counted in bytes_in_use until a request of the same size reuses it, a
request beyond the device's limit reclaims it, or an executable with
remapped command buffers is destroyed, so without this the last groups'
inputs and the programs' temporaries would stay held after the load.
Platform.releaseFreedDeviceMemory asks each device for one byte beyond
its limit through pjrt's new Client.createUninitializedBufferSilent,
which logs its error at debug only: the allocator completes the deferred
frees of the device's memory, not those of collective or pinned host
memory, then fails with ResourceExhausted, as expected. Any other answer
is logged once and ends the VMM drain on that platform. The VMM drain
does nothing on platforms created with another allocator, which frees at
once (BFC on oneAPI and Metal, CPU, Furiosa), and on a device that
reports no limit. A GPU test checks that a freed buffer stops counting
in bytes_in_use once drained.
wait calls it once the programs are destroyed, when a load was started.
A failed or canceled loader does not: its caller frees most of the
load's memory afterwards, and after a device fault the request could
fail with that fault, logging a second line, or block. Through llmd on
two GB300, an earlier build of the stack, with this drain patched in at
the end of wait, brought the device memory in use after the load to
llmd's on monorepo master at 945cf926, to the byte, on MiMo-V2.6-Flash,
Qwen3.8-Flash-Next-FP8 and DeepSeek-V4.1-Flash, where the same build
without the drain held 2 to 17 GiB more per device. That drain took at
most 63 ms per call for both devices (76 ms for the same request made in
llmd right after the load).
On ROCm the drain also makes the HIP runtime give back the device memory
it keeps outside the allocator, which the plugin's memory stats never
count: chiefly the scratch memory of kernels that spill registers, such
as XLA's autotuning candidates. ROCr keeps that scratch bound to its
queue, up to 3 GiB per XCC by default, and keeps what a queue gives up
mapped in its scratch cache, until the driver refuses one of its
allocations. releaseRuntimeDeviceMemory in platforms/rocm therefore asks
each device, through the libamdhip64.so.7 the plugin loaded, for its
whole memory rounded down to 2 MiB: the driver refuses, ROCr reclaims
every queue's scratch and frees its caches, and hipMalloc fails with
hipErrorOutOfMemory, as expected. This runs whatever the allocator,
before the VMM drain, and makes running work on the device wait while
ROCr suspends and resumes each queue holding scratch. A missing HIP
runtime, or a device that does not refuse, is logged once as a warning
and ends it for the process. zml depends on //platforms/rocm directly
for this. A GPU test checks that every device refuses and still moves
data afterwards; the release itself shows only in the driver's counters.
Through llmd on MI300X, an earlier build of the stack's last commit ran
interleaved with one without this HIP release (its VMM drain in place).
On two GPUs, MiMo-V2.6-Flash's autotuning left 7.4 to 21.3 GiB of it at
ready on the GPU that ran it without the release (4 of 4 processes, on
both GPUs in one); with it both GPUs were within 238 MiB of llmd on
master (4 of 4), the autotuning GPU, 25.5 to 28.4 GiB above the other at
the end of the compiles in the 3 processes where only one GPU autotuned,
back at the other's level by the time llmd listened. On four GPUs,
LFM3-1.7B-FP8's 2.5 to 7.7 GiB on three of them was gone likewise.
Start-up time did not grow measurably, and decode speed and answers were
unchanged. A probe making the same HIP calls in llmd's process timed
each call at 3.3 to 4.5 ms on a GPU whose queue held scratch and under
0.5 ms on one with nothing to free.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Loader.init now checks its options and returns at once. A task of the
loader's own warms the device allocators and then creates the backend,
so a caller that creates its loader before building the model overlaps
that work with its own. When io cannot run the task concurrently, init
runs it in place. The first start waits for the task and fails the
loader with its error; deinit cancels a set-up no start waited for and
destroys what it built.
Platform.warmupDeviceAllocators forces lazy device allocators to reserve
their pools: a preallocated BFC arena materializes only on its first
allocation. It creates and frees a one-byte buffer per device, the
devices warmed concurrently or in place when io cannot spawn a task, and
returns the first error.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two helpers for tests of the new loader. storeOf builds a store over a
copy of a registry, since a store takes its registry's entries and a
test may want several stores over one checkpoint. load bufferizes a
model, loads it from a store with a loader of its own and returns the
buffers, for the caller to free with mem.deinitBufferized.
zml does not export the new loader, so testing.zig imports its files by
path. The loader's tests gain one that starts loads from two stores
before one wait.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zml/io/loader.zig is the new loader's front end, on the buffered backend.
zml does not export it: io.zig's old Loader still serves every caller,
and only io.zig's test block reaches the new file.
Loader.init checks the read width (1 to 128) and creates the backend.
start takes a whole model, its buffer tree and the store that bound it.
It pairs tensors with buffers by structure (mem.pairBufferized), so
buffers bufferized from another tree are InvalidBinding with the field
path where they differ, and it empties every buffer before anything is
written. Each bound tensor then becomes one LoadSpec, checked once: its
source is lowered to byte parts, an empty source or a run inside a byte
is refused, the placement must pack sub-byte elements into whole bytes,
divide over its mesh and name only the platform's devices, the memory
kind must be one the platform takes (device memory), and the parts go
through backend.checkParts. A source of strided runs (a transpose, a
broadcast, an interleave or interior padding) and a tensor a program
computes are refused: this loader assembles contiguous parts and runs no
program. Every check failure is InvalidBinding, logged with the tensor's
field path and its source's key. The specs are published as one
submission; several starts, from one store or several, may come before
one wait.
wait retires the submissions in publish order and logs the bytes, time
and rate once; it is idempotent. Every error of start or wait fails the
loader: the backend is aborted, the pending submissions are retired, and
every later call returns the first error. cancel does the same with
Canceled. After any failure each buffer is empty or whole, and
mem.deinitBufferized frees the tree. deinit panics on loads still
started, whose store and buffers may be gone.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The new loader's front end reaches its backend through backend.Backend:
init, submit, await, abort and destroy, dispatched with inline else, and
Submission, the batch a submit published. The union has one arm, the
buffered backend, so every load of the new loader is buffered.
supportsMemory says which delivery kinds the loader takes on a platform:
default device memory only, since the buffered backend places whole
tensors with Buffer.from. LoadSpec's doc now names it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
pairBufferized walks a model and the buffer tree bufferize made from it
together, in meta.visit order, and returns one Pair per tensor: the
tensor, the buffer at its place and its field path from the root, such as
.layers[1].ffn.weight. The pairs and paths live in the caller's arena.
The two trees must have the structure bufferize gives them. At the first
place they differ (another union arm, an optional set on one side only,
slices of other lengths) the walk stops with error.Mismatch, and the
report writer holds the path there and what differs. Comparing tensor
counts alone would let two trees of equal counts and different shapes
pair the wrong buffers.
The new loader's front end is its first caller.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The buffered backend read each part with one positional read, so its read
concurrency was only its worker count, and one large tensor read as a
single stream. A remote source answers a request after a round trip
however small it is, so a tensor split over parallel range requests
loads faster there.
ReadPermits bounds the reads in flight across every task of the backend
to the read width. A tensor's read takes one permit, then, when the VFS
read hints mark the source high-latency, grabs the permits that are free,
without waiting, and starts a helper on each. The owner and its helpers
claim spans from a shared byte cursor (ChunkedRead), each the tensor split
as many ways as it has readers but never below the hints'
read_chunk_size, so a lone reader still takes the whole tensor in one
read. A helper that the io does not start gives its permit back. Permits
are released before the tensor's transfer, so a tensor placing on the
devices lends its read budget to one still reading. On a local source a
tensor stays one read: splitting there only spends tasks.
The tests check the claim sizes, run overlapping batches with 32-byte
chunks with and without concurrency, and load a tensor from a MockServer
HTTP origin in exactly four range GETs over four permits.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The buffered backend allocated each tensor's host staging and freed it
once placed, with nothing bounding the bytes but the worker count.
Mapping, faulting and unmapping a whole tensor's pages for every tensor
takes half of a serial buffered load of a 14.96 GiB checkpoint, and a
fifth of a concurrent one.
Staging keeps released buffers for reuse under one budget for the live
and kept bytes alike: min(read width, 12) times the largest staging
request submitted so far. A request takes the smallest kept buffer that
fits. Otherwise it waits until its size fits beside the live bytes, frees
kept buffers that are too small, smallest first, and allocates. Each
submission can only raise the budget. A worker that waited for its
staging loads nothing if the loader failed meanwhile. Reused staging
holds another tensor's bytes, so the zeroing of gaps is now load-bearing;
the composite test loads a bigger tensor first to check it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zml/io/buffered_loader.zig is the new loader's backend for the platforms
that place a whole host tensor with Buffer.from: TPU, neuron and metal. It
does what the old loader's BufferedMemoryWriter does, on the seam's
LoadSpec. Nothing calls it yet; io.zig's test block reaches it.
submit starts up to the read width of workers on a batch, which claim its
specs from a shared cursor; on an io that starts no worker the batch loads
inside submit. Each tensor allocates its host staging, zeroes the bytes no
part covers and reads each contiguous part at its offset with
Tensor.readRange. A strided part reads its source range once, past the
tensor, and copyRuns puts each run where it lands. Buffer.from(wait) then
places the tensor on its shape's mesh. Several batches can be in flight,
each with its own workers.
The first failure is kept and logged once, with the tensor's file, offset
and sizes; workers then start no other tensor, every later submit returns
it, and each output is left as it was or holds its whole buffer. abort
fails the loader with Canceled. awaitBatch waits for the workers without
a cancellation point, since they borrow the batch's specs.
The tests submit specs directly on CPU devices. No machine available runs
TPU, neuron or metal, so it has not run on one.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A tensor can now be computed while loading instead of filled from a
source. TensorStore.Program is a program value: a plain-data struct,
checked at compile time to hold no pointer, whose type declares
forward(self, output, inputs). Program.init copies the value into the
store's arena behind a per-type vtable; as reads it back and name, the
type's name, names the compiled module.
Binding gains its compute arm, a program and its placed inputs in order.
Compute.sameExecutable tells whether two bindings compile to one
executable: programs of one type with equal values, over inputs and
outputs whose shapes are equal with their tags and partitioning.
Compute.compile compiles the program on the store's meshes.
Source.bindProgram binds one over placed sources of one store. The loader
that compiles and runs these programs comes later.
View.setMemory is the one way to change where the loader delivers a
bound tensor after binding; it panics on a program's output, which lands
in default device memory. View.hostMappedBytes sums, over the whole
store, one device's shard of every tensor delivered host_mapped: what
XLA counts against device memory for those views, which a model passes
back through Compiler.Options.extra_device_memory_bytes, whose doc now
points to it. A tensor whose mesh does not divide its dimensions is
IncompatibleSharding, logged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bitCast reinterprets bits like Tensor.bitCast. To a narrower type, each
element becomes the elements of a new last axis .bitcast. To a wider
type, the last axis must hold one element of the new type, and its bits
must be one contiguous run in the checkpoint, which panics otherwise.
Sub-byte types that land inside a byte fail at lowering with
SubBytePosition. FP8 and NVFP4 block scales, including the 128x4
swizzle, are laid out as sources this way: the swizzle case pads, splits,
transposes and merges, checked against Tensor.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
broad is Tensor.broad: the elements repeat along the axes the target shape
adds or grows from 1, as dimensions whose checkpoint stride is zero. The
loader reads a few large repetitions once each, and copies many small
ones from one read of the source. Shared scales are bound this way.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
pad is Tensor.pad with zero as the only padding value: low, high and
interior padding are nonnegative, so it never crops. Each piece's origin
and steps move by the padding along every axis, and the elements no piece
covers are the zeros. Rows appended after the last one move no element,
so that case only grows the shape. It replaces padding a table by hand
over the old store's writers.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
transpose is Tensor.transpose, and no axes reverses them. When the axes
that move all hold one element and the others keep their order, the
row-major order is unchanged and the transpose is a reshape that leaves
the pieces alone. Otherwise each piece's origin and dimensions are
permuted, and a transposed tensor lowers to strided parts of single
elements. Placing a transposed source with as keeps its element order.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
slice is Tensor.slice for a static, contiguous slice: it clips each
piece's lattice to the slice's range along the axis, keeping the steps
whose whole sub-lattice is inside as one piece and recursing into the
partial ones, and shifts the result's origin to the slice's start. A
singleton slice drops the axis, and .len takes precedence over the
default end as Tensor.Slice resolves it. split and chunkExact are slices;
split's result lives in the store's arena. They split fused checkpoint
tensors such as gate_up and qkv, and expert slices.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Ops that move elements rewrite a source's pieces. A piece is kept flat;
an op that addresses axes decomposes it into pieces in the coordinates of
the input shape, each dimension moving one axis, cutting it where a
dimension would carry from one axis into the next (at row boundaries when
the lattice steps regularly, else one piece per step). The op then moves
those coordinates and emits each piece flattened in the result's shape,
its dimensions outermost first and merged where one continues another.
Pieces are allocated in the store's arena, the scratch freed per op.
The first ops on it are concatenate and stack, like Tensor's. Along the
outermost axis the pieces only shift their origin. Fused QKV, gate_up and
expert stacks are interleaves built from splitAxis, concatenate and
merge, which keep one piece per checkpoint tensor at checkpoint sizes.
Sources of different stores panic.
Each op case is checked against Tensor: the same shape-op expression is
compiled and run as a program over zml.Tensor, and its bytes and tags are
compared with the source's binding lowered and replayed on the host,
which also fails on any byte written twice.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The new TensorStore now binds model tensors. A Binding is what the loader
will read of a tensor: its load arm holds the placed source the tensor is
filled from and the memory it is delivered to, default device memory.
bindingOf looks a tensor's binding up by id; the store owns its bindings
and every source in its arena, and frees them in deinit.
View gains source and maybeSource, which give a checkpoint tensor as a
Source with optional tags, and createTensor, maybeCreateTensor and
createReplicatedTensor, which are source(...).withPartitioning(...).bind().
A replicated tensor has no mesh and takes only .{} as its partitioning.
A missing key panics with the key, so binding cannot fail.
io.zig's test block reaches Source.zig, whose tests now run.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zml/io/Source.zig describes a tensor the new loader assembles by moving
checkpoint bytes, never by computing them. A source is a shape and a list
of pieces: each piece reads the bits of one checkpoint tensor through an
affine lattice, mapping the source's row-major elements to bit offsets in
the tensor, and elements no piece covers are zero. Pieces live in the
store's arena.
This change adds the source of one checkpoint tensor (init through a
View), its accessors, and the ops that only relabel or reshape it
(withTags, rename, renameTag, splitAxis, merge, reshape, flatten,
insertAxes, appendAxes, squeeze), which keep the row-major order and so
never touch a piece. Placement comes last: withPartitioning and as return
a Placed, the only thing that binds. Placed.bind records the model tensor
through a private bindNew, the one writer of bindings, so pieces stay
disjoint by construction. Placed.lower turns each piece into one
backend.LoadPart by one rule: dimensions outermost first, the innermost
run contiguous on both sides as the width, and SubBytePosition for a run
that starts or ends inside a byte. withPartitioning(.replicated, ...)
takes only .{}, checked at compile time.
The tests replay a binding's lowered parts on the host and fail on any
byte written twice. Nothing imports the file yet: the store wiring that
binds through it follows.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zml/io/backend.zig holds the contract between the new loader's front end
and its backends, which later changes build on from both sides. Nothing
calls it yet; io.zig's test block reaches it.
A LoadSpec is one destination: its shape, the memory it is delivered to,
the caller's output buffer and the parts that fill it. A LoadPart is the
bytes of one checkpoint tensor that a Lattice moves: a run of bytes,
repeated over up to 2 * MAX_RANK dimensions with a source and a
destination stride each, a zero source stride repeating the same bytes.
Lattice.Cursor walks a range of runs without a division per run, and
copyRuns copies them between two addresses. checkParts is the check the
front end runs before a backend sees a spec: at least one part, each
non-empty and inside its source and the destination, in ascending
destination order, strided runs landing in index order, interleaving
strided parts covering no more than the destination, and every file end
far enough below maxInt(u64) that a read widened to the direct-I/O
alignment does not overflow. Spans are checked for overflow before the
order, so a part with huge strides is refused instead of overflowing.
requestSize gives the source request for a DMA block: the read chunk
rounded up to whole blocks, at least one, at most 32 MiB
(max_read_request_size). A block that is not a nonzero multiple of the
direct-I/O alignment, or a request too short to hold a read widened at
both ends, has none. readAlignment is that alignment when the io is a VFS
and direct I/O is not off, else 0. backend.Options carries the checked
loader options; its dmaBlockSizes is the pinned block, else one fixed
16 MiB block.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Buffer.from, Buffer.uninitialized and HostAccessible.init created one
shard for every device of the platform, and toSlice and toSliceAlloc read
as many, whatever mesh the shape was placed on. A buffer on a mesh over
fewer devices then got shards on the wrong devices, or read past its
shards. They now walk the placement's mesh in its canonical device order,
and the readbacks walk the buffer's own shards. buffer.deviceMemory finds
a mesh device's memory of a kind, and panics naming the device when it has
none, where only device 0 was checked. placementOrPanic places a shape
without a mesh through Platform.placedShape. On a mesh over every device
nothing changes.
The test puts a buffer on two of four devices, out of the platform's
order, in default and pinned host memory. It needs test infrastructure:
- testing.Devices gives N devices, the accelerator's when it has as many,
else a CPU platform of its own, with a one-axis physical mesh over
exactly them;
- testing.expectShards reads every shard back in mesh order and checks its
memory space and its box of the tensor's row-major bytes, for a
delivery kind of the new loader's store;
- Buffer.Shard.toHost copies one shard to the host and waits.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zml/io/TensorStore.zig is the new loader's store. zml does not export it:
zml.io.TensorStore in io.zig stays the store models and the current loader
use, and keeps borrowing its registry. The new store takes the registry's
entries instead: init leaves the caller's registry empty, so its deferred
deinit stays correct, and locks the map's pointers until deinit, so an
entry's address identifies its checkpoint tensor and a later insert
panics in safe builds.
It has the old store's caller API: init with Options{meshes,
quantization}, deinit, view and mesh, and View's root, withPrefix,
withLayer, prefix, get, count, mesh and quantization. Memory names where
the loader delivers a tensor (default, host_pinned or host_mapped), with
isHost and the memory space of each device's buffer.
zml/io/log.zig adds the new loader's log scopes, load, io and mem. Their
err logs at debug in test builds, where the test runner counts a logged
error as a failure, and failure appends the error's name and logs
cancellation and OutOfMemory at debug.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ShardGrid is the first file of the new loader under zml/io/, which zml
does not export yet: io.zig's test block is the only thing that reaches
it. It places a destination's logical bytes on its devices by arithmetic,
never by listing rows: init takes the shape as PJRT allocates it, placed
on the platform and with sub-byte storage packed, and computes the even
box grid; locate maps a logical byte to its box and the offset inside it,
rankIn counts a box's bytes before a logical offset, runEnd and runCount
cut a range into the runs that stay in one box, and replicas gives the
devices per box.
Platform.placedShape names the default a buffer already applies to a shape
without a mesh: replicated over every device.
The tests walk every fixed-step byte range of replicated, packed sub-byte,
1D mirrored and folded, 2D, 3D and scalar placements on 4- and 8-device
CPU platforms and compare each byte with Placement.slices. They skip where
no CPU plugin is available.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A throttle was a charged retry: each reader backed off on its own and
retried, so a rate-limited backend kept seeing about as many requests and
a long limit exhausted every reader's retries. A throttle (429 everywhere,
503 on a backend that names it so) now arms a hold over every request of
the backend. Each request waits out the hold before it goes, waking at the
deadline plus jitter of its own, and the wait is cancelable. A throttle
arms the hold for the delay the server named or for a full-jitter backoff
on the throttles of the episode, whichever is longer, clamped between the
initial delay and max_hold (2 minutes); a later answer can extend a running
hold but never shorten it. A clean window as long as the hold ends the
episode, and an episode longer than throttle_budget (5 minutes) fails the
backend's requests with RateLimited until one does. Throttles are no longer
charged to max_retries. The governor counts holds and the time requests
spent waiting them out. Requests carry their URI's authority as the hold's
key, though one hold covers the whole backend.
Tested against MockServer only; no real throttling source was available.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A peer that stops sending without closing its connection holds the reader
waiting on it, and the load waiting on that reader, forever. Each exchange
now runs on a task of its own while the attempt watches it: range_read
counts the body bytes it receives in Attempt.progress, the attempt checks
that count twelve times per Retry.stall_timeout (60 s by default), and
after a whole timeout without progress it cancels the exchange and retries
it as a timeout. An exchange that had already finished when it was
canceled keeps its value. Where the io cannot run the exchange
concurrently, it runs unwatched. A request that a hook sends before its
exchange, such as a re-resolve, is watched by its own exchange, so waiting
on it is no stall.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
HTTP's reads and lookups now run through request.zig. HTTP.init takes
Options{protocol, retry} instead of a bare protocol, and its governor,
named http, treats a 503 as a server failure. fetchSize walks redirects
with HeadChain, which follows the relative Location that Hugging Face
answers a resolve URL with, where HTTP's own walk failed. A positional
read is one range_read GET that follows redirects and fills every buffer;
it filled only the first. Opening, stat-ing and probing a path follow the
open error contract, and access sends a HEAD where it succeeded for any
path. A failed positional read is Canceled or Unexpected and a failed
streaming read InputOutput, each logged once. A seek saturates at 0
instead of overflowing, and deinit logs the read totals at debug. Remotes
passes .{ .protocol = .https }. VFSBase's io is aligned for the i96 fields
of the governor HTTP now embeds, so recovering a backend from its base
needs no @alignCast.
HTTP has no parallel_read pool, so master's loader still reads https://
with one request per read. This breaks HTTP.init callers; llmd has none
once it uses VFS.Remotes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
std.http.Client does not follow a redirect for a HEAD, so a backend that
sizes an object with a HEAD walks the chain itself. HTTP's walk stopped at
a relative Location (Hugging Face answers a resolve URL with one), and HF
carries a walk of its own. HeadChain follows absolute and relative
Locations through three rotating 8 KiB buffers, up to a hop limit the
caller sets, and returns the URL that answered a 2xx with its
Content-Length. Each hop is one governed HEAD with open semantics, so a
missing or denied object fails the walk with FileNotFound or
PermissionDenied, and a callback chooses the credentials sent to each URL.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
performRangeRead turns one positional read into one GET with a Range
header for what the caller's buffers hold of the object, run through the
governed loop, and scatters the response into every buffer. Nothing is
sent at the end of the object. A 206 must carry a Content-Range that
covers the request, parsed strictly; a 200 that ignored the Range is
positioned by discarding up to the offset. The backend supplies the
request through a prepare hook, called once per attempt, since an S3
signature covers the Range header and a timestamp and a token or a signed
URL may expire between attempts; static serves a plain GET. It replaces
parallel_read's chunk reader and HTTP's own Content-Range parsing once the
backends move to it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each remote backend sends its requests with code of its own: HTTP retries
nothing, and HF, S3 and GCS each carry a copy of the same retry loop that
differs in which failures it retries and how it reports the rest.
request.zig holds one loop for all of them. A backend owns a Governor,
with its name, its retry knobs and its request statistics, and runs each
request through perform, which retries a retryable failure after a
full-jitter backoff, or after the delay the server named in Retry-After or
HF's RateLimit header, up to max_delay, and gives up after max_retries.
exchange owns the std.http plumbing of one attempt: it forces identity
encoding, keeps its own redirect buffer, closes a connection after a
partial send, and tells a canceled read from a failed one. A 408 is a
timeout, a 429 a throttle, a 5xx a server failure, and a 503 either a
throttle or a server failure as the backend says; any other status fails
the request, logged once at err.
Open, stat and access requests share one error contract: 404 and 410 are
FileNotFound, 401 and 403 PermissionDenied, a cancel stays Canceled, and
anything else is Unexpected, logged once by the governor. A failed read is
Canceled or Unexpected. No backend uses the loop yet; its tests drive it,
one of them against MockServer.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The remote backends' failure paths (retries, refusals, redirects, rate
limiting) cannot be tested against a real origin. MockServer serves an
object per path over a real loopback socket, with Range GETs and HEADs, and
can be told to fail the first GETs, hold GETs at a barrier until enough
arrive, throttle (the first requests, every GET, or past a rate window,
with a status and a Retry-After), refuse a path with a status, or redirect
it. It counts HEADs, GETs and the peak of concurrent GETs, and keeps the
first error it met through stdx.FirstError. It is exported as
VFS.MockServer so tests outside vfs can use it too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Tasks that run concurrently and may each fail often need to keep the first
failure and let the others see it. FirstError(E) keeps it in one atomic
word: set stores an error unless one is already kept and says whether it
kept this one, and get reads it back.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Neither read mode wins everywhere: a direct read comes from the disk even
when the page cache holds the file, and a buffered one pays a copy that
competes with DMA for host memory bandwidth. DirectIo.auto reads a file
directly when at most half of 32 pages sampled across it are cached, and
buffered otherwise. A sample is a preadv2 with RWF_NOWAIT, which answers
EAGAIN for an uncached page, needs no write permission and costs the same
whatever the file's size; unknown residency counts as cached. A sample
that misses queues the read of its page, so each probe shifts its samples
by a random offset, and a cold file read directly is still found cold by
the next probe unless its samples land in what those misses read.
predictDirectIo gives the answer useDirectIo would give for a local path,
and the file's size, on a descriptor of its own: it decides no handle and
logs nothing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A reader that reads a local file in large aligned requests can ask the VFS
to read it past the page cache: useDirectIo(file, policy) sets O_DIRECT on
the descriptor under .on, once per handle, and from then on the VFS issues
that handle's positional reads itself, with preadv in batches of at most
IOV_MAX buffers. The kernel answers a direct read it cannot serve with
EINVAL or EFAULT, which the threaded io treats as a programmer error; here
such a refusal clears the flag and the read, and every later one, goes
through the inner io buffered. It is logged at warn when the read was
aligned and at err when it was not. Streaming reads and file copies take
the handle off direct reads first. The flag goes on and off through
fcntl, retried on EINTR, and only on a descriptor opened read-only.
vfs/direct_io.zig holds the mechanism; the VFS exports DirectIo and
direct_io_alignment (4096, or 0 where O_DIRECT does not exist).
Nothing calls useDirectIo yet, so every handle stays undecided and reads
as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Creating or deleting through the VFS went through VFSBase's forwarders,
which handed the inner io a VFS handle index as a descriptor and a URI as a
local path: creating a directory from a directory opened through the VFS
used the wrong descriptor, and createFile on s3://bucket/key wrote
bucket/key on local disk. Creates and deletes (directories, plain and
atomic files) now resolve their directory through the VFS's handle table
and strip file://, and the directory dirCreateDirPathOpen opens gets a
handle of its own. A remote path, given whole or relative to a directory
opened on a remote, is refused with ReadOnlyFileSystem before any request,
and so is opening a remote file for anything but reading. The refusal is
logged through base.scopedLog, which logs err at debug in test builds, so
the tests of these failure paths pass under the test runner.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every program that reads remote checkpoints wired the same remote schemes
by hand: one std.http.Client, then the HTTP, HF, S3 and GCS backends and
their registration. The copies had drifted: examples/llm had no https and
ignored the proxy variables. VFS.Remotes builds one client on the VFS's base
io, with proxies from the environment, sets up https, hf, s3 and gs through
each backend's auto constructor and registers all four, or none on error.
It is initialized in place, since the backends point at its client.
examples/io and examples/llm use it, so examples/llm gains https and
proxies.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Local files are the VFS's own: a file:// URI now resolves to the inner io,
like a bare path, and registerBackend asserts that nobody registers the
file scheme. vfs/file.zig goes. It was a forwarding backend with a second
handle table that, for file:// URIs, also opened a second O_DIRECT
descriptor and read through it transparently whenever a read happened to
be aligned. Bare paths, which llmd and the examples use, never had that,
so only file:// URIs change: they now read buffered. The examples stop
creating and registering a VFS.File.
This breaks llmd's use of VFS.File in main.zig and zoo.zig.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A backend is now registered as a Backend: its io together with ReadHints,
the smallest request worth sending and whether the backend's latency calls
for many requests in flight. Local files read best in 8 MiB requests, the
https, s3 and gs backends in 16 MiB and hf in 32 MiB, the remote ones all
high-latency. registerBackend replaces register and unregister, and VFSBase
is no longer exported. readHints and isLocal answer for a path without
opening it, and fromIo finds the VFS behind an Io, so a reader that holds
only an Io can size its reads. examples/io and examples/llm register
through registerBackend.
This breaks llmd's register calls in main.zig and zoo.zig.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
TensorStore.fromRegistry(allocator, ®istry, meshes) becomes
init(allocator, ®istry, Options{ meshes, quantization }). The store
still borrows the registry: entries registered after init, such as
aliases of a fused tensor, are found, and each bound tensor holds a copy
of its entry.
Options.quantization is the platform's Quantization.Target, which code
binding block-FP8 weights reads through View.quantization; that panics
when the store has none, so weights cannot be laid out for a different
FP8 encoding than the activations.
View.get(subkey) returns the checkpoint tensor, or null, with one hash
lookup. It replaces getShape, getShapeOpts and hasKey; hasKey scanned
every key for one starting with the subkey, so it also matched a
tensor's prefix. View.prefix() returns "" at the root instead of null.
The examples, the torch2zml howto and testing.testLayer move to init,
and the lfm2 and llama layer tests read tensors with get and
Tensor.read. llmd's fromRegistry, hasKey and getShape calls follow in
its next zml bump.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Opening a checkpoint had several faults. fromPath opened a file path
relative to the file's own directory, so a relative path such as
models/x/model.safetensors looked for models/x/models/x/model.safetensors.
resolveModelRepo accepted a file and opened its directory, and mapped
every error but FileNotFound to InvalidPath, including a denied directory
and a canceled open. resolveModelEntrypoint ignored why each probe failed,
and fetchRegistry leaked the registry when a file failed to parse.
- fromPath opens a file by its base name in its own directory.
- resolveModelRepo takes directories only; anything else is a logged
InvalidPath, and PermissionDenied and Canceled pass through.
ModelPathResolutionError no longer lists OutOfMemory.
- resolveModelEntrypoint becomes private. It still tries
model.safetensors after the index, and reports PermissionDenied when
neither opens and one was denied: S3 answers 403 for a missing key to
a principal that may not list the bucket.
- fetchRegistry becomes the private addCheckpoint and the registry is
freed when parsing fails. FileType and resolveFiletype are deleted.
TensorRegistry.addRepo(io, repo, name) adds a second checkpoint's
tensors under name and a dot, so a speculative drafter can share its
model's registry; it refuses a name a tensor is already under before
reading anything. parseSafetensors takes that prefix and allocates from
the registry's allocator.
checkpointFiles(arena, io, path) names the safetensors files fromPath
reads for a path, in their directory, which it returns open. It reads
no header and logs nothing, for a reader that prepares a load before
the registry exists. fromPath and it share how a path locates its
checkpoint (locateCheckpoint), the directory's entrypoint
(resolveModelEntrypoint, which also names it) and the files an index
lists (indexFiles).
safetensors.Tensor gains read, which reads a whole tensor from its file,
and readRange, which reads a tensor-relative range of a file the caller
opened with positional, bounds-checked reads. TensorReader stays for
now. testing.writeSafetensors writes a fixture file and registers it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
registerTensor put each tensor in the registry, so when two files of a
checkpoint named the same tensor, the last one read silently won and
which bytes were loaded depended on the index's file order. It now
refuses a name already registered with DuplicateTensor, logging the
tensor and both files. A checkpoint that repeats a name across its
shards no longer loads.
llmd checks the registry before registering a tensor of its own, so it
is unaffected.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A checkpoint's header is untrusted input, but parseSafetensors read it
as raw JSON values and unwrapped fields with .?, so a malformed header
panicked, or was undefined behavior in release builds. It also cast
negative offsets to u64, allocated a header length read from the file
without checking it against the file, and never checked that a tensor's
data_offsets span its shape.
The header is now parsed as typed entries that ignore unknown fields.
Its length must fit the file, and each tensor's data_offsets must span
exactly the bytes of its shape within the file's data, the byte count
checked for overflow. Anything else, including a file shorter than the
8-byte length field, is InvalidSafetensorsHeader, logged with the file
and tensor. stringToDtype returns null on an unknown dtype and its
caller logs the file and tensor, and the TooManyDimensions log names
the file.
The index is parsed as a typed {weight_map}. Parsing it as raw values
with alloc_if_needed left strings pointing into the reader's buffer,
which the next refill overwrote, so an index longer than 64 KiB could
name the wrong files. An index without a weight_map is refused rather
than read as an empty checkpoint, and a read error returns its cause.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The registry parsed every header's __metadata__ and every index's
metadata into a map of typed values, about 300 lines, and nothing in zml
or llmd reads it. Metadata, Metadatas, TensorRegistry.metadata,
initWithMetadata, mergeMetadata and the parsing helpers are deleted,
along with the registry's unused mutex. A header's __metadata__ is
skipped, and the torch2zml howto's sample output loses the debug line
that metadata parsing logged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Tests that read a checkpoint need a safetensors file with known tensors.
writeSafetensorsFile writes one from (name, dtype, dims, bytes) entries,
laying the tensors out in order after the header, and pattern fills a
tensor with distinct, nonzero bytes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A partitioning written as a struct of tags, such as .{ .out = .model },
could only be parsed at comptime: Partitioning.parse captured it in a
comptime Logical{src, tgt}. Code that picks a partitioning at runtime,
such as a weight binder that chooses row or column parallelism, had no
value to pass.
Logical is now a runtime value, {entries, default}, keyed by tensor axis
tags. Each entry holds a Spec: a logical mesh axis, .replicated or
.open. Logical.init takes a struct of specs (null optional specs are
skipped), a Logical or .replicated; get(tag) returns an axis's spec, and
resolve(tags, mesh, default) turns it into a positional Partitioning for
one tensor on one mesh, so one value applies to tensors whose axes are
ordered differently and to different meshes. Shape.withPartitioning,
Tensor.withPartitioning and the old TensorStore's maybeCreateTensor
resolve through it, and maybeCreateHostPinnedTensor no longer needs a
comptime partitioning. Partitioning.parse, Logical.parse, apply and
applyUseAll are deleted.
Shape.eqlWithTagsAndPartitioning also compares the mesh and each axis's
partition spec.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Block-FP8 weights must be bound in the FP8 encoding and scale layout the
platform's GEMMs read, and activations quantized to the same encoding.
quantizeInput chose the encoding with its own platform switch, and
weights had nothing to follow. Quantization.Target{fp8, swizzled_scales}
and Target.of(platform) are now the one switch: FN on CUDA with e8m0
scales swizzled on sm_10x, FNUZ on CDNA3, FN on CDNA4 and RDNA4, and no
FP8 elsewhere or when a ROCm device's architecture is unknown.
quantizeInput reads Target.of(platform).fp8, so its result is unchanged;
on CUDA it now queries the compute capability on each call. The block128
scaled dot test takes its dtype from Target as well.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bufferize(allocator, T, &model) repeated the model's type next to a
pointer that already carries it. It becomes bufferize(allocator, &model)
and returns Bufferized of the pointed-to type. The callers in
examples/io, examples/llm and both mnist examples move with it.
Buffer.deinitAll keeps its type argument. llmd's bufferize calls follow
in its next zml bump.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
lfm2, llama, qwen3_5 and qwen3_5_moe each freed their weights with a
tree of hand-written unloadBuffers functions, one per layer struct, and
each LoadedModel carried its own copy of the same loadBuffers. A walker
that missed a field leaked its device memory.
models.LoadedModel.loadBuffers now bufferizes and loads any model arm
with one body, and main, lfm2's session and the lfm2 and llama layer
tests free buffers with mem.deinitBufferized, which follows the Bufferized
type. The per-model loaders and every unloadBuffers walker are deleted,
so no example calls nn.Linear.unloadBuffers anymore.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Exe.call panicked when PJRT failed to launch an executable or when a
waited execution event reported an error, so a caller could not recover
from, say, a device running out of memory. It now returns
pjrt.ApiError. A failed launch writes no outputs. With wait, every
partition's event is awaited and destroyed, and a failed execution
destroys the outputs it wrote, so the results hold no buffer after any
error. The neuron and non-neuron event loops become one; their behavior
is unchanged.
Runner.run and FnExe's Runner.run keep their panic. Direct callers in
the examples, the tutorial and zml.testing add try, and the old
Loader.loadExecute reports a failed call as LoadFailed; PJRT has already
logged its cause. llmd's one direct call follows in its next zml bump.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each model frees its buffers with a hand-written walker that has to
follow every field of its Bufferized struct, and leaks when a field is
added without updating it. mem.deinitBufferized frees any tree bufferize
returns: it deinits every Buffer and frees the slices bufferize
allocated, through structs, unions, optionals, arrays and nested
slices. Passing it the model instead of its buffers does not compile.
When an allocation failed partway through a model, bufferize returned
the error and leaked the slices it had already allocated. It now frees
what it built before returning the error. It also fills the slice it
allocates before storing it, so models with []const Tensor fields
bufferize too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Bufferized maps a [N]Tensor field to [N]Buffer, but bufferize had no
case for arrays and reached `unreachable` at runtime on any model with
one. It now bufferizes arrays element by element.
bufferize also walked Bufferized's field names and looked each one up
in the model. Bufferized drops the fields that hold no tensor and, for
a tuple, renumbers the ones it keeps, so bufferizing a tuple with a
tensor-free field before a tensor did not compile. bufferize now walks
the model's fields and maps each to the field Bufferized keeps for it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
pjrtx.Client.compile went through pjrt.Client.compile, which started
compileRaw with io.async and awaited it at once. Under zio that ran
XLA's compile on an 8 MiB coroutine stack, even though Compiler.compile
had already left the coroutine with zio.blockInPlace; this is the
likely cause of the stack overflow seen compiling Laguna.
pjrtx.Client.compile now calls compileRaw directly and no longer takes
an io, Compiler stops passing one, and pjrt.Client.compile, which had no
other caller, is deleted. Under threaded io the compile now runs on the
caller's thread instead of a pool thread.
Under zio, Compiler.compile now leaves the coroutine with
zio.blockInPlaceReserved rather than zio.blockInPlace. blockInPlace
queues the compile behind the blocking pool's other jobs, and the pool
adds a worker only when none is idle and at least twice as many jobs as
workers are queued, so a burst of compiles shared one to three threads:
llmd's six model compiles took 6 to 15 s under zio on the B70 against
4.5 s under threaded io, and DeepSeek's two on a GB300 ran one after the
other (73-77 s against 50 s). A compile holds its thread for its whole
run, so each now takes a pool worker of its own, idle or new, and
concurrent compiles run in parallel as they do under threaded io.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
gcsUri set an object's path to the VFS path, bucket/key, without the
leading slash, so every GCS request targeted `bucket/key` instead of
`/bucket/key`. storage.googleapis.com accepts that, but it is not a
valid origin-form target and other endpoints need not. The path is now
rooted, with a test, and gcs.zig's tests run under //vfs:test.
A streaming read whose request failed returned EndOfStream on http, hf,
s3 and gs, so a reader saw a short file instead of an error. It now
returns InputOutput; the error is still logged where it happens.
The GCS backend also loses code nothing reaches: the JWT assertion and
its commented-out signing, getTimestamp, the commented-out authHeaders,
EndpointUrl, and the `region` option, which no caller sets.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The VFS seeded its handle table with the cwd placeholder at index 0,
then stdin, stdout and stderr, so std.Io.File.stdout() (handle 1)
resolved to descriptor 0. Writing to stdout through the VFS went to
stdin instead, which only works when both are the same terminal. The
standard streams now take indices 0 to 2 and the cwd placeholder
follows them.
A local name with a colon, such as notes:old.txt, was parsed as a URI:
from the cwd it failed as an unregistered scheme, and stripScheme cut
the name after the colon. A path is now remote only when it contains
"://", stripScheme removes only up to "://", and dirRealPathFile passes
the stripped path to the backend, so a remote real path no longer
carries its scheme twice.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Docs that no longer match the code:
- huggingface_access_token.md names HF_TOKEN, the variable vfs/hf.zig
reads, instead of HUGGINGFACE_TOKEN.
- write_first_model.md calls Buffer.fromSlice with its three arguments
and explains that compile() reads each tensor's sharding from its
shape, instead of a `sharding` variable the tutorial never defines.
- howto_torch2zml.md calls Platform.deinit with its io, and replaces
maybeCreateReplicatedTensor, which does not exist, with
maybeCreateTensor(..., .replicated, .{}).
In examples/llm, lfm2, llama, qwen3_5 and qwen3_5_moe each kept a second
loader on their inner Model (loadBuffers or load) that nothing calls and
that would not compile if called; they are deleted. LoadedModel.loadBuffers
stays the only loader.
qwen3_5_moe binds its .replicated tensors with an empty partitioning,
which the store ignores for the replicated mesh anyway, and examples/io
iterates the registry's tensors directly. Neither changes behavior.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
expectClose's doc names zml.HostBuffer, which no longer exists; it
accepts a zml.Slice. BufferizedWithArgs has no user and is deleted.
Buffer's doc likewise pointed to zml.aio.loadBuffers and
HostBuffer.toDevice, neither of which exists; it now names fromSlice and
fromBytes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
quantizeMxfp8 has no caller in zml or llmd. It also kept its own copy of
the platform's FP8 encoding switch, which quantizeInput already holds.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Platform.init warned when a plugin reported more than MAX_NUM_DEVICES
devices, then used them all, although per-device state such as Exe's
events lives in arrays of MAX_NUM_DEVICES entries. validateDeviceCount
now logs and returns InvalidDeviceCount instead.
Platform.init also leaked the PJRT client, and on CUDA the FlashInfer
state, when it failed after creating them. Both are now released on
error.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`visit` walks only a bounded array's first `len` elements, but the type
MapRestrict maps it to holds the whole backing buffer. Pairing a model's
tensors with its buffers by visit order would then miss or misplace
elements, so mapping a bounded array that contains the source type is now
a compile error that points to slices and arrays instead.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Slice.absolute tested `.end == to_the_end` before `.len`. Since `.end`
defaults to `to_the_end`, a slice given as `.{ .start, .len }` took the
rest of the axis instead of `len` elements, contrary to the documented
examples. `.len` is now checked first, and the slice test covers it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Duration.hz divides the i96 nanosecond count with `/`, which does not
compile for signed integers once the function is called; it now uses
@divTrunc, with a test that calls it.
AllocatingLimitedConcurrentGroup has no user in zml or llmd and is
deleted. LimitedGroup stays: zml/io.zig's Loader uses it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 10:22:53 +01:00
174 changed files with 26314 additions and 15967 deletions