IREE writes a token's text whole or not at all, so `feedOne` asserted
whenever the caller's buffer was smaller than the token, e.g. Laguna's
415-byte whitespace tokens in a 128-byte buffer. The decoder now owns
its output buffer and grows it the way `decode` already did, so
`feedOne` and `finalize` no longer take a buffer.
Move manual regions out of flash attention kernels and use
`manualComputation2` for every shared attention backend.
Also fully replicate the embedding executable outputs in Qwen and Llama.
Partial graph constraints leave their batch and sequence dimensions
open, which is invalid at executable boundaries.
Fix two further blockers exposed by the Qwen run that we forgot to move
recently: softplus comparisons must use the input dtype and ROCm DNN
runfiles must include the MIOpen kernel archives.
The name can appear in the IR and thanks to #1003 it shows in the web
view of the model making all compute blocks pretty obvious.
The screenshot below without names would be quite confusing,
but now we can see this is the QKV of first layer of a Llama model.
<img width="2211" height="1208" alt="image"
src="https://github.com/user-attachments/assets/1917edb0-a772-4a1d-9aa7-ad5aaf7b5260"
/>
Accept a logical mesh and typed input/output partition specs at manual
computation boundaries.
Will replace `zml.ops.manualComputation` in a future PR.
Ordinary graph results should have an unspecified layout until
constrained or inferred by the partitioner.
Previously, we could report replication simply because a tensor had no
mesh, even when its stored partition specs were open. Other paths
preserved selected operations’ sharding metadata, making the frontend’s
interpretation depend on how a tensor was produced.
This change gives those results a consistent open state while requiring
concrete layouts at executable boundaries.
Shardy explicitly defines the intended semantics: [“no sharding
attribute on a tensor is equivalent to a fully open tensor
sharding.”](https://openxla.org/shardy/sharding_representation#open)
Open dimensions remain available for further partitioning; closed
dimensions fix the specified layout.
Replication is therefore a constraint, whereas an open layout lets the
partitioner choose either sharding or replication:
This distinction also applies to unused mesh axes. Marking an axis
explicitly replicated prevents propagation from using it, even when a
tensor dimension is open. For example, `[{?}, {}]` must leave unused
axes available; adding replicated={"data", "model"} removes those
choices.
Keeping intermediates open lets the partitioner use constraints from
producers and consumers, with operation-specific rules for
transformations such as dot and reshape.
The LM head normalized and projected every prompt position at prefill,
and the session read back the whole token buffer just to keep the last
token.
Every model now slices the hidden state at `active_length - 1` before
the norm, so prefill computes logits for a single position and returns a
single token. Decode writes the next token over the one it read, so it
allocates no buffer per step.
lfm2's `actual_seq_len` is renamed `active_length`, the name the other
models use.
Akin to #926, this PR introduces a restructuration of the moe kernels in
ZML.
No need to re-explaing the organizatiopn because it follows what has
been done in the mentioned PR.
Here is the detailed changes of this PR:
* Kernels reorganization. `zml/kernels/moe` is the place to put all the
moe implementation. There is one sub-folder per backend.
* Changed the expected shape and tags when calling the moe forward
function. It's now expecting a 2D tensor tagged `.{.t , .d}`.
Tests still passes without key modifications. The only small tweak that
was done is related to fp32 -> fp8 triton quantization on sm89. Due to
how triton lowers the convert, it was producing slightly incorrect
result. A small preparation was added to ensure the conversion is
identical on this platform.
**Future TODO**:
* Ideally, this conversion on sm89 should be gated so that we don't pay
the cost on other platforms. We don't have a way to detect the platform
in the triton `Builder` yet but it could probably be added
* Settle on a complete API that encodes all the scenario we could
encounter. One example of that is what is required for Deepseek V4.1
Flash, the moe gemms should be done in `mxfp8 x mxfp4` and we should
have a way to request that in a better way than `quantize_input: bool`.
## Motivation
This is the big sharding rewrite we need to achieve complex sharding.
Currently we always resolve a mesh logical axis to a mesh physical axis
(or several of them).
But on GPU we always have 1D topology, so we can't have 2D logical
meshes.
So this PR drops the logical -> physical axis bindings, and instead
store a logical -> devices binding.
This is more flexible, and has the nice side effect of simplifying
`Buffer.from`
On stablehlo side this changes how we generate the MLIR because we now
use logical meshes, and `manualComputation` uses logical axes.
(In shardy, GSPM is not affected AFAICT because it uses device ID
directly)
## Buffer.from
This also accelerates `Buffer.from` that I rewrote to be FAST.
Version | Fixed cost | Per device
-- | -- | --
origin/master | 109.6 ns | 35.39 ns
Rewrite | 35.8 ns | 4.38 ns
Improvement | 3.1× | 8.1×
## Creating mesh
I'm adding a new syntax to create LogicalMesh, to allow the more complex
meshes we want to do, and in particular, I now allow to force a specific
number of dimensions.
You can see it in action in `mnist.zig`.
```
const dp_mp = try platform.registerMesh("DP-MP", try .init(
.{ .data, .model },
&.{ .low_bandwidth, .high_bandwidth },
&.{ 2, -1 },
));
info: topology used: Mesh(name=DP-MP, partitions=4, replicas=1)
┌ Mappings (logical coordinates from physical coordinates):
├── data (low_bandwidth) -> (bus / 2)
└── model (high_bandwidth) -> (bus % 2)
┌ PhysicalMesh(platform=cpu shardable_axes={ .bus } num_devices=4)
└── [bus] 0..4 (tree)
├── Device(id=0, coords={ 0 })
├── Device(id=1, coords={ 1 })
├── Device(id=2, coords={ 2 })
└── Device(id=3, coords={ 3 })
┌ Logical coordinates:
└── data=0
└── model=0,1 -> devices=0,1
└── data=1
└── model=0,1 -> devices=2,3
```
The previous syntax is still available, and ZML will align logical axes
to physical axes like we do before this PR.
This part as unit tests, and I checked it work as I expect on CUDA but I
would need to double check the logics still works on TPU/Neuron, because
there is a few platform specific branches in that code.
## Test plan
As part of the benchmark Codex generated a test suite with hard-coded
offsets, allowing to check that Buffer.from didn't change behavior
through the commits, checking we correctly compute each individual
shard.
TODO check that LLMD perf isn't degraded.
here I'm promoting a pattern I would like to extend to more of ZML:
function that takes `anytype` argument should do as little work as
possible and called a fully typed function.
I kept `manualComputation` signature to not break existing code,
I moved the code to `zml/ops/manual_computation.zig` to better
encapsulate the required helper functions.
(and `zml/ops.zig` is pretty big atm)
Similarly `Sharding.Partitioning.parse` now calls `const logical =
comptime Logical.parse(partitioning);` then `logical.applyUseAll(tags,
mesh, default_value);` which is fully typed.
I'm also documenting `manualComputation` better
This PR introduces a restructuration of the kernels in ZML. This is done
to simplify and speedup bringup on new platforms.
From now on each "kernel" will have its own directory under
`zml/kernels`. "Kernel" is in quote because we need to think about it
being more of an "operation in a graph" than an actual kernel. For
example, `moe` will eventually be part of this `kernels` directory
despite being an operation potentially launching multiple actual kernels
on some hardware.
Also it's probable that in the future, this `kernels` directory will be
put outside of zml or, at least, properly separated so that we don't
have circular imports in the codebase. Currently, we have
`@import("../../../zml.zig")` in the kernel code and this is a recipe
for disaster.
On top of the reorganization, it also adds a level of indirection for
attention. Where previously `attention` was a generic term, we now have
`attention` and `flash_attention` because `flash_attention` is an
implementation of `attention`.
Here is a detailed summary of the changes introduced by this PR:
* Kernels reorganization. `flash_attention` and `paged_attention` are
now separated. Each folder has a subfolder for each backend we support.
`flash_attention` is also now a `BackendType` of `zml.attention` with
its own implementations.
* Removed `cuda_fa2` and `cuda_fa3` backend because they were obsolete.
They have been replaced by a `cute` backend that has been generated
automatically following the operation documentation, showing that this
approach of scoping kernels per DSL actually scales.
* Removed `attnd` support for now
Fix a memory leak in Qwen 3.5 Dense and MoE examples:
`layer_index` was never freed.
Before (74562d5) / After
Dense 0.8B VRAM : 11079 MB vs 5461 MB
Dense 0.8B tok/s : 169 vs 198
tok/s increases because it reduces `cuMemCreate` calls from 26 to 2
per tokens
In Buffer code we already have explicit sharding, but not for tensor.
I'm adding a `zml.Sharding` to `.withPartitioning` to make it explicit
which mesh we are working with.
To simplify the migration, `.withPartitioning` in Tensor code can use
`.model` syntax to lookup a sharding by name.
This replace the previous logic of `selectSharding(shape)` which would
look up a sharding that matches the axes of the given partitioning which
could yield surprising behavior.
This also modifies how sharding are stored in zml.Shape.
Before it was storing a list of axis names + a tag (2 * 8 * 64bits), now
I'm storing a zml.Sharding (64 bits) and a compact PartitionArray (32
bits) that references the logical axis of the sharding by offset instead
of by name.
There is an impact on the `TensorRegistry` because it also needs to be
aware of the possible shardings, instead of delegating the matching
partition spec -> sharding to loading. This means error will be detected
sooner.
`store.createTensor` accepts `@enumLiteral()` to lookup the relevant
sharding.
This also simplifies `zml.Buffer` because the full sharding is now
stored in the shape, so it doesn't need to be passed independently. It
also means that it will propagate from the tensor who are graph inputs,
to the buffers that are eg inputs.
`Placement` logic also got a bit simpler.
This PR removes a lot of slop in the MoE part of ZML, as well as some
global improvements.
Among the improvements:
* Unify activation functions and parameters across backends
* Unify activation computation across backends
* Replace error bubbling with explicit asserts (because we have no
meanignful way to handle them)
* Add a StableHLO MoE backend
Record donated input indices during compilation and destroy their
PJRT_Buffer handles once execution is enqueued for `zml.Exe.Runner`.
I'm not doing this automatically in `call()` because it's a bit
invasive. Not all the code uses the pattern of overwriting an input
buffer like `run(.{cache}, .{&cache})`, but most do. So I'm cleaning it
up on easy path and leave `call` as is for finer control.
Removing callOpts & runOpts to simplify the API and changing `io` to be
required for API clarity. I'm replacing `zml.testing.autoCall` with
`Exe.eval` which solves most needs in a simpler way IMHO
This PR adds support for models downloaded via `hf download` without the
`--local-dir` flag.
This is done by keeping the given path instead of getting it from the
`std.Io.File`. The problem with `File.realpath` is that in the case of
models downloaded from HF, the files points to blobs whose name are hex
strings, thus losing the required file extension. Unfortunately, we rely
on the file extension to detect the kind of repository the path
represents.
I also added `//examples/io:playground` to the `//:completion` target so
that we have completion when working on it.
Refactoring of the MoE selection and add repacking contract required by
the model inits.
This solve a design problem in the model init :
- Before, backend selection required the resulting MoE after repacking
(dtype), and repacking required to know the selected backend.
- Now, backend selection is based on raw weights loaded from HF
(quantization scheme and dtype). Then it induces the packing.
Add useful structs for packing that are specific to MoE
(`ExpertsLayout`).
Two things, the second one falling out of the first.
**`fp8_block32`**: E4M3FN weights with one scale per 32x32 tile, the
layout
DeepSeek V4.1 ships. `quantizeBlockFp8` takes a block size and a scale
dtype;
the E8M0 path floors amax at 1e-4 and rounds the exponent up, so the
activation
side is MXFP8: one E8M0 scale per row per 32 contracted values. That
pairing is
what XLA's fp8_block_gemv arm needs to hand both scales to
`tt.dot_scaled`
(`Fp8BlockGemvScaleInMma` requires E8M0 on *both* sides); F32 activation
scales
would quietly fall to the per-block multiply. The Triton MoE path gets
the same
scheme at `block_size_k = 32`.
`Linear.forward` also stops rounding the activation to BF16 before
quantizing,
so amax is computed on the input as given.
**An explicit output dtype** on `scaledDot` and `Linear.forward`.
Inference
cannot answer what an FP8 x FP8 product should return: the accumulation
is F32
either way, so BF16 or F32 back is the caller's choice. `forwardScaled`
now
hands the caller's type to `scaledDot` rather than rounding through BF16
and
converting after. Narrow outputs are rejected.
Separately, `Linear.forward` converts the weight and bias to the
activation
dtype. `dotGeneral` asserts both operands share one, which broke the
moment a
model raised its activation to F32 against a BF16 weight; the wrappers
this API
replaced converted for exactly that reason.
**Breaking:** `Linear.forward` takes a second argument and `scaledDot` a
fifth.
---------
Co-authored-by: Raphael Nahoum <raphael@zml.ai>
- The current Fp8 scaling wasn't properly finished. We didn't use the
scales at all.
- Adding a `quantize_input` option that GLM5.3-Flash uses
- `applyActivation` wasn't properly using the threshold for relu
- The bias was silently ignored for the MoE.
Fixing an issue raised in https://github.com/zml/zml/pull/754, to bound
the GDN recurrence in the Qwen3.5 example model to the number of active
tokens.
We also make qwen3.5 and qwen3.5-moe slightly more consistent.
Simplify `manualComputation` by merging `inputs` and `body_context` into a single `input` argument.
Internally, collect all and shard all tensors (including `?Tensor`) from `input` before invoking the body function.
A binding created through `maybeCreateBinding` is computed from its
sources by an executable and must never be streamed from the checkpoint.
`Loader.defaultCallback` inferred that from `sources.len != 1`, which a
single-source binding fails resulting to the raw bytes being loaded and
producing wrong results
Model discovery leaves its safetensors entrypoint and repository handles
open after building the tensor registry, including when parsing fails.
Close handles owned by `fromRepo` and `fromPath`, and close repositories
at the existing `fromRepo` call sites.
`forwardMoe` is effectively consuming a `gate_up` and `down` linear MLP,
it "just" only applies parts of it.
I've renamed:
```
Quant -> Quantization
QuantSchem -> Quantization.Scheme
TensorScale -> Quantization.GlobalScale
Direction { multiplier, divisor } -> Operation { multiply, divide }
```
and moved it outside of zml.nn.Linear
This makes it easier in LLMD to just load zml.nn.Linear
The XLA API assumes that all sub-bytes data is given to it unpacked.
But when you read safetensors files the data is packed.
My proposition is the following: when we have a zml.fp4 tensor/buffer we
say to pjrt it's a u8 buffer instead.
Since PJRT infers the input/output shape from stablehlo I had to
introduce some bitcast at the edge of the compilation:
- fp4 input are declared as u8, but I insert bitcast operand before
calling the zig code, so the model code only see the fp4 tensors.
- when the model code returns a fp4 output, I bitcast to u8 before
exiting the final compilation
Breaking changes: none
Currently fp4 can't cross the pjrt boundary, so all fp4 code already
manually lie about the data being u8, and it will keep working. In the
future this code will be able to be "upgraded" by explicitly tagging the
input as fp4 and remove the "unpacking" code from their model code.
Note: the tracking XLA issue seems pretty cold
https://github.com/openxla/xla/issues/16795
Migration instructions:
zml.module.CompilationContext -> zml.Compiler
zml.module.CompileError -> zml.Compiler.Error
zml.module.CompilationOptions -> zml.Compiler.Options
zml.module.compile -> zml.Platform.compiler or zml.Compiler.compile
Breaking changes: none, I've kept a small compatibility layer at the top
level, with upgrade instructions.
We can remove it in a second time.
## Overview
* merge `dynamicSlice` and `slice` into `slices`
* merge `dynamicSlice1d` and `slice1d` into `slice`
* drop `choose`, `choose1d`
* handle singleton for dynamic slices too
* typed API so that it's easier to specify arguments
## Migration guide:
```diff
// new .slice accepts all old .slice1d inputs
- const x1 = x.slice1d(-1, .{ .start = 0, .end = half_dim });
+ const x1 = x.slice(-1, .{ .start = 0, .end = half_dim });
- k.dynamicSlice(.{ .layer = zml.Tensor.DynSlice{ .start = cache_index, .len = 1 } }).squeeze(.layer);
+ k.slice(.layer, .dynSingle(cache_index));
- k.dynamicSlice(.{
- .layer = zml.Tensor.DynSlice{ .start = cache_index, .len = 1 },
- }).squeeze(.layer).choose1d(.k, kv_head);
+ k.slice(.{ .layer, .k }, &.{ .dynSingle(cache_index), .single(kv_head) });
- o.choose1d(ax, 5),
+ o.slice(ax, .single(5))
```
Now the `Tensor.Slice` struct is bigger and now as a Tensor in it, which
may break user code if your model was storing a `Slice` inside it,
because now `zml.Bufferize(Model)` will contain one more `?Buffer`. You
can either pass an extra `null` to the `zml.Exe` or refactor to remove
`Slice` out of `Model`. I haven't seen this happen so far. The idea is
that `Slice` will mostly be a stack only struct, only created to call
`slice`
This PR add the support of nvfp4 x nvfp4 moe backend on sm120 / sm100
(natively supported nvfp4 hardware). It :
- retrieves the released artifact cutlass moe flashinfer :
https://github.com/zml/flashinfer/releases/tag/cutlass-moe-v0.1.0
- loads the .so
- implements the nvfp4 x nvfp4 backend (added to existing triton and
mosaic tpu).
It's a bit of a mess to transfer ownership of the Exe to the runner as
that needs careful handling of defers. Fundamentally there is also no
particular reason why the Runner should own the Exe and it was not
consistent across zml.Exe and zml.TypedExe. Now it's always borrowed.
Renaming `zml.TypedExe` to `zml.FnExe` because it's shorter. I've
removed all the aliases `type LayerExe = zml.TypedExe(...)` it's useless
with ZLS, it's a lot easier to have `zml.FnExe(<function>)`
Emits `__metal$moe_gemm` / `$f4` / `$f8` so a MoE layer can run bf16,
NVFP4 or DeepSeek block-FP8 (`f8e4m3fn` weights + bf16 128×128 scales)
`forwardMoe` takes two new optional arguments, `w1_global_scale` and
`w2_global_scale`. The per-expert global (`weight_scale_2`) is applied
in-graph after the GEMM.
in LLMD and //examples/llm, we compile multiple executable, re-use the
args & results. We do all of this manually, in addition we don't have
any type-safety on the arguments/outputs which is a bit too easy to
misuse IMHO.
So adding two abstractions:
- Runner: Packs the args, exe and results together to avoid calling
`args.set`, `exe.run` and `results.fill` with similar arguments. Today
it owns the Exe. It doensn't strictly needs it, but it's simpler and
works for us for now.
- TypedExe: Type-safe version of an Exe, expecting a single input &
ouptut to have keyed arguments to have fully unambiguous arguments &
results mapping.
I've updated //example/llm with this new pattern.
Also adding an explicit `zml.module.CompileError`.
Simplify vfs architecture: merge `//zml/io` and `//zml/io/vfs`. They move to top level `@zml//vfs`
The PR also include two bug fix:
- S3: when no content length we were trying to read exactly 1M, while we
should read up to 1M
- GCP: restore the commented code for parsing bucket info
## Test plan:
```
bazel run --config=debug //examples/io:playground -- ls gs://gcp-public-data-landsat/
bazel run --config=debug //examples/io:playground -- tree s3://noaa-goes19/ABI-Flood-Day-Shapefiles/2025/08
```
The old `zml.io.load` was plagued with problems. This PR is an attempt
at improving the state of things.
One major problem of `zml.io.load` was that it was not very composable.
It was either using it to load all the model or doing everything by
hand.
Another problem was that it didn't offer a way to apply transformations
at load time to the tensors stored in a safetensors file. For example,
it happens that for MoE models, instead of the experts weights to be
tightly packed in a nice 3D tensor, the weight file contain one tensor
for each expert. This is nice until your implementation required a 3D
tensor as input. Using the old `zml.io.load` API this resulted in very
convoluted way to do concatenation at load time. With this new API, we
offer convenience so that it can be handled nicely and as fast as
possible.
The main change is the introduction of the `Loader` struct. It's a
struct that you instantiate once and reuse whenever you want to load
tensors. It relies on `Binding`, which are a way to store which tensor
are required and how they need to be combined (if it's a complex binding
involving more than one source tensor).
One advantage of this `Loader` is that it's composable. Loading a struct
using `Loader.auto(Struct)` or loading each individual field of `Struct`
has essentially the same behavior. This solved the major problem of the
old solution.
For simplicity I kept the old model loading code almost identical, as
this new solution is a drop-in replacement.
# Update
After some discussions with @gwenzek I've changed my mind. Instead of
having a declarative API, we'll provide helper to do transformation
easily. The main changes that this PR now bring are:
* Introduce `Loader` struct
* `Loader` exposes `load` which is the composable replacement of
`zml.io.load`
* `Loader` exposes `loadExecute` that allows you to execute an
executable with the buffers loaded from the sources associated with a
tensor. This allows for example to concatenate tensors.
We'll continue to work on it, this is just a preliminary work that
allows us to do transformation at load time in LLMD.
A 1-chip TPU mesh collapses to rank 1 (only `link_x`), so the
unconditional `link_y` fold tripped InvalidPhysicalAxis. Only fold
`link_y` when the physical mesh actually has it.
- Bump Metal plugin version
- Drop the attention metadata fields from the llama and lfm2 Sessions. Prefill and decode now each build their own metadata inline and free it.
# Summary
This PR extends the support to chunked parallel read introduced
initially for HF in the VFS, to GCS and S3.
It:
- Adds a shared `zml/io/vfs/parallel_read.zig` helper for chunked
parallel HTTP range reads
- Add configurable read pool options for S3/GCS initialization
- Register the `gs://` VFS backend in `examples/llm`.
# Tests
## S3
We used [MINIO](https://hub.docker.com/r/minio/minio) docker image from
the `5090` GPU machine. From there, we mirror the
`meta-llama/Llama-3.1-8B-Instruct` model through the S3 server.
Then from the `4090` GPU machine, we run
```bash
bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=s3://zml-test/Llama-3.1-8B-Instruct --prompt="hello, how are you ?"
```
It works both on 1 and 2 GPU.
## GCS
In the same spirit, we put the `meta-llama/Llama-3.1-8B-Instruct` in a
GCS bucket we own. Then, on the `4090`, we run
```bash
bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct --prompt="hello, how are you ?"
```
It works both on 1 and 2 GPU.
### Performance
As a reference, we run:
```bash
gcloud storage cp gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00001-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00002-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00003-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00004-of-00004.safetensors /tmp/gcs-gcloud-llama-shards/
```
It gives:
```
Completed files 4/4 | 15.0GiB/15.0GiB | 942.7MiB/s
Average throughput: 918.8MiB/s
```
Using the VFS in various setups:
* `dma_chunks = 32`, `dma_chunk_size = 128`, `parallelism = 16`,
`chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64`
```bash
info (llama): Loaded weights [14.96GiB, 21.302s, 719.02MiB/s]
```
* `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`,
`chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64`
```bash
info (llama): Loaded weights [14.96GiB, 18.227s, 840.32MiB/s]
```
* `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`,
`chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64`
```bash
info (llama): Loaded weights [14.96GiB, 17.376s, 881.43MiB/s]
```
* `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`,
`chunk_size = 16 MiB`, `num_workers = 32`, `queue_capacity = 64`
```bash
info (llama): Loaded weights [14.96GiB, 17.28s, 886.36MiB/s]
```
* `dma_chunks = 32`, `dma_chunk_size = 512`, `parallelism = 16`,
`chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64`
```bash
info (llama): Loaded weights [14.96GiB, 18.658s, 820.90MiB/s]
```