2026-01-04 19:37:42 +00:00
|
|
|
|
2025-12-15 13:39:48 +01:00
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
|
|
|
|
|
zml/io/vfs: add parallel read to GCS and S3 (#604)
# Summary
This PR extends the support to chunked parallel read introduced
initially for HF in the VFS, to GCS and S3.
It:
- Adds a shared `zml/io/vfs/parallel_read.zig` helper for chunked
parallel HTTP range reads
- Add configurable read pool options for S3/GCS initialization
- Register the `gs://` VFS backend in `examples/llm`.
# Tests
## S3
We used [MINIO](https://hub.docker.com/r/minio/minio) docker image from
the `5090` GPU machine. From there, we mirror the
`meta-llama/Llama-3.1-8B-Instruct` model through the S3 server.
Then from the `4090` GPU machine, we run
```bash
bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=s3://zml-test/Llama-3.1-8B-Instruct --prompt="hello, how are you ?"
```
It works both on 1 and 2 GPU.
## GCS
In the same spirit, we put the `meta-llama/Llama-3.1-8B-Instruct` in a
GCS bucket we own. Then, on the `4090`, we run
```bash
bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct --prompt="hello, how are you ?"
```
It works both on 1 and 2 GPU.
### Performance
As a reference, we run:
```bash
gcloud storage cp gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00001-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00002-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00003-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00004-of-00004.safetensors /tmp/gcs-gcloud-llama-shards/
```
It gives:
```
Completed files 4/4 | 15.0GiB/15.0GiB | 942.7MiB/s
Average throughput: 918.8MiB/s
```
Using the VFS in various setups:
* `dma_chunks = 32`, `dma_chunk_size = 128`, `parallelism = 16`,
`chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64`
```bash
info (llama): Loaded weights [14.96GiB, 21.302s, 719.02MiB/s]
```
* `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`,
`chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64`
```bash
info (llama): Loaded weights [14.96GiB, 18.227s, 840.32MiB/s]
```
* `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`,
`chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64`
```bash
info (llama): Loaded weights [14.96GiB, 17.376s, 881.43MiB/s]
```
* `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`,
`chunk_size = 16 MiB`, `num_workers = 32`, `queue_capacity = 64`
```bash
info (llama): Loaded weights [14.96GiB, 17.28s, 886.36MiB/s]
```
* `dma_chunks = 32`, `dma_chunk_size = 512`, `parallelism = 16`,
`chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64`
```bash
info (llama): Loaded weights [14.96GiB, 18.658s, 820.90MiB/s]
```
2026-06-26 19:24:31 +02:00
|
|
|
|
2026-07-10 11:55:38 +02:00
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
2026-07-30 14:21:36 +02:00
|
|
|
|
2025-12-15 13:39:48 +01:00
|
|
|
|
2026-06-24 17:45:39 +02:00
|
|
|
|
zml/io/vfs: add parallel read to GCS and S3 (#604)
# Summary
This PR extends the support to chunked parallel read introduced
initially for HF in the VFS, to GCS and S3.
It:
- Adds a shared `zml/io/vfs/parallel_read.zig` helper for chunked
parallel HTTP range reads
- Add configurable read pool options for S3/GCS initialization
- Register the `gs://` VFS backend in `examples/llm`.
# Tests
## S3
We used [MINIO](https://hub.docker.com/r/minio/minio) docker image from
the `5090` GPU machine. From there, we mirror the
`meta-llama/Llama-3.1-8B-Instruct` model through the S3 server.
Then from the `4090` GPU machine, we run
```bash
bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=s3://zml-test/Llama-3.1-8B-Instruct --prompt="hello, how are you ?"
```
It works both on 1 and 2 GPU.
## GCS
In the same spirit, we put the `meta-llama/Llama-3.1-8B-Instruct` in a
GCS bucket we own. Then, on the `4090`, we run
```bash
bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct --prompt="hello, how are you ?"
```
It works both on 1 and 2 GPU.
### Performance
As a reference, we run:
```bash
gcloud storage cp gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00001-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00002-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00003-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00004-of-00004.safetensors /tmp/gcs-gcloud-llama-shards/
```
It gives:
```
Completed files 4/4 | 15.0GiB/15.0GiB | 942.7MiB/s
Average throughput: 918.8MiB/s
```
Using the VFS in various setups:
* `dma_chunks = 32`, `dma_chunk_size = 128`, `parallelism = 16`,
`chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64`
```bash
info (llama): Loaded weights [14.96GiB, 21.302s, 719.02MiB/s]
```
* `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`,
`chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64`
```bash
info (llama): Loaded weights [14.96GiB, 18.227s, 840.32MiB/s]
```
* `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`,
`chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64`
```bash
info (llama): Loaded weights [14.96GiB, 17.376s, 881.43MiB/s]
```
* `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`,
`chunk_size = 16 MiB`, `num_workers = 32`, `queue_capacity = 64`
```bash
info (llama): Loaded weights [14.96GiB, 17.28s, 886.36MiB/s]
```
* `dma_chunks = 32`, `dma_chunk_size = 512`, `parallelism = 16`,
`chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64`
```bash
info (llama): Loaded weights [14.96GiB, 18.658s, 820.90MiB/s]
```
2026-06-26 19:24:31 +02:00
|
|
|
|
2026-06-24 17:45:39 +02:00
|
|
|
|
|
|
|
|
|
2026-07-10 11:55:38 +02:00
|
|
|
|
2026-06-24 17:45:39 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-07-10 11:55:38 +02:00
|
|
|
|
2026-06-24 17:45:39 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-07-10 11:55:38 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-06-24 17:45:39 +02:00
|
|
|
|
|
|
|
|
|
2026-07-10 11:55:38 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-06-24 17:45:39 +02:00
|
|
|
|
|
|
|
|
|
2026-07-10 11:55:38 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-06-24 17:45:39 +02:00
|
|
|
|
|
|
|
|
|
2026-07-10 11:55:38 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-06-24 17:45:39 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
zml/io/vfs: add parallel read to GCS and S3 (#604)
# Summary
This PR extends the support to chunked parallel read introduced
initially for HF in the VFS, to GCS and S3.
It:
- Adds a shared `zml/io/vfs/parallel_read.zig` helper for chunked
parallel HTTP range reads
- Add configurable read pool options for S3/GCS initialization
- Register the `gs://` VFS backend in `examples/llm`.
# Tests
## S3
We used [MINIO](https://hub.docker.com/r/minio/minio) docker image from
the `5090` GPU machine. From there, we mirror the
`meta-llama/Llama-3.1-8B-Instruct` model through the S3 server.
Then from the `4090` GPU machine, we run
```bash
bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=s3://zml-test/Llama-3.1-8B-Instruct --prompt="hello, how are you ?"
```
It works both on 1 and 2 GPU.
## GCS
In the same spirit, we put the `meta-llama/Llama-3.1-8B-Instruct` in a
GCS bucket we own. Then, on the `4090`, we run
```bash
bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct --prompt="hello, how are you ?"
```
It works both on 1 and 2 GPU.
### Performance
As a reference, we run:
```bash
gcloud storage cp gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00001-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00002-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00003-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00004-of-00004.safetensors /tmp/gcs-gcloud-llama-shards/
```
It gives:
```
Completed files 4/4 | 15.0GiB/15.0GiB | 942.7MiB/s
Average throughput: 918.8MiB/s
```
Using the VFS in various setups:
* `dma_chunks = 32`, `dma_chunk_size = 128`, `parallelism = 16`,
`chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64`
```bash
info (llama): Loaded weights [14.96GiB, 21.302s, 719.02MiB/s]
```
* `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`,
`chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64`
```bash
info (llama): Loaded weights [14.96GiB, 18.227s, 840.32MiB/s]
```
* `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`,
`chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64`
```bash
info (llama): Loaded weights [14.96GiB, 17.376s, 881.43MiB/s]
```
* `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`,
`chunk_size = 16 MiB`, `num_workers = 32`, `queue_capacity = 64`
```bash
info (llama): Loaded weights [14.96GiB, 17.28s, 886.36MiB/s]
```
* `dma_chunks = 32`, `dma_chunk_size = 512`, `parallelism = 16`,
`chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64`
```bash
info (llama): Loaded weights [14.96GiB, 18.658s, 820.90MiB/s]
```
2026-06-26 19:24:31 +02:00
|
|
|
|
2026-06-24 17:45:39 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-07-10 11:55:38 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-06-24 17:45:39 +02:00
|
|
|
|
2026-07-10 11:55:38 +02:00
|
|
|
|
2026-06-24 17:45:39 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-12-19 14:00:26 +01:00
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
2025-12-19 14:00:26 +01:00
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
2025-12-19 14:00:26 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
2025-12-19 14:00:26 +01:00
|
|
|
|
2026-01-15 15:07:07 +01:00
|
|
|
|
2025-12-19 14:00:26 +01:00
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-12-19 14:00:26 +01:00
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-12-19 14:00:26 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-12-19 14:00:26 +01:00
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
|
|
|
|
|
2025-12-19 14:00:26 +01:00
|
|
|
|
|
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
|
|
|
|
|
2025-12-19 14:00:26 +01:00
|
|
|
|
|
|
|
|
|
2025-12-15 13:39:48 +01:00
|
|
|
|
|
|
|
|
|
2026-06-24 17:45:39 +02:00
|
|
|
|
2026-07-10 13:21:30 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-06-24 17:45:39 +02:00
|
|
|
|
|
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-12-15 13:39:48 +01:00
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-12-15 13:39:48 +01:00
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-12-15 13:39:48 +01:00
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
2026-01-15 15:07:07 +01:00
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
|
|
|
|
|
2025-12-19 14:00:26 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-12-15 13:39:48 +01:00
|
|
|
|
|
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
|
|
|
|
|
2025-12-19 14:00:26 +01:00
|
|
|
|
2025-12-15 13:39:48 +01:00
|
|
|
|
|
|
|
|
|
2025-12-19 14:00:26 +01:00
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
2025-12-19 14:00:26 +01:00
|
|
|
|
2026-01-15 15:07:07 +01:00
|
|
|
|
2026-06-24 17:45:39 +02:00
|
|
|
|
2026-01-15 15:07:07 +01:00
|
|
|
|
2026-04-13 11:56:32 +02:00
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-12-19 14:00:26 +01:00
|
|
|
|
2026-06-24 17:45:39 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
2026-01-15 15:07:07 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-12-15 13:39:48 +01:00
|
|
|
|
2026-06-24 17:45:39 +02:00
|
|
|
|
2026-01-15 15:07:07 +01:00
|
|
|
|
2026-06-24 17:45:39 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-01-15 15:07:07 +01:00
|
|
|
|
2025-12-15 13:39:48 +01:00
|
|
|
|
2026-02-19 10:48:15 +01:00
|
|
|
|
|
|
|
|
|
2026-02-24 09:52:40 +01:00
|
|
|
|
2026-02-19 10:48:15 +01:00
|
|
|
|
|
|
|
|
|
2026-01-15 15:07:07 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-02-24 09:52:40 +01:00
|
|
|
|
2026-01-15 15:07:07 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-06-24 17:45:39 +02:00
|
|
|
|
2025-12-15 13:39:48 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-06-24 17:45:39 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-12-15 13:39:48 +01:00
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-12-15 13:39:48 +01:00
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-01-15 15:07:07 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-12-15 13:39:48 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
|
|
|
|
|
2026-02-19 10:48:15 +01:00
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-12-15 13:39:48 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-12-15 13:39:48 +01:00
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
|
|
|
|
|
2025-12-19 14:00:26 +01:00
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
|
|
|
|
|
2025-12-19 14:00:26 +01:00
|
|
|
|
|
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-12-19 14:00:26 +01:00
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
|
|
|
|
|
2025-12-19 14:00:26 +01:00
|
|
|
|
|
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-12-19 14:00:26 +01:00
|
|
|
|
|
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-12-15 13:39:48 +01:00
|
|
|
|
|
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
2025-12-19 14:00:26 +01:00
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
2025-12-15 13:39:48 +01:00
|
|
|
|
|
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-12-15 13:39:48 +01:00
|
|
|
|
|
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
|
|
|
|
|
2025-12-19 14:00:26 +01:00
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
2025-12-19 14:00:26 +01:00
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
2025-12-19 14:00:26 +01:00
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
2025-12-19 14:00:26 +01:00
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
|
|
|
|
|
2025-12-19 14:00:26 +01:00
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
|
|
|
|
|
2025-12-19 14:00:26 +01:00
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
|
|
|
|
|
2025-12-19 14:00:26 +01:00
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-12-19 14:00:26 +01:00
|
|
|
|
|
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
|
|
|
|
|
2025-12-19 14:00:26 +01:00
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
2025-12-19 14:00:26 +01:00
|
|
|
|
|
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-12-19 14:00:26 +01:00
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-12-19 14:00:26 +01:00
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
2026-04-29 10:38:19 +02:00
|
|
|
|
2025-12-19 14:00:26 +01:00
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
2026-03-10 12:51:08 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-04-13 11:56:32 +02:00
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-12-22 11:29:55 +01:00
|
|
|
|
|
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-12-22 11:29:55 +01:00
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
2025-12-22 11:29:55 +01:00
|
|
|
|
2025-12-19 14:00:26 +01:00
|
|
|
|
|
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
2025-12-19 14:00:26 +01:00
|
|
|
|
|
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
|
|
|
|
|
2025-12-15 13:39:48 +01:00
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
|
|
|
|
|
2025-12-15 13:39:48 +01:00
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
|
|
|
|
|
2025-12-15 13:39:48 +01:00
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-12-15 13:39:48 +01:00
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-12-15 13:39:48 +01:00
|
|
|
|
|
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
2025-12-15 13:39:48 +01:00
|
|
|
|
|
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-12-15 13:39:48 +01:00
|
|
|
|
|
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-04-13 11:56:32 +02:00
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-04-13 11:56:32 +02:00
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-04-13 11:56:32 +02:00
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-12-15 13:39:48 +01:00
|
|
|
|
|
|
|
|
|
2026-02-19 10:48:15 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-04-13 11:56:32 +02:00
|
|
|
|
2026-02-19 10:48:15 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-04-13 11:56:32 +02:00
|
|
|
|
2026-02-19 10:48:15 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-10-13 17:11:38 +02:00
|
|
|
|
2026-02-19 10:48:15 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-12-15 13:39:48 +01:00
|
|
|
|
|
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-12-15 13:39:48 +01:00
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-02-19 10:48:15 +01:00
|
|
|
|
2025-12-19 14:00:26 +01:00
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-02-19 10:48:15 +01:00
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-10-13 17:11:38 +02:00
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-02-19 10:48:15 +01:00
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-12-15 13:39:48 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
2025-12-19 14:00:26 +01:00
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-02-19 10:48:15 +01:00
|
|
|
|
2025-12-15 13:39:48 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-12-15 13:39:48 +01:00
|
|
|
|
|
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-12-15 13:39:48 +01:00
|
|
|
|
|
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-04-13 11:56:32 +02:00
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
2025-12-15 13:39:48 +01:00
|
|
|
|
|
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-12-15 13:39:48 +01:00
|
|
|
|
|
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-12-15 13:39:48 +01:00
|
|
|
|
|
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2025-12-15 13:39:48 +01:00
|
|
|
|
|
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-06-24 17:45:39 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
zml/io/vfs: add parallel read to GCS and S3 (#604)
# Summary
This PR extends the support to chunked parallel read introduced
initially for HF in the VFS, to GCS and S3.
It:
- Adds a shared `zml/io/vfs/parallel_read.zig` helper for chunked
parallel HTTP range reads
- Add configurable read pool options for S3/GCS initialization
- Register the `gs://` VFS backend in `examples/llm`.
# Tests
## S3
We used [MINIO](https://hub.docker.com/r/minio/minio) docker image from
the `5090` GPU machine. From there, we mirror the
`meta-llama/Llama-3.1-8B-Instruct` model through the S3 server.
Then from the `4090` GPU machine, we run
```bash
bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=s3://zml-test/Llama-3.1-8B-Instruct --prompt="hello, how are you ?"
```
It works both on 1 and 2 GPU.
## GCS
In the same spirit, we put the `meta-llama/Llama-3.1-8B-Instruct` in a
GCS bucket we own. Then, on the `4090`, we run
```bash
bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct --prompt="hello, how are you ?"
```
It works both on 1 and 2 GPU.
### Performance
As a reference, we run:
```bash
gcloud storage cp gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00001-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00002-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00003-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00004-of-00004.safetensors /tmp/gcs-gcloud-llama-shards/
```
It gives:
```
Completed files 4/4 | 15.0GiB/15.0GiB | 942.7MiB/s
Average throughput: 918.8MiB/s
```
Using the VFS in various setups:
* `dma_chunks = 32`, `dma_chunk_size = 128`, `parallelism = 16`,
`chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64`
```bash
info (llama): Loaded weights [14.96GiB, 21.302s, 719.02MiB/s]
```
* `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`,
`chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64`
```bash
info (llama): Loaded weights [14.96GiB, 18.227s, 840.32MiB/s]
```
* `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`,
`chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64`
```bash
info (llama): Loaded weights [14.96GiB, 17.376s, 881.43MiB/s]
```
* `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`,
`chunk_size = 16 MiB`, `num_workers = 32`, `queue_capacity = 64`
```bash
info (llama): Loaded weights [14.96GiB, 17.28s, 886.36MiB/s]
```
* `dma_chunks = 32`, `dma_chunk_size = 512`, `parallelism = 16`,
`chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64`
```bash
info (llama): Loaded weights [14.96GiB, 18.658s, 820.90MiB/s]
```
2026-06-26 19:24:31 +02:00
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-06-24 17:45:39 +02:00
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
|
|
|
|
|
2026-06-24 17:45:39 +02:00
|
|
|
|
2026-07-10 11:55:38 +02:00
|
|
|
|
2026-06-24 17:45:39 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
|
|
|
|
|
2026-06-24 17:45:39 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
|
|
|
|
|
2026-06-24 17:45:39 +02:00
|
|
|
|
2026-07-10 11:55:38 +02:00
|
|
|
|
|
|
|
|
|
2026-01-07 12:08:31 +01:00
|
|
|
|
2026-06-24 17:45:39 +02:00
|
|
|
|
2025-12-15 13:39:48 +01:00
|
|
|
|
|
|
|
|
|