2026-03-24 16:55:29 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-03-26 15:58:03 +01:00
|
|
|
|
2026-10-12 20:31:17 +02:00
|
|
|
|
2026-04-22 16:23:36 +02:00
|
|
|
|
2026-03-24 16:55:29 +01:00
|
|
|
|
|
|
|
|
|
2026-03-25 15:14:22 +01:00
|
|
|
|
2026-03-24 16:55:29 +01:00
|
|
|
|
2026-03-25 15:14:22 +01:00
|
|
|
|
2026-03-24 16:55:29 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-03-26 15:58:03 +01:00
|
|
|
|
2026-10-12 20:31:17 +02:00
|
|
|
|
2026-04-22 16:23:36 +02:00
|
|
|
|
2026-03-24 16:55:29 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-03-25 15:14:22 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-03-24 16:55:29 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
zml/io/vfs: add parallel read to GCS and S3 (#604)
# Summary
This PR extends the support to chunked parallel read introduced
initially for HF in the VFS, to GCS and S3.
It:
- Adds a shared `zml/io/vfs/parallel_read.zig` helper for chunked
parallel HTTP range reads
- Add configurable read pool options for S3/GCS initialization
- Register the `gs://` VFS backend in `examples/llm`.
# Tests
## S3
We used [MINIO](https://hub.docker.com/r/minio/minio) docker image from
the `5090` GPU machine. From there, we mirror the
`meta-llama/Llama-3.1-8B-Instruct` model through the S3 server.
Then from the `4090` GPU machine, we run
```bash
bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=s3://zml-test/Llama-3.1-8B-Instruct --prompt="hello, how are you ?"
```
It works both on 1 and 2 GPU.
## GCS
In the same spirit, we put the `meta-llama/Llama-3.1-8B-Instruct` in a
GCS bucket we own. Then, on the `4090`, we run
```bash
bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct --prompt="hello, how are you ?"
```
It works both on 1 and 2 GPU.
### Performance
As a reference, we run:
```bash
gcloud storage cp gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00001-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00002-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00003-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00004-of-00004.safetensors /tmp/gcs-gcloud-llama-shards/
```
It gives:
```
Completed files 4/4 | 15.0GiB/15.0GiB | 942.7MiB/s
Average throughput: 918.8MiB/s
```
Using the VFS in various setups:
* `dma_chunks = 32`, `dma_chunk_size = 128`, `parallelism = 16`,
`chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64`
```bash
info (llama): Loaded weights [14.96GiB, 21.302s, 719.02MiB/s]
```
* `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`,
`chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64`
```bash
info (llama): Loaded weights [14.96GiB, 18.227s, 840.32MiB/s]
```
* `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`,
`chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64`
```bash
info (llama): Loaded weights [14.96GiB, 17.376s, 881.43MiB/s]
```
* `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`,
`chunk_size = 16 MiB`, `num_workers = 32`, `queue_capacity = 64`
```bash
info (llama): Loaded weights [14.96GiB, 17.28s, 886.36MiB/s]
```
* `dma_chunks = 32`, `dma_chunk_size = 512`, `parallelism = 16`,
`chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64`
```bash
info (llama): Loaded weights [14.96GiB, 18.658s, 820.90MiB/s]
```
2026-06-26 19:24:31 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-03-24 16:55:29 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
zml/io/vfs: add parallel read to GCS and S3 (#604)
# Summary
This PR extends the support to chunked parallel read introduced
initially for HF in the VFS, to GCS and S3.
It:
- Adds a shared `zml/io/vfs/parallel_read.zig` helper for chunked
parallel HTTP range reads
- Add configurable read pool options for S3/GCS initialization
- Register the `gs://` VFS backend in `examples/llm`.
# Tests
## S3
We used [MINIO](https://hub.docker.com/r/minio/minio) docker image from
the `5090` GPU machine. From there, we mirror the
`meta-llama/Llama-3.1-8B-Instruct` model through the S3 server.
Then from the `4090` GPU machine, we run
```bash
bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=s3://zml-test/Llama-3.1-8B-Instruct --prompt="hello, how are you ?"
```
It works both on 1 and 2 GPU.
## GCS
In the same spirit, we put the `meta-llama/Llama-3.1-8B-Instruct` in a
GCS bucket we own. Then, on the `4090`, we run
```bash
bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct --prompt="hello, how are you ?"
```
It works both on 1 and 2 GPU.
### Performance
As a reference, we run:
```bash
gcloud storage cp gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00001-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00002-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00003-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00004-of-00004.safetensors /tmp/gcs-gcloud-llama-shards/
```
It gives:
```
Completed files 4/4 | 15.0GiB/15.0GiB | 942.7MiB/s
Average throughput: 918.8MiB/s
```
Using the VFS in various setups:
* `dma_chunks = 32`, `dma_chunk_size = 128`, `parallelism = 16`,
`chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64`
```bash
info (llama): Loaded weights [14.96GiB, 21.302s, 719.02MiB/s]
```
* `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`,
`chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64`
```bash
info (llama): Loaded weights [14.96GiB, 18.227s, 840.32MiB/s]
```
* `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`,
`chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64`
```bash
info (llama): Loaded weights [14.96GiB, 17.376s, 881.43MiB/s]
```
* `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`,
`chunk_size = 16 MiB`, `num_workers = 32`, `queue_capacity = 64`
```bash
info (llama): Loaded weights [14.96GiB, 17.28s, 886.36MiB/s]
```
* `dma_chunks = 32`, `dma_chunk_size = 512`, `parallelism = 16`,
`chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64`
```bash
info (llama): Loaded weights [14.96GiB, 18.658s, 820.90MiB/s]
```
2026-06-26 19:24:31 +02:00
|
|
|
|
2026-03-24 16:55:29 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-03-24 18:16:25 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-03-24 16:55:29 +01:00
|
|
|
|
2026-04-13 11:39:08 +02:00
|
|
|
|
2026-03-24 16:55:29 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-10-12 20:31:17 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-07-17 14:10:41 +02:00
|
|
|
|
2026-05-12 17:27:37 +02:00
|
|
|
|
2026-03-24 18:16:25 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-03-24 16:55:29 +01:00
|
|
|
|
|
|
|
|
|
2026-09-02 10:10:18 +03:00
|
|
|
|
2026-03-24 16:55:29 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-10-06 15:37:22 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-03-24 16:55:29 +01:00
|
|
|
|
|
|
|
|
|
2026-03-26 15:58:03 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-03-24 16:55:29 +01:00
|
|
|
|
|
|
|
|
|
2026-03-24 18:16:25 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-03-24 16:55:29 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-03-24 18:16:25 +01:00
|
|
|
|
|
|
|
|
|
2026-03-24 16:55:29 +01:00
|
|
|
|
2026-08-20 10:56:32 +02:00
|
|
|
|
|
|
|
|
|
2026-10-06 15:37:22 +02:00
|
|
|
|
2026-03-24 18:16:25 +01:00
|
|
|
|
2026-03-24 16:55:29 +01:00
|
|
|
|
2026-04-20 18:24:01 +02:00
|
|
|
|
2026-10-06 15:37:22 +02:00
|
|
|
|
2026-04-20 18:24:01 +02:00
|
|
|
|
|
|
|
|
|
2026-03-24 16:55:29 +01:00
|
|
|
|
|
|
|
|
|
2026-03-24 18:16:25 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-07-09 14:02:57 +02:00
|
|
|
|
2026-03-24 16:55:29 +01:00
|
|
|
|
|
|
|
|
|
2026-08-18 16:36:38 +02:00
|
|
|
|
|
|
|
|
|
2026-03-24 18:16:25 +01:00
|
|
|
|
2026-03-24 16:55:29 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-03-24 18:16:25 +01:00
|
|
|
|
2026-03-24 16:55:29 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-03-24 18:16:25 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-08-20 10:56:32 +02:00
|
|
|
|
2026-03-24 18:16:25 +01:00
|
|
|
|
|
|
|
|
|
2026-03-24 16:55:29 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-04-22 16:23:36 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-03-24 16:55:29 +01:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-04-07 16:42:55 +02:00
|
|
|
|
2026-03-24 16:55:29 +01:00
|
|
|
|
|
|
|
|
|
2026-07-09 14:02:57 +02:00
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2026-03-24 16:55:29 +01:00
|
|
|
|
|
|
|
|
|