0
Fork 0
mirror of https://github.com/zml/zml.git synced 2026-11-16 19:11:09 +00:00
zml/vfs/gcs.zig

1138 lines
46 KiB

zml/io/vfs: add parallel read to GCS and S3 (#604) # Summary This PR extends the support to chunked parallel read introduced initially for HF in the VFS, to GCS and S3. It: - Adds a shared `zml/io/vfs/parallel_read.zig` helper for chunked parallel HTTP range reads - Add configurable read pool options for S3/GCS initialization - Register the `gs://` VFS backend in `examples/llm`. # Tests ## S3 We used [MINIO](https://hub.docker.com/r/minio/minio) docker image from the `5090` GPU machine. From there, we mirror the `meta-llama/Llama-3.1-8B-Instruct` model through the S3 server. Then from the `4090` GPU machine, we run ```bash bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=s3://zml-test/Llama-3.1-8B-Instruct --prompt="hello, how are you ?" ``` It works both on 1 and 2 GPU. ## GCS In the same spirit, we put the `meta-llama/Llama-3.1-8B-Instruct` in a GCS bucket we own. Then, on the `4090`, we run ```bash bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct --prompt="hello, how are you ?" ``` It works both on 1 and 2 GPU. ### Performance As a reference, we run: ```bash gcloud storage cp gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00001-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00002-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00003-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00004-of-00004.safetensors /tmp/gcs-gcloud-llama-shards/ ``` It gives: ``` Completed files 4/4 | 15.0GiB/15.0GiB | 942.7MiB/s Average throughput: 918.8MiB/s ``` Using the VFS in various setups: * `dma_chunks = 32`, `dma_chunk_size = 128`, `parallelism = 16`, `chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 21.302s, 719.02MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 18.227s, 840.32MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 17.376s, 881.43MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 32`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 17.28s, 886.36MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 512`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 18.658s, 820.90MiB/s] ```
2026-06-26 19:24:31 +02:00
zml/io/vfs: add parallel read to GCS and S3 (#604) # Summary This PR extends the support to chunked parallel read introduced initially for HF in the VFS, to GCS and S3. It: - Adds a shared `zml/io/vfs/parallel_read.zig` helper for chunked parallel HTTP range reads - Add configurable read pool options for S3/GCS initialization - Register the `gs://` VFS backend in `examples/llm`. # Tests ## S3 We used [MINIO](https://hub.docker.com/r/minio/minio) docker image from the `5090` GPU machine. From there, we mirror the `meta-llama/Llama-3.1-8B-Instruct` model through the S3 server. Then from the `4090` GPU machine, we run ```bash bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=s3://zml-test/Llama-3.1-8B-Instruct --prompt="hello, how are you ?" ``` It works both on 1 and 2 GPU. ## GCS In the same spirit, we put the `meta-llama/Llama-3.1-8B-Instruct` in a GCS bucket we own. Then, on the `4090`, we run ```bash bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct --prompt="hello, how are you ?" ``` It works both on 1 and 2 GPU. ### Performance As a reference, we run: ```bash gcloud storage cp gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00001-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00002-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00003-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00004-of-00004.safetensors /tmp/gcs-gcloud-llama-shards/ ``` It gives: ``` Completed files 4/4 | 15.0GiB/15.0GiB | 942.7MiB/s Average throughput: 918.8MiB/s ``` Using the VFS in various setups: * `dma_chunks = 32`, `dma_chunk_size = 128`, `parallelism = 16`, `chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 21.302s, 719.02MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 18.227s, 840.32MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 17.376s, 881.43MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 32`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 17.28s, 886.36MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 512`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 18.658s, 820.90MiB/s] ```
2026-06-26 19:24:31 +02:00
zml/io/vfs: add parallel read to GCS and S3 (#604) # Summary This PR extends the support to chunked parallel read introduced initially for HF in the VFS, to GCS and S3. It: - Adds a shared `zml/io/vfs/parallel_read.zig` helper for chunked parallel HTTP range reads - Add configurable read pool options for S3/GCS initialization - Register the `gs://` VFS backend in `examples/llm`. # Tests ## S3 We used [MINIO](https://hub.docker.com/r/minio/minio) docker image from the `5090` GPU machine. From there, we mirror the `meta-llama/Llama-3.1-8B-Instruct` model through the S3 server. Then from the `4090` GPU machine, we run ```bash bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=s3://zml-test/Llama-3.1-8B-Instruct --prompt="hello, how are you ?" ``` It works both on 1 and 2 GPU. ## GCS In the same spirit, we put the `meta-llama/Llama-3.1-8B-Instruct` in a GCS bucket we own. Then, on the `4090`, we run ```bash bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct --prompt="hello, how are you ?" ``` It works both on 1 and 2 GPU. ### Performance As a reference, we run: ```bash gcloud storage cp gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00001-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00002-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00003-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00004-of-00004.safetensors /tmp/gcs-gcloud-llama-shards/ ``` It gives: ``` Completed files 4/4 | 15.0GiB/15.0GiB | 942.7MiB/s Average throughput: 918.8MiB/s ``` Using the VFS in various setups: * `dma_chunks = 32`, `dma_chunk_size = 128`, `parallelism = 16`, `chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 21.302s, 719.02MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 18.227s, 840.32MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 17.376s, 881.43MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 32`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 17.28s, 886.36MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 512`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 18.658s, 820.90MiB/s] ```
2026-06-26 19:24:31 +02:00
zml/io/vfs: add parallel read to GCS and S3 (#604) # Summary This PR extends the support to chunked parallel read introduced initially for HF in the VFS, to GCS and S3. It: - Adds a shared `zml/io/vfs/parallel_read.zig` helper for chunked parallel HTTP range reads - Add configurable read pool options for S3/GCS initialization - Register the `gs://` VFS backend in `examples/llm`. # Tests ## S3 We used [MINIO](https://hub.docker.com/r/minio/minio) docker image from the `5090` GPU machine. From there, we mirror the `meta-llama/Llama-3.1-8B-Instruct` model through the S3 server. Then from the `4090` GPU machine, we run ```bash bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=s3://zml-test/Llama-3.1-8B-Instruct --prompt="hello, how are you ?" ``` It works both on 1 and 2 GPU. ## GCS In the same spirit, we put the `meta-llama/Llama-3.1-8B-Instruct` in a GCS bucket we own. Then, on the `4090`, we run ```bash bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct --prompt="hello, how are you ?" ``` It works both on 1 and 2 GPU. ### Performance As a reference, we run: ```bash gcloud storage cp gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00001-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00002-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00003-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00004-of-00004.safetensors /tmp/gcs-gcloud-llama-shards/ ``` It gives: ``` Completed files 4/4 | 15.0GiB/15.0GiB | 942.7MiB/s Average throughput: 918.8MiB/s ``` Using the VFS in various setups: * `dma_chunks = 32`, `dma_chunk_size = 128`, `parallelism = 16`, `chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 21.302s, 719.02MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 18.227s, 840.32MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 17.376s, 881.43MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 32`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 17.28s, 886.36MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 512`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 18.658s, 820.90MiB/s] ```
2026-06-26 19:24:31 +02:00
zml/io/vfs: add parallel read to GCS and S3 (#604) # Summary This PR extends the support to chunked parallel read introduced initially for HF in the VFS, to GCS and S3. It: - Adds a shared `zml/io/vfs/parallel_read.zig` helper for chunked parallel HTTP range reads - Add configurable read pool options for S3/GCS initialization - Register the `gs://` VFS backend in `examples/llm`. # Tests ## S3 We used [MINIO](https://hub.docker.com/r/minio/minio) docker image from the `5090` GPU machine. From there, we mirror the `meta-llama/Llama-3.1-8B-Instruct` model through the S3 server. Then from the `4090` GPU machine, we run ```bash bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=s3://zml-test/Llama-3.1-8B-Instruct --prompt="hello, how are you ?" ``` It works both on 1 and 2 GPU. ## GCS In the same spirit, we put the `meta-llama/Llama-3.1-8B-Instruct` in a GCS bucket we own. Then, on the `4090`, we run ```bash bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct --prompt="hello, how are you ?" ``` It works both on 1 and 2 GPU. ### Performance As a reference, we run: ```bash gcloud storage cp gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00001-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00002-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00003-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00004-of-00004.safetensors /tmp/gcs-gcloud-llama-shards/ ``` It gives: ``` Completed files 4/4 | 15.0GiB/15.0GiB | 942.7MiB/s Average throughput: 918.8MiB/s ``` Using the VFS in various setups: * `dma_chunks = 32`, `dma_chunk_size = 128`, `parallelism = 16`, `chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 21.302s, 719.02MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 18.227s, 840.32MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 17.376s, 881.43MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 32`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 17.28s, 886.36MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 512`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 18.658s, 820.90MiB/s] ```
2026-06-26 19:24:31 +02:00
zml/io/vfs: add parallel read to GCS and S3 (#604) # Summary This PR extends the support to chunked parallel read introduced initially for HF in the VFS, to GCS and S3. It: - Adds a shared `zml/io/vfs/parallel_read.zig` helper for chunked parallel HTTP range reads - Add configurable read pool options for S3/GCS initialization - Register the `gs://` VFS backend in `examples/llm`. # Tests ## S3 We used [MINIO](https://hub.docker.com/r/minio/minio) docker image from the `5090` GPU machine. From there, we mirror the `meta-llama/Llama-3.1-8B-Instruct` model through the S3 server. Then from the `4090` GPU machine, we run ```bash bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=s3://zml-test/Llama-3.1-8B-Instruct --prompt="hello, how are you ?" ``` It works both on 1 and 2 GPU. ## GCS In the same spirit, we put the `meta-llama/Llama-3.1-8B-Instruct` in a GCS bucket we own. Then, on the `4090`, we run ```bash bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct --prompt="hello, how are you ?" ``` It works both on 1 and 2 GPU. ### Performance As a reference, we run: ```bash gcloud storage cp gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00001-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00002-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00003-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00004-of-00004.safetensors /tmp/gcs-gcloud-llama-shards/ ``` It gives: ``` Completed files 4/4 | 15.0GiB/15.0GiB | 942.7MiB/s Average throughput: 918.8MiB/s ``` Using the VFS in various setups: * `dma_chunks = 32`, `dma_chunk_size = 128`, `parallelism = 16`, `chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 21.302s, 719.02MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 18.227s, 840.32MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 17.376s, 881.43MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 32`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 17.28s, 886.36MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 512`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 18.658s, 820.90MiB/s] ```
2026-06-26 19:24:31 +02:00
zml/io/vfs: add parallel read to GCS and S3 (#604) # Summary This PR extends the support to chunked parallel read introduced initially for HF in the VFS, to GCS and S3. It: - Adds a shared `zml/io/vfs/parallel_read.zig` helper for chunked parallel HTTP range reads - Add configurable read pool options for S3/GCS initialization - Register the `gs://` VFS backend in `examples/llm`. # Tests ## S3 We used [MINIO](https://hub.docker.com/r/minio/minio) docker image from the `5090` GPU machine. From there, we mirror the `meta-llama/Llama-3.1-8B-Instruct` model through the S3 server. Then from the `4090` GPU machine, we run ```bash bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=s3://zml-test/Llama-3.1-8B-Instruct --prompt="hello, how are you ?" ``` It works both on 1 and 2 GPU. ## GCS In the same spirit, we put the `meta-llama/Llama-3.1-8B-Instruct` in a GCS bucket we own. Then, on the `4090`, we run ```bash bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct --prompt="hello, how are you ?" ``` It works both on 1 and 2 GPU. ### Performance As a reference, we run: ```bash gcloud storage cp gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00001-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00002-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00003-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00004-of-00004.safetensors /tmp/gcs-gcloud-llama-shards/ ``` It gives: ``` Completed files 4/4 | 15.0GiB/15.0GiB | 942.7MiB/s Average throughput: 918.8MiB/s ``` Using the VFS in various setups: * `dma_chunks = 32`, `dma_chunk_size = 128`, `parallelism = 16`, `chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 21.302s, 719.02MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 18.227s, 840.32MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 17.376s, 881.43MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 32`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 17.28s, 886.36MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 512`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 18.658s, 820.90MiB/s] ```
2026-06-26 19:24:31 +02:00
zml/io/vfs: add parallel read to GCS and S3 (#604) # Summary This PR extends the support to chunked parallel read introduced initially for HF in the VFS, to GCS and S3. It: - Adds a shared `zml/io/vfs/parallel_read.zig` helper for chunked parallel HTTP range reads - Add configurable read pool options for S3/GCS initialization - Register the `gs://` VFS backend in `examples/llm`. # Tests ## S3 We used [MINIO](https://hub.docker.com/r/minio/minio) docker image from the `5090` GPU machine. From there, we mirror the `meta-llama/Llama-3.1-8B-Instruct` model through the S3 server. Then from the `4090` GPU machine, we run ```bash bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=s3://zml-test/Llama-3.1-8B-Instruct --prompt="hello, how are you ?" ``` It works both on 1 and 2 GPU. ## GCS In the same spirit, we put the `meta-llama/Llama-3.1-8B-Instruct` in a GCS bucket we own. Then, on the `4090`, we run ```bash bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct --prompt="hello, how are you ?" ``` It works both on 1 and 2 GPU. ### Performance As a reference, we run: ```bash gcloud storage cp gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00001-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00002-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00003-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00004-of-00004.safetensors /tmp/gcs-gcloud-llama-shards/ ``` It gives: ``` Completed files 4/4 | 15.0GiB/15.0GiB | 942.7MiB/s Average throughput: 918.8MiB/s ``` Using the VFS in various setups: * `dma_chunks = 32`, `dma_chunk_size = 128`, `parallelism = 16`, `chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 21.302s, 719.02MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 18.227s, 840.32MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 17.376s, 881.43MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 32`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 17.28s, 886.36MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 512`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 18.658s, 820.90MiB/s] ```
2026-06-26 19:24:31 +02:00
zml/io/vfs: add parallel read to GCS and S3 (#604) # Summary This PR extends the support to chunked parallel read introduced initially for HF in the VFS, to GCS and S3. It: - Adds a shared `zml/io/vfs/parallel_read.zig` helper for chunked parallel HTTP range reads - Add configurable read pool options for S3/GCS initialization - Register the `gs://` VFS backend in `examples/llm`. # Tests ## S3 We used [MINIO](https://hub.docker.com/r/minio/minio) docker image from the `5090` GPU machine. From there, we mirror the `meta-llama/Llama-3.1-8B-Instruct` model through the S3 server. Then from the `4090` GPU machine, we run ```bash bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=s3://zml-test/Llama-3.1-8B-Instruct --prompt="hello, how are you ?" ``` It works both on 1 and 2 GPU. ## GCS In the same spirit, we put the `meta-llama/Llama-3.1-8B-Instruct` in a GCS bucket we own. Then, on the `4090`, we run ```bash bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct --prompt="hello, how are you ?" ``` It works both on 1 and 2 GPU. ### Performance As a reference, we run: ```bash gcloud storage cp gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00001-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00002-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00003-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00004-of-00004.safetensors /tmp/gcs-gcloud-llama-shards/ ``` It gives: ``` Completed files 4/4 | 15.0GiB/15.0GiB | 942.7MiB/s Average throughput: 918.8MiB/s ``` Using the VFS in various setups: * `dma_chunks = 32`, `dma_chunk_size = 128`, `parallelism = 16`, `chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 21.302s, 719.02MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 18.227s, 840.32MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 17.376s, 881.43MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 32`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 17.28s, 886.36MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 512`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 18.658s, 820.90MiB/s] ```
2026-06-26 19:24:31 +02:00
zml/io/vfs: add parallel read to GCS and S3 (#604) # Summary This PR extends the support to chunked parallel read introduced initially for HF in the VFS, to GCS and S3. It: - Adds a shared `zml/io/vfs/parallel_read.zig` helper for chunked parallel HTTP range reads - Add configurable read pool options for S3/GCS initialization - Register the `gs://` VFS backend in `examples/llm`. # Tests ## S3 We used [MINIO](https://hub.docker.com/r/minio/minio) docker image from the `5090` GPU machine. From there, we mirror the `meta-llama/Llama-3.1-8B-Instruct` model through the S3 server. Then from the `4090` GPU machine, we run ```bash bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=s3://zml-test/Llama-3.1-8B-Instruct --prompt="hello, how are you ?" ``` It works both on 1 and 2 GPU. ## GCS In the same spirit, we put the `meta-llama/Llama-3.1-8B-Instruct` in a GCS bucket we own. Then, on the `4090`, we run ```bash bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct --prompt="hello, how are you ?" ``` It works both on 1 and 2 GPU. ### Performance As a reference, we run: ```bash gcloud storage cp gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00001-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00002-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00003-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00004-of-00004.safetensors /tmp/gcs-gcloud-llama-shards/ ``` It gives: ``` Completed files 4/4 | 15.0GiB/15.0GiB | 942.7MiB/s Average throughput: 918.8MiB/s ``` Using the VFS in various setups: * `dma_chunks = 32`, `dma_chunk_size = 128`, `parallelism = 16`, `chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 21.302s, 719.02MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 18.227s, 840.32MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 17.376s, 881.43MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 32`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 17.28s, 886.36MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 512`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 18.658s, 820.90MiB/s] ```
2026-06-26 19:24:31 +02:00
zml/io/vfs: add parallel read to GCS and S3 (#604) # Summary This PR extends the support to chunked parallel read introduced initially for HF in the VFS, to GCS and S3. It: - Adds a shared `zml/io/vfs/parallel_read.zig` helper for chunked parallel HTTP range reads - Add configurable read pool options for S3/GCS initialization - Register the `gs://` VFS backend in `examples/llm`. # Tests ## S3 We used [MINIO](https://hub.docker.com/r/minio/minio) docker image from the `5090` GPU machine. From there, we mirror the `meta-llama/Llama-3.1-8B-Instruct` model through the S3 server. Then from the `4090` GPU machine, we run ```bash bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=s3://zml-test/Llama-3.1-8B-Instruct --prompt="hello, how are you ?" ``` It works both on 1 and 2 GPU. ## GCS In the same spirit, we put the `meta-llama/Llama-3.1-8B-Instruct` in a GCS bucket we own. Then, on the `4090`, we run ```bash bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct --prompt="hello, how are you ?" ``` It works both on 1 and 2 GPU. ### Performance As a reference, we run: ```bash gcloud storage cp gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00001-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00002-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00003-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00004-of-00004.safetensors /tmp/gcs-gcloud-llama-shards/ ``` It gives: ``` Completed files 4/4 | 15.0GiB/15.0GiB | 942.7MiB/s Average throughput: 918.8MiB/s ``` Using the VFS in various setups: * `dma_chunks = 32`, `dma_chunk_size = 128`, `parallelism = 16`, `chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 21.302s, 719.02MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 18.227s, 840.32MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 17.376s, 881.43MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 32`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 17.28s, 886.36MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 512`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 18.658s, 820.90MiB/s] ```
2026-06-26 19:24:31 +02:00
zml/io/vfs: add parallel read to GCS and S3 (#604) # Summary This PR extends the support to chunked parallel read introduced initially for HF in the VFS, to GCS and S3. It: - Adds a shared `zml/io/vfs/parallel_read.zig` helper for chunked parallel HTTP range reads - Add configurable read pool options for S3/GCS initialization - Register the `gs://` VFS backend in `examples/llm`. # Tests ## S3 We used [MINIO](https://hub.docker.com/r/minio/minio) docker image from the `5090` GPU machine. From there, we mirror the `meta-llama/Llama-3.1-8B-Instruct` model through the S3 server. Then from the `4090` GPU machine, we run ```bash bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=s3://zml-test/Llama-3.1-8B-Instruct --prompt="hello, how are you ?" ``` It works both on 1 and 2 GPU. ## GCS In the same spirit, we put the `meta-llama/Llama-3.1-8B-Instruct` in a GCS bucket we own. Then, on the `4090`, we run ```bash bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct --prompt="hello, how are you ?" ``` It works both on 1 and 2 GPU. ### Performance As a reference, we run: ```bash gcloud storage cp gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00001-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00002-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00003-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00004-of-00004.safetensors /tmp/gcs-gcloud-llama-shards/ ``` It gives: ``` Completed files 4/4 | 15.0GiB/15.0GiB | 942.7MiB/s Average throughput: 918.8MiB/s ``` Using the VFS in various setups: * `dma_chunks = 32`, `dma_chunk_size = 128`, `parallelism = 16`, `chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 21.302s, 719.02MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 18.227s, 840.32MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 17.376s, 881.43MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 32`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 17.28s, 886.36MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 512`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 18.658s, 820.90MiB/s] ```
2026-06-26 19:24:31 +02:00
zml/io/vfs: add parallel read to GCS and S3 (#604) # Summary This PR extends the support to chunked parallel read introduced initially for HF in the VFS, to GCS and S3. It: - Adds a shared `zml/io/vfs/parallel_read.zig` helper for chunked parallel HTTP range reads - Add configurable read pool options for S3/GCS initialization - Register the `gs://` VFS backend in `examples/llm`. # Tests ## S3 We used [MINIO](https://hub.docker.com/r/minio/minio) docker image from the `5090` GPU machine. From there, we mirror the `meta-llama/Llama-3.1-8B-Instruct` model through the S3 server. Then from the `4090` GPU machine, we run ```bash bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=s3://zml-test/Llama-3.1-8B-Instruct --prompt="hello, how are you ?" ``` It works both on 1 and 2 GPU. ## GCS In the same spirit, we put the `meta-llama/Llama-3.1-8B-Instruct` in a GCS bucket we own. Then, on the `4090`, we run ```bash bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct --prompt="hello, how are you ?" ``` It works both on 1 and 2 GPU. ### Performance As a reference, we run: ```bash gcloud storage cp gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00001-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00002-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00003-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00004-of-00004.safetensors /tmp/gcs-gcloud-llama-shards/ ``` It gives: ``` Completed files 4/4 | 15.0GiB/15.0GiB | 942.7MiB/s Average throughput: 918.8MiB/s ``` Using the VFS in various setups: * `dma_chunks = 32`, `dma_chunk_size = 128`, `parallelism = 16`, `chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 21.302s, 719.02MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 18.227s, 840.32MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 17.376s, 881.43MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 32`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 17.28s, 886.36MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 512`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 18.658s, 820.90MiB/s] ```
2026-06-26 19:24:31 +02:00
zml/io/vfs: add parallel read to GCS and S3 (#604) # Summary This PR extends the support to chunked parallel read introduced initially for HF in the VFS, to GCS and S3. It: - Adds a shared `zml/io/vfs/parallel_read.zig` helper for chunked parallel HTTP range reads - Add configurable read pool options for S3/GCS initialization - Register the `gs://` VFS backend in `examples/llm`. # Tests ## S3 We used [MINIO](https://hub.docker.com/r/minio/minio) docker image from the `5090` GPU machine. From there, we mirror the `meta-llama/Llama-3.1-8B-Instruct` model through the S3 server. Then from the `4090` GPU machine, we run ```bash bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=s3://zml-test/Llama-3.1-8B-Instruct --prompt="hello, how are you ?" ``` It works both on 1 and 2 GPU. ## GCS In the same spirit, we put the `meta-llama/Llama-3.1-8B-Instruct` in a GCS bucket we own. Then, on the `4090`, we run ```bash bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct --prompt="hello, how are you ?" ``` It works both on 1 and 2 GPU. ### Performance As a reference, we run: ```bash gcloud storage cp gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00001-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00002-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00003-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00004-of-00004.safetensors /tmp/gcs-gcloud-llama-shards/ ``` It gives: ``` Completed files 4/4 | 15.0GiB/15.0GiB | 942.7MiB/s Average throughput: 918.8MiB/s ``` Using the VFS in various setups: * `dma_chunks = 32`, `dma_chunk_size = 128`, `parallelism = 16`, `chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 21.302s, 719.02MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 18.227s, 840.32MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 17.376s, 881.43MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 32`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 17.28s, 886.36MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 512`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 18.658s, 820.90MiB/s] ```
2026-06-26 19:24:31 +02:00
zml/io/vfs: add parallel read to GCS and S3 (#604) # Summary This PR extends the support to chunked parallel read introduced initially for HF in the VFS, to GCS and S3. It: - Adds a shared `zml/io/vfs/parallel_read.zig` helper for chunked parallel HTTP range reads - Add configurable read pool options for S3/GCS initialization - Register the `gs://` VFS backend in `examples/llm`. # Tests ## S3 We used [MINIO](https://hub.docker.com/r/minio/minio) docker image from the `5090` GPU machine. From there, we mirror the `meta-llama/Llama-3.1-8B-Instruct` model through the S3 server. Then from the `4090` GPU machine, we run ```bash bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=s3://zml-test/Llama-3.1-8B-Instruct --prompt="hello, how are you ?" ``` It works both on 1 and 2 GPU. ## GCS In the same spirit, we put the `meta-llama/Llama-3.1-8B-Instruct` in a GCS bucket we own. Then, on the `4090`, we run ```bash bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct --prompt="hello, how are you ?" ``` It works both on 1 and 2 GPU. ### Performance As a reference, we run: ```bash gcloud storage cp gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00001-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00002-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00003-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00004-of-00004.safetensors /tmp/gcs-gcloud-llama-shards/ ``` It gives: ``` Completed files 4/4 | 15.0GiB/15.0GiB | 942.7MiB/s Average throughput: 918.8MiB/s ``` Using the VFS in various setups: * `dma_chunks = 32`, `dma_chunk_size = 128`, `parallelism = 16`, `chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 21.302s, 719.02MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 18.227s, 840.32MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 17.376s, 881.43MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 32`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 17.28s, 886.36MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 512`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 18.658s, 820.90MiB/s] ```
2026-06-26 19:24:31 +02:00
zml/io/vfs: add parallel read to GCS and S3 (#604) # Summary This PR extends the support to chunked parallel read introduced initially for HF in the VFS, to GCS and S3. It: - Adds a shared `zml/io/vfs/parallel_read.zig` helper for chunked parallel HTTP range reads - Add configurable read pool options for S3/GCS initialization - Register the `gs://` VFS backend in `examples/llm`. # Tests ## S3 We used [MINIO](https://hub.docker.com/r/minio/minio) docker image from the `5090` GPU machine. From there, we mirror the `meta-llama/Llama-3.1-8B-Instruct` model through the S3 server. Then from the `4090` GPU machine, we run ```bash bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=s3://zml-test/Llama-3.1-8B-Instruct --prompt="hello, how are you ?" ``` It works both on 1 and 2 GPU. ## GCS In the same spirit, we put the `meta-llama/Llama-3.1-8B-Instruct` in a GCS bucket we own. Then, on the `4090`, we run ```bash bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct --prompt="hello, how are you ?" ``` It works both on 1 and 2 GPU. ### Performance As a reference, we run: ```bash gcloud storage cp gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00001-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00002-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00003-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00004-of-00004.safetensors /tmp/gcs-gcloud-llama-shards/ ``` It gives: ``` Completed files 4/4 | 15.0GiB/15.0GiB | 942.7MiB/s Average throughput: 918.8MiB/s ``` Using the VFS in various setups: * `dma_chunks = 32`, `dma_chunk_size = 128`, `parallelism = 16`, `chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 21.302s, 719.02MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 18.227s, 840.32MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 17.376s, 881.43MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 32`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 17.28s, 886.36MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 512`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 18.658s, 820.90MiB/s] ```
2026-06-26 19:24:31 +02:00
zml/io/vfs: add parallel read to GCS and S3 (#604) # Summary This PR extends the support to chunked parallel read introduced initially for HF in the VFS, to GCS and S3. It: - Adds a shared `zml/io/vfs/parallel_read.zig` helper for chunked parallel HTTP range reads - Add configurable read pool options for S3/GCS initialization - Register the `gs://` VFS backend in `examples/llm`. # Tests ## S3 We used [MINIO](https://hub.docker.com/r/minio/minio) docker image from the `5090` GPU machine. From there, we mirror the `meta-llama/Llama-3.1-8B-Instruct` model through the S3 server. Then from the `4090` GPU machine, we run ```bash bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=s3://zml-test/Llama-3.1-8B-Instruct --prompt="hello, how are you ?" ``` It works both on 1 and 2 GPU. ## GCS In the same spirit, we put the `meta-llama/Llama-3.1-8B-Instruct` in a GCS bucket we own. Then, on the `4090`, we run ```bash bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct --prompt="hello, how are you ?" ``` It works both on 1 and 2 GPU. ### Performance As a reference, we run: ```bash gcloud storage cp gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00001-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00002-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00003-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00004-of-00004.safetensors /tmp/gcs-gcloud-llama-shards/ ``` It gives: ``` Completed files 4/4 | 15.0GiB/15.0GiB | 942.7MiB/s Average throughput: 918.8MiB/s ``` Using the VFS in various setups: * `dma_chunks = 32`, `dma_chunk_size = 128`, `parallelism = 16`, `chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 21.302s, 719.02MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 18.227s, 840.32MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 17.376s, 881.43MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 32`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 17.28s, 886.36MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 512`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 18.658s, 820.90MiB/s] ```
2026-06-26 19:24:31 +02:00
zml/io/vfs: add parallel read to GCS and S3 (#604) # Summary This PR extends the support to chunked parallel read introduced initially for HF in the VFS, to GCS and S3. It: - Adds a shared `zml/io/vfs/parallel_read.zig` helper for chunked parallel HTTP range reads - Add configurable read pool options for S3/GCS initialization - Register the `gs://` VFS backend in `examples/llm`. # Tests ## S3 We used [MINIO](https://hub.docker.com/r/minio/minio) docker image from the `5090` GPU machine. From there, we mirror the `meta-llama/Llama-3.1-8B-Instruct` model through the S3 server. Then from the `4090` GPU machine, we run ```bash bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=s3://zml-test/Llama-3.1-8B-Instruct --prompt="hello, how are you ?" ``` It works both on 1 and 2 GPU. ## GCS In the same spirit, we put the `meta-llama/Llama-3.1-8B-Instruct` in a GCS bucket we own. Then, on the `4090`, we run ```bash bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct --prompt="hello, how are you ?" ``` It works both on 1 and 2 GPU. ### Performance As a reference, we run: ```bash gcloud storage cp gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00001-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00002-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00003-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00004-of-00004.safetensors /tmp/gcs-gcloud-llama-shards/ ``` It gives: ``` Completed files 4/4 | 15.0GiB/15.0GiB | 942.7MiB/s Average throughput: 918.8MiB/s ``` Using the VFS in various setups: * `dma_chunks = 32`, `dma_chunk_size = 128`, `parallelism = 16`, `chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 21.302s, 719.02MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 18.227s, 840.32MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 17.376s, 881.43MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 32`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 17.28s, 886.36MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 512`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 18.658s, 820.90MiB/s] ```
2026-06-26 19:24:31 +02:00
zml/io/vfs: add parallel read to GCS and S3 (#604) # Summary This PR extends the support to chunked parallel read introduced initially for HF in the VFS, to GCS and S3. It: - Adds a shared `zml/io/vfs/parallel_read.zig` helper for chunked parallel HTTP range reads - Add configurable read pool options for S3/GCS initialization - Register the `gs://` VFS backend in `examples/llm`. # Tests ## S3 We used [MINIO](https://hub.docker.com/r/minio/minio) docker image from the `5090` GPU machine. From there, we mirror the `meta-llama/Llama-3.1-8B-Instruct` model through the S3 server. Then from the `4090` GPU machine, we run ```bash bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=s3://zml-test/Llama-3.1-8B-Instruct --prompt="hello, how are you ?" ``` It works both on 1 and 2 GPU. ## GCS In the same spirit, we put the `meta-llama/Llama-3.1-8B-Instruct` in a GCS bucket we own. Then, on the `4090`, we run ```bash bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct --prompt="hello, how are you ?" ``` It works both on 1 and 2 GPU. ### Performance As a reference, we run: ```bash gcloud storage cp gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00001-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00002-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00003-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00004-of-00004.safetensors /tmp/gcs-gcloud-llama-shards/ ``` It gives: ``` Completed files 4/4 | 15.0GiB/15.0GiB | 942.7MiB/s Average throughput: 918.8MiB/s ``` Using the VFS in various setups: * `dma_chunks = 32`, `dma_chunk_size = 128`, `parallelism = 16`, `chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 21.302s, 719.02MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 18.227s, 840.32MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 17.376s, 881.43MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 32`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 17.28s, 886.36MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 512`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 18.658s, 820.90MiB/s] ```
2026-06-26 19:24:31 +02:00
zml/io/vfs: add parallel read to GCS and S3 (#604) # Summary This PR extends the support to chunked parallel read introduced initially for HF in the VFS, to GCS and S3. It: - Adds a shared `zml/io/vfs/parallel_read.zig` helper for chunked parallel HTTP range reads - Add configurable read pool options for S3/GCS initialization - Register the `gs://` VFS backend in `examples/llm`. # Tests ## S3 We used [MINIO](https://hub.docker.com/r/minio/minio) docker image from the `5090` GPU machine. From there, we mirror the `meta-llama/Llama-3.1-8B-Instruct` model through the S3 server. Then from the `4090` GPU machine, we run ```bash bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=s3://zml-test/Llama-3.1-8B-Instruct --prompt="hello, how are you ?" ``` It works both on 1 and 2 GPU. ## GCS In the same spirit, we put the `meta-llama/Llama-3.1-8B-Instruct` in a GCS bucket we own. Then, on the `4090`, we run ```bash bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct --prompt="hello, how are you ?" ``` It works both on 1 and 2 GPU. ### Performance As a reference, we run: ```bash gcloud storage cp gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00001-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00002-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00003-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00004-of-00004.safetensors /tmp/gcs-gcloud-llama-shards/ ``` It gives: ``` Completed files 4/4 | 15.0GiB/15.0GiB | 942.7MiB/s Average throughput: 918.8MiB/s ``` Using the VFS in various setups: * `dma_chunks = 32`, `dma_chunk_size = 128`, `parallelism = 16`, `chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 21.302s, 719.02MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 18.227s, 840.32MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 17.376s, 881.43MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 32`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 17.28s, 886.36MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 512`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 18.658s, 820.90MiB/s] ```
2026-06-26 19:24:31 +02:00
zml/io/vfs: add parallel read to GCS and S3 (#604) # Summary This PR extends the support to chunked parallel read introduced initially for HF in the VFS, to GCS and S3. It: - Adds a shared `zml/io/vfs/parallel_read.zig` helper for chunked parallel HTTP range reads - Add configurable read pool options for S3/GCS initialization - Register the `gs://` VFS backend in `examples/llm`. # Tests ## S3 We used [MINIO](https://hub.docker.com/r/minio/minio) docker image from the `5090` GPU machine. From there, we mirror the `meta-llama/Llama-3.1-8B-Instruct` model through the S3 server. Then from the `4090` GPU machine, we run ```bash bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=s3://zml-test/Llama-3.1-8B-Instruct --prompt="hello, how are you ?" ``` It works both on 1 and 2 GPU. ## GCS In the same spirit, we put the `meta-llama/Llama-3.1-8B-Instruct` in a GCS bucket we own. Then, on the `4090`, we run ```bash bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct --prompt="hello, how are you ?" ``` It works both on 1 and 2 GPU. ### Performance As a reference, we run: ```bash gcloud storage cp gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00001-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00002-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00003-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00004-of-00004.safetensors /tmp/gcs-gcloud-llama-shards/ ``` It gives: ``` Completed files 4/4 | 15.0GiB/15.0GiB | 942.7MiB/s Average throughput: 918.8MiB/s ``` Using the VFS in various setups: * `dma_chunks = 32`, `dma_chunk_size = 128`, `parallelism = 16`, `chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 21.302s, 719.02MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 18.227s, 840.32MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 17.376s, 881.43MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 32`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 17.28s, 886.36MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 512`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 18.658s, 820.90MiB/s] ```
2026-06-26 19:24:31 +02:00
zml/io/vfs: add parallel read to GCS and S3 (#604) # Summary This PR extends the support to chunked parallel read introduced initially for HF in the VFS, to GCS and S3. It: - Adds a shared `zml/io/vfs/parallel_read.zig` helper for chunked parallel HTTP range reads - Add configurable read pool options for S3/GCS initialization - Register the `gs://` VFS backend in `examples/llm`. # Tests ## S3 We used [MINIO](https://hub.docker.com/r/minio/minio) docker image from the `5090` GPU machine. From there, we mirror the `meta-llama/Llama-3.1-8B-Instruct` model through the S3 server. Then from the `4090` GPU machine, we run ```bash bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=s3://zml-test/Llama-3.1-8B-Instruct --prompt="hello, how are you ?" ``` It works both on 1 and 2 GPU. ## GCS In the same spirit, we put the `meta-llama/Llama-3.1-8B-Instruct` in a GCS bucket we own. Then, on the `4090`, we run ```bash bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct --prompt="hello, how are you ?" ``` It works both on 1 and 2 GPU. ### Performance As a reference, we run: ```bash gcloud storage cp gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00001-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00002-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00003-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00004-of-00004.safetensors /tmp/gcs-gcloud-llama-shards/ ``` It gives: ``` Completed files 4/4 | 15.0GiB/15.0GiB | 942.7MiB/s Average throughput: 918.8MiB/s ``` Using the VFS in various setups: * `dma_chunks = 32`, `dma_chunk_size = 128`, `parallelism = 16`, `chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 21.302s, 719.02MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 18.227s, 840.32MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 17.376s, 881.43MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 32`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 17.28s, 886.36MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 512`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 18.658s, 820.90MiB/s] ```
2026-06-26 19:24:31 +02:00
zml/io/vfs: add parallel read to GCS and S3 (#604) # Summary This PR extends the support to chunked parallel read introduced initially for HF in the VFS, to GCS and S3. It: - Adds a shared `zml/io/vfs/parallel_read.zig` helper for chunked parallel HTTP range reads - Add configurable read pool options for S3/GCS initialization - Register the `gs://` VFS backend in `examples/llm`. # Tests ## S3 We used [MINIO](https://hub.docker.com/r/minio/minio) docker image from the `5090` GPU machine. From there, we mirror the `meta-llama/Llama-3.1-8B-Instruct` model through the S3 server. Then from the `4090` GPU machine, we run ```bash bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=s3://zml-test/Llama-3.1-8B-Instruct --prompt="hello, how are you ?" ``` It works both on 1 and 2 GPU. ## GCS In the same spirit, we put the `meta-llama/Llama-3.1-8B-Instruct` in a GCS bucket we own. Then, on the `4090`, we run ```bash bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct --prompt="hello, how are you ?" ``` It works both on 1 and 2 GPU. ### Performance As a reference, we run: ```bash gcloud storage cp gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00001-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00002-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00003-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00004-of-00004.safetensors /tmp/gcs-gcloud-llama-shards/ ``` It gives: ``` Completed files 4/4 | 15.0GiB/15.0GiB | 942.7MiB/s Average throughput: 918.8MiB/s ``` Using the VFS in various setups: * `dma_chunks = 32`, `dma_chunk_size = 128`, `parallelism = 16`, `chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 21.302s, 719.02MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 18.227s, 840.32MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 17.376s, 881.43MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 32`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 17.28s, 886.36MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 512`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 18.658s, 820.90MiB/s] ```
2026-06-26 19:24:31 +02:00
zml/io/vfs: add parallel read to GCS and S3 (#604) # Summary This PR extends the support to chunked parallel read introduced initially for HF in the VFS, to GCS and S3. It: - Adds a shared `zml/io/vfs/parallel_read.zig` helper for chunked parallel HTTP range reads - Add configurable read pool options for S3/GCS initialization - Register the `gs://` VFS backend in `examples/llm`. # Tests ## S3 We used [MINIO](https://hub.docker.com/r/minio/minio) docker image from the `5090` GPU machine. From there, we mirror the `meta-llama/Llama-3.1-8B-Instruct` model through the S3 server. Then from the `4090` GPU machine, we run ```bash bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=s3://zml-test/Llama-3.1-8B-Instruct --prompt="hello, how are you ?" ``` It works both on 1 and 2 GPU. ## GCS In the same spirit, we put the `meta-llama/Llama-3.1-8B-Instruct` in a GCS bucket we own. Then, on the `4090`, we run ```bash bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct --prompt="hello, how are you ?" ``` It works both on 1 and 2 GPU. ### Performance As a reference, we run: ```bash gcloud storage cp gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00001-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00002-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00003-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00004-of-00004.safetensors /tmp/gcs-gcloud-llama-shards/ ``` It gives: ``` Completed files 4/4 | 15.0GiB/15.0GiB | 942.7MiB/s Average throughput: 918.8MiB/s ``` Using the VFS in various setups: * `dma_chunks = 32`, `dma_chunk_size = 128`, `parallelism = 16`, `chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 21.302s, 719.02MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 18.227s, 840.32MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 17.376s, 881.43MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 32`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 17.28s, 886.36MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 512`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 18.658s, 820.90MiB/s] ```
2026-06-26 19:24:31 +02:00
zml/io/vfs: add parallel read to GCS and S3 (#604) # Summary This PR extends the support to chunked parallel read introduced initially for HF in the VFS, to GCS and S3. It: - Adds a shared `zml/io/vfs/parallel_read.zig` helper for chunked parallel HTTP range reads - Add configurable read pool options for S3/GCS initialization - Register the `gs://` VFS backend in `examples/llm`. # Tests ## S3 We used [MINIO](https://hub.docker.com/r/minio/minio) docker image from the `5090` GPU machine. From there, we mirror the `meta-llama/Llama-3.1-8B-Instruct` model through the S3 server. Then from the `4090` GPU machine, we run ```bash bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=s3://zml-test/Llama-3.1-8B-Instruct --prompt="hello, how are you ?" ``` It works both on 1 and 2 GPU. ## GCS In the same spirit, we put the `meta-llama/Llama-3.1-8B-Instruct` in a GCS bucket we own. Then, on the `4090`, we run ```bash bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct --prompt="hello, how are you ?" ``` It works both on 1 and 2 GPU. ### Performance As a reference, we run: ```bash gcloud storage cp gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00001-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00002-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00003-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00004-of-00004.safetensors /tmp/gcs-gcloud-llama-shards/ ``` It gives: ``` Completed files 4/4 | 15.0GiB/15.0GiB | 942.7MiB/s Average throughput: 918.8MiB/s ``` Using the VFS in various setups: * `dma_chunks = 32`, `dma_chunk_size = 128`, `parallelism = 16`, `chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 21.302s, 719.02MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 18.227s, 840.32MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 17.376s, 881.43MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 32`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 17.28s, 886.36MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 512`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 18.658s, 820.90MiB/s] ```
2026-06-26 19:24:31 +02:00
zml/io/vfs: add parallel read to GCS and S3 (#604) # Summary This PR extends the support to chunked parallel read introduced initially for HF in the VFS, to GCS and S3. It: - Adds a shared `zml/io/vfs/parallel_read.zig` helper for chunked parallel HTTP range reads - Add configurable read pool options for S3/GCS initialization - Register the `gs://` VFS backend in `examples/llm`. # Tests ## S3 We used [MINIO](https://hub.docker.com/r/minio/minio) docker image from the `5090` GPU machine. From there, we mirror the `meta-llama/Llama-3.1-8B-Instruct` model through the S3 server. Then from the `4090` GPU machine, we run ```bash bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=s3://zml-test/Llama-3.1-8B-Instruct --prompt="hello, how are you ?" ``` It works both on 1 and 2 GPU. ## GCS In the same spirit, we put the `meta-llama/Llama-3.1-8B-Instruct` in a GCS bucket we own. Then, on the `4090`, we run ```bash bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct --prompt="hello, how are you ?" ``` It works both on 1 and 2 GPU. ### Performance As a reference, we run: ```bash gcloud storage cp gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00001-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00002-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00003-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00004-of-00004.safetensors /tmp/gcs-gcloud-llama-shards/ ``` It gives: ``` Completed files 4/4 | 15.0GiB/15.0GiB | 942.7MiB/s Average throughput: 918.8MiB/s ``` Using the VFS in various setups: * `dma_chunks = 32`, `dma_chunk_size = 128`, `parallelism = 16`, `chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 21.302s, 719.02MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 18.227s, 840.32MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 17.376s, 881.43MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 32`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 17.28s, 886.36MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 512`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 18.658s, 820.90MiB/s] ```
2026-06-26 19:24:31 +02:00
zml/io/vfs: add parallel read to GCS and S3 (#604) # Summary This PR extends the support to chunked parallel read introduced initially for HF in the VFS, to GCS and S3. It: - Adds a shared `zml/io/vfs/parallel_read.zig` helper for chunked parallel HTTP range reads - Add configurable read pool options for S3/GCS initialization - Register the `gs://` VFS backend in `examples/llm`. # Tests ## S3 We used [MINIO](https://hub.docker.com/r/minio/minio) docker image from the `5090` GPU machine. From there, we mirror the `meta-llama/Llama-3.1-8B-Instruct` model through the S3 server. Then from the `4090` GPU machine, we run ```bash bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=s3://zml-test/Llama-3.1-8B-Instruct --prompt="hello, how are you ?" ``` It works both on 1 and 2 GPU. ## GCS In the same spirit, we put the `meta-llama/Llama-3.1-8B-Instruct` in a GCS bucket we own. Then, on the `4090`, we run ```bash bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct --prompt="hello, how are you ?" ``` It works both on 1 and 2 GPU. ### Performance As a reference, we run: ```bash gcloud storage cp gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00001-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00002-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00003-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00004-of-00004.safetensors /tmp/gcs-gcloud-llama-shards/ ``` It gives: ``` Completed files 4/4 | 15.0GiB/15.0GiB | 942.7MiB/s Average throughput: 918.8MiB/s ``` Using the VFS in various setups: * `dma_chunks = 32`, `dma_chunk_size = 128`, `parallelism = 16`, `chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 21.302s, 719.02MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 18.227s, 840.32MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 17.376s, 881.43MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 32`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 17.28s, 886.36MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 512`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 18.658s, 820.90MiB/s] ```
2026-06-26 19:24:31 +02:00