0
Fork 0
mirror of https://github.com/zml/zml.git synced 2026-11-16 19:11:09 +00:00
zml/examples/llm/main.zig

205 lines
7 KiB

zml: re-structure attention kernels (#926) This PR introduces a restructuration of the kernels in ZML. This is done to simplify and speedup bringup on new platforms. From now on each "kernel" will have its own directory under `zml/kernels`. "Kernel" is in quote because we need to think about it being more of an "operation in a graph" than an actual kernel. For example, `moe` will eventually be part of this `kernels` directory despite being an operation potentially launching multiple actual kernels on some hardware. Also it's probable that in the future, this `kernels` directory will be put outside of zml or, at least, properly separated so that we don't have circular imports in the codebase. Currently, we have `@import("../../../zml.zig")` in the kernel code and this is a recipe for disaster. On top of the reorganization, it also adds a level of indirection for attention. Where previously `attention` was a generic term, we now have `attention` and `flash_attention` because `flash_attention` is an implementation of `attention`. Here is a detailed summary of the changes introduced by this PR: * Kernels reorganization. `flash_attention` and `paged_attention` are now separated. Each folder has a subfolder for each backend we support. `flash_attention` is also now a `BackendType` of `zml.attention` with its own implementations. * Removed `cuda_fa2` and `cuda_fa3` backend because they were obsolete. They have been replaced by a `cute` backend that has been generated automatically following the operation documentation, showing that this approach of scoping kernels per DSL actually scales. * Removed `attnd` support for now
2026-10-12 20:31:17 +02:00
zml: re-structure attention kernels (#926) This PR introduces a restructuration of the kernels in ZML. This is done to simplify and speedup bringup on new platforms. From now on each "kernel" will have its own directory under `zml/kernels`. "Kernel" is in quote because we need to think about it being more of an "operation in a graph" than an actual kernel. For example, `moe` will eventually be part of this `kernels` directory despite being an operation potentially launching multiple actual kernels on some hardware. Also it's probable that in the future, this `kernels` directory will be put outside of zml or, at least, properly separated so that we don't have circular imports in the codebase. Currently, we have `@import("../../../zml.zig")` in the kernel code and this is a recipe for disaster. On top of the reorganization, it also adds a level of indirection for attention. Where previously `attention` was a generic term, we now have `attention` and `flash_attention` because `flash_attention` is an implementation of `attention`. Here is a detailed summary of the changes introduced by this PR: * Kernels reorganization. `flash_attention` and `paged_attention` are now separated. Each folder has a subfolder for each backend we support. `flash_attention` is also now a `BackendType` of `zml.attention` with its own implementations. * Removed `cuda_fa2` and `cuda_fa3` backend because they were obsolete. They have been replaced by a `cute` backend that has been generated automatically following the operation documentation, showing that this approach of scoping kernels per DSL actually scales. * Removed `attnd` support for now
2026-10-12 20:31:17 +02:00
zml/io/vfs: add parallel read to GCS and S3 (#604) # Summary This PR extends the support to chunked parallel read introduced initially for HF in the VFS, to GCS and S3. It: - Adds a shared `zml/io/vfs/parallel_read.zig` helper for chunked parallel HTTP range reads - Add configurable read pool options for S3/GCS initialization - Register the `gs://` VFS backend in `examples/llm`. # Tests ## S3 We used [MINIO](https://hub.docker.com/r/minio/minio) docker image from the `5090` GPU machine. From there, we mirror the `meta-llama/Llama-3.1-8B-Instruct` model through the S3 server. Then from the `4090` GPU machine, we run ```bash bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=s3://zml-test/Llama-3.1-8B-Instruct --prompt="hello, how are you ?" ``` It works both on 1 and 2 GPU. ## GCS In the same spirit, we put the `meta-llama/Llama-3.1-8B-Instruct` in a GCS bucket we own. Then, on the `4090`, we run ```bash bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct --prompt="hello, how are you ?" ``` It works both on 1 and 2 GPU. ### Performance As a reference, we run: ```bash gcloud storage cp gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00001-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00002-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00003-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00004-of-00004.safetensors /tmp/gcs-gcloud-llama-shards/ ``` It gives: ``` Completed files 4/4 | 15.0GiB/15.0GiB | 942.7MiB/s Average throughput: 918.8MiB/s ``` Using the VFS in various setups: * `dma_chunks = 32`, `dma_chunk_size = 128`, `parallelism = 16`, `chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 21.302s, 719.02MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 18.227s, 840.32MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 17.376s, 881.43MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 32`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 17.28s, 886.36MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 512`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 18.658s, 820.90MiB/s] ```
2026-06-26 19:24:31 +02:00
zml/io/vfs: add parallel read to GCS and S3 (#604) # Summary This PR extends the support to chunked parallel read introduced initially for HF in the VFS, to GCS and S3. It: - Adds a shared `zml/io/vfs/parallel_read.zig` helper for chunked parallel HTTP range reads - Add configurable read pool options for S3/GCS initialization - Register the `gs://` VFS backend in `examples/llm`. # Tests ## S3 We used [MINIO](https://hub.docker.com/r/minio/minio) docker image from the `5090` GPU machine. From there, we mirror the `meta-llama/Llama-3.1-8B-Instruct` model through the S3 server. Then from the `4090` GPU machine, we run ```bash bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=s3://zml-test/Llama-3.1-8B-Instruct --prompt="hello, how are you ?" ``` It works both on 1 and 2 GPU. ## GCS In the same spirit, we put the `meta-llama/Llama-3.1-8B-Instruct` in a GCS bucket we own. Then, on the `4090`, we run ```bash bazel run --@zml//platforms:cuda=true --@zml//platforms:cpu=false --config=release --config=native //examples/llm -- --model=gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct --prompt="hello, how are you ?" ``` It works both on 1 and 2 GPU. ### Performance As a reference, we run: ```bash gcloud storage cp gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00001-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00002-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00003-of-00004.safetensors gs://zml-mirror2/models/meta-llama/Llama-3.1-8B-Instruct/model-00004-of-00004.safetensors /tmp/gcs-gcloud-llama-shards/ ``` It gives: ``` Completed files 4/4 | 15.0GiB/15.0GiB | 942.7MiB/s Average throughput: 918.8MiB/s ``` Using the VFS in various setups: * `dma_chunks = 32`, `dma_chunk_size = 128`, `parallelism = 16`, `chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 21.302s, 719.02MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 10 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 18.227s, 840.32MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 17.376s, 881.43MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 256`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 32`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 17.28s, 886.36MiB/s] ``` * `dma_chunks = 32`, `dma_chunk_size = 512`, `parallelism = 16`, `chunk_size = 16 MiB`, `num_workers = 16`, `queue_capacity = 64` ```bash info (llama): Loaded weights [14.96GiB, 18.658s, 820.90MiB/s] ```
2026-06-26 19:24:31 +02:00
2026-03-24 18:16:25 +01:00
zml: re-structure attention kernels (#926) This PR introduces a restructuration of the kernels in ZML. This is done to simplify and speedup bringup on new platforms. From now on each "kernel" will have its own directory under `zml/kernels`. "Kernel" is in quote because we need to think about it being more of an "operation in a graph" than an actual kernel. For example, `moe` will eventually be part of this `kernels` directory despite being an operation potentially launching multiple actual kernels on some hardware. Also it's probable that in the future, this `kernels` directory will be put outside of zml or, at least, properly separated so that we don't have circular imports in the codebase. Currently, we have `@import("../../../zml.zig")` in the kernel code and this is a recipe for disaster. On top of the reorganization, it also adds a level of indirection for attention. Where previously `attention` was a generic term, we now have `attention` and `flash_attention` because `flash_attention` is an implementation of `attention`. Here is a detailed summary of the changes introduced by this PR: * Kernels reorganization. `flash_attention` and `paged_attention` are now separated. Each folder has a subfolder for each backend we support. `flash_attention` is also now a `BackendType` of `zml.attention` with its own implementations. * Removed `cuda_fa2` and `cuda_fa3` backend because they were obsolete. They have been replaced by a `cute` backend that has been generated automatically following the operation documentation, showing that this approach of scoping kernels per DSL actually scales. * Removed `attnd` support for now
2026-10-12 20:31:17 +02:00
2026-03-24 18:16:25 +01:00
2026-03-24 18:16:25 +01:00
2026-03-24 18:16:25 +01:00
2026-03-24 18:16:25 +01:00
2026-03-24 18:16:25 +01:00
2026-03-24 18:16:25 +01:00
2026-03-24 18:16:25 +01:00
2026-03-24 18:16:25 +01:00
2026-03-24 18:16:25 +01:00
zml/tokenizer: add IREE tokenizer (#416) Adding the [IREE tokenizer](https://github.com/iree-org/iree/blob/main/runtime/src/iree/tokenizer/README.md).Being made from the ground up with streaming (de-)tokenization in mind, I've extended the APIs to expose a `std.Io.Writer` for the tokenization. I've quickly benchmarked IREE's tokenizer vs HuggingFace's one with a request I retrieved from a coding agent (opencode) in the middle of a session. The request body was 360KB encoded in 95K tokens. I also did another test with a similarly sized request with the same results as shown hereafter. I've tested two cases: - oneshot: the template is rendered all at once and only then we encode the whole string. - stream: the template rendering and encoding are pipelined to avoid the intermediate allocation. There is a slight difference in the generated tokens between HuggingFace and IREE. Some tokens like `introdu` were encoded as `i`, `n`, `t`, etc. Decoding them returns the same text. It's very close in terms of total token count: - hf: 95779 - iree/stream: 95705 - iree/oneshot: 95587 | Model | IREE avg (oneshot / stream) ms | HuggingFace avg (ms) | Uplift vs HF (oneshot / stream) | |---|---:|---:|---| | liquidai/LFM2.5-1.2B-Instruct | 8.5 / 7.3 | 40.5 | 4.8x / 5.5x | | Qwen/Qwen3.5-9B | 6.0 / 8.2 | 59.4 | 9.9x / 7.2x | | mistralai/Ministral-3-3B-Reasoning-2512 | 7.4 / 6.9 | 59.2 | 8x / 8.6x | It's unclear why the Qwen's chat template is somehow slower in the stream case, maybe because there are more writes. Adding non-empty buffer in the interface helps a bit, but tends to slow down mistral case. So well --------- Co-authored-by: Corentin Kerisit <corentin.kerisit@gmail.com>
2026-04-07 16:42:55 +02:00