This website requires JavaScript.
About
Status
₊˚⊹Discord
Explore
Register
Sign in
Reference in a new issue
Blame
Copy permalink
Reference in a new issue
Copy permalink
DKB
/
zml
Star
0
Fork
You've already forked zml
0
mirror of
https://github.com/zml/zml.git
synced
2026-11-16 19:11:09 +00:00
Code
Issues
23
Releases
1
Packages
Activity
raphael/decoder-buffer
zml
/
examples
/
llm
/
models
/
common.zig
88 lines
3.1 KiB
Raw
Permalink
Normal View
History
examples/llm: Common model example (#425)
2026-03-24 16:55:29 +01:00
zml/attention: implement batching in cu_fa2 backend (#641)
2026-07-17 14:10:41 +02:00
examples/llm: Common model example (#425)
2026-03-24 16:55:29 +01:00
examples/llm: better single prompt UX + --topk (#438) - Same UX as the chat - Add `--topk` argument
2026-03-26 15:58:03 +01:00
examples/llm: split Llama and Qwen 3.5 with LFM cleanup (#555) This PR splits Llama in `examples/llm` into separate `embed_tokens`, `layer`, and `lm_head` kernels for both prefill and decode, and brings LFM2 onto the same composed execution path. And done same work for Qwen 3.5 . It removes the old LFM2 `--single` mode, reworks Llama KV-cache access to use explicit layer indices, and reuses more decode-time state through a persistent runner and prebuilt token index buffers. It also tightens sharding/partitioning in LFM2 for multi-device runs, adds clearer per-phase compile logging/program names, removes the unused `--single` CLI flag.
2026-05-28 10:10:29 +02:00
zml/Tensor: explicit sharding in Tensor code (#868) In Buffer code we already have explicit sharding, but not for tensor. I'm adding a `zml.Sharding` to `.withPartitioning` to make it explicit which mesh we are working with. To simplify the migration, `.withPartitioning` in Tensor code can use `.model` syntax to lookup a sharding by name. This replace the previous logic of `selectSharding(shape)` which would look up a sharding that matches the axes of the given partitioning which could yield surprising behavior. This also modifies how sharding are stored in zml.Shape. Before it was storing a list of axis names + a tag (2 * 8 * 64bits), now I'm storing a zml.Sharding (64 bits) and a compact PartitionArray (32 bits) that references the logical axis of the sharding by offset instead of by name. There is an impact on the `TensorRegistry` because it also needs to be aware of the possible shardings, instead of delegating the matching partition spec -> sharding to loading. This means error will be detected sooner. `store.createTensor` accepts `@enumLiteral()` to lookup the relevant sharding. This also simplifies `zml.Buffer` because the full sharding is now stored in the shape, so it doesn't need to be passed independently. It also means that it will propagate from the tensor who are graph inputs, to the buffers that are eg inputs. `Placement` logic also got a bit simpler.
2026-10-06 15:37:22 +02:00
examples/llm: Common model example (#425)
2026-03-24 16:55:29 +01:00
zml/Tensor: explicit sharding in Tensor code (#868) In Buffer code we already have explicit sharding, but not for tensor. I'm adding a `zml.Sharding` to `.withPartitioning` to make it explicit which mesh we are working with. To simplify the migration, `.withPartitioning` in Tensor code can use `.model` syntax to lookup a sharding by name. This replace the previous logic of `selectSharding(shape)` which would look up a sharding that matches the axes of the given partitioning which could yield surprising behavior. This also modifies how sharding are stored in zml.Shape. Before it was storing a list of axis names + a tag (2 * 8 * 64bits), now I'm storing a zml.Sharding (64 bits) and a compact PartitionArray (32 bits) that references the logical axis of the sharding by offset instead of by name. There is an impact on the `TensorRegistry` because it also needs to be aware of the possible shardings, instead of delegating the matching partition spec -> sharding to loading. This means error will be detected sooner. `store.createTensor` accepts `@enumLiteral()` to lookup the relevant sharding. This also simplifies `zml.Buffer` because the full sharding is now stored in the shape, so it doesn't need to be passed independently. It also means that it will propagate from the tensor who are graph inputs, to the buffers that are eg inputs. `Placement` logic also got a bit simpler.
2026-10-06 15:37:22 +02:00
zml/moe: Mosaic TPU MoE backend. (#571) This PR add the TPU backend for MoE. It uses the GMM Expert parallelism kernel to perform the grouped gemms on gate_up and down weights (these weights must have a precise shape to run optimally). This backend is currently optimized for small batches. A specialized mosaic tpu kernel will be introduced to handle bigger ones. This implementation come with few fixes concerning the TPU sharding and the Qwen 3.5 moe model which uses the backend : - TPU sharding : PJRT reports TPU device coordinates in a max rank form. So it breaks folding. To avoid this we compute the Rank by not counting where coords =0. - Linear attention was reading padding token in the Qwen 3.5 model. So we introduce masking of them. n.b. : The kv head repetition to satisfy the sharding will be handled a more generic way in a next PR --------- Co-authored-by: Corentin Godeau <corentin@zml.ai>
2026-06-16 10:06:52 +02:00
examples/llm: fix single-chip TPU sharding crash (#627) A 1-chip TPU mesh collapses to rank 1 (only `link_x`), so the unconditional `link_y` fold tripped InvalidPhysicalAxis. Only fold `link_y` when the physical mesh actually has it.
2026-07-03 14:32:41 +02:00
zml/moe: Mosaic TPU MoE backend. (#571) This PR add the TPU backend for MoE. It uses the GMM Expert parallelism kernel to perform the grouped gemms on gate_up and down weights (these weights must have a precise shape to run optimally). This backend is currently optimized for small batches. A specialized mosaic tpu kernel will be introduced to handle bigger ones. This implementation come with few fixes concerning the TPU sharding and the Qwen 3.5 moe model which uses the backend : - TPU sharding : PJRT reports TPU device coordinates in a max rank form. So it breaks folding. To avoid this we compute the Rank by not counting where coords =0. - Linear attention was reading padding token in the Qwen 3.5 model. So we introduce masking of them. n.b. : The kv head repetition to satisfy the sharding will be handled a more generic way in a next PR --------- Co-authored-by: Corentin Godeau <corentin@zml.ai>
2026-06-16 10:06:52 +02:00
examples/llm: fix single-chip TPU sharding crash (#627) A 1-chip TPU mesh collapses to rank 1 (only `link_x`), so the unconditional `link_y` fold tripped InvalidPhysicalAxis. Only fold `link_y` when the physical mesh actually has it.
2026-07-03 14:32:41 +02:00
zml/moe: Mosaic TPU MoE backend. (#571) This PR add the TPU backend for MoE. It uses the GMM Expert parallelism kernel to perform the grouped gemms on gate_up and down weights (these weights must have a precise shape to run optimally). This backend is currently optimized for small batches. A specialized mosaic tpu kernel will be introduced to handle bigger ones. This implementation come with few fixes concerning the TPU sharding and the Qwen 3.5 moe model which uses the backend : - TPU sharding : PJRT reports TPU device coordinates in a max rank form. So it breaks folding. To avoid this we compute the Rank by not counting where coords =0. - Linear attention was reading padding token in the Qwen 3.5 model. So we introduce masking of them. n.b. : The kv head repetition to satisfy the sharding will be handled a more generic way in a next PR --------- Co-authored-by: Corentin Godeau <corentin@zml.ai>
2026-06-16 10:06:52 +02:00
zml/Tensor: explicit sharding in Tensor code (#868) In Buffer code we already have explicit sharding, but not for tensor. I'm adding a `zml.Sharding` to `.withPartitioning` to make it explicit which mesh we are working with. To simplify the migration, `.withPartitioning` in Tensor code can use `.model` syntax to lookup a sharding by name. This replace the previous logic of `selectSharding(shape)` which would look up a sharding that matches the axes of the given partitioning which could yield surprising behavior. This also modifies how sharding are stored in zml.Shape. Before it was storing a list of axis names + a tag (2 * 8 * 64bits), now I'm storing a zml.Sharding (64 bits) and a compact PartitionArray (32 bits) that references the logical axis of the sharding by offset instead of by name. There is an impact on the `TensorRegistry` because it also needs to be aware of the possible shardings, instead of delegating the matching partition spec -> sharding to loading. This means error will be detected sooner. `store.createTensor` accepts `@enumLiteral()` to lookup the relevant sharding. This also simplifies `zml.Buffer` because the full sharding is now stored in the shape, so it doesn't need to be passed independently. It also means that it will propagate from the tensor who are graph inputs, to the buffers that are eg inputs. `Placement` logic also got a bit simpler.
2026-10-06 15:37:22 +02:00
zml/moe: Mosaic TPU MoE backend. (#571) This PR add the TPU backend for MoE. It uses the GMM Expert parallelism kernel to perform the grouped gemms on gate_up and down weights (these weights must have a precise shape to run optimally). This backend is currently optimized for small batches. A specialized mosaic tpu kernel will be introduced to handle bigger ones. This implementation come with few fixes concerning the TPU sharding and the Qwen 3.5 moe model which uses the backend : - TPU sharding : PJRT reports TPU device coordinates in a max rank form. So it breaks folding. To avoid this we compute the Rank by not counting where coords =0. - Linear attention was reading padding token in the Qwen 3.5 model. So we introduce masking of them. n.b. : The kv head repetition to satisfy the sharding will be handled a more generic way in a next PR --------- Co-authored-by: Corentin Godeau <corentin@zml.ai>
2026-06-16 10:06:52 +02:00
zml/platform: Add furiosa platform (#871) Co-authored-by: Raphael Nahoum <raphael@zml.ai>
2026-10-09 12:21:16 +02:00
zml/Tensor: explicit sharding in Tensor code (#868) In Buffer code we already have explicit sharding, but not for tensor. I'm adding a `zml.Sharding` to `.withPartitioning` to make it explicit which mesh we are working with. To simplify the migration, `.withPartitioning` in Tensor code can use `.model` syntax to lookup a sharding by name. This replace the previous logic of `selectSharding(shape)` which would look up a sharding that matches the axes of the given partitioning which could yield surprising behavior. This also modifies how sharding are stored in zml.Shape. Before it was storing a list of axis names + a tag (2 * 8 * 64bits), now I'm storing a zml.Sharding (64 bits) and a compact PartitionArray (32 bits) that references the logical axis of the sharding by offset instead of by name. There is an impact on the `TensorRegistry` because it also needs to be aware of the possible shardings, instead of delegating the matching partition spec -> sharding to loading. This means error will be detected sooner. `store.createTensor` accepts `@enumLiteral()` to lookup the relevant sharding. This also simplifies `zml.Buffer` because the full sharding is now stored in the shape, so it doesn't need to be passed independently. It also means that it will propagate from the tensor who are graph inputs, to the buffers that are eg inputs. `Placement` logic also got a bit simpler.
2026-10-06 15:37:22 +02:00
zml/moe: Mosaic TPU MoE backend. (#571) This PR add the TPU backend for MoE. It uses the GMM Expert parallelism kernel to perform the grouped gemms on gate_up and down weights (these weights must have a precise shape to run optimally). This backend is currently optimized for small batches. A specialized mosaic tpu kernel will be introduced to handle bigger ones. This implementation come with few fixes concerning the TPU sharding and the Qwen 3.5 moe model which uses the backend : - TPU sharding : PJRT reports TPU device coordinates in a max rank form. So it breaks folding. To avoid this we compute the Rank by not counting where coords =0. - Linear attention was reading padding token in the Qwen 3.5 model. So we introduce masking of them. n.b. : The kv head repetition to satisfy the sharding will be handled a more generic way in a next PR --------- Co-authored-by: Corentin Godeau <corentin@zml.ai>
2026-06-16 10:06:52 +02:00
examples/llm: Simplifications (#430)
2026-03-24 18:16:25 +01:00
zml/Tensor: explicit sharding in Tensor code (#868) In Buffer code we already have explicit sharding, but not for tensor. I'm adding a `zml.Sharding` to `.withPartitioning` to make it explicit which mesh we are working with. To simplify the migration, `.withPartitioning` in Tensor code can use `.model` syntax to lookup a sharding by name. This replace the previous logic of `selectSharding(shape)` which would look up a sharding that matches the axes of the given partitioning which could yield surprising behavior. This also modifies how sharding are stored in zml.Shape. Before it was storing a list of axis names + a tag (2 * 8 * 64bits), now I'm storing a zml.Sharding (64 bits) and a compact PartitionArray (32 bits) that references the logical axis of the sharding by offset instead of by name. There is an impact on the `TensorRegistry` because it also needs to be aware of the possible shardings, instead of delegating the matching partition spec -> sharding to loading. This means error will be detected sooner. `store.createTensor` accepts `@enumLiteral()` to lookup the relevant sharding. This also simplifies `zml.Buffer` because the full sharding is now stored in the shape, so it doesn't need to be passed independently. It also means that it will propagate from the tensor who are graph inputs, to the buffers that are eg inputs. `Placement` logic also got a bit simpler.
2026-10-06 15:37:22 +02:00
zml/moe: Add sharding for triton MoE (#552) This PR introduces the sharding as expert parallelism for MoE : - Introduce a .experts logical sharding (required to express an other semantic than TP (.model)) - Wrap the whole MoE impl into a manualComputation - Implement a manual stableHlo all reduce since shardy can not infer the correct communication after a manualComputation (blackbox) - Partition qwen 3.5 moe example with TP for the rest of the model (non MoE) --------- Co-authored-by: loupicaaa <louis@zml.ai>
2026-05-27 10:13:38 +02:00
examples/llm: Common model example (#425)
2026-03-24 16:55:29 +01:00