This website requires JavaScript.
About
Status
₊˚⊹Discord
Explore
Register
Sign in
Reference in a new issue
Blame
Copy permalink
Reference in a new issue
Copy permalink
DKB
/
zml
Star
0
Fork
You've already forked zml
0
mirror of
https://github.com/zml/zml.git
synced
2026-11-16 19:11:09 +00:00
Code
Issues
23
Releases
1
Packages
Activity
raphael/decoder-buffer
zml
/
platforms
/
neuron
/
topology.zig
222 lines
7.7 KiB
Raw
Permalink
Normal View
History
platforms/neuron: bump toolchain and stabilize support (#476) - bump Neuron dependencies in `platforms/neuron/requirements.lock.txt` - `libneuronxla` `2.2.15515.0+50c26cbd` -> `2.2.16408.0+50c26cbd` - `neuronx-cc` `2.23.6484.0+3b612583` -> `2.24.8799.0+6f62ff7c` - make `neuronx-cc` invocation more robust by fixing path resolution, releasing the Python GIL during compilation, checking the compiler process exit status, and propagating NEFF wrapping/read failures - resolve PJRT devices and memories by PJRT handles instead of assuming stable numeric ids during platform initialization, including Neuron cases where memory ids are reused across devices - avoid unsupported Neuron synchronization/event paths by always requesting execution events on Neuron, deinitializing them without awaiting, and removing the explicit await from the sharding example - avoid unsupported Neuron runtime paths by: - falling back to host-backed buffer initialization for `Buffer.uninitialized` - using passthrough DMA allocation on Neuron - skipping `Tensor.print()` custom calls on Neuron - add NRT-backed Neuron physical mesh topology construction, including chip/core placement, ring/torus/point-to-point topology inference, and Neuron local-hardware-id based shard placement - let the sharding example pick the default partitioner from the current target
2026-05-12 09:58:32 +02:00
platforms/neuron: introduce NKI stack with SDPA kernel (#528) **Summary** This PR update to Neuron v2.30, introduces the Neuron NKI stack into ZML and wires it into the LLM attention path. The main goal is to make Neuron attention run through NKI kernels. It adds the compiler/runtime plumbing needed to embed NKI kernels as Neuron custom-native calls, then uses that path for SDPA-style prefill/decode attention. Paged Attention will come later. On inf2.8xlarge it runs : - 25 tok/s for Llama 3.1 8B BF16 - 63.5 tok/s for LFM2.5-1.2B-Instruct BF16 - 19 tok/s for Qwen3.5-9B BF16
2026-06-16 10:57:00 +02:00
platforms/neuron: bump toolchain and stabilize support (#476) - bump Neuron dependencies in `platforms/neuron/requirements.lock.txt` - `libneuronxla` `2.2.15515.0+50c26cbd` -> `2.2.16408.0+50c26cbd` - `neuronx-cc` `2.23.6484.0+3b612583` -> `2.24.8799.0+6f62ff7c` - make `neuronx-cc` invocation more robust by fixing path resolution, releasing the Python GIL during compilation, checking the compiler process exit status, and propagating NEFF wrapping/read failures - resolve PJRT devices and memories by PJRT handles instead of assuming stable numeric ids during platform initialization, including Neuron cases where memory ids are reused across devices - avoid unsupported Neuron synchronization/event paths by always requesting execution events on Neuron, deinitializing them without awaiting, and removing the explicit await from the sharding example - avoid unsupported Neuron runtime paths by: - falling back to host-backed buffer initialization for `Buffer.uninitialized` - using passthrough DMA allocation on Neuron - skipping `Tensor.print()` custom calls on Neuron - add NRT-backed Neuron physical mesh topology construction, including chip/core placement, ring/torus/point-to-point topology inference, and Neuron local-hardware-id based shard placement - let the sharding example pick the default partitioner from the current target
2026-05-12 09:58:32 +02:00
platforms/neuron: introduce NKI stack with SDPA kernel (#528) **Summary** This PR update to Neuron v2.30, introduces the Neuron NKI stack into ZML and wires it into the LLM attention path. The main goal is to make Neuron attention run through NKI kernels. It adds the compiler/runtime plumbing needed to embed NKI kernels as Neuron custom-native calls, then uses that path for SDPA-style prefill/decode attention. Paged Attention will come later. On inf2.8xlarge it runs : - 25 tok/s for Llama 3.1 8B BF16 - 63.5 tok/s for LFM2.5-1.2B-Instruct BF16 - 19 tok/s for Qwen3.5-9B BF16
2026-06-16 10:57:00 +02:00
platforms/neuron: bump toolchain and stabilize support (#476) - bump Neuron dependencies in `platforms/neuron/requirements.lock.txt` - `libneuronxla` `2.2.15515.0+50c26cbd` -> `2.2.16408.0+50c26cbd` - `neuronx-cc` `2.23.6484.0+3b612583` -> `2.24.8799.0+6f62ff7c` - make `neuronx-cc` invocation more robust by fixing path resolution, releasing the Python GIL during compilation, checking the compiler process exit status, and propagating NEFF wrapping/read failures - resolve PJRT devices and memories by PJRT handles instead of assuming stable numeric ids during platform initialization, including Neuron cases where memory ids are reused across devices - avoid unsupported Neuron synchronization/event paths by always requesting execution events on Neuron, deinitializing them without awaiting, and removing the explicit await from the sharding example - avoid unsupported Neuron runtime paths by: - falling back to host-backed buffer initialization for `Buffer.uninitialized` - using passthrough DMA allocation on Neuron - skipping `Tensor.print()` custom calls on Neuron - add NRT-backed Neuron physical mesh topology construction, including chip/core placement, ring/torus/point-to-point topology inference, and Neuron local-hardware-id based shard placement - let the sharding example pick the default partitioner from the current target
2026-05-12 09:58:32 +02:00
platforms/neuron: introduce NKI stack with SDPA kernel (#528) **Summary** This PR update to Neuron v2.30, introduces the Neuron NKI stack into ZML and wires it into the LLM attention path. The main goal is to make Neuron attention run through NKI kernels. It adds the compiler/runtime plumbing needed to embed NKI kernels as Neuron custom-native calls, then uses that path for SDPA-style prefill/decode attention. Paged Attention will come later. On inf2.8xlarge it runs : - 25 tok/s for Llama 3.1 8B BF16 - 63.5 tok/s for LFM2.5-1.2B-Instruct BF16 - 19 tok/s for Qwen3.5-9B BF16
2026-06-16 10:57:00 +02:00
platforms/neuron: bump toolchain and stabilize support (#476) - bump Neuron dependencies in `platforms/neuron/requirements.lock.txt` - `libneuronxla` `2.2.15515.0+50c26cbd` -> `2.2.16408.0+50c26cbd` - `neuronx-cc` `2.23.6484.0+3b612583` -> `2.24.8799.0+6f62ff7c` - make `neuronx-cc` invocation more robust by fixing path resolution, releasing the Python GIL during compilation, checking the compiler process exit status, and propagating NEFF wrapping/read failures - resolve PJRT devices and memories by PJRT handles instead of assuming stable numeric ids during platform initialization, including Neuron cases where memory ids are reused across devices - avoid unsupported Neuron synchronization/event paths by always requesting execution events on Neuron, deinitializing them without awaiting, and removing the explicit await from the sharding example - avoid unsupported Neuron runtime paths by: - falling back to host-backed buffer initialization for `Buffer.uninitialized` - using passthrough DMA allocation on Neuron - skipping `Tensor.print()` custom calls on Neuron - add NRT-backed Neuron physical mesh topology construction, including chip/core placement, ring/torus/point-to-point topology inference, and Neuron local-hardware-id based shard placement - let the sharding example pick the default partitioner from the current target
2026-05-12 09:58:32 +02:00
platforms/neuron: introduce NKI stack with SDPA kernel (#528) **Summary** This PR update to Neuron v2.30, introduces the Neuron NKI stack into ZML and wires it into the LLM attention path. The main goal is to make Neuron attention run through NKI kernels. It adds the compiler/runtime plumbing needed to embed NKI kernels as Neuron custom-native calls, then uses that path for SDPA-style prefill/decode attention. Paged Attention will come later. On inf2.8xlarge it runs : - 25 tok/s for Llama 3.1 8B BF16 - 63.5 tok/s for LFM2.5-1.2B-Instruct BF16 - 19 tok/s for Qwen3.5-9B BF16
2026-06-16 10:57:00 +02:00
platforms/neuron: bump toolchain and stabilize support (#476) - bump Neuron dependencies in `platforms/neuron/requirements.lock.txt` - `libneuronxla` `2.2.15515.0+50c26cbd` -> `2.2.16408.0+50c26cbd` - `neuronx-cc` `2.23.6484.0+3b612583` -> `2.24.8799.0+6f62ff7c` - make `neuronx-cc` invocation more robust by fixing path resolution, releasing the Python GIL during compilation, checking the compiler process exit status, and propagating NEFF wrapping/read failures - resolve PJRT devices and memories by PJRT handles instead of assuming stable numeric ids during platform initialization, including Neuron cases where memory ids are reused across devices - avoid unsupported Neuron synchronization/event paths by always requesting execution events on Neuron, deinitializing them without awaiting, and removing the explicit await from the sharding example - avoid unsupported Neuron runtime paths by: - falling back to host-backed buffer initialization for `Buffer.uninitialized` - using passthrough DMA allocation on Neuron - skipping `Tensor.print()` custom calls on Neuron - add NRT-backed Neuron physical mesh topology construction, including chip/core placement, ring/torus/point-to-point topology inference, and Neuron local-hardware-id based shard placement - let the sharding example pick the default partitioner from the current target
2026-05-12 09:58:32 +02:00
platforms/neuron: introduce NKI stack with SDPA kernel (#528) **Summary** This PR update to Neuron v2.30, introduces the Neuron NKI stack into ZML and wires it into the LLM attention path. The main goal is to make Neuron attention run through NKI kernels. It adds the compiler/runtime plumbing needed to embed NKI kernels as Neuron custom-native calls, then uses that path for SDPA-style prefill/decode attention. Paged Attention will come later. On inf2.8xlarge it runs : - 25 tok/s for Llama 3.1 8B BF16 - 63.5 tok/s for LFM2.5-1.2B-Instruct BF16 - 19 tok/s for Qwen3.5-9B BF16
2026-06-16 10:57:00 +02:00
platforms/neuron: bump toolchain and stabilize support (#476) - bump Neuron dependencies in `platforms/neuron/requirements.lock.txt` - `libneuronxla` `2.2.15515.0+50c26cbd` -> `2.2.16408.0+50c26cbd` - `neuronx-cc` `2.23.6484.0+3b612583` -> `2.24.8799.0+6f62ff7c` - make `neuronx-cc` invocation more robust by fixing path resolution, releasing the Python GIL during compilation, checking the compiler process exit status, and propagating NEFF wrapping/read failures - resolve PJRT devices and memories by PJRT handles instead of assuming stable numeric ids during platform initialization, including Neuron cases where memory ids are reused across devices - avoid unsupported Neuron synchronization/event paths by always requesting execution events on Neuron, deinitializing them without awaiting, and removing the explicit await from the sharding example - avoid unsupported Neuron runtime paths by: - falling back to host-backed buffer initialization for `Buffer.uninitialized` - using passthrough DMA allocation on Neuron - skipping `Tensor.print()` custom calls on Neuron - add NRT-backed Neuron physical mesh topology construction, including chip/core placement, ring/torus/point-to-point topology inference, and Neuron local-hardware-id based shard placement - let the sharding example pick the default partitioner from the current target
2026-05-12 09:58:32 +02:00
platforms/neuron: introduce NKI stack with SDPA kernel (#528) **Summary** This PR update to Neuron v2.30, introduces the Neuron NKI stack into ZML and wires it into the LLM attention path. The main goal is to make Neuron attention run through NKI kernels. It adds the compiler/runtime plumbing needed to embed NKI kernels as Neuron custom-native calls, then uses that path for SDPA-style prefill/decode attention. Paged Attention will come later. On inf2.8xlarge it runs : - 25 tok/s for Llama 3.1 8B BF16 - 63.5 tok/s for LFM2.5-1.2B-Instruct BF16 - 19 tok/s for Qwen3.5-9B BF16
2026-06-16 10:57:00 +02:00
platforms/neuron: bump toolchain and stabilize support (#476) - bump Neuron dependencies in `platforms/neuron/requirements.lock.txt` - `libneuronxla` `2.2.15515.0+50c26cbd` -> `2.2.16408.0+50c26cbd` - `neuronx-cc` `2.23.6484.0+3b612583` -> `2.24.8799.0+6f62ff7c` - make `neuronx-cc` invocation more robust by fixing path resolution, releasing the Python GIL during compilation, checking the compiler process exit status, and propagating NEFF wrapping/read failures - resolve PJRT devices and memories by PJRT handles instead of assuming stable numeric ids during platform initialization, including Neuron cases where memory ids are reused across devices - avoid unsupported Neuron synchronization/event paths by always requesting execution events on Neuron, deinitializing them without awaiting, and removing the explicit await from the sharding example - avoid unsupported Neuron runtime paths by: - falling back to host-backed buffer initialization for `Buffer.uninitialized` - using passthrough DMA allocation on Neuron - skipping `Tensor.print()` custom calls on Neuron - add NRT-backed Neuron physical mesh topology construction, including chip/core placement, ring/torus/point-to-point topology inference, and Neuron local-hardware-id based shard placement - let the sharding example pick the default partitioner from the current target
2026-05-12 09:58:32 +02:00
platforms/neuron: introduce NKI stack with SDPA kernel (#528) **Summary** This PR update to Neuron v2.30, introduces the Neuron NKI stack into ZML and wires it into the LLM attention path. The main goal is to make Neuron attention run through NKI kernels. It adds the compiler/runtime plumbing needed to embed NKI kernels as Neuron custom-native calls, then uses that path for SDPA-style prefill/decode attention. Paged Attention will come later. On inf2.8xlarge it runs : - 25 tok/s for Llama 3.1 8B BF16 - 63.5 tok/s for LFM2.5-1.2B-Instruct BF16 - 19 tok/s for Qwen3.5-9B BF16
2026-06-16 10:57:00 +02:00
platforms/neuron: bump toolchain and stabilize support (#476) - bump Neuron dependencies in `platforms/neuron/requirements.lock.txt` - `libneuronxla` `2.2.15515.0+50c26cbd` -> `2.2.16408.0+50c26cbd` - `neuronx-cc` `2.23.6484.0+3b612583` -> `2.24.8799.0+6f62ff7c` - make `neuronx-cc` invocation more robust by fixing path resolution, releasing the Python GIL during compilation, checking the compiler process exit status, and propagating NEFF wrapping/read failures - resolve PJRT devices and memories by PJRT handles instead of assuming stable numeric ids during platform initialization, including Neuron cases where memory ids are reused across devices - avoid unsupported Neuron synchronization/event paths by always requesting execution events on Neuron, deinitializing them without awaiting, and removing the explicit await from the sharding example - avoid unsupported Neuron runtime paths by: - falling back to host-backed buffer initialization for `Buffer.uninitialized` - using passthrough DMA allocation on Neuron - skipping `Tensor.print()` custom calls on Neuron - add NRT-backed Neuron physical mesh topology construction, including chip/core placement, ring/torus/point-to-point topology inference, and Neuron local-hardware-id based shard placement - let the sharding example pick the default partitioner from the current target
2026-05-12 09:58:32 +02:00