mirror of
https://github.com/zml/zml.git
synced 2026-11-16 19:11:09 +00:00
**Summary** This PR update to Neuron v2.30, introduces the Neuron NKI stack into ZML and wires it into the LLM attention path. The main goal is to make Neuron attention run through NKI kernels. It adds the compiler/runtime plumbing needed to embed NKI kernels as Neuron custom-native calls, then uses that path for SDPA-style prefill/decode attention. Paged Attention will come later. On inf2.8xlarge it runs : - 25 tok/s for Llama 3.1 8B BF16 - 63.5 tok/s for LFM2.5-1.2B-Instruct BF16 - 19 tok/s for Qwen3.5-9B BF16
179 lines
5.1 KiB
Executable file
179 lines
5.1 KiB
Executable file