0
Fork 0
mirror of https://github.com/zml/zml.git synced 2026-11-16 19:11:09 +00:00
Commit graph

2 commits

Author SHA1 Message Date
Hugo Mano
1fead1c014
platforms/neuron: introduce NKI stack with SDPA kernel (#528)
**Summary**

This PR update to Neuron v2.30, introduces the Neuron NKI stack into ZML and wires it into the
LLM attention path.

The main goal is to make Neuron attention run through NKI kernels.
It adds the compiler/runtime plumbing needed to embed NKI kernels as
Neuron custom-native calls, then uses that path for SDPA-style
prefill/decode attention. Paged Attention will come later.

On inf2.8xlarge it runs :
- 25 tok/s for Llama 3.1 8B BF16
- 63.5 tok/s for LFM2.5-1.2B-Instruct BF16
- 19 tok/s for Qwen3.5-9B BF16
2026-06-16 10:57:00 +02:00
Steeve Morin
9e2c09066b runtimes: rename to platforms 2026-01-19 17:41:38 +01:00
Renamed from runtimes/neuron/packages.lock.json (Browse further)