0
Fork 0
mirror of https://github.com/zml/zml.git synced 2026-11-16 19:11:09 +00:00
zml/examples/llm/models/lfm2/inference.zig
Raphael Nahoum 24f8c2b986
examples/llm: sample only the last prompt token (#965)
The LM head normalized and projected every prompt position at prefill,
and the session read back the whole token buffer just to keep the last
token.

Every model now slices the hidden state at `active_length - 1` before
the norm, so prefill computes logits for a single position and returns a
single token. Decode writes the next token over the one it read, so it
allocates no buffer per step.

lfm2's `actual_seq_len` is renamed `active_length`, the name the other
models use.
2026-10-21 11:47:42 +02:00

321 lines
11 KiB