mirror of
https://github.com/zml/zml.git
synced 2026-11-16 19:11:09 +00:00
The LM head normalized and projected every prompt position at prefill, and the session read back the whole token buffer just to keep the last token. Every model now slices the hidden state at `active_length - 1` before the norm, so prefill computes logits for a single position and returns a single token. Decode writes the next token over the one it read, so it allocates no buffer per step. lfm2's `actual_seq_len` is renamed `active_length`, the name the other models use.
321 lines
11 KiB
321 lines
11 KiB