0
Fork 0
mirror of https://github.com/zml/zml.git synced 2026-11-16 19:11:09 +00:00

[LLMD] LFM3-1.7B outputs corrupted tokens on RX 7900 XTX (gfx1100, ROCm), BF16 and FP8 #1006

Closed
opened 2026-11-02 15:20:30 +00:00 by ghz-max · 1 comment
ghz-max commented 2026-11-02 15:20:30 +00:00 (Migrated from github.com)

Describe the bug

On an RX 7900 XTX (gfx1100), zmlai/llmd:20261029.0-rocm loads and serves LiquidAI/LFM3-1.7B and LiquidAI/LFM3-1.7B-FP8, but the generated text is corrupted: wrong or merged tokens and stray Cyrillic/CJK/Latin-extended characters appear in otherwise sensible sentences. It happens with greedy decoding (temperature: 0), with both the BF16 and the FP8 checkpoints, on single requests with no concurrency.

The same model as a GGUF (LiquidAI/LFM3-1.7B-GGUF:Q8_0) on the same GPU with llama.cpp b12040 (Vulkan/RADV) produces clean output, so the weights and the GPU look fine.

Examples, prompt What is the capital of France? Answer in one sentence., temperature: 0:

Run content start of reasoning_content
llmd BF16 Paris is7 the capital of France. The titan asks: Koreja: made up? Theuro? The user asks: "What is the capital of France? Answer iñ one sentence." This is aov. Simpleе. The capital ofat France is Paris.
llmd FP8 oot capital of France is **Paris**. We need toop LFM? The user asks: "捠 What is the capital of France? Answer in one sentence. " Actually apologies,ɡ? Theодio?
llama.cpp Q8_0 (reference) The capital of France is Paris. We need to answer: "What is the capital of France?" and the instruction: "Answer in one sentence." So just give a single sentence.

A longer prompt (In 3 short bullet points: what is a heat pump?) gives the same pattern on llmd: a heat pumpвина device that transfers heat ..., it uses a refrigerant循环 to move thermal energy.

Speed on llmd is reasonable (65–110 tok/s decode), so the kernels run; the numerics look wrong.

To Reproduce

  1. Start the server with only the dGPU render node passed (the machine also has a Raphael iGPU, gfx1036, which I excluded):
    docker run -d --name llmd-lfm3 -p 127.0.0.1:8000:8000 --shm-size=32GB \
      --device=/dev/kfd --device=/dev/dri/renderD128 \
      zmlai/llmd:20261029.0-rocm \
      --model=hf://LiquidAI/LFM3-1.7B --model-name=lfm3 --gpu-memory-fraction=0.8
    
    The log shows Devices: - AMD Radeon RX 7900 XTX (rocm:0), Compiled all models [11.884s], Loaded weights [3.22GiB ...], Listening on 0.0.0.0:8000.
  2. Send:
    curl -s http://127.0.0.1:8000/v1/chat/completions -H 'content-type: application/json' \
      -d '{"model":"lfm3","messages":[{"role":"user","content":"What is the capital of France? Answer in one sentence."}],"temperature":0,"max_tokens":300}'
    
  3. See corrupted tokens in content and reasoning_content.
  4. Same result with --model=hf://LiquidAI/LFM3-1.7B-FP8 (see the FP8 note below for the flags needed to get it to answer at all).

Expected behavior

Coherent output, matching what llama.cpp produces for the same model on the same GPU.

Configuration:

  • Model: LiquidAI/LFM3-1.7B (BF16) and LiquidAI/LFM3-1.7B-FP8
  • OS: Arch Linux, kernel 7.2.11-arch1-1, in-tree amdgpu, linux-firmware-amdgpu 20260916, Docker 29.9.2
  • Platform: rocm
  • Accelerator: AMD Radeon RX 7900 XTX, 24 GB (Navi 31, gfx1100)
  • Driver version: in-tree amdgpu of kernel 7.2.11 (host ROCm userspace 7.2.4, not mounted into the container)
  • Version: zmlai/llmd:20261029.0-rocm, zmlai/llmd@sha256:a1966cc48f4f814e00edea2ffec36c87db4016d3788cdac882a5d7cdd8be84a2
  • Command line: see step 1

Additional context

FP8, separate problem: with the default --cache-memory-fraction (0.95), the FP8 server crashes on the first request, at both --gpu-memory-fraction=0.6 and 0.85:

warning(pjrt): [PJRT_Event_Await] ResourceExhausted: Not enough PA budget: pa_allocated=21300772864B, allocation_size=2516582400B, pa_budget=21890073190B
thread 32 panic: Device I/O error

The container exits with code 139. The cache seems to fill the budget, leaving no room for a 2.4 GB allocation at the first request. With --gpu-memory-fraction=0.6 --cache-memory-fraction=0.6 it serves, and the output is corrupted as above. FP8 compilation also takes 3m28s on this GPU, against 12s for BF16; gfx1100 has no FP8 hardware, so I assume an emulated path.

llama.cpp runs the same LFM3 model correctly on this GPU, so the GPU, driver and weights look fine. What I have not tried is a different model architecture on llmd on this GPU, so I can't tell whether the bug is in llmd's LFM3 layers (gated short convolutions, sliding-window attention) or in llmd on gfx1100 in general. I can run another model or a debug build if that helps.

**Describe the bug** On an RX 7900 XTX (gfx1100), `zmlai/llmd:20261029.0-rocm` loads and serves `LiquidAI/LFM3-1.7B` and `LiquidAI/LFM3-1.7B-FP8`, but the generated text is corrupted: wrong or merged tokens and stray Cyrillic/CJK/Latin-extended characters appear in otherwise sensible sentences. It happens with greedy decoding (`temperature: 0`), with both the BF16 and the FP8 checkpoints, on single requests with no concurrency. The same model as a GGUF (`LiquidAI/LFM3-1.7B-GGUF:Q8_0`) on the same GPU with llama.cpp b12040 (Vulkan/RADV) produces clean output, so the weights and the GPU look fine. Examples, prompt `What is the capital of France? Answer in one sentence.`, `temperature: 0`: | Run | `content` | start of `reasoning_content` | | --- | --- | --- | | llmd BF16 | `Paris is7 the capital of France.` | `The titan asks: Koreja: made up? Theuro? The user asks: "What is the capital of France? Answer iñ one sentence." This is aov. Simpleе. The capital ofat France is Paris.` | | llmd FP8 | `oot capital of France is **Paris**.` | `We need toop LFM? The user asks: "捠 What is the capital of France? Answer in one sentence. " Actually apologies,ɡ? Theодio?` | | llama.cpp Q8_0 (reference) | `The capital of France is Paris.` | `We need to answer: "What is the capital of France?" and the instruction: "Answer in one sentence." So just give a single sentence.` | A longer prompt (`In 3 short bullet points: what is a heat pump?`) gives the same pattern on llmd: `a heat pumpвина device that transfers heat ..., it uses a refrigerant循环 to move thermal energy`. Speed on llmd is reasonable (65–110 tok/s decode), so the kernels run; the numerics look wrong. **To Reproduce** 1. Start the server with only the dGPU render node passed (the machine also has a Raphael iGPU, gfx1036, which I excluded): ``` docker run -d --name llmd-lfm3 -p 127.0.0.1:8000:8000 --shm-size=32GB \ --device=/dev/kfd --device=/dev/dri/renderD128 \ zmlai/llmd:20261029.0-rocm \ --model=hf://LiquidAI/LFM3-1.7B --model-name=lfm3 --gpu-memory-fraction=0.8 ``` The log shows `Devices: - AMD Radeon RX 7900 XTX (rocm:0)`, `Compiled all models [11.884s]`, `Loaded weights [3.22GiB ...]`, `Listening on 0.0.0.0:8000`. 2. Send: ``` curl -s http://127.0.0.1:8000/v1/chat/completions -H 'content-type: application/json' \ -d '{"model":"lfm3","messages":[{"role":"user","content":"What is the capital of France? Answer in one sentence."}],"temperature":0,"max_tokens":300}' ``` 3. See corrupted tokens in `content` and `reasoning_content`. 4. Same result with `--model=hf://LiquidAI/LFM3-1.7B-FP8` (see the FP8 note below for the flags needed to get it to answer at all). **Expected behavior** Coherent output, matching what llama.cpp produces for the same model on the same GPU. **Configuration:** - Model: `LiquidAI/LFM3-1.7B` (BF16) and `LiquidAI/LFM3-1.7B-FP8` - OS: Arch Linux, kernel `7.2.11-arch1-1`, in-tree `amdgpu`, `linux-firmware-amdgpu 20260916`, Docker 29.9.2 - Platform: `rocm` - Accelerator: AMD Radeon RX 7900 XTX, 24 GB (Navi 31, gfx1100) - Driver version: in-tree `amdgpu` of kernel 7.2.11 (host ROCm userspace 7.2.4, not mounted into the container) - Version: `zmlai/llmd:20261029.0-rocm`, `zmlai/llmd@sha256:a1966cc48f4f814e00edea2ffec36c87db4016d3788cdac882a5d7cdd8be84a2` - Command line: see step 1 **Additional context** FP8, separate problem: with the default `--cache-memory-fraction` (0.95), the FP8 server crashes on the first request, at both `--gpu-memory-fraction=0.6` and `0.85`: ``` warning(pjrt): [PJRT_Event_Await] ResourceExhausted: Not enough PA budget: pa_allocated=21300772864B, allocation_size=2516582400B, pa_budget=21890073190B thread 32 panic: Device I/O error ``` The container exits with code 139. The cache seems to fill the budget, leaving no room for a 2.4 GB allocation at the first request. With `--gpu-memory-fraction=0.6 --cache-memory-fraction=0.6` it serves, and the output is corrupted as above. FP8 compilation also takes 3m28s on this GPU, against 12s for BF16; gfx1100 has no FP8 hardware, so I assume an emulated path. llama.cpp runs the same LFM3 model correctly on this GPU, so the GPU, driver and weights look fine. What I have not tried is a different model architecture on llmd on this GPU, so I can't tell whether the bug is in llmd's LFM3 layers (gated short convolutions, sliding-window attention) or in llmd on gfx1100 in general. I can run another model or a debug build if that helps.
dmehala commented 2026-11-02 16:58:54 +00:00 (Migrated from github.com)

Hi @ghz-max, thank you for flagging this issue and the detailed reproduction. I can confirm that I’m able to reproduce it, and I’m looking into it.

Hi @ghz-max, thank you for flagging this issue and the detailed reproduction. I can confirm that I’m able to reproduce it, and I’m looking into it.
Sign in to join this conversation.
No milestone
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
DKB/zml#1006
No description provided.