0
Fork 0
mirror of https://github.com/zml/zml.git synced 2026-11-16 19:11:09 +00:00

examples/llm: Qwen3.5-0.8B crashes due to duplicate KV-head partitioning #1090

Open
opened 2026-11-16 17:24:48 +00:00 by TheBCann · 0 comments
TheBCann commented 2026-11-16 17:24:48 +00:00 (Migrated from github.com)

[Bug] Qwen3.5-0.8B crashes due to duplicate KV-head partitioning

Summary

Qwen3.5-0.8B fails during full-attention graph compilation because the incoming key/value tensors contain 8 KV heads, while the KV cache expects 4.

The issue appears to originate in examples/llm/models/qwen3_5/model.zig, where partitionProjectedKv() is called twice on both the K and V tensors.

I've reproduced the same failure on macOS ARM64 and Linux x86-64, and independently verified the tensor dimensions using LLDB and GDB.

Steps to reproduce

On an unmodified ZML checkout:

bazel run //examples/llm -- \
  --model=hf://Qwen/Qwen3.5-0.8B \
  --prompt="What is the capital of France?"

The Bazel build succeeds, but model initialization fails during prefill full-attention graph compilation.

Actual behavior

dynamicUpdateSlice expects 'update_' dimensions to be less
than or equal to their corresponding dimension in input tensor,
got 8 and 4 for axis .h

The Linux stack trace includes:

zml/ops.zig:1967
  scatter()

zml/tensor.zig:3419
scatterSlices()

examples/llm/models/qwen3_5/model.zig:1085
KvCache.SelfAttnCache.update()

examples/llm/models/qwen3_5/model.zig:653
SelfAttn.forward()

Expected behavior

The incoming K/V tensors should have a head dimension compatible with the KV cache, allowing compilation and inference to complete.

Investigation and suspected root cause

In SelfAttn.forward(), partitionProjectedKv() is called before normalization and then again after RoPE.

When kv_head_sharding is .sharded with heads.factor = 2, the helper calls:

kv.stutter1d(kv.axis(.h), heads.factor)

This repeats the KV heads each time it is invoked:

Original KV heads:          2
After first partition: 4
After second partition: 8
KV cache expected: 4

Both incoming K and V tensors therefore exceed the cache's expected head dimension.

Debugger evidence

Using LLDB on macOS and GDB on Linux, I inspected the arguments entering KvCache.SelfAttnCache.update().

Tensor Shape Head count
Cached K [1, 6, 2048, 4, 256] 4
Cached V [1, 6, 2048, 4, 256] 4
Incoming K [1, 2048, 8, 256] 8
Incoming V [1, 2048, 8, 256] 8

The cached tensors have an additional layer dimension, which is expected.

The mismatch was observed before the failing scatter/update operation, during symbolic graph construction.

Preliminary fix

Removing the second pair of partitionProjectedKv() calls, immediately after RoPE, resolves the original shape mismatch.

 q = self.rotary_embed.applyRope(q, cos, sin);
 k = self.rotary_embed.applyRope(k, cos, sin);
- k = partitionProjectedKv(k, kv_head_sharding);
- v = partitionProjectedKv(v, kv_head_sharding);

const new_kv_cache = kv_cache.update(k, v, token_index.convert(.u32));

This minimal change previously allowed Qwen3.5-0.8B inference to complete on macOS ARM64.

Additional validation is still needed to determine whether K should receive an explicit sharding constraint after RoPE, and to establish numerical correctness and regression coverage.

Environment

macOS

  • macOS 27.2, build 26B123

  • ARM64

  • Bazel 9.3.0

  • Bazel-managed Zig 0.17.0

  • ZML commit: 1042cba3

  • StableHLO / PJRT CPU backend

Linux

  • Omarchy 3.8.2, Arch Linux-based, x86-64

  • Intel Core i7-9700K

  • Bazel 9.3.0

  • Bazel-managed Zig 0.17.0

  • ZML branch: master

  • ZML commit: 8fcff8c

  • StableHLO / PJRT CPU backend

  • Four CPU devices

  • GDB 18.1

Both machines also have Zig 0.18 development builds installed locally, but the Bazel-managed Zig 0.17.0 toolchain was used for these reproductions.

Current status

  • Reproduced on macOS ARM64

  • Reproduced on Linux x86-64

  • Confirmed shape mismatch through LLDB and GDB

  • Identified duplicate KV-head expansion

  • Demonstrated successful inference after the minimal correction on macOS

  • Verify the correction on Linux master

  • Verify sharding behavior following RoPE

  • Add regression tests

I can follow up with a pull request for the fix.

# [Bug] Qwen3.5-0.8B crashes due to duplicate KV-head partitioning

Summary

Qwen3.5-0.8B fails during full-attention graph compilation because the incoming key/value tensors contain 8 KV heads, while the KV cache expects 4.

The issue appears to originate in examples/llm/models/qwen3_5/model.zig, where partitionProjectedKv() is called twice on both the K and V tensors.

I've reproduced the same failure on macOS ARM64 and Linux x86-64, and independently verified the tensor dimensions using LLDB and GDB.

Steps to reproduce

On an unmodified ZML checkout:

bazel run //examples/llm -- \
  --model=hf://Qwen/Qwen3.5-0.8B \
  --prompt="What is the capital of France?"

The Bazel build succeeds, but model initialization fails during prefill full-attention graph compilation.

Actual behavior

dynamicUpdateSlice expects 'update_' dimensions to be less
than or equal to their corresponding dimension in input tensor,
got 8 and 4 for axis .h

The Linux stack trace includes:

zml/ops.zig:1967
  scatter()

zml/tensor.zig:3419
  scatterSlices()

examples/llm/models/qwen3_5/model.zig:1085
  KvCache.SelfAttnCache.update()

examples/llm/models/qwen3_5/model.zig:653
  SelfAttn.forward()

Expected behavior

The incoming K/V tensors should have a head dimension compatible with the KV cache, allowing compilation and inference to complete.

Investigation and suspected root cause

In SelfAttn.forward(), partitionProjectedKv() is called before normalization and then again after RoPE.

When kv_head_sharding is .sharded with heads.factor = 2, the helper calls:

kv.stutter1d(kv.axis(.h), heads.factor)

This repeats the KV heads each time it is invoked:

Original KV heads:          2
After first partition:      4
After second partition:     8
KV cache expected:          4

Both incoming K and V tensors therefore exceed the cache's expected head dimension.

Debugger evidence

Using LLDB on macOS and GDB on Linux, I inspected the arguments entering KvCache.SelfAttnCache.update().

Tensor Shape Head count
Cached K [1, 6, 2048, 4, 256] 4
Cached V [1, 6, 2048, 4, 256] 4
Incoming K [1, 2048, 8, 256] 8
Incoming V [1, 2048, 8, 256] 8

The cached tensors have an additional layer dimension, which is expected.

The mismatch was observed before the failing scatter/update operation, during symbolic graph construction.

Preliminary fix

Removing the second pair of partitionProjectedKv() calls, immediately after RoPE, resolves the original shape mismatch.

 q = self.rotary_embed.applyRope(q, cos, sin);
 k = self.rotary_embed.applyRope(k, cos, sin);
- k = partitionProjectedKv(k, kv_head_sharding);
- v = partitionProjectedKv(v, kv_head_sharding);

 const new_kv_cache = kv_cache.update(k, v, token_index.convert(.u32));

This minimal change previously allowed Qwen3.5-0.8B inference to complete on macOS ARM64.

Additional validation is still needed to determine whether K should receive an explicit sharding constraint after RoPE, and to establish numerical correctness and regression coverage.

Environment

macOS

Linux

Both machines also have Zig 0.18 development builds installed locally, but the Bazel-managed Zig 0.17.0 toolchain was used for these reproductions.

Current status

  • Reproduced on macOS ARM64
  • Reproduced on Linux x86-64
  • Confirmed shape mismatch through LLDB and GDB
  • Identified duplicate KV-head expansion
  • Demonstrated successful inference after the minimal correction on macOS
  • Verify the correction on Linux master
  • Verify sharding behavior following RoPE
  • Add regression tests

I can follow up with a pull request for the fix.

<h1><span>[Bug] Qwen3.5-0.8B crashes due to duplicate KV-head partitioning</span></h1><h2><span>Summary</span></h2><p><span>Qwen3.5-0.8B fails during full-attention graph compilation because the incoming key/value tensors contain 8 KV heads, while the KV cache expects 4.</span></p><p><span>The issue appears to originate in </span><code><span>examples/llm/models/qwen3_5/model.zig</span></code><span>, where </span><code><span>partitionProjectedKv()</span></code><span> is called twice on both the K and V tensors.</span></p><p><span>I've reproduced the same failure on macOS ARM64 and Linux x86-64, and independently verified the tensor dimensions using LLDB and GDB.</span></p><h2><span>Steps to reproduce</span></h2><p><span>On an unmodified ZML checkout:</span></p><pre><code><span>bazel run //examples/llm -- \ --model=hf://Qwen/Qwen3.5-0.8B \ --prompt="What is the capital of France?"</span></code></pre><p><span>The Bazel build succeeds, but model initialization fails during prefill full-attention graph compilation.</span></p><h2><span>Actual behavior</span></h2><pre><code><span>dynamicUpdateSlice expects 'update_' dimensions to be less than or equal to their corresponding dimension in input tensor, got 8 and 4 for axis .h</span></code></pre><p><span>The Linux stack trace includes:</span></p><pre><code><span>zml/ops.zig:1967 scatter() zml/tensor.zig:3419 scatterSlices() examples/llm/models/qwen3_5/model.zig:1085 KvCache.SelfAttnCache.update() examples/llm/models/qwen3_5/model.zig:653 SelfAttn.forward()</span></code></pre><h2><span>Expected behavior</span></h2><p><span>The incoming K/V tensors should have a head dimension compatible with the KV cache, allowing compilation and inference to complete.</span></p><h2><span>Investigation and suspected root cause</span></h2><p><span>In </span><code><span>SelfAttn.forward()</span></code><span>, </span><code><span>partitionProjectedKv()</span></code><span> is called before normalization and then again after RoPE.</span></p><p><span>When </span><code><span>kv_head_sharding</span></code><span> is </span><code><span>.sharded</span></code><span> with </span><code><span>heads.factor = 2</span></code><span>, the helper calls:</span></p><pre><code><span>kv.stutter1d(kv.axis(.h), heads.factor)</span></code></pre><p><span>This repeats the KV heads each time it is invoked:</span></p><pre><code><span>Original KV heads: 2 After first partition: 4 After second partition: 8 KV cache expected: 4</span></code></pre><p><span>Both incoming K and V tensors therefore exceed the cache's expected head dimension.</span></p><h3><span>Debugger evidence</span></h3><p><span>Using LLDB on macOS and GDB on Linux, I inspected the arguments entering </span><code><span>KvCache.SelfAttnCache.update()</span></code><span>.</span></p> Tensor | Shape | Head count -- | -- | -- Cached K | [1, 6, 2048, 4, 256] | 4 Cached V | [1, 6, 2048, 4, 256] | 4 Incoming K | [1, 2048, 8, 256] | 8 Incoming V | [1, 2048, 8, 256] | 8 <p><span>The cached tensors have an additional </span><code><span>layer</span></code><span> dimension, which is expected.</span></p><p><span>The mismatch was observed before the failing scatter/update operation, during symbolic graph construction.</span></p><h2><span>Preliminary fix</span></h2><p><span>Removing the second pair of </span><code><span>partitionProjectedKv()</span></code><span> calls, immediately after RoPE, resolves the original shape mismatch.</span></p><pre><code><span> q = self.rotary_embed.applyRope(q, cos, sin); k = self.rotary_embed.applyRope(k, cos, sin); - k = partitionProjectedKv(k, kv_head_sharding); - v = partitionProjectedKv(v, kv_head_sharding); const new_kv_cache = kv_cache.update(k, v, token_index.convert(.u32));</span></code></pre><p><span>This minimal change previously allowed Qwen3.5-0.8B inference to complete on macOS ARM64.</span></p><p><span>Additional validation is still needed to determine whether K should receive an explicit sharding constraint after RoPE, and to establish numerical correctness and regression coverage.</span></p><h2><span>Environment</span></h2><h3><span>macOS</span></h3><ul><li><p><span>macOS 27.2, build </span><code><span>26B123</span></code></p></li><li><p><span>ARM64</span></p></li><li><p><span>Bazel 9.3.0</span></p></li><li><p><span>Bazel-managed Zig 0.17.0</span></p></li><li><p><span>ZML commit: </span><a href="https://github.com/zml/zml/commit/1042cba3ff15c1a852e3c2cf8f57fce877e8df52"><code><span>1042cba3</span></code></a></p></li><li><p><span>StableHLO / PJRT CPU backend</span></p></li></ul><h3><span>Linux</span></h3><ul><li><p><span>Omarchy 3.8.2, Arch Linux-based, x86-64</span></p></li><li><p><span>Intel Core i7-9700K</span></p></li><li><p><span>Bazel 9.3.0</span></p></li><li><p><span>Bazel-managed Zig 0.17.0</span></p></li><li><p><span>ZML branch: </span><code><span>master</span></code></p></li><li><p><span>ZML commit: </span><a href="https://github.com/zml/zml/commit/8fcff8c462070d06056f8a85ba0c8a81f0f3a636"><code><span>8fcff8c</span></code></a></p></li><li><p><span>StableHLO / PJRT CPU backend</span></p></li><li><p><span>Four CPU devices</span></p></li><li><p><span>GDB 18.1</span></p></li></ul><p><span>Both machines also have Zig 0.18 development builds installed locally, but the Bazel-managed Zig 0.17.0 toolchain was used for these reproductions.</span></p><h2><span>Current status</span></h2><ul><li><span><input type="checkbox" disabled="" checked=""></span><div><p><span>Reproduced on macOS ARM64</span></p></div></li><li><span><input type="checkbox" disabled="" checked=""></span><div><p><span>Reproduced on Linux x86-64</span></p></div></li><li><span><input type="checkbox" disabled="" checked=""></span><div><p><span>Confirmed shape mismatch through LLDB and GDB</span></p></div></li><li><span><input type="checkbox" disabled="" checked=""></span><div><p><span>Identified duplicate KV-head expansion</span></p></div></li><li><span><input type="checkbox" disabled="" checked=""></span><div><p><span>Demonstrated successful inference after the minimal correction on macOS</span></p></div></li><li><span><input type="checkbox" disabled=""></span><div><p><span>Verify the correction on Linux </span><code><span>master</span></code></p></div></li><li><span><input type="checkbox" disabled=""></span><div><p><span>Verify sharding behavior following RoPE</span></p></div></li><li><span><input type="checkbox" disabled=""></span><div><p><span>Add regression tests</span></p></div></li></ul><p><span>I can follow up with a pull request for the fix.</span></p># [Bug] Qwen3.5-0.8B crashes due to duplicate KV-head partitioning ## Summary Qwen3.5-0.8B fails during full-attention graph compilation because the incoming key/value tensors contain 8 KV heads, while the KV cache expects 4. The issue appears to originate in `examples/llm/models/qwen3_5/model.zig`, where `partitionProjectedKv()` is called twice on both the K and V tensors. I've reproduced the same failure on macOS ARM64 and Linux x86-64, and independently verified the tensor dimensions using LLDB and GDB. ## Steps to reproduce On an unmodified ZML checkout: ```bash bazel run //examples/llm -- \ --model=hf://Qwen/Qwen3.5-0.8B \ --prompt="What is the capital of France?" ``` The Bazel build succeeds, but model initialization fails during prefill full-attention graph compilation. ## Actual behavior ```text dynamicUpdateSlice expects 'update_' dimensions to be less than or equal to their corresponding dimension in input tensor, got 8 and 4 for axis .h ``` The Linux stack trace includes: ```text zml/ops.zig:1967 scatter() zml/tensor.zig:3419 scatterSlices() examples/llm/models/qwen3_5/model.zig:1085 KvCache.SelfAttnCache.update() examples/llm/models/qwen3_5/model.zig:653 SelfAttn.forward() ``` ## Expected behavior The incoming K/V tensors should have a head dimension compatible with the KV cache, allowing compilation and inference to complete. ## Investigation and suspected root cause In `SelfAttn.forward()`, `partitionProjectedKv()` is called before normalization and then again after RoPE. When `kv_head_sharding` is `.sharded` with `heads.factor = 2`, the helper calls: ```zig kv.stutter1d(kv.axis(.h), heads.factor) ``` This repeats the KV heads each time it is invoked: ```text Original KV heads: 2 After first partition: 4 After second partition: 8 KV cache expected: 4 ``` Both incoming K and V tensors therefore exceed the cache's expected head dimension. ### Debugger evidence Using LLDB on macOS and GDB on Linux, I inspected the arguments entering `KvCache.SelfAttnCache.update()`. | Tensor | Shape | Head count | |---|---|---:| | Cached K | `[1, 6, 2048, 4, 256]` | 4 | | Cached V | `[1, 6, 2048, 4, 256]` | 4 | | Incoming K | `[1, 2048, 8, 256]` | 8 | | Incoming V | `[1, 2048, 8, 256]` | 8 | The cached tensors have an additional `layer` dimension, which is expected. The mismatch was observed before the failing scatter/update operation, during symbolic graph construction. ## Preliminary fix Removing the second pair of `partitionProjectedKv()` calls, immediately after RoPE, resolves the original shape mismatch. ```diff q = self.rotary_embed.applyRope(q, cos, sin); k = self.rotary_embed.applyRope(k, cos, sin); - k = partitionProjectedKv(k, kv_head_sharding); - v = partitionProjectedKv(v, kv_head_sharding); const new_kv_cache = kv_cache.update(k, v, token_index.convert(.u32)); ``` This minimal change previously allowed Qwen3.5-0.8B inference to complete on macOS ARM64. Additional validation is still needed to determine whether K should receive an explicit sharding constraint after RoPE, and to establish numerical correctness and regression coverage. ## Environment ### macOS - macOS 27.2, build `26B123` - ARM64 - Bazel 9.3.0 - Bazel-managed Zig 0.17.0 - ZML commit: [`[1042cba3](https://github.com/zml/zml/commit/1042cba3ff15c1a852e3c2cf8f57fce877e8df52)`](https://github.com/zml/zml/commit/1042cba3ff15c1a852e3c2cf8f57fce877e8df52) - StableHLO / PJRT CPU backend ### Linux - Omarchy 3.8.2, Arch Linux-based, x86-64 - Intel Core i7-9700K - Bazel 9.3.0 - Bazel-managed Zig 0.17.0 - ZML branch: `master` - ZML commit: [`[8fcff8c](https://github.com/zml/zml/commit/8fcff8c462070d06056f8a85ba0c8a81f0f3a636)`](https://github.com/zml/zml/commit/8fcff8c462070d06056f8a85ba0c8a81f0f3a636) - StableHLO / PJRT CPU backend - Four CPU devices - GDB 18.1 Both machines also have Zig 0.18 development builds installed locally, but the Bazel-managed Zig 0.17.0 toolchain was used for these reproductions. ## Current status - [x] Reproduced on macOS ARM64 - [x] Reproduced on Linux x86-64 - [x] Confirmed shape mismatch through LLDB and GDB - [x] Identified duplicate KV-head expansion - [x] Demonstrated successful inference after the minimal correction on macOS - [ ] Verify the correction on Linux `master` - [ ] Verify sharding behavior following RoPE - [ ] Add regression tests I can follow up with a pull request for the fix.
Sign in to join this conversation.
No milestone
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
DKB/zml#1090
No description provided.