mirror of
https://github.com/zml/zml.git
synced 2026-11-16 19:11:09 +00:00
examples/llm: Qwen3.5-0.8B crashes due to duplicate KV-head partitioning #1090
Open
opened 2026-11-16 17:24:48 +00:00 by TheBCann
·
0 comments
No Branch/Tag specified
master
cmc/phonon4phun
corendos/modularization/14-moe-module
corendos/modularization/13-paged-attention-module
corendos/modularization/12-flash-attention-module
brabier/remove-arena-io
raphael/smi-next
corendos/modularization/10-introduce-nn-quantization-modules
corendos/modularization/11-package-for-modules
corendos/modularization/7-introduce-meta-constants-modules
corendos/modularization/8-introduce-profiling-module
corendos/modularization/9-introduce-core-module
raphael/oneapi-prefill-attention
dmehala/cute-sparse-attn
steeve/intel/eager
corendos/update-nvim-config
dmehala/improve-cute-sparse-attn
malo/nemotron-3.5-transcribe
brabier/intel-fp8
evi/flags
oboulant/minimax-h3-t2v-rebase
codex/pjrt-zig-compatibility
brabier/profiling-fixes
brabier/extend-pjrt-memory-stats
brabier/ds41-deep-infer-rebuild
oboulant/minimax-h3-zml-rebase
corendos/modularization/5-remove-custom-call-helpers
raphael/max-token-length
dmehala/metal-sparse-attention
raphael/decoder-buffer
brabier/loader-rework
oboulant/llmd-runfiles-groups
gw/codex/upgrade-hlo2json-to-zig-016
brabier/zls
raphael/furiosa-next
codex/quantized-tensor-representation
gw/sharding_2
brabier/ds41-deep-infer
hugomano/manual-computation-boundaries
brabier/explicit-quantization-schemes
hugomano/manual-computation-migration
hugomano/manual-computation
brabier/adaptive-concurrency
dmehala/cute-attn-improv
dmehala/cute-improvements
gw/codex/zig-incremental-worker
brabier/zml-nn-rope
evi/qwen-image-2.1
kevin/furiosa-performance
steeve/new_tt
gw/codex/zig_build_py_017
kevin/vanilla-attention
dmehala/cute-exp
brabier/fix-somehow-zls
raphael/furiosa-decode-select
brabier/improved-cute-moe
brabier/megamoe
brabier/ds41-prefill-v3
raphael/furiosa-paged-perf
louis/cute-moe-perf
corendos/gemma-4-improvement
dmehala/fix-moe-nvfp4-sharding
brabier/ds41-perf-v2
dmehala/moe-triton-local-experts
brabier/platform-capabilities-base
brabier/bake-arguments
brabier/ds41-prefill-perf
brabier/specify-dot-general-result-dtype
raphael/qaic
dmehala/sparse-attention-improvements
dmehala/compiler-enable-cudnn-fusion
tristan/constant_buffer
tristan/tmp/dma_tpu_fix
raphael/tenstorrent-next
ajv/feature/minimax-h3-t2v-video
ajv/feature/minimax-h3-zml
brabier/shape-partitioning
louis/nccl-init
clanker/smi-svg-logo
dmehala/respect-nn-tag-contract
hugomano/fix-sharding
brabier/host-overhead
brabier/propagate-sharding
brabier/oneapi-zio-sigsegv
brabier/undonatable-c-api
raphael/tcl
gw/codex/zig-017-refresh
gw/reshard-bis
clanker/smi-remote-host-data
brabier/perf-host
steeve/wip/furiosa
raphael/llvm-dialect-rename
dmehala/sparse-unified-attention
gw/data_parallel
brabier/new-base-ds41
hugomano/bench-tui
clanker/tui-safetensors-explorer
kevin/s4000
gw/deepseek
brabier/expose-memory-stats
brabier/platform-capabilities
hugomano/ds41-fp8-ready
hugomano/ds41-fp8-linear
hugomano/ds41-03-cache-lifetimes
gw/codex/sharding
clanker/data-parallelism
brabier/vanilla-glm5.3-flash
corendos/unified-cuda-fa2-fa3
louis/cutile-attention
brabier/moe-parallel-expert
dmehala/zml-smi-add-cpu-name
brabier/appendDim
brabier/pr-rework
brabier/scaled-dot-128
corendos/llama-gb300-explo
brabier/glm
brabier/glm-fp8
dmehala/breeze-tts
raphael/bump_cuda
brabier/use-new-autotune-parameter
brabier/qwen38
hugomano/rocm-0.11.1
kevin/distributed
corendos/deepseek-v4-perf-exploration
raphael/catch-error
brabier/io-rework-static-concurrency
louis/deepseek-v4-nvfp4-final-rebase
brabier/avoid-schedule-churn
brabier/glm53-flash
brabier/mla2
corendos/debug-locations
dmehala/manual-computation-improvements
simon-tokens
gw/tensor_id
brabier/autotune
kevin/kimi-k3
louis/deepseek-v4-nvfp4
corendos/triton-mha-kernel
brabier/glm5.2
reese/vulkan
raphael/quant-linear
raphael/metal-moe-quant
reese/vulkan-latest
hugomano/neuron-paged-attention
gw/codex/build.zig
gw/codex/cudaz
hugomano/neuron-2.31
raphael/cuda_quantization
gw/triton2
corendos/dsv4-requirements-perf-exploration
gw/walsh
oboulant/investiguate-ci-timeout-fail
gw/llama-walsh
hugomano/rocm-hrx
raphael/metal-quantization
dmehala/hlo-explorer
oboulant/xet-on-rework-io
corendos/fix-for-laguna-loader
corendos/improve-nn-rope
brabier/proper-sharding-qwen3
oboulant/profile-vfs-hf
louis/chain_exec
hf-xet-core
reese/step-3p5-flash
reese/step-3p5-flash-restore-layer-tests
tristan/dspark
reese/step-3p5-flash-78b5f98f
corendos/loader-toolbox-with-transform
rocm-jaxlib-v0.10.2-hrx
reese/step-3p5-flash-debug-generation
zml-prometheus
reese/step-3p5-flash-wip-2
tristan/fix_xprof_server
corendos/realpath-alloc-io-dir
reese/step-3p5-flash-fix-cli
reese/step-3p5-flash-regress
cerisier/ci-remote-cache-opts
louis/compilation-change
reese/step-3p5-flash-fix-progress-indicator
reese/step-3p5-flash-un-hardcode-executables
reese/step-3p5-flash-cuda_fa2
louis/gemma4-transpose
corendos/fp8-quantization
raphael/metal
raphael/oneapi-2026
oboulant/dumb-xet-no-reader
louis/fused-moe-tpu
tristan/dflash_gemma4
louis/gdn
hugomano/nki-paged-attention
oboulant/dumb-xet
oboulant/xet-pull-mode
reese/step-3.5-flash
oboulant/xet-merge
hugomano/xet
oboulant/xet-reconstruction-cleanup
oboulant/xet-reconstruction
steeve/oneapideb
louis/tpu-moe
raphael/tt-new-smi
raphael/smi-improved
raphael/tenstorrent-lfm
gw/simplify_platform
raphael/tenstorrent
tristan/tmp_debug
tristan/fix_gemma_4_attention
simon-acestep
tristan/dflash
louis/moe-ep
oboulant/ltx-two-gpus
codex/zig-local-execution
raphael/tt
codex/zig-toolchain-variants
louis/moe_triton_sharding
raphael/dsl-harness
raphael/GDN
louis/gmm-mosaic
oboulant/ltx-attn-dispatch-cleanup
gw/mini-build-zig
oboulant/ltx-full-mi300x
oboulant/fa-rocm-from-scratch
oboulant/fa-rocm-working
hugomano/neuron/nki-attention
raphael/mosaic-gdn
gw/opt
gw/zig_llvm_forks
louis/nemotron-moe
translate-c-standalone-upstreamable
oboulant/ltx-cleanup
oboulant/xla-autotuning
raphael/mosaic
louis/moe-laguna
raphael/triton
oboulant/ltx-cleanup-optims
gw/cleanup2
gw/attnd-client
brabier/ci-separate-zml-smi
hugomano/neuron-nki-custom-call
brabier/use-new-custom-call-for-fa
corendos/triton-fp8-support
louis/gemma4-nvfp4-quantized
hugomano/splits-split
nvfp4-matmul
corendos/gemma4-unified-attention
corendos/quick-fix-llama-crash
corendos/zig-crash
tristan/qwen3_5
tristan/tmp/gated_delta_net_tail_fix
brabier/zml-attention-attention
tristan/qwen3_5_image
tristan/qwen3_5_text
steeve/zml-smi-pkg
oboulant/ltx
steeve/tracy
louis/gpt_oss_moe_backend
brabier/inject-appropriate-buildexecroot
tmp/comparison_vllm
hugomano/qwen_3_5_rocm
tmp/investigate_compilation_issues
flash-attention-llvm
brabier/ci-build-buddy
corendos/fix-ci
corendos/bump-translate-c
kevin/flux2
steeve/cuda_compat
brabier/shitshow-debug
brabier/fix-zls
hugomano/v2-2
tristan/simple_ttir_test
brabier/hugomano-v2-2-debug
louis/gpt-oss-v2
brabier/fix-zls-imports
raphael/voxtral
tristan/zai-glm5
brabier/jinja2
tristan/tune_llama_for_nanbeige
brabier/llmd-v2-fixes
corendos/llmd-v2
iree
steeve/metal-v2
steeve/ttv2
hugomano/zml/v2/refacto-io
steeve/zml/v2-buffered-writer
hugomano/zml-v2/rework-io
louis/llama-layers
corendos/zml-v2-layout-explo
gw/isaac
steeve/zig-0.16-fs
corendos/zml-v2-llama-pipeline
hugomano/vfs-2
corendos/zml-v2-zig-0.16
libneuronxla-2.2.14584.0+06ac23d1
remote-exec
hugomano/perf-llama
2.2.14584.0+06ac23d1
gw/buffer.io
steeve/zig-0.16
corendos/zml-v2
corendos/llmd-xla-rocm
hugomano/efficient-weights-io
hugomano/pjrt-profiler
corendos/wip-compile-rework
hugomano/cuda-13
gw/dev
steeve/neuron_bump
gw/flat_bufferized
gw/tokenizer_bench
gw/qwen_8b
steeve/tt
gw/gpt_oss
hugomano/custom_call_v2
cryptodeal/mlx_plugin_impl
steeve/mlir2
steeve/zig_binary
hugomano/explo-partitioning
gw/snapshot
cerisier/neuronx-cc
corentin/clean-sandboxes
corendos/llmd
gw/llmd
gw/build.zig
sodeps
corendos/bump-rocm
jbaptiste/bazel-deb-patch
steeve/lto
corendos/paged-attention-2-flashattn-cleanup
corendos/paged-attention-2-flashattn
corendos/paged-attention-2-flashattn-sharded
hugoano/splitexe-customcall
hugomano/rope
steeve/async_buffers
corendos/paged-attention-2-flashinfer
hugomano/custom_call
hugomano/ffi-async-cb
corendos/paged-attention-2-future-cudnn
corendos/paged-attention-2-triton
jbaptiste/udp_fixes_2
corendos/nvtx
corendos/paged-attention-2-cudnn
gw/triton_cc
steeve/llmd3
hugomano/triton
rene/xfer
steeve/triton
gw/triton
gw/block_name
hugomano/dma2
jbaptiste/async-udp
gw/dev2
corendos/llama-split-experiment
corendos/test-prefill
corendos/test-paged
rene/tinyllama
corendos/llmd3-temp
hugomano/dma
corendos/llmd3-chunked-paged
renerenerene
corendos/scatter-maximum
steeve/llmd2
steeve/async-buffers-hf-tokenizers
steeve/tokenizer
corendos/amd-profiling-frankenstein
corendos/amd-profiling
corendos/tokenizer
hugomano/llama-trn1
corendos/meta-merged-tuple
corendos/ffi
steeve/http
steeve/rocm_rules_distroless
steeve/neuron_debian_snapshot
corendos/synapse
corendos/dot_algorithm
steeve/pybinary
gw/sdxl
corendos/llama-sharded-tests
misc/improved-benchmarks
corendos/benchmark-tests
corendos/libxev-epoll-backend
corendos/rocm-crash
misc/stablehlo-import-cleanup
zml-smi-v0.3.0
Labels
Clear labels
Something isn't working
Improvements or additions to documentation
This issue or pull request already exists
New feature or request
Good for newcomers
Extra attention is needed
This doesn't seem right
Further information is requested
This will not be worked on
bug
Something isn't working
documentation
Improvements or additions to documentation
duplicate
This issue or pull request already exists
enhancement
New feature or request
good first issue
Good for newcomers
help wanted
Extra attention is needed
invalid
This doesn't seem right
LLMD
platform:metal
question
Further information is requested
runtimes
runtimes/cuda
runtimes/metal
runtimes/oneapi
runtimes/rocm
runtimes/tpu
wontfix
This will not be worked on
No labels
bug
documentation
duplicate
enhancement
good first issue
help wanted
invalid
LLMD
platform:metal
question
runtimes
runtimes/cuda
runtimes/metal
runtimes/oneapi
runtimes/rocm
runtimes/tpu
wontfix
Milestone
Clear milestone
No items
No milestone
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".
No due date set.
Dependencies
No dependencies set.
Reference
DKB/zml#1090
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
[Bug] Qwen3.5-0.8B crashes due to duplicate KV-head partitioning
Summary
Qwen3.5-0.8B fails during full-attention graph compilation because the incoming key/value tensors contain 8 KV heads, while the KV cache expects 4.
The issue appears to originate in
examples/llm/models/qwen3_5/model.zig, wherepartitionProjectedKv()is called twice on both the K and V tensors.I've reproduced the same failure on macOS ARM64 and Linux x86-64, and independently verified the tensor dimensions using LLDB and GDB.
Steps to reproduce
On an unmodified ZML checkout:
The Bazel build succeeds, but model initialization fails during prefill full-attention graph compilation.
Actual behavior
The Linux stack trace includes:
Expected behavior
The incoming K/V tensors should have a head dimension compatible with the KV cache, allowing compilation and inference to complete.
Investigation and suspected root cause
In
SelfAttn.forward(),partitionProjectedKv()is called before normalization and then again after RoPE.When
kv_head_shardingis.shardedwithheads.factor = 2, the helper calls:This repeats the KV heads each time it is invoked:
Both incoming K and V tensors therefore exceed the cache's expected head dimension.
Debugger evidence
Using LLDB on macOS and GDB on Linux, I inspected the arguments entering
KvCache.SelfAttnCache.update().The cached tensors have an additional
layerdimension, which is expected.The mismatch was observed before the failing scatter/update operation, during symbolic graph construction.
Preliminary fix
Removing the second pair of
partitionProjectedKv()calls, immediately after RoPE, resolves the original shape mismatch.This minimal change previously allowed Qwen3.5-0.8B inference to complete on macOS ARM64.
Additional validation is still needed to determine whether K should receive an explicit sharding constraint after RoPE, and to establish numerical correctness and regression coverage.
Environment
macOS
macOS 27.2, build
26B123ARM64
Bazel 9.3.0
Bazel-managed Zig 0.17.0
ZML commit:
1042cba3StableHLO / PJRT CPU backend
Linux
Omarchy 3.8.2, Arch Linux-based, x86-64
Intel Core i7-9700K
Bazel 9.3.0
Bazel-managed Zig 0.17.0
ZML branch:
masterZML commit:
8fcff8cStableHLO / PJRT CPU backend
Four CPU devices
GDB 18.1
Both machines also have Zig 0.18 development builds installed locally, but the Bazel-managed Zig 0.17.0 toolchain was used for these reproductions.
Current status
Reproduced on macOS ARM64
Reproduced on Linux x86-64
Confirmed shape mismatch through LLDB and GDB
Identified duplicate KV-head expansion
Demonstrated successful inference after the minimal correction on macOS
Verify the correction on Linux
masterVerify sharding behavior following RoPE
Add regression tests
I can follow up with a pull request for the fix.
# [Bug] Qwen3.5-0.8B crashes due to duplicate KV-head partitioningSummary
Qwen3.5-0.8B fails during full-attention graph compilation because the incoming key/value tensors contain 8 KV heads, while the KV cache expects 4.
The issue appears to originate in
examples/llm/models/qwen3_5/model.zig, wherepartitionProjectedKv()is called twice on both the K and V tensors.I've reproduced the same failure on macOS ARM64 and Linux x86-64, and independently verified the tensor dimensions using LLDB and GDB.
Steps to reproduce
On an unmodified ZML checkout:
The Bazel build succeeds, but model initialization fails during prefill full-attention graph compilation.
Actual behavior
The Linux stack trace includes:
Expected behavior
The incoming K/V tensors should have a head dimension compatible with the KV cache, allowing compilation and inference to complete.
Investigation and suspected root cause
In
SelfAttn.forward(),partitionProjectedKv()is called before normalization and then again after RoPE.When
kv_head_shardingis.shardedwithheads.factor = 2, the helper calls:This repeats the KV heads each time it is invoked:
Both incoming K and V tensors therefore exceed the cache's expected head dimension.
Debugger evidence
Using LLDB on macOS and GDB on Linux, I inspected the arguments entering
KvCache.SelfAttnCache.update().[1, 6, 2048, 4, 256][1, 6, 2048, 4, 256][1, 2048, 8, 256][1, 2048, 8, 256]The cached tensors have an additional
layerdimension, which is expected.The mismatch was observed before the failing scatter/update operation, during symbolic graph construction.
Preliminary fix
Removing the second pair of
partitionProjectedKv()calls, immediately after RoPE, resolves the original shape mismatch.This minimal change previously allowed Qwen3.5-0.8B inference to complete on macOS ARM64.
Additional validation is still needed to determine whether K should receive an explicit sharding constraint after RoPE, and to establish numerical correctness and regression coverage.
Environment
macOS
26B123[1042cba3](https://github.com/zml/zml/commit/1042cba3ff15c1a852e3c2cf8f57fce877e8df52)Linux
master[8fcff8c](https://github.com/zml/zml/commit/8fcff8c462070d06056f8a85ba0c8a81f0f3a636)Both machines also have Zig 0.18 development builds installed locally, but the Bazel-managed Zig 0.17.0 toolchain was used for these reproductions.
Current status
masterI can follow up with a pull request for the fix.