0
Fork 0
mirror of https://github.com/zml/zml.git synced 2026-11-16 19:11:09 +00:00

Llama 3.1 8B execution issues #170

Closed
opened 2025-01-25 15:26:56 +00:00 by allisoneer · 3 comments
allisoneer commented 2025-01-25 15:26:56 +00:00 (Migrated from github.com)

This is on current master.

Happens with both of the following commands (cuda and rocm):

bazel run -c opt //llama:Llama-3.1-8B-Instruct --@zml//runtimes:cuda=true --@zml//runtimes:cpu=false -- --prompt="Once upon a time,"
# and
bazel run -c opt //llama:Llama-3.1-8B-Instruct --@zml//runtimes:rocm=true --@zml//runtimes:cpu=false -- --prompt="Once upon a time,"

Here is a quick view of the error:

Executable expected shape u32[1,256]{1,0} for argument 291 but got incompatible shape u32[256]{0}

Here are the full logs of both machines:

Nvidia cuda
 bazel run -c opt //llama:Llama-3.1-8B-Instruct --@zml//runtimes:cuda=true --@zml//runtimes:cpu=false -- --prompt="Once upon a time,"
INFO: Analyzed target //llama:Llama-3.1-8B-Instruct (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target //llama:Llama-3.1-8B-Instruct up-to-date:
  bazel-bin/llama/Llama-3.1-8B-Instruct
INFO: Elapsed time: 0.482s, Critical Path: 0.00s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
INFO: Running command line: bazel-bin/llama/Llama-3.1-8B-Instruct '--config=../zml++huggingface+Meta-Llama-3.1-8B-Instruct/config.json' '--weights=../zml++huggingface+Meta-Llama-3.1-8B-Instruct/model.safetensors.index.json' '--tokenizer=../zml++huggingface+Meta-Llama-3.1-8B-Instruct/tokenizer.json' <args omitted>
info(llama):    LLama was compiled with builtin.OptimizeMode.ReleaseSafe
info(pjrt): Loaded library: libpjrt_cuda.so
info(zml/context): Available Platforms:
info(zml/context):   •  cpu
info(zml/context):   ✅ cuda (AUTO-SELECTED)
info(zml/context):        ◦ #0: NVIDIA L40S
info(zml/context):        ◦ #1: NVIDIA L40S
info(zml/context):        ◦ #2: NVIDIA L40S
info(zml/context):        ◦ #3: NVIDIA L40S
info(zml/context):        ◦ #4: NVIDIA L40S
info(zml/context):        ◦ #5: NVIDIA L40S
info(zml/context):        ◦ #6: NVIDIA L40S
info(zml/context):        ◦ #7: NVIDIA L40S
info(zml/context):   •  rocm
info(zml/context):   •  tpu
info(zml/context):   •  neuron
info(zml): Compiling llama.LlamaLM.forward with { Shape({b=1,s=256,u32}), Shape({b=1,u32}), meta.MapType(tensor.Tensor,shape.Shape).map(llama.KvCache){ .k = Shape({layer=32,b=1,k=256,h=8!,hd=128,bf16}), .v = Shape({layer=32,b=1,k=256,h=8!,hd=128,bf16}), .layer_index = Shape({u32}) }, meta.MapType(tensor.Tensor,shape.Shape).map(tensor.Tensor.Rng){ ._state = Shape({2,u64}), .algorithm = stablehlo.RngAlgorithm.Type.DEFAULT } }
info(zml): Compiling llama.LlamaLM.forward with { Shape({b=1,s=1,u32}), Shape({b=1,u32}), meta.MapType(tensor.Tensor,shape.Shape).map(llama.KvCache){ .k = Shape({layer=32,b=1,k=256,h=8!,hd=128,bf16}), .v = Shape({layer=32,b=1,k=256,h=8!,hd=128,bf16}), .layer_index = Shape({u32}) }, meta.MapType(tensor.Tensor,shape.Shape).map(tensor.Tensor.Rng){ ._state = Shape({2,u64}), .algorithm = stablehlo.RngAlgorithm.Type.DEFAULT } }
info(llama):    Loading Llama weights from ../zml++huggingface+Meta-Llama-3.1-8B-Instruct/model.safetensors.index.json...
info(zml/module): Wrote MLIR to /tmp/zml/llama/cuda/llama.LlamaLM.forward_4125b1dbb1372210/module.mlir
info(zml/module): Wrote MLIR to /tmp/zml/llama/cuda/llama.LlamaLM.forward_98293ef88d8df614/module.mlir
[libprotobuf ERROR external/com_google_protobuf/src/google/protobuf/wire_format_lite.cc:618] String field 'xla.gpu.CompilationResultProto.DnnCompiledGraphsEntry.value' contains invalid UTF-8 data when parsing a protocol buffer. Use the 'bytes' type if you intend to send raw bytes.
error(pjrt): [PJRT_Executable_DeserializeAndLoad] Failed to parse serialized GpuThunkAotCompilationResult.
warning(zml/module): Failed to load pre-compiled module: error.Internal at /tmp/zml/llama/cuda/llama.LlamaLM.forward_4125b1dbb1372210/module.pjrt
[libprotobuf ERROR external/com_google_protobuf/src/google/protobuf/wire_format_lite.cc:618] String field 'xla.gpu.CompilationResultProto.DnnCompiledGraphsEntry.value' contains invalid UTF-8 data when parsing a protocol buffer. Use the 'bytes' type if you intend to send raw bytes.
error(pjrt): [PJRT_Executable_DeserializeAndLoad] Failed to parse serialized GpuThunkAotCompilationResult.
warning(zml/module): Failed to load pre-compiled module: error.Internal at /tmp/zml/llama/cuda/llama.LlamaLM.forward_98293ef88d8df614/module.pjrt
2025-01-25 15:22:43.181413: I external/xla/xla/service/dump.cc:561] HloModule dump enabled with path prefix: , suffix: before_optimizations
2025-01-25 15:22:51.536152: I external/xla/xla/service/latency_hiding_scheduler.cc:2696] LatencyHidingScheduler current memory usage: 5441136 bytes. Current limit: 35895907468
2025-01-25 15:22:51.917438: I external/xla/xla/service/latency_hiding_scheduler.cc:2696] LatencyHidingScheduler current memory usage: 50135080 bytes. Current limit: 35895906499
info(llama): ✅ Loaded weights in 10.561s
2025-01-25 15:22:53.363756: I external/xla/xla/service/llvm_ir/llvm_command_line_options.cc:50] XLA (re)initializing LLVM with options fingerprint: 11054585593084613546
2025-01-25 15:22:53.556148: I external/xla/xla/stream_executor/cuda/subprocess_compilation.cc:488] Using nvlink for parallel linking
[libprotobuf ERROR external/com_google_protobuf/src/google/protobuf/wire_format_lite.cc:618] String field 'xla.gpu.CompilationResultProto.DnnCompiledGraphsEntry.value' contains invalid UTF-8 data when serializing a protocol buffer. Use the 'bytes' type if you intend to send raw bytes.
info(zml/module): Compilation took 12.181s
[libprotobuf ERROR external/com_google_protobuf/src/google/protobuf/wire_format_lite.cc:618] String field 'xla.gpu.CompilationResultProto.DnnCompiledGraphsEntry.value' contains invalid UTF-8 data when serializing a protocol buffer. Use the 'bytes' type if you intend to send raw bytes.
info(zml/module): Compilation took 13.213s
info(llama): ✅ Compiled model in 13.215s
info(llama): Creating KvCache
info(llama): Loading tokenizer from ../zml++huggingface+Meta-Llama-3.1-8B-Instruct/tokenizer.json
info(llama): Loaded tokenizer from ../zml++huggingface+Meta-Llama-3.1-8B-Instruct/tokenizer.json [508.921ms]
info(llama): ✅ Prompt: Once upon a time,
E0125 15:22:55.460935 1310419 pjrt_stream_executor_client.cc:3086] Execution of replica 0 failed: INVALID_ARGUMENT: Executable expected shape u32[1,256]{1,0} for argument 291 but got incompatible shape u32[256]{0}
E0125 15:22:55.460982 1310437 pjrt_stream_executor_client.cc:3086] Execution of replica 0 failed: INVALID_ARGUMENT: Executable expected shape u32[1,256]{1,0} for argument 291 but got incompatible shape u32[256]{0}
E0125 15:22:55.461025 1310434 pjrt_stream_executor_client.cc:3086] Execution of replica 0 failed: INVALID_ARGUMENT: Executable expected shape u32[1,256]{1,0} for argument 291 but got incompatible shape u32[256]{0}
E0125 15:22:55.461047 1310422 pjrt_stream_executor_client.cc:3086] Execution of replica 0 failed: INVALID_ARGUMENT: Executable expected shape u32[1,256]{1,0} for argument 291 but got incompatible shape u32[256]{0}
E0125 15:22:55.461060 1310425 pjrt_stream_executor_client.cc:3086] Execution of replica 0 failed: INVALID_ARGUMENT: Executable expected shape u32[1,256]{1,0} for argument 291 but got incompatible shape u32[256]{0}
E0125 15:22:55.461095 1310428 pjrt_stream_executor_client.cc:3086] Execution of replica 0 failed: INVALID_ARGUMENT: Executable expected shape u32[1,256]{1,0} for argument 291 but got incompatible shape u32[256]{0}
E0125 15:22:55.461102 1310440 pjrt_stream_executor_client.cc:3086] Execution of replica 0 failed: INVALID_ARGUMENT: Executable expected shape u32[1,256]{1,0} for argument 291 but got incompatible shape u32[256]{0}
E0125 15:22:55.461219 1310431 pjrt_stream_executor_client.cc:3086] Execution of replica 0 failed: INVALID_ARGUMENT: Executable expected shape u32[1,256]{1,0} for argument 291 but got incompatible shape u32[256]{0}
error(pjrt): [PJRT_LoadedExecutable_Execute] Executable expected shape u32[1,256]{1,0} for argument 291 but got incompatible shape u32[256]{0}: while running replica 0 and partition 0 of a replicated computation (other replicas may have failed as well).
thread 1309923 panic: attempt to unwrap error
/home/allison/.cache/bazel/_bazel_allison/36225b4c74aea3b4d2da3f021042a9ba/sandbox/linux-sandbox/3164/execroot/_main/external/zml+/zml/exe.zig:0:0: 0x15f9f3a in call (llama_lib)
/home/allison/.cache/bazel/_bazel_allison/36225b4c74aea3b4d2da3f021042a9ba/sandbox/linux-sandbox/3164/execroot/_main/llama/main.zig:93:57: 0x15f8a44 in generateText (llama_lib)
/home/allison/.cache/bazel/_bazel_allison/36225b4c74aea3b4d2da3f021042a9ba/sandbox/linux-sandbox/3164/execroot/_main/llama/main.zig:328:44: 0x15ffb68 in asyncMain (llama_lib)
/home/allison/.cache/bazel/_bazel_allison/36225b4c74aea3b4d2da3f021042a9ba/sandbox/linux-sandbox/3164/execroot/_main/external/zml+/async/coro.zig:263:28: 0x15b7e7b in wrapfn (llama_lib)
/home/allison/.cache/bazel/_bazel_allison/36225b4c74aea3b4d2da3f021042a9ba/sandbox/linux-sandbox/3164/execroot/_main/external/zml+/async/coro.zig:164:31: 0x15b7e30 in runcoro (llama_lib)
???:?:?: 0xaaaaaaaaaaaaaaa9 in ??? (???)
Unwind information for `???:0xaaaaaaaaaaaaaaa9` was not available, trace may be incomplete
AMD rocm
bazel run -c opt //llama:Llama-3.1-8B-Instruct --@zml//runtimes:rocm=true --@zml//runtimes:cpu=false -- --prompt="Once upon a time,"
INFO: Analyzed target //llama:Llama-3.1-8B-Instruct (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target //llama:Llama-3.1-8B-Instruct up-to-date:
  bazel-bin/llama/Llama-3.1-8B-Instruct
INFO: Elapsed time: 0.213s, Critical Path: 0.00s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
INFO: Running command line: bazel-bin/llama/Llama-3.1-8B-Instruct '--config=../zml++huggingface+Meta-Llama-3.1-8B-Instruct/config.json' '--weights=../zml++huggingface+Meta-Llama-3.1-8B-Instruct/model.safetensors.index.json' '--tokenizer=../zml++huggingface+Meta-Llama-3.1-8B-Instruct/tokenizer.json' <args omitted>
info(llama):    LLama was compiled with builtin.OptimizeMode.ReleaseSafe
info(pjrt): Loaded library: libpjrt_rocm.so
info(zml/context): Available Platforms:
info(zml/context):   •  cpu
info(zml/context):   •  cuda
info(zml/context):   ✅ rocm (AUTO-SELECTED)
info(zml/context):        ◦ #0:
info(zml/context):        ◦ #1:
info(zml/context):        ◦ #2:
info(zml/context):        ◦ #3:
info(zml/context):        ◦ #4:
info(zml/context):        ◦ #5:
info(zml/context):        ◦ #6:
info(zml/context):        ◦ #7:
info(zml/context):   •  tpu
info(zml/context):   •  neuron
info(zml): Compiling llama.LlamaLM.forward with { Shape({b=1,s=256,u32}), Shape({b=1,u32}), meta.MapType(tensor.Tensor,shape.Shape).map(llama.KvCache){ .k = Shape({layer=32,b=1,k=256,h=8!,hd=128,bf16}), .v = Shape({layer=32,b=1,k=256,h=8!,hd=128,bf16}), .layer_index = Shape({u32}) }, meta.MapType(tensor.Tensor,shape.Shape).map(tensor.Tensor.Rng){ ._state = Shape({2,u64}), .algorithm = stablehlo.RngAlgorithm.Type.DEFAULT } }
info(llama):    Loading Llama weights from ../zml++huggingface+Meta-Llama-3.1-8B-Instruct/model.safetensors.index.json...
info(zml): Compiling llama.LlamaLM.forward with { Shape({b=1,s=1,u32}), Shape({b=1,u32}), meta.MapType(tensor.Tensor,shape.Shape).map(llama.KvCache){ .k = Shape({layer=32,b=1,k=256,h=8!,hd=128,bf16}), .v = Shape({layer=32,b=1,k=256,h=8!,hd=128,bf16}), .layer_index = Shape({u32}) }, meta.MapType(tensor.Tensor,shape.Shape).map(tensor.Tensor.Rng){ ._state = Shape({2,u64}), .algorithm = stablehlo.RngAlgorithm.Type.DEFAULT } }
info(zml/module): Wrote MLIR to /tmp/zml/llama/rocm/llama.LlamaLM.forward_f9e6ea06e80ef7b2/module.mlir
info(zml/module): Wrote MLIR to /tmp/zml/llama/rocm/llama.LlamaLM.forward_59c4dcdc5bf68be0/module.mlir
info(zml/module): Loaded pre-compiled module from /tmp/zml/llama/rocm/llama.LlamaLM.forward_f9e6ea06e80ef7b2/module.pjrt
info(zml/module): Loaded pre-compiled module from /tmp/zml/llama/rocm/llama.LlamaLM.forward_59c4dcdc5bf68be0/module.pjrt
info(zml/module): Compilation took 1.290s
info(llama): ✅ Loaded weights in 7.289s
info(llama): ✅ Compiled model in 7.29s
info(llama): Creating KvCache
info(llama): Loading tokenizer from ../zml++huggingface+Meta-Llama-3.1-8B-Instruct/tokenizer.json
info(llama): Loaded tokenizer from ../zml++huggingface+Meta-Llama-3.1-8B-Instruct/tokenizer.json [289.68ms]
info(llama): ✅ Prompt: Once upon a time,
E0125 15:09:19.613079  782419 pjrt_stream_executor_client.cc:3045] Execution of replica 0 failed: INVALID_ARGUMENT: Executable expected shape u32[1,256]{1,0} for argument 291 but got incompatible shape u32[256]{0}
E0125 15:09:19.613276  782422 pjrt_stream_executor_client.cc:3045] Execution of replica 0 failed: INVALID_ARGUMENT: Executable expected shape u32[1,256]{1,0} for argument 291 but got incompatible shape u32[256]{0}
E0125 15:09:19.613309  782416 pjrt_stream_executor_client.cc:3045] Execution of replica 0 failed: INVALID_ARGUMENT: Executable expected shape u32[1,256]{1,0} for argument 291 but got incompatible shape u32[256]{0}
E0125 15:09:19.613365  782428 pjrt_stream_executor_client.cc:3045] Execution of replica 0 failed: INVALID_ARGUMENT: Executable expected shape u32[1,256]{1,0} for argument 291 but got incompatible shape u32[256]{0}
E0125 15:09:19.613441  782431 pjrt_stream_executor_client.cc:3045] Execution of replica 0 failed: INVALID_ARGUMENT: Executable expected shape u32[1,256]{1,0} for argument 291 but got incompatible shape u32[256]{0}
E0125 15:09:19.613448  782434 pjrt_stream_executor_client.cc:3045] Execution of replica 0 failed: INVALID_ARGUMENT: Executable expected shape u32[1,256]{1,0} for argument 291 but got incompatible shape u32[256]{0}
E0125 15:09:19.613657  782425 pjrt_stream_executor_client.cc:3045] Execution of replica 0 failed: INVALID_ARGUMENT: Executable expected shape u32[1,256]{1,0} for argument 291 but got incompatible shape u32[256]{0}
E0125 15:09:19.613712  782437 pjrt_stream_executor_client.cc:3045] Execution of replica 0 failed: INVALID_ARGUMENT: Executable expected shape u32[1,256]{1,0} for argument 291 but got incompatible shape u32[256]{0}
error(pjrt): [PJRT_LoadedExecutable_Execute] Executable expected shape u32[1,256]{1,0} for argument 291 but got incompatible shape u32[256]{0}: while running replica 0 and partition 0 of a replicated computation (other replicas may have failed as well).
thread 781957 panic: attempt to unwrap error
/home/allison/.cache/bazel/_bazel_allison/13b37383329cb8e9be33fcec42a1459a/sandbox/linux-sandbox/2/execroot/_main/external/zml+/zml/exe.zig:0:0: 0x1608e2a in call (llama_lib)
/home/allison/.cache/bazel/_bazel_allison/13b37383329cb8e9be33fcec42a1459a/sandbox/linux-sandbox/2/execroot/_main/llama/main.zig:93:57: 0x1607934 in generateText (llama_lib)
/home/allison/.cache/bazel/_bazel_allison/13b37383329cb8e9be33fcec42a1459a/sandbox/linux-sandbox/2/execroot/_main/llama/main.zig:328:44: 0x160ea58 in asyncMain (llama_lib)
/home/allison/.cache/bazel/_bazel_allison/13b37383329cb8e9be33fcec42a1459a/sandbox/linux-sandbox/2/execroot/_main/external/zml+/async/coro.zig:263:28: 0x15c6d6b in wrapfn (llama_lib)
/home/allison/.cache/bazel/_bazel_allison/13b37383329cb8e9be33fcec42a1459a/sandbox/linux-sandbox/2/execroot/_main/external/zml+/async/coro.zig:164:31: 0x15c6d20 in runcoro (llama_lib)
???:?:?: 0xaaaaaaaaaaaaaaa9 in ??? (???)
Unwind information for `???:0xaaaaaaaaaaaaaaa9` was not available, trace may be incomplete

It looks like a simple enough bug where I could go fix it myself, but don't have time at this very moment to do that so I'm posting in case it's helpful. Also it may be less relevant once the tokenizer changes get merged, but again just covering the bases while I'm here.

I did a quick check 3 commits ago to see if it was an easy blame but I had the same error on that one. Again, probably easy fix, but again covering bases. Gotta run now, cheers.

This is on [current master](https://github.com/zml/zml/commit/d76e5b9cd487b5ab584a1b80c397081f94b8b0c4). Happens with both of the following commands (cuda and rocm): ``` bazel run -c opt //llama:Llama-3.1-8B-Instruct --@zml//runtimes:cuda=true --@zml//runtimes:cpu=false -- --prompt="Once upon a time," # and bazel run -c opt //llama:Llama-3.1-8B-Instruct --@zml//runtimes:rocm=true --@zml//runtimes:cpu=false -- --prompt="Once upon a time," ``` Here is a quick view of the error: ``` Executable expected shape u32[1,256]{1,0} for argument 291 but got incompatible shape u32[256]{0} ``` Here are the full logs of both machines: <details> <summary>Nvidia cuda</summary> ``` bazel run -c opt //llama:Llama-3.1-8B-Instruct --@zml//runtimes:cuda=true --@zml//runtimes:cpu=false -- --prompt="Once upon a time," INFO: Analyzed target //llama:Llama-3.1-8B-Instruct (0 packages loaded, 0 targets configured). INFO: Found 1 target... Target //llama:Llama-3.1-8B-Instruct up-to-date: bazel-bin/llama/Llama-3.1-8B-Instruct INFO: Elapsed time: 0.482s, Critical Path: 0.00s INFO: 1 process: 1 internal. INFO: Build completed successfully, 1 total action INFO: Running command line: bazel-bin/llama/Llama-3.1-8B-Instruct '--config=../zml++huggingface+Meta-Llama-3.1-8B-Instruct/config.json' '--weights=../zml++huggingface+Meta-Llama-3.1-8B-Instruct/model.safetensors.index.json' '--tokenizer=../zml++huggingface+Meta-Llama-3.1-8B-Instruct/tokenizer.json' <args omitted> info(llama): LLama was compiled with builtin.OptimizeMode.ReleaseSafe info(pjrt): Loaded library: libpjrt_cuda.so info(zml/context): Available Platforms: info(zml/context): • cpu info(zml/context): ✅ cuda (AUTO-SELECTED) info(zml/context): ◦ #0: NVIDIA L40S info(zml/context): ◦ #1: NVIDIA L40S info(zml/context): ◦ #2: NVIDIA L40S info(zml/context): ◦ #3: NVIDIA L40S info(zml/context): ◦ #4: NVIDIA L40S info(zml/context): ◦ #5: NVIDIA L40S info(zml/context): ◦ #6: NVIDIA L40S info(zml/context): ◦ #7: NVIDIA L40S info(zml/context): • rocm info(zml/context): • tpu info(zml/context): • neuron info(zml): Compiling llama.LlamaLM.forward with { Shape({b=1,s=256,u32}), Shape({b=1,u32}), meta.MapType(tensor.Tensor,shape.Shape).map(llama.KvCache){ .k = Shape({layer=32,b=1,k=256,h=8!,hd=128,bf16}), .v = Shape({layer=32,b=1,k=256,h=8!,hd=128,bf16}), .layer_index = Shape({u32}) }, meta.MapType(tensor.Tensor,shape.Shape).map(tensor.Tensor.Rng){ ._state = Shape({2,u64}), .algorithm = stablehlo.RngAlgorithm.Type.DEFAULT } } info(zml): Compiling llama.LlamaLM.forward with { Shape({b=1,s=1,u32}), Shape({b=1,u32}), meta.MapType(tensor.Tensor,shape.Shape).map(llama.KvCache){ .k = Shape({layer=32,b=1,k=256,h=8!,hd=128,bf16}), .v = Shape({layer=32,b=1,k=256,h=8!,hd=128,bf16}), .layer_index = Shape({u32}) }, meta.MapType(tensor.Tensor,shape.Shape).map(tensor.Tensor.Rng){ ._state = Shape({2,u64}), .algorithm = stablehlo.RngAlgorithm.Type.DEFAULT } } info(llama): Loading Llama weights from ../zml++huggingface+Meta-Llama-3.1-8B-Instruct/model.safetensors.index.json... info(zml/module): Wrote MLIR to /tmp/zml/llama/cuda/llama.LlamaLM.forward_4125b1dbb1372210/module.mlir info(zml/module): Wrote MLIR to /tmp/zml/llama/cuda/llama.LlamaLM.forward_98293ef88d8df614/module.mlir [libprotobuf ERROR external/com_google_protobuf/src/google/protobuf/wire_format_lite.cc:618] String field 'xla.gpu.CompilationResultProto.DnnCompiledGraphsEntry.value' contains invalid UTF-8 data when parsing a protocol buffer. Use the 'bytes' type if you intend to send raw bytes. error(pjrt): [PJRT_Executable_DeserializeAndLoad] Failed to parse serialized GpuThunkAotCompilationResult. warning(zml/module): Failed to load pre-compiled module: error.Internal at /tmp/zml/llama/cuda/llama.LlamaLM.forward_4125b1dbb1372210/module.pjrt [libprotobuf ERROR external/com_google_protobuf/src/google/protobuf/wire_format_lite.cc:618] String field 'xla.gpu.CompilationResultProto.DnnCompiledGraphsEntry.value' contains invalid UTF-8 data when parsing a protocol buffer. Use the 'bytes' type if you intend to send raw bytes. error(pjrt): [PJRT_Executable_DeserializeAndLoad] Failed to parse serialized GpuThunkAotCompilationResult. warning(zml/module): Failed to load pre-compiled module: error.Internal at /tmp/zml/llama/cuda/llama.LlamaLM.forward_98293ef88d8df614/module.pjrt 2025-01-25 15:22:43.181413: I external/xla/xla/service/dump.cc:561] HloModule dump enabled with path prefix: , suffix: before_optimizations 2025-01-25 15:22:51.536152: I external/xla/xla/service/latency_hiding_scheduler.cc:2696] LatencyHidingScheduler current memory usage: 5441136 bytes. Current limit: 35895907468 2025-01-25 15:22:51.917438: I external/xla/xla/service/latency_hiding_scheduler.cc:2696] LatencyHidingScheduler current memory usage: 50135080 bytes. Current limit: 35895906499 info(llama): ✅ Loaded weights in 10.561s 2025-01-25 15:22:53.363756: I external/xla/xla/service/llvm_ir/llvm_command_line_options.cc:50] XLA (re)initializing LLVM with options fingerprint: 11054585593084613546 2025-01-25 15:22:53.556148: I external/xla/xla/stream_executor/cuda/subprocess_compilation.cc:488] Using nvlink for parallel linking [libprotobuf ERROR external/com_google_protobuf/src/google/protobuf/wire_format_lite.cc:618] String field 'xla.gpu.CompilationResultProto.DnnCompiledGraphsEntry.value' contains invalid UTF-8 data when serializing a protocol buffer. Use the 'bytes' type if you intend to send raw bytes. info(zml/module): Compilation took 12.181s [libprotobuf ERROR external/com_google_protobuf/src/google/protobuf/wire_format_lite.cc:618] String field 'xla.gpu.CompilationResultProto.DnnCompiledGraphsEntry.value' contains invalid UTF-8 data when serializing a protocol buffer. Use the 'bytes' type if you intend to send raw bytes. info(zml/module): Compilation took 13.213s info(llama): ✅ Compiled model in 13.215s info(llama): Creating KvCache info(llama): Loading tokenizer from ../zml++huggingface+Meta-Llama-3.1-8B-Instruct/tokenizer.json info(llama): Loaded tokenizer from ../zml++huggingface+Meta-Llama-3.1-8B-Instruct/tokenizer.json [508.921ms] info(llama): ✅ Prompt: Once upon a time, E0125 15:22:55.460935 1310419 pjrt_stream_executor_client.cc:3086] Execution of replica 0 failed: INVALID_ARGUMENT: Executable expected shape u32[1,256]{1,0} for argument 291 but got incompatible shape u32[256]{0} E0125 15:22:55.460982 1310437 pjrt_stream_executor_client.cc:3086] Execution of replica 0 failed: INVALID_ARGUMENT: Executable expected shape u32[1,256]{1,0} for argument 291 but got incompatible shape u32[256]{0} E0125 15:22:55.461025 1310434 pjrt_stream_executor_client.cc:3086] Execution of replica 0 failed: INVALID_ARGUMENT: Executable expected shape u32[1,256]{1,0} for argument 291 but got incompatible shape u32[256]{0} E0125 15:22:55.461047 1310422 pjrt_stream_executor_client.cc:3086] Execution of replica 0 failed: INVALID_ARGUMENT: Executable expected shape u32[1,256]{1,0} for argument 291 but got incompatible shape u32[256]{0} E0125 15:22:55.461060 1310425 pjrt_stream_executor_client.cc:3086] Execution of replica 0 failed: INVALID_ARGUMENT: Executable expected shape u32[1,256]{1,0} for argument 291 but got incompatible shape u32[256]{0} E0125 15:22:55.461095 1310428 pjrt_stream_executor_client.cc:3086] Execution of replica 0 failed: INVALID_ARGUMENT: Executable expected shape u32[1,256]{1,0} for argument 291 but got incompatible shape u32[256]{0} E0125 15:22:55.461102 1310440 pjrt_stream_executor_client.cc:3086] Execution of replica 0 failed: INVALID_ARGUMENT: Executable expected shape u32[1,256]{1,0} for argument 291 but got incompatible shape u32[256]{0} E0125 15:22:55.461219 1310431 pjrt_stream_executor_client.cc:3086] Execution of replica 0 failed: INVALID_ARGUMENT: Executable expected shape u32[1,256]{1,0} for argument 291 but got incompatible shape u32[256]{0} error(pjrt): [PJRT_LoadedExecutable_Execute] Executable expected shape u32[1,256]{1,0} for argument 291 but got incompatible shape u32[256]{0}: while running replica 0 and partition 0 of a replicated computation (other replicas may have failed as well). thread 1309923 panic: attempt to unwrap error /home/allison/.cache/bazel/_bazel_allison/36225b4c74aea3b4d2da3f021042a9ba/sandbox/linux-sandbox/3164/execroot/_main/external/zml+/zml/exe.zig:0:0: 0x15f9f3a in call (llama_lib) /home/allison/.cache/bazel/_bazel_allison/36225b4c74aea3b4d2da3f021042a9ba/sandbox/linux-sandbox/3164/execroot/_main/llama/main.zig:93:57: 0x15f8a44 in generateText (llama_lib) /home/allison/.cache/bazel/_bazel_allison/36225b4c74aea3b4d2da3f021042a9ba/sandbox/linux-sandbox/3164/execroot/_main/llama/main.zig:328:44: 0x15ffb68 in asyncMain (llama_lib) /home/allison/.cache/bazel/_bazel_allison/36225b4c74aea3b4d2da3f021042a9ba/sandbox/linux-sandbox/3164/execroot/_main/external/zml+/async/coro.zig:263:28: 0x15b7e7b in wrapfn (llama_lib) /home/allison/.cache/bazel/_bazel_allison/36225b4c74aea3b4d2da3f021042a9ba/sandbox/linux-sandbox/3164/execroot/_main/external/zml+/async/coro.zig:164:31: 0x15b7e30 in runcoro (llama_lib) ???:?:?: 0xaaaaaaaaaaaaaaa9 in ??? (???) Unwind information for `???:0xaaaaaaaaaaaaaaa9` was not available, trace may be incomplete ``` </details> <details> <summary>AMD rocm</summary> ``` bazel run -c opt //llama:Llama-3.1-8B-Instruct --@zml//runtimes:rocm=true --@zml//runtimes:cpu=false -- --prompt="Once upon a time," INFO: Analyzed target //llama:Llama-3.1-8B-Instruct (0 packages loaded, 0 targets configured). INFO: Found 1 target... Target //llama:Llama-3.1-8B-Instruct up-to-date: bazel-bin/llama/Llama-3.1-8B-Instruct INFO: Elapsed time: 0.213s, Critical Path: 0.00s INFO: 1 process: 1 internal. INFO: Build completed successfully, 1 total action INFO: Running command line: bazel-bin/llama/Llama-3.1-8B-Instruct '--config=../zml++huggingface+Meta-Llama-3.1-8B-Instruct/config.json' '--weights=../zml++huggingface+Meta-Llama-3.1-8B-Instruct/model.safetensors.index.json' '--tokenizer=../zml++huggingface+Meta-Llama-3.1-8B-Instruct/tokenizer.json' <args omitted> info(llama): LLama was compiled with builtin.OptimizeMode.ReleaseSafe info(pjrt): Loaded library: libpjrt_rocm.so info(zml/context): Available Platforms: info(zml/context): • cpu info(zml/context): • cuda info(zml/context): ✅ rocm (AUTO-SELECTED) info(zml/context): ◦ #0: info(zml/context): ◦ #1: info(zml/context): ◦ #2: info(zml/context): ◦ #3: info(zml/context): ◦ #4: info(zml/context): ◦ #5: info(zml/context): ◦ #6: info(zml/context): ◦ #7: info(zml/context): • tpu info(zml/context): • neuron info(zml): Compiling llama.LlamaLM.forward with { Shape({b=1,s=256,u32}), Shape({b=1,u32}), meta.MapType(tensor.Tensor,shape.Shape).map(llama.KvCache){ .k = Shape({layer=32,b=1,k=256,h=8!,hd=128,bf16}), .v = Shape({layer=32,b=1,k=256,h=8!,hd=128,bf16}), .layer_index = Shape({u32}) }, meta.MapType(tensor.Tensor,shape.Shape).map(tensor.Tensor.Rng){ ._state = Shape({2,u64}), .algorithm = stablehlo.RngAlgorithm.Type.DEFAULT } } info(llama): Loading Llama weights from ../zml++huggingface+Meta-Llama-3.1-8B-Instruct/model.safetensors.index.json... info(zml): Compiling llama.LlamaLM.forward with { Shape({b=1,s=1,u32}), Shape({b=1,u32}), meta.MapType(tensor.Tensor,shape.Shape).map(llama.KvCache){ .k = Shape({layer=32,b=1,k=256,h=8!,hd=128,bf16}), .v = Shape({layer=32,b=1,k=256,h=8!,hd=128,bf16}), .layer_index = Shape({u32}) }, meta.MapType(tensor.Tensor,shape.Shape).map(tensor.Tensor.Rng){ ._state = Shape({2,u64}), .algorithm = stablehlo.RngAlgorithm.Type.DEFAULT } } info(zml/module): Wrote MLIR to /tmp/zml/llama/rocm/llama.LlamaLM.forward_f9e6ea06e80ef7b2/module.mlir info(zml/module): Wrote MLIR to /tmp/zml/llama/rocm/llama.LlamaLM.forward_59c4dcdc5bf68be0/module.mlir info(zml/module): Loaded pre-compiled module from /tmp/zml/llama/rocm/llama.LlamaLM.forward_f9e6ea06e80ef7b2/module.pjrt info(zml/module): Loaded pre-compiled module from /tmp/zml/llama/rocm/llama.LlamaLM.forward_59c4dcdc5bf68be0/module.pjrt info(zml/module): Compilation took 1.290s info(llama): ✅ Loaded weights in 7.289s info(llama): ✅ Compiled model in 7.29s info(llama): Creating KvCache info(llama): Loading tokenizer from ../zml++huggingface+Meta-Llama-3.1-8B-Instruct/tokenizer.json info(llama): Loaded tokenizer from ../zml++huggingface+Meta-Llama-3.1-8B-Instruct/tokenizer.json [289.68ms] info(llama): ✅ Prompt: Once upon a time, E0125 15:09:19.613079 782419 pjrt_stream_executor_client.cc:3045] Execution of replica 0 failed: INVALID_ARGUMENT: Executable expected shape u32[1,256]{1,0} for argument 291 but got incompatible shape u32[256]{0} E0125 15:09:19.613276 782422 pjrt_stream_executor_client.cc:3045] Execution of replica 0 failed: INVALID_ARGUMENT: Executable expected shape u32[1,256]{1,0} for argument 291 but got incompatible shape u32[256]{0} E0125 15:09:19.613309 782416 pjrt_stream_executor_client.cc:3045] Execution of replica 0 failed: INVALID_ARGUMENT: Executable expected shape u32[1,256]{1,0} for argument 291 but got incompatible shape u32[256]{0} E0125 15:09:19.613365 782428 pjrt_stream_executor_client.cc:3045] Execution of replica 0 failed: INVALID_ARGUMENT: Executable expected shape u32[1,256]{1,0} for argument 291 but got incompatible shape u32[256]{0} E0125 15:09:19.613441 782431 pjrt_stream_executor_client.cc:3045] Execution of replica 0 failed: INVALID_ARGUMENT: Executable expected shape u32[1,256]{1,0} for argument 291 but got incompatible shape u32[256]{0} E0125 15:09:19.613448 782434 pjrt_stream_executor_client.cc:3045] Execution of replica 0 failed: INVALID_ARGUMENT: Executable expected shape u32[1,256]{1,0} for argument 291 but got incompatible shape u32[256]{0} E0125 15:09:19.613657 782425 pjrt_stream_executor_client.cc:3045] Execution of replica 0 failed: INVALID_ARGUMENT: Executable expected shape u32[1,256]{1,0} for argument 291 but got incompatible shape u32[256]{0} E0125 15:09:19.613712 782437 pjrt_stream_executor_client.cc:3045] Execution of replica 0 failed: INVALID_ARGUMENT: Executable expected shape u32[1,256]{1,0} for argument 291 but got incompatible shape u32[256]{0} error(pjrt): [PJRT_LoadedExecutable_Execute] Executable expected shape u32[1,256]{1,0} for argument 291 but got incompatible shape u32[256]{0}: while running replica 0 and partition 0 of a replicated computation (other replicas may have failed as well). thread 781957 panic: attempt to unwrap error /home/allison/.cache/bazel/_bazel_allison/13b37383329cb8e9be33fcec42a1459a/sandbox/linux-sandbox/2/execroot/_main/external/zml+/zml/exe.zig:0:0: 0x1608e2a in call (llama_lib) /home/allison/.cache/bazel/_bazel_allison/13b37383329cb8e9be33fcec42a1459a/sandbox/linux-sandbox/2/execroot/_main/llama/main.zig:93:57: 0x1607934 in generateText (llama_lib) /home/allison/.cache/bazel/_bazel_allison/13b37383329cb8e9be33fcec42a1459a/sandbox/linux-sandbox/2/execroot/_main/llama/main.zig:328:44: 0x160ea58 in asyncMain (llama_lib) /home/allison/.cache/bazel/_bazel_allison/13b37383329cb8e9be33fcec42a1459a/sandbox/linux-sandbox/2/execroot/_main/external/zml+/async/coro.zig:263:28: 0x15c6d6b in wrapfn (llama_lib) /home/allison/.cache/bazel/_bazel_allison/13b37383329cb8e9be33fcec42a1459a/sandbox/linux-sandbox/2/execroot/_main/external/zml+/async/coro.zig:164:31: 0x15c6d20 in runcoro (llama_lib) ???:?:?: 0xaaaaaaaaaaaaaaa9 in ??? (???) Unwind information for `???:0xaaaaaaaaaaaaaaa9` was not available, trace may be incomplete ``` </details> It looks like a simple enough bug where I could go fix it myself, but don't have time at this very moment to do that so I'm posting in case it's helpful. Also it may be less relevant once [the tokenizer changes](https://github.com/zml/zml/pull/158) get merged, but again just covering the bases while I'm here. I did a quick check [3 commits ago](https://github.com/zml/zml/commit/8394632b19fb7876ec13ec7fdfaafe4e3d2fbc90) to see if it was an easy blame but I had the same error on that one. Again, probably easy fix, but again covering bases. Gotta run now, cheers.
gwenzek commented 2025-01-26 09:36:54 +00:00 (Migrated from github.com)

Probably due to https://github.com/zml/zml/pull/162

@renerocksai

Probably due to https://github.com/zml/zml/pull/162 @renerocksai
steeve commented 2025-01-27 08:24:26 +00:00 (Migrated from github.com)

ccing @renerocksai

ccing @renerocksai
renerocksai commented 2025-01-27 14:24:59 +00:00 (Migrated from github.com)

Thx, @AdjectiveAllison, for bringing this up. I fixed it in #171, should be merged soon.

Thx, @AdjectiveAllison, for bringing this up. I fixed it in #171, should be merged soon.
Sign in to join this conversation.
No milestone
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
DKB/zml#170
No description provided.