0
Fork 0
mirror of https://github.com/zml/zml.git synced 2026-11-16 19:11:09 +00:00

Mnist failing on master #137

Closed
opened 2024-12-27 17:07:41 +00:00 by allisoneer · 1 comment
allisoneer commented 2024-12-27 17:07:41 +00:00 (Migrated from github.com)

Failing to execute on cuda, cpu, and rocm on current master

ROCM Ubuntu mi300x:

❮ bazel run -c opt //mnist --@zml//runtimes:rocm=true --@zml//runtimes:cpu=false
INFO: Analyzed target //mnist:mnist (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target //mnist:mnist up-to-date:
  bazel-bin/mnist/mnist
INFO: Elapsed time: 0.377s, Critical Path: 0.00s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
INFO: Running command line: bazel-bin/mnist/mnist ../_main~_repo_rules~com_github_ggerganov_ggml_mnist/file/mnist.pt ../_main~_repo_rules~com_github_ggerganov_ggml_mnist_data/file/mnist.ylc

CPU Ubuntu mi300x:

❮ bazel run -c opt //mnist --@zml//runtimes:rocm=false --@zml//runtimes:cpu=true
INFO: Analyzed target //mnist:mnist (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target //mnist:mnist up-to-date:
  bazel-bin/mnist/mnist
INFO: Elapsed time: 0.352s, Critical Path: 0.00s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
INFO: Running command line: bazel-bin/mnist/mnist ../_main~_repo_rules~com_github_ggerganov_ggml_mnist/file/mnist.pt ../_main~_repo_rules~com_github_ggerganov_ggml_mnist_data/file/mnist.ylc
info(pjrt): Loaded library: libpjrt_cpu.so
info(zml/context): Available Platforms:
info(zml/context):   ✅ cpu (AUTO-SELECTED)
info(zml/context):        ◦ #0: cpu
info(zml/context):   •  cuda
info(zml/context):   •  rocm
info(zml/context):   •  tpu
info(zml/context):   •  neuron
info(mnist): Reading model shapes from PyTorch file ../_main~_repo_rules~com_github_ggerganov_ggml_mnist/file/mnist.pt...
info(mnist): Compiling model to MLIR....
info(zml): Compiling mnist.Mnist.forward with { Shape({28,28}, dtype=.u8) }
thread 424329 panic: attempt to use null value
/home/allison/.cache/bazel/_bazel_allison/36225b4c74aea3b4d2da3f021042a9ba/sandbox/linux-sandbox/12/execroot/_main/external/zml~/zml/module.zig:0:0: 0x147c8a9 in compileInternal__anon_13522 (mnist_lib)
/home/allison/.cache/bazel/_bazel_allison/36225b4c74aea3b4d2da3f021042a9ba/sandbox/linux-sandbox/12/execroot/_main/external/zml~/zml/exe.zig:66:51: 0x146f1d0 in wrapfn (mnist_lib)
/home/allison/.cache/bazel/_bazel_allison/36225b4c74aea3b4d2da3f021042a9ba/sandbox/linux-sandbox/12/execroot/_main/external/zigcoro~/src/coro.zig:219:31: 0x144ced3 in runcoro (mnist_lib)
???:?:?: 0xaaaaaaaaaaaaaaa9 in ??? (???)
Unwind information for `???:0xaaaaaaaaaaaaaaa9` was not available, trace may be incomplete

Cuda arch linux l40s:

❮ bazel run -c opt //mnist --@zml//runtimes:cuda=true --@zml//runtimes:cpu=false
INFO: Analyzed target //mnist:mnist (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target //mnist:mnist up-to-date:
  bazel-bin/mnist/mnist
INFO: Elapsed time: 0.319s, Critical Path: 0.00s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
INFO: Running command line: bazel-bin/mnist/mnist ../_main~_repo_rules~com_github_ggerganov_ggml_mnist/file/mnist.pt ../_main~_repo_rules~com_github_ggerganov_ggml_mnist_data/file/mnist.ylc
info(pjrt): Loaded library: libpjrt_cuda.so
info(zml/context): Available Platforms:
info(zml/context):   •  cpu
info(zml/context):   ✅ cuda (AUTO-SELECTED)
info(zml/context):        ◦ #0: NVIDIA L40S
info(zml/context):        ◦ #1: NVIDIA L40S
info(zml/context):        ◦ #2: NVIDIA L40S
info(zml/context):        ◦ #3: NVIDIA L40S
info(zml/context):        ◦ #4: NVIDIA L40S
info(zml/context):        ◦ #5: NVIDIA L40S
info(zml/context):        ◦ #6: NVIDIA L40S
info(zml/context):        ◦ #7: NVIDIA L40S
info(zml/context):   •  rocm
info(zml/context):   •  tpu
info(zml/context):   •  neuron
info(mnist): Reading model shapes from PyTorch file ../_main~_repo_rules~com_github_ggerganov_ggml_mnist/file/mnist.pt...
info(mnist): Compiling model to MLIR....
info(zml): Compiling mnist.Mnist.forward with { Shape({28,28}, dtype=.u8) }
thread 453336 panic: attempt to use null value
/home/allison/.cache/bazel/_bazel_allison/36225b4c74aea3b4d2da3f021042a9ba/sandbox/linux-sandbox/2/execroot/_main/external/zml~/zml/module.zig:0:0: 0x14832a9 in compileInternal__anon_13522 (mnist_lib)
/home/allison/.cache/bazel/_bazel_allison/36225b4c74aea3b4d2da3f021042a9ba/sandbox/linux-sandbox/2/execroot/_main/external/zml~/zml/exe.zig:66:51: 0x1475bd0 in wrapfn (mnist_lib)
/home/allison/.cache/bazel/_bazel_allison/36225b4c74aea3b4d2da3f021042a9ba/sandbox/linux-sandbox/2/execroot/_main/external/zigcoro~/src/coro.zig:219:31: 0x14538d3 in runcoro (mnist_lib)
???:?:?: 0xaaaaaaaaaaaaaaa9 in ??? (???)
Unwind information for `???:0xaaaaaaaaaaaaaaa9` was not available, trace may be incomplete
Failing to execute on cuda, cpu, and rocm on [current master](https://github.com/zml/zml/commit/cf135bd4281c75110fa33501ac87f216c7c3edb2) ROCM Ubuntu mi300x: ```bash ❮ bazel run -c opt //mnist --@zml//runtimes:rocm=true --@zml//runtimes:cpu=false INFO: Analyzed target //mnist:mnist (0 packages loaded, 0 targets configured). INFO: Found 1 target... Target //mnist:mnist up-to-date: bazel-bin/mnist/mnist INFO: Elapsed time: 0.377s, Critical Path: 0.00s INFO: 1 process: 1 internal. INFO: Build completed successfully, 1 total action INFO: Running command line: bazel-bin/mnist/mnist ../_main~_repo_rules~com_github_ggerganov_ggml_mnist/file/mnist.pt ../_main~_repo_rules~com_github_ggerganov_ggml_mnist_data/file/mnist.ylc ``` CPU Ubuntu mi300x: ```bash ❮ bazel run -c opt //mnist --@zml//runtimes:rocm=false --@zml//runtimes:cpu=true INFO: Analyzed target //mnist:mnist (0 packages loaded, 0 targets configured). INFO: Found 1 target... Target //mnist:mnist up-to-date: bazel-bin/mnist/mnist INFO: Elapsed time: 0.352s, Critical Path: 0.00s INFO: 1 process: 1 internal. INFO: Build completed successfully, 1 total action INFO: Running command line: bazel-bin/mnist/mnist ../_main~_repo_rules~com_github_ggerganov_ggml_mnist/file/mnist.pt ../_main~_repo_rules~com_github_ggerganov_ggml_mnist_data/file/mnist.ylc info(pjrt): Loaded library: libpjrt_cpu.so info(zml/context): Available Platforms: info(zml/context): ✅ cpu (AUTO-SELECTED) info(zml/context): ◦ #0: cpu info(zml/context): • cuda info(zml/context): • rocm info(zml/context): • tpu info(zml/context): • neuron info(mnist): Reading model shapes from PyTorch file ../_main~_repo_rules~com_github_ggerganov_ggml_mnist/file/mnist.pt... info(mnist): Compiling model to MLIR.... info(zml): Compiling mnist.Mnist.forward with { Shape({28,28}, dtype=.u8) } thread 424329 panic: attempt to use null value /home/allison/.cache/bazel/_bazel_allison/36225b4c74aea3b4d2da3f021042a9ba/sandbox/linux-sandbox/12/execroot/_main/external/zml~/zml/module.zig:0:0: 0x147c8a9 in compileInternal__anon_13522 (mnist_lib) /home/allison/.cache/bazel/_bazel_allison/36225b4c74aea3b4d2da3f021042a9ba/sandbox/linux-sandbox/12/execroot/_main/external/zml~/zml/exe.zig:66:51: 0x146f1d0 in wrapfn (mnist_lib) /home/allison/.cache/bazel/_bazel_allison/36225b4c74aea3b4d2da3f021042a9ba/sandbox/linux-sandbox/12/execroot/_main/external/zigcoro~/src/coro.zig:219:31: 0x144ced3 in runcoro (mnist_lib) ???:?:?: 0xaaaaaaaaaaaaaaa9 in ??? (???) Unwind information for `???:0xaaaaaaaaaaaaaaa9` was not available, trace may be incomplete ``` Cuda arch linux l40s: ```bash ❮ bazel run -c opt //mnist --@zml//runtimes:cuda=true --@zml//runtimes:cpu=false INFO: Analyzed target //mnist:mnist (0 packages loaded, 0 targets configured). INFO: Found 1 target... Target //mnist:mnist up-to-date: bazel-bin/mnist/mnist INFO: Elapsed time: 0.319s, Critical Path: 0.00s INFO: 1 process: 1 internal. INFO: Build completed successfully, 1 total action INFO: Running command line: bazel-bin/mnist/mnist ../_main~_repo_rules~com_github_ggerganov_ggml_mnist/file/mnist.pt ../_main~_repo_rules~com_github_ggerganov_ggml_mnist_data/file/mnist.ylc info(pjrt): Loaded library: libpjrt_cuda.so info(zml/context): Available Platforms: info(zml/context): • cpu info(zml/context): ✅ cuda (AUTO-SELECTED) info(zml/context): ◦ #0: NVIDIA L40S info(zml/context): ◦ #1: NVIDIA L40S info(zml/context): ◦ #2: NVIDIA L40S info(zml/context): ◦ #3: NVIDIA L40S info(zml/context): ◦ #4: NVIDIA L40S info(zml/context): ◦ #5: NVIDIA L40S info(zml/context): ◦ #6: NVIDIA L40S info(zml/context): ◦ #7: NVIDIA L40S info(zml/context): • rocm info(zml/context): • tpu info(zml/context): • neuron info(mnist): Reading model shapes from PyTorch file ../_main~_repo_rules~com_github_ggerganov_ggml_mnist/file/mnist.pt... info(mnist): Compiling model to MLIR.... info(zml): Compiling mnist.Mnist.forward with { Shape({28,28}, dtype=.u8) } thread 453336 panic: attempt to use null value /home/allison/.cache/bazel/_bazel_allison/36225b4c74aea3b4d2da3f021042a9ba/sandbox/linux-sandbox/2/execroot/_main/external/zml~/zml/module.zig:0:0: 0x14832a9 in compileInternal__anon_13522 (mnist_lib) /home/allison/.cache/bazel/_bazel_allison/36225b4c74aea3b4d2da3f021042a9ba/sandbox/linux-sandbox/2/execroot/_main/external/zml~/zml/exe.zig:66:51: 0x1475bd0 in wrapfn (mnist_lib) /home/allison/.cache/bazel/_bazel_allison/36225b4c74aea3b4d2da3f021042a9ba/sandbox/linux-sandbox/2/execroot/_main/external/zigcoro~/src/coro.zig:219:31: 0x14538d3 in runcoro (mnist_lib) ???:?:?: 0xaaaaaaaaaaaaaaa9 in ??? (???) Unwind information for `???:0xaaaaaaaaaaaaaaa9` was not available, trace may be incomplete ```
deependujha commented 2024-12-30 18:24:46 +00:00 (Migrated from github.com)

simple_layer also fails (mac m1)

Screenshot 2024-12-30 at 11 51 59 PM
`simple_layer` also fails (mac m1) <img width="764" alt="Screenshot 2024-12-30 at 11 51 59 PM" src="https://github.com/user-attachments/assets/784b6507-57e2-4dd2-a48b-f9af2d2feee7" />
Sign in to join this conversation.
No milestone
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
DKB/zml#137
No description provided.