0
Fork 0
mirror of https://github.com/zml/zml.git synced 2026-11-16 19:11:09 +00:00

Unable to run llama3.1 on RTX3090 #19

Closed
opened 2024-09-20 15:38:47 +00:00 by rockwotj · 15 comments
rockwotj commented 2024-09-20 15:38:47 +00:00 (Migrated from github.com)

Thanks for ZML - I'm pretty eager to play with it. I have successfully run llama3.1 on my CPU, but attempting to run on my GPU(s) causes some issues. The stacktrace I get:

F0920 14:59:22.055848 3011543 stream_executor_util.cc:504] Could not create RepeatBufferKernel: INTERNAL: Failed call to cudaGetFuncBySymbol: no kernel image i
s available for execution on the device
*** Check failure stack trace: ***
    @     0x7508497b8704  absl::lts_20230802::log_internal::LogMessage::SendToLog()
    @     0x7508497b8604  absl::lts_20230802::log_internal::LogMessage::Flush()
    @     0x7508497b8aa9  absl::lts_20230802::log_internal::LogMessageFatal::~LogMessageFatal()
    @     0x75084820834d  xla::primitive_util::FloatingPointTypeSwitch<>()
    @     0x7508482062e9  xla::gpu::InitializeBuffer()
    @     0x750845aabef5  xla::gpu::AutotunerUtil::CreateBuffer()
    @     0x750845aaaac7  xla::gpu::RedzoneBuffers::CreateInputs()
    @     0x750845aaa70f  xla::gpu::RedzoneBuffers::FromInstruction()
    @     0x750845a93cec  xla::gpu::(anonymous namespace)::GemmAutotuner::operator()()
    @     0x750845a93aeb  std::_Function_handler<>::_M_invoke()
    @     0x750845aae5d6  xla::gpu::AutotunerUtil::Autotune()
    @     0x750845a92e76  xla::gpu::GemmAlgorithmPicker::Run()
    @     0x750845a6d36f  xla::HloPassPipeline::RunHelper()
    @     0x750845a6a740  xla::HloPassPipeline::RunPassesInternal<>()
    @     0x750845a6a113  xla::HloPassPipeline::Run()
    @     0x7508429c122f  xla::gpu::GpuCompiler::OptimizeHloPostLayoutAssignment()
    @     0x7508429aa6f3  xla::gpu::NVPTXCompiler::OptimizeHloPostLayoutAssignment()
    @     0x7508429bb44d  xla::gpu::GpuCompiler::OptimizeHloModule()
    @     0x7508429c432b  xla::gpu::GpuCompiler::RunHloPasses()
    @     0x75084298e230  xla::Service::BuildExecutable()
    @     0x75084293e682  xla::LocalService::CompileExecutables()
    @     0x7508429314b4  xla::LocalClient::Compile()
    @     0x7508428d9770  xla::PjRtStreamExecutorClient::Compile()
    @     0x7508426cfda2  xla::StreamExecutorGpuClient::Compile()
    @     0x7508428da89a  xla::PjRtStreamExecutorClient::Compile()
    @     0x750842698818  std::__detail::__variant::__gen_vtable_impl<>::__visit_invoke()
    @     0x750842688488  pjrt::PJRT_Client_Compile()
    @          0x136d892  async.callGeneric__anon_18562.TaskT.run

Output of nvidia-smi:

$ nvidia-smi
Fri Sep 20 15:35:42 2024
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 560.35.03              Driver Version: 560.35.03      CUDA Version: 12.6     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 3090        Off |   00000000:21:00.0 Off |                  N/A |
|  0%   51C    P8             28W /  420W |       2MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA GeForce RTX 3090        Off |   00000000:49:00.0 Off |                  N/A |
|  0%   57C    P8             29W /  420W |       2MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

Is there something missing here? Or do I need newer drivers/cuda?

Thanks for ZML - I'm pretty eager to play with it. I have successfully run llama3.1 on my CPU, but attempting to run on my GPU(s) causes some issues. The stacktrace I get: ``` F0920 14:59:22.055848 3011543 stream_executor_util.cc:504] Could not create RepeatBufferKernel: INTERNAL: Failed call to cudaGetFuncBySymbol: no kernel image i s available for execution on the device *** Check failure stack trace: *** @ 0x7508497b8704 absl::lts_20230802::log_internal::LogMessage::SendToLog() @ 0x7508497b8604 absl::lts_20230802::log_internal::LogMessage::Flush() @ 0x7508497b8aa9 absl::lts_20230802::log_internal::LogMessageFatal::~LogMessageFatal() @ 0x75084820834d xla::primitive_util::FloatingPointTypeSwitch<>() @ 0x7508482062e9 xla::gpu::InitializeBuffer() @ 0x750845aabef5 xla::gpu::AutotunerUtil::CreateBuffer() @ 0x750845aaaac7 xla::gpu::RedzoneBuffers::CreateInputs() @ 0x750845aaa70f xla::gpu::RedzoneBuffers::FromInstruction() @ 0x750845a93cec xla::gpu::(anonymous namespace)::GemmAutotuner::operator()() @ 0x750845a93aeb std::_Function_handler<>::_M_invoke() @ 0x750845aae5d6 xla::gpu::AutotunerUtil::Autotune() @ 0x750845a92e76 xla::gpu::GemmAlgorithmPicker::Run() @ 0x750845a6d36f xla::HloPassPipeline::RunHelper() @ 0x750845a6a740 xla::HloPassPipeline::RunPassesInternal<>() @ 0x750845a6a113 xla::HloPassPipeline::Run() @ 0x7508429c122f xla::gpu::GpuCompiler::OptimizeHloPostLayoutAssignment() @ 0x7508429aa6f3 xla::gpu::NVPTXCompiler::OptimizeHloPostLayoutAssignment() @ 0x7508429bb44d xla::gpu::GpuCompiler::OptimizeHloModule() @ 0x7508429c432b xla::gpu::GpuCompiler::RunHloPasses() @ 0x75084298e230 xla::Service::BuildExecutable() @ 0x75084293e682 xla::LocalService::CompileExecutables() @ 0x7508429314b4 xla::LocalClient::Compile() @ 0x7508428d9770 xla::PjRtStreamExecutorClient::Compile() @ 0x7508426cfda2 xla::StreamExecutorGpuClient::Compile() @ 0x7508428da89a xla::PjRtStreamExecutorClient::Compile() @ 0x750842698818 std::__detail::__variant::__gen_vtable_impl<>::__visit_invoke() @ 0x750842688488 pjrt::PJRT_Client_Compile() @ 0x136d892 async.callGeneric__anon_18562.TaskT.run ``` Output of `nvidia-smi`: ``` $ nvidia-smi Fri Sep 20 15:35:42 2024 +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 560.35.03 Driver Version: 560.35.03 CUDA Version: 12.6 | |-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA GeForce RTX 3090 Off | 00000000:21:00.0 Off | N/A | | 0% 51C P8 28W / 420W | 2MiB / 24576MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ | 1 NVIDIA GeForce RTX 3090 Off | 00000000:49:00.0 Off | N/A | | 0% 57C P8 29W / 420W | 2MiB / 24576MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ ``` Is there something missing here? Or do I need newer drivers/cuda?
rockwotj commented 2024-09-20 15:39:59 +00:00 (Migrated from github.com)

Maybe this is related to https://github.com/zml/zml/issues/10?

Maybe this is related to https://github.com/zml/zml/issues/10?
steeve commented 2024-09-20 16:40:57 +00:00 (Migrated from github.com)

Interesting, it seems you're on the latest driver so probably not #10
Perhaps related to 3090? Let us find one and try.

Interesting, it seems you're on the latest driver so probably not #10 Perhaps related to 3090? Let us find one and try.
rockwotj commented 2024-09-21 00:41:02 +00:00 (Migrated from github.com)

Alternatively, is there vetted hardware? I am assuming I could rent a T4 in the cloud and that should work?

Alternatively, is there vetted hardware? I am assuming I could rent a T4 in the cloud and that should work?
steeve commented 2024-09-22 17:57:33 +00:00 (Migrated from github.com)

It should be any NVIDIA hardware, but I can tell you that we tested it with RTX 4090, A100 and H100

It should be any NVIDIA hardware, but I can tell you that we tested it with RTX 4090, A100 and H100
steeve commented 2024-09-22 21:15:07 +00:00 (Migrated from github.com)

Let me see if I can grab a 3090 somewhere

Let me see if I can grab a 3090 somewhere
steeve commented 2024-09-23 13:56:16 +00:00 (Migrated from github.com)

We think this might fix you: zml/pjrt-artifacts@f148eccdd1

We're gonna bump soon.

We think this might fix you: https://github.com/zml/pjrt-artifacts/commit/f148eccdd179b6c70020bc4feeaa17ba951525d8 We're gonna bump soon.
rockwotj commented 2024-09-23 15:05:13 +00:00 (Migrated from github.com)

Nice thanks. I'll give it a spin after #21 goes in

zml/xla@177464c630 (diff-d5b0d91612)

Nice thanks. I'll give it a spin after #21 goes in https://github.com/zml/xla/commit/177464c630b8acd2ba9795f81c0248bdaedfb3cf#diff-d5b0d91612cf29b3369c834c0e8bb36e368b422eb49697307f355bf9f022b90eR143
rockwotj commented 2024-09-24 15:24:48 +00:00 (Migrated from github.com)

Hrm, still getting issues on the tip of master:

info(llama): ✅ Prompt: Once upon a time, there was a little girl named Lily.

debug(llama): Tokenized Prompt { 128000, 9843, 5304, 264, 892, 11, 1070, 574, 264, 2697, 3828, 7086, 48390, 13 }
E0924 15:22:25.581933 3039929 pjrt_stream_executor_client.cc:3056] Execution of replica 0 failed: INTERNAL: Failed call to cudaGetFuncBySymbol: no kernel image is available for execution on the device
error(pjrt): [PJRT_LoadedExecutable_Execute] Failed call to cudaGetFuncBySymbol: no kernel image is available for execution on the device
thread 3039929 panic: attempt to unwrap error
/home/rockwood/.cache/bazel/_bazel_rockwood/820f45ca8d82020506d653ce665c1430/sandbox/linux-sandbox/1859/execroot/_main/external/zml~/zml/module.zig:0:0: 0x133fc32 in call (llama_lib)
/home/rockwood/.cache/bazel/_bazel_rockwood/820f45ca8d82020506d653ce665c1430/sandbox/linux-sandbox/1859/execroot/_main/llama/main.zig:65:62: 0x133d97d in generateText (llama_lib)
/home/rockwood/.cache/bazel/_bazel_rockwood/820f45ca8d82020506d653ce665c1430/sandbox/linux-sandbox/1859/execroot/_main/llama/main.zig:263:35: 0x12f33f6 in wrapfn (llama_lib)
/home/rockwood/.cache/bazel/_bazel_rockwood/820f45ca8d82020506d653ce665c1430/sandbox/linux-sandbox/1859/execroot/_main/external/zigcoro~/src/coro.zig:219:31: 0x12ee913 in runcoro (llama_lib)
???:?:?: 0xaaaaaaaaaaaaaaa9 in ??? (???)
Unwind information for `???:0xaaaaaaaaaaaaaaa9` was not available, trace may be incomplete
Hrm, still getting issues on the tip of master: ``` info(llama): ✅ Prompt: Once upon a time, there was a little girl named Lily. debug(llama): Tokenized Prompt { 128000, 9843, 5304, 264, 892, 11, 1070, 574, 264, 2697, 3828, 7086, 48390, 13 } E0924 15:22:25.581933 3039929 pjrt_stream_executor_client.cc:3056] Execution of replica 0 failed: INTERNAL: Failed call to cudaGetFuncBySymbol: no kernel image is available for execution on the device error(pjrt): [PJRT_LoadedExecutable_Execute] Failed call to cudaGetFuncBySymbol: no kernel image is available for execution on the device thread 3039929 panic: attempt to unwrap error /home/rockwood/.cache/bazel/_bazel_rockwood/820f45ca8d82020506d653ce665c1430/sandbox/linux-sandbox/1859/execroot/_main/external/zml~/zml/module.zig:0:0: 0x133fc32 in call (llama_lib) /home/rockwood/.cache/bazel/_bazel_rockwood/820f45ca8d82020506d653ce665c1430/sandbox/linux-sandbox/1859/execroot/_main/llama/main.zig:65:62: 0x133d97d in generateText (llama_lib) /home/rockwood/.cache/bazel/_bazel_rockwood/820f45ca8d82020506d653ce665c1430/sandbox/linux-sandbox/1859/execroot/_main/llama/main.zig:263:35: 0x12f33f6 in wrapfn (llama_lib) /home/rockwood/.cache/bazel/_bazel_rockwood/820f45ca8d82020506d653ce665c1430/sandbox/linux-sandbox/1859/execroot/_main/external/zigcoro~/src/coro.zig:219:31: 0x12ee913 in runcoro (llama_lib) ???:?:?: 0xaaaaaaaaaaaaaaa9 in ??? (???) Unwind information for `???:0xaaaaaaaaaaaaaaa9` was not available, trace may be incomplete ```
steeve commented 2024-09-24 21:48:20 +00:00 (Migrated from github.com)

Damn, we couldn't get some 3090 but I have one at home I'll bring it

Damn, we couldn't get some 3090 but I have one at home I'll bring it
rockwotj commented 2024-09-24 21:52:53 +00:00 (Migrated from github.com)

I appreciate the effort & hustle!

I appreciate the effort & hustle!
steren commented 2024-10-05 03:47:04 +00:00 (Migrated from github.com)

Same error on GTX 1080 in WSL running ./bazel.sh run -c opt //llama:TinyLlama-Stories-15M --@zml//runtimes:cuda=true

No problem when running mnist

F1004 20:45:26.065507   29683 stream_executor_util.cc:504] Could not create RepeatBufferKernel: INTERNAL: Failed call to cudaGetFuncBySymbol: no kernel image is available for execution on the device
*** Check failure stack trace: ***
    @     0x7f782aef79e4  absl::lts_20230802::log_internal::LogMessage::SendToLog()
    @     0x7f782aef78e4  absl::lts_20230802::log_internal::LogMessage::Flush()
    @     0x7f782aef7d89  absl::lts_20230802::log_internal::LogMessageFatal::~LogMessageFatal()
    @     0x7f7829944d3c  xla::gpu::InitializeTypedBuffer<>()
    @     0x7f7829940e02  xla::primitive_util::FloatingPointTypeSwitch<>()
    @     0x7f782993f279  xla::gpu::InitializeBuffer()
    @     0x7f782715d5f5  xla::gpu::AutotunerUtil::CreateBuffer()
    @     0x7f782715c1c7  xla::gpu::RedzoneBuffers::CreateInputs()
    @     0x7f782715be0f  xla::gpu::RedzoneBuffers::FromInstruction()
    @     0x7f782714528c  xla::gpu::(anonymous namespace)::GemmAutotuner::operator()()
    @     0x7f782714508b  std::_Function_handler<>::_M_invoke()
    @     0x7f782715fc16  xla::gpu::AutotunerUtil::Autotune()
    @     0x7f7827144416  xla::gpu::GemmAlgorithmPicker::Run()
    @     0x7f782711424f  xla::HloPassPipeline::RunHelper()
    @     0x7f7827111620  xla::HloPassPipeline::RunPassesInternal<>()
    @     0x7f7827110ff3  xla::HloPassPipeline::Run()
    @     0x7f782405a76d  xla::gpu::GpuCompiler::OptimizeHloPostLayoutAssignment()
    @     0x7f78240437c1  xla::gpu::NVPTXCompiler::OptimizeHloPostLayoutAssignment()
    @     0x7f7824055030  xla::gpu::GpuCompiler::OptimizeHloModule()
    @     0x7f782405e654  xla::gpu::GpuCompiler::RunHloPasses()
    @     0x7f7824030530  xla::Service::BuildExecutable()
    @     0x7f7823fe00d3  xla::LocalService::CompileExecutables()
    @     0x7f7823fd2f24  xla::LocalClient::Compile()
    @     0x7f7823f7af5e  xla::PjRtStreamExecutorClient::CompileInternal()
    @     0x7f7823f7c00a  xla::PjRtStreamExecutorClient::Compile()
    @     0x7f7823d39ff8  std::__detail::__variant::__gen_vtable_impl<>::__visit_invoke()
    @     0x7f7823d29d48  pjrt::PJRT_Client_Compile()
    @          0x1366e37  async.callGeneric__anon_16976.TaskT.run
 $ nvidia-smi
Fri Oct  4 20:47:39 2024
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 555.42.03              Driver Version: 555.85         CUDA Version: 12.5     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce GTX 1080        On  |   00000000:01:00.0  On |                  N/A |
| 31%   47C    P8             11W /  180W |     730MiB /   8192MiB |      1%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A        23      G   /Xwayland                                   N/A      |
+-----------------------------------------------------------------------------------------+
Same error on GTX 1080 in WSL running ` ./bazel.sh run -c opt //llama:TinyLlama-Stories-15M --@zml//runtimes:cuda=true` No problem when running mnist ``` F1004 20:45:26.065507 29683 stream_executor_util.cc:504] Could not create RepeatBufferKernel: INTERNAL: Failed call to cudaGetFuncBySymbol: no kernel image is available for execution on the device *** Check failure stack trace: *** @ 0x7f782aef79e4 absl::lts_20230802::log_internal::LogMessage::SendToLog() @ 0x7f782aef78e4 absl::lts_20230802::log_internal::LogMessage::Flush() @ 0x7f782aef7d89 absl::lts_20230802::log_internal::LogMessageFatal::~LogMessageFatal() @ 0x7f7829944d3c xla::gpu::InitializeTypedBuffer<>() @ 0x7f7829940e02 xla::primitive_util::FloatingPointTypeSwitch<>() @ 0x7f782993f279 xla::gpu::InitializeBuffer() @ 0x7f782715d5f5 xla::gpu::AutotunerUtil::CreateBuffer() @ 0x7f782715c1c7 xla::gpu::RedzoneBuffers::CreateInputs() @ 0x7f782715be0f xla::gpu::RedzoneBuffers::FromInstruction() @ 0x7f782714528c xla::gpu::(anonymous namespace)::GemmAutotuner::operator()() @ 0x7f782714508b std::_Function_handler<>::_M_invoke() @ 0x7f782715fc16 xla::gpu::AutotunerUtil::Autotune() @ 0x7f7827144416 xla::gpu::GemmAlgorithmPicker::Run() @ 0x7f782711424f xla::HloPassPipeline::RunHelper() @ 0x7f7827111620 xla::HloPassPipeline::RunPassesInternal<>() @ 0x7f7827110ff3 xla::HloPassPipeline::Run() @ 0x7f782405a76d xla::gpu::GpuCompiler::OptimizeHloPostLayoutAssignment() @ 0x7f78240437c1 xla::gpu::NVPTXCompiler::OptimizeHloPostLayoutAssignment() @ 0x7f7824055030 xla::gpu::GpuCompiler::OptimizeHloModule() @ 0x7f782405e654 xla::gpu::GpuCompiler::RunHloPasses() @ 0x7f7824030530 xla::Service::BuildExecutable() @ 0x7f7823fe00d3 xla::LocalService::CompileExecutables() @ 0x7f7823fd2f24 xla::LocalClient::Compile() @ 0x7f7823f7af5e xla::PjRtStreamExecutorClient::CompileInternal() @ 0x7f7823f7c00a xla::PjRtStreamExecutorClient::Compile() @ 0x7f7823d39ff8 std::__detail::__variant::__gen_vtable_impl<>::__visit_invoke() @ 0x7f7823d29d48 pjrt::PJRT_Client_Compile() @ 0x1366e37 async.callGeneric__anon_16976.TaskT.run ``` ``` $ nvidia-smi Fri Oct 4 20:47:39 2024 +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 555.42.03 Driver Version: 555.85 CUDA Version: 12.5 | |-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA GeForce GTX 1080 On | 00000000:01:00.0 On | N/A | | 31% 47C P8 11W / 180W | 730MiB / 8192MiB | 1% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ +-----------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=========================================================================================| | 0 N/A N/A 23 G /Xwayland N/A | +-----------------------------------------------------------------------------------------+ ```
allisoneer commented 2024-11-08 01:25:45 +00:00 (Migrated from github.com)

Very similar error on a virtual half of an A40. It seems to be happening on tinyllama and llama3.1 8b examples but not on openllama 3b.

Works as expected:

bazel run -c opt //llama:OpenLLaMA-3B        \
                --@zml//runtimes:cuda=true         \
                -- --prompt="Once upon a time,"
                ...
info(llama): ✅ Generated 256 tokens in 4.962e0s: 51.587tok/s

Doesn't:

//llama:TinyLlama-Stories-15M and //llama:Llama-3.1-8B-Instruct

Looks like I have libunwind errors but here it is just in case it's useful:

info(llama): ✅ Done loading weights
info(llama): ✅ Llama model loaded from ../zml~~huggingface~Meta-Llama-3.1-8B-Instruct/model.safetensors.index.json
info(llama): ✅ Compiled model in 4246 milliseconds!

info(llama): ✅ Prompt: Once upon a time,

debug(llama): Tokenized Prompt { 128000, 9843, 5304, 264, 892, 11 }
E1108 01:21:27.092124  104458 pjrt_stream_executor_client.cc:3056] Execution of replica 0 failed: INTERNAL: Failed call to cudaGetFuncBySymbol: no kernel image is available for execution on the device
error(pjrt): [PJRT_LoadedExecutable_Execute] Failed call to cudaGetFuncBySymbol: no kernel image is available for execution on the device
thread 104458 panic: attempt to unwrap error
/home/allison/.cache/bazel/_bazel_allison/36225b4c74aea3b4d2da3f021042a9ba/sandbox/linux-sandbox/1736/execroot/_main/external/zml~/zml/module.zig:0:0: 0x14cb309 in call (llama_lib)
/home/allison/.cache/bazel/_bazel_allison/36225b4c74aea3b4d2da3f021042a9ba/sandbox/linux-sandbox/1736/execroot/_main/llama/main.zig:63:62: 0x14c93f2 in generateText (llama_lib)
/home/allison/.cache/bazel/_bazel_allison/36225b4c74aea3b4d2da3f021042a9ba/sandbox/linux-sandbox/1736/execroot/_main/llama/main.zig:262:35: 0x14cf9de in asyncMain (llama_lib)
/home/allison/.cache/bazel/_bazel_allison/36225b4c74aea3b4d2da3f021042a9ba/sandbox/linux-sandbox/1736/execroot/_main/external/zml~/async/threaded.zig:92:9: 0x14d35a7 in main (llama_lib)
???:?:?: 0x789f69034e07 in ??? (libc.so.6)
Unwind information for `libc.so.6:0x789f69034e07` was not available, trace may be incomplete
❯ nvidia-smi
Fri Nov  8 01:20:30 2024
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.90.07              Driver Version: 550.90.07      CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA A40-24Q                 Off |   00000000:06:00.0 Off |                    0 |
| N/A   N/A    P8             N/A /  N/A  |       1MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+
Very similar error on a virtual half of an A40. It seems to be happening on tinyllama and llama3.1 8b examples but not on openllama 3b. Works as expected: ``` bazel run -c opt //llama:OpenLLaMA-3B \ --@zml//runtimes:cuda=true \ -- --prompt="Once upon a time," ... info(llama): ✅ Generated 256 tokens in 4.962e0s: 51.587tok/s ``` Doesn't: ``` //llama:TinyLlama-Stories-15M and //llama:Llama-3.1-8B-Instruct ``` Looks like I have libunwind errors but here it is just in case it's useful: ``` info(llama): ✅ Done loading weights info(llama): ✅ Llama model loaded from ../zml~~huggingface~Meta-Llama-3.1-8B-Instruct/model.safetensors.index.json info(llama): ✅ Compiled model in 4246 milliseconds! info(llama): ✅ Prompt: Once upon a time, debug(llama): Tokenized Prompt { 128000, 9843, 5304, 264, 892, 11 } E1108 01:21:27.092124 104458 pjrt_stream_executor_client.cc:3056] Execution of replica 0 failed: INTERNAL: Failed call to cudaGetFuncBySymbol: no kernel image is available for execution on the device error(pjrt): [PJRT_LoadedExecutable_Execute] Failed call to cudaGetFuncBySymbol: no kernel image is available for execution on the device thread 104458 panic: attempt to unwrap error /home/allison/.cache/bazel/_bazel_allison/36225b4c74aea3b4d2da3f021042a9ba/sandbox/linux-sandbox/1736/execroot/_main/external/zml~/zml/module.zig:0:0: 0x14cb309 in call (llama_lib) /home/allison/.cache/bazel/_bazel_allison/36225b4c74aea3b4d2da3f021042a9ba/sandbox/linux-sandbox/1736/execroot/_main/llama/main.zig:63:62: 0x14c93f2 in generateText (llama_lib) /home/allison/.cache/bazel/_bazel_allison/36225b4c74aea3b4d2da3f021042a9ba/sandbox/linux-sandbox/1736/execroot/_main/llama/main.zig:262:35: 0x14cf9de in asyncMain (llama_lib) /home/allison/.cache/bazel/_bazel_allison/36225b4c74aea3b4d2da3f021042a9ba/sandbox/linux-sandbox/1736/execroot/_main/external/zml~/async/threaded.zig:92:9: 0x14d35a7 in main (llama_lib) ???:?:?: 0x789f69034e07 in ??? (libc.so.6) Unwind information for `libc.so.6:0x789f69034e07` was not available, trace may be incomplete ``` ``` ❯ nvidia-smi Fri Nov 8 01:20:30 2024 +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 550.90.07 Driver Version: 550.90.07 CUDA Version: 12.4 | |-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA A40-24Q Off | 00000000:06:00.0 Off | 0 | | N/A N/A P8 N/A / N/A | 1MiB / 24576MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ +-----------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=========================================================================================| | No running processes found | +-----------------------------------------------------------------------------------------+ ```
allisoneer commented 2024-11-13 18:37:23 +00:00 (Migrated from github.com)

To add onto my troubleshooting:

Models that fail:

  • TinyLlama-15M (f32, 6/6 heads)
  • LLaMA 3.1 8B (bf16, 32/8 heads)
  • TinyLlama 1.1B (bf16, 32/4 heads)

Model that works:

  • OpenLLaMA 3B (f16, 32/32 heads)

I (think I) ruled out several potential causes:

  1. GQA (Group Query Attention) ruled out because:
  • TinyLlama-15M fails but doesn't use GQA (6/6 heads)
  • OpenLLaMA works but doesn't use GQA (32/32 heads)
  1. ROPE implementation ruled out because:
  • LLaMA 3.1 fails with sequential ROPE
  • OpenLLaMA works with sequential ROPE
  1. Data type emerges as the only consistent differentiator:
  • Working model uses f16
  • Failing models use bf16 or f32

My next troubleshooting step was going to be attempting to convert openllama completely to another data type(bf16 from f16) before compiling and seeing if I get the same failure. I had some issues getting that to work the way I wanted to, so I'm posting results as-is for now.

To add onto my troubleshooting: Models that fail: - TinyLlama-15M (f32, 6/6 heads) - LLaMA 3.1 8B (bf16, 32/8 heads) - TinyLlama 1.1B (bf16, 32/4 heads) Model that works: - OpenLLaMA 3B (f16, 32/32 heads) I (think I) ruled out several potential causes: 1. GQA (Group Query Attention) ruled out because: - TinyLlama-15M fails but doesn't use GQA (6/6 heads) - OpenLLaMA works but doesn't use GQA (32/32 heads) 2. ROPE implementation ruled out because: - LLaMA 3.1 fails with sequential ROPE - OpenLLaMA works with sequential ROPE 3. Data type emerges as the only consistent differentiator: - Working model uses f16 - Failing models use bf16 or f32 My next troubleshooting step was going to be attempting to convert openllama completely to another data type(bf16 from f16) before compiling and seeing if I get the same failure. I had some issues getting that to work the way I wanted to, so I'm posting results as-is for now.
steeve commented 2024-11-27 18:40:07 +00:00 (Migrated from github.com)

Hey folks, we have just landed a fix for Failed call to cudaGetFuncBySymbol: no kernel image is available for execution on the device, please try it !

That was a interesting journey :)

Hey folks, we have just landed a fix for `Failed call to cudaGetFuncBySymbol: no kernel image is available for execution on the device`, please try it ! That was a interesting journey :)
rockwotj commented 2024-11-29 01:48:28 +00:00 (Migrated from github.com)

Works for me! Thanks @steeve

bazel run -c opt //llama:OpenLLaMA-3B        \
          --@zml//runtimes:cuda=true         \
          -- --prompt="Once upon a time,"
info(llama):    LLama was compiled with builtin.OptimizeMode.ReleaseSafe
info(pjrt): Loaded library: libpjrt_cpu.so
info(pjrt): Loaded library: libpjrt_cuda.so
info(llama): Model file: ../zml~~huggingface~OpenLM-Research-OpenLLaMA-3B/model.safetensors
info(llama): ✅ Parsed llama config: llama.LlamaOptions{ .gen_opts = nn.SamplingStrategy{ .topk = 2, .temperature = 1e0 }, .max_seq_len = 256, .num_heads = 32, .num_kv_heads = 32, .rms_norm_eps = 1e-5, .rope_opts = nn.RopeOpts{ .impl = nn.RopeOpts.Implementation.sequential, .freq_base = 1e4 } }
info(llama):    Loading tokenizer from ../zml~~huggingface~OpenLM-Research-OpenLLaMA-3B/tokenizer.pb
info(llama): ✅ Loaded tokenizer from ../zml~~huggingface~OpenLM-Research-OpenLLaMA-3B/tokenizer.pb
info(llama):    Loading Llama weights from ../zml~~huggingface~OpenLM-Research-OpenLLaMA-3B/model.safetensors...
2024-11-29 01:45:45.656988: I external/xla/xla/service/dump.cc:556] HloModule dump enabled with path prefix: , suffix: before_optimizations
info(llama): ✅ Loaded weights in 1163ms
info(llama): ✅ Compiled model in 4274ms
info(llama): ✅ Prompt: Once upon a time,
debug(llama): Tokenized Prompt { 1, 4612, 2619, 260, 647, 31844 }
there was a little girl who was born with a very rare condition. She was born with a very small heart and was unable to breathe without the help of a ventilator. She had to be fed through a feeding tube and had a tracheostomy to help with her breathing.
Her parents were told that she would not be able to live a long life. They were told that she would never walk, talk, or even smile.
But, this little girl was different.
She had a big heart and a big smile and she was determined to live her life to the fullest. She was determined to prove that she was not a “little” girl. She was determined to prove that she could do anything that she set her mind to do. She was determined to prove that she was not a “little” girl.
Works for me! Thanks @steeve ``` bazel run -c opt //llama:OpenLLaMA-3B \ --@zml//runtimes:cuda=true \ -- --prompt="Once upon a time," info(llama): LLama was compiled with builtin.OptimizeMode.ReleaseSafe info(pjrt): Loaded library: libpjrt_cpu.so info(pjrt): Loaded library: libpjrt_cuda.so info(llama): Model file: ../zml~~huggingface~OpenLM-Research-OpenLLaMA-3B/model.safetensors info(llama): ✅ Parsed llama config: llama.LlamaOptions{ .gen_opts = nn.SamplingStrategy{ .topk = 2, .temperature = 1e0 }, .max_seq_len = 256, .num_heads = 32, .num_kv_heads = 32, .rms_norm_eps = 1e-5, .rope_opts = nn.RopeOpts{ .impl = nn.RopeOpts.Implementation.sequential, .freq_base = 1e4 } } info(llama): Loading tokenizer from ../zml~~huggingface~OpenLM-Research-OpenLLaMA-3B/tokenizer.pb info(llama): ✅ Loaded tokenizer from ../zml~~huggingface~OpenLM-Research-OpenLLaMA-3B/tokenizer.pb info(llama): Loading Llama weights from ../zml~~huggingface~OpenLM-Research-OpenLLaMA-3B/model.safetensors... 2024-11-29 01:45:45.656988: I external/xla/xla/service/dump.cc:556] HloModule dump enabled with path prefix: , suffix: before_optimizations info(llama): ✅ Loaded weights in 1163ms info(llama): ✅ Compiled model in 4274ms info(llama): ✅ Prompt: Once upon a time, debug(llama): Tokenized Prompt { 1, 4612, 2619, 260, 647, 31844 } there was a little girl who was born with a very rare condition. She was born with a very small heart and was unable to breathe without the help of a ventilator. She had to be fed through a feeding tube and had a tracheostomy to help with her breathing. Her parents were told that she would not be able to live a long life. They were told that she would never walk, talk, or even smile. But, this little girl was different. She had a big heart and a big smile and she was determined to live her life to the fullest. She was determined to prove that she was not a “little” girl. She was determined to prove that she could do anything that she set her mind to do. She was determined to prove that she was not a “little” girl. ```
Sign in to join this conversation.
No milestone
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
DKB/zml#19
No description provided.