DexterousHands icon indicating copy to clipboard operation
DexterousHands copied to clipboard

RuntimeError: CUDA error: no kernel image is available for execution on the device

Open BunyaminEfe opened this issue 2 years ago • 16 comments

ex Why am I getting this error you see? python3 joint_monkey.py command works fine. isaacGYM works fine

BunyaminEfe avatar Oct 27 '22 22:10 BunyaminEfe

Hi @BunyaminEfe ,

This is always due to a mismatch between the pytorch version and your GPU cuda driver: 198411612-a766784f-85bf-4334-a0cf-926cd917a9da

According to the above error message, you can try to upgrade the pytorch version corresponding to the cuda driver.

Hope this can help you.

cypypccpy avatar Oct 28 '22 01:10 cypypccpy

sor2 English version of the fault text in the last line Parçalama arızası (çekirdek döküldü) = Shredding failure (core spilled) What is the solution of the line here?

BunyaminEfe avatar Oct 28 '22 11:10 BunyaminEfe

Hi @BunyaminEfe ,

This may be because of this line that reports the error:

MESA-INTEL: warning: Performance support disabled, consider sysctl dev.i915.perf_stream_paranoid=0

I found a post similar to your error on the official forum of Isaac gym, You might be able to refer to the answer in it: image

Anyway, since Isaac Gym is still in development, empirically it will have kinds of issues. Sometimes I also look for answers on the official forums I mentioned above, which are very helpful.

Hope this can help you.

cypypccpy avatar Oct 28 '22 11:10 cypypccpy

I looked at the place you mentioned at the moment, on the 470 installed computer, but now I'm upgrading to 520. I'm thinking of trying this shape.

BunyaminEfe avatar Oct 28 '22 11:10 BunyaminEfe

Upgrading version 520 was not a solution unfortunately @cypypccpy

BunyaminEfe avatar Oct 28 '22 11:10 BunyaminEfe

How about using this setting? It looks like the questioner above solved it this way:

Oh, I understand,
Using this setting :
export VK_ICD_FILENAMES=/usr/share/vulkan/icd.d/nvidia_icd.json

cypypccpy avatar Oct 28 '22 13:10 cypypccpy

i solved that problem but now i am facing this problem sor3

BunyaminEfe avatar Oct 28 '22 14:10 BunyaminEfe

Oh, we have also encountered this problem before. It seems that you are using Cloud Container, Our experience suggests that Isaac Gym doesn't seem to support this well, so we solved it by switching to Cloud Hosting.

cypypccpy avatar Oct 29 '22 10:10 cypypccpy

@cypypccpy I am not using Cloud Container. My computer is giving the error right now.

BunyaminEfe avatar Oct 29 '22 11:10 BunyaminEfe

Okay, sorry because we've only encountered this at Cloud Container before. I'm sure this is an environment configuration problem, can you show me the result of typing nvidia-smi on the command line? I hope to help you solve this problem. @BunyaminEfe

cypypccpy avatar Oct 29 '22 14:10 cypypccpy

How about these post? These errors are similar to yours, I don't know if it helps: https://forums.developer.nvidia.com/t/weird-bug/210419, https://forums.developer.nvidia.com/t/cuda-and-driver-version/209140, https://forums.developer.nvidia.com/t/ubuntu-20-04-libpython3-7m-so-1-0-error-segmentation-fault/163948

image

Or you can try to use this my cuda driver that works well, maybe 520 has some problems because it is too new: image

cypypccpy avatar Oct 29 '22 14:10 cypypccpy

I updated it to version 520 because it gave error on version 470 but it's still the same GPU = NVIDIA GeForce RTX 3050 Laptop GPU nvidia-smi

BunyaminEfe avatar Oct 30 '22 10:10 BunyaminEfe

Hi @BunyaminEfe ,

I'm so sorry that I couldn't solve your problem, but we and some other people use bi-dexhands well in different computer environments, so I think it may not be a problem with the code. Maybe you can go to the official NVIDIA forum and ask a question about this issue, or try another computer.

cypypccpy avatar Oct 30 '22 13:10 cypypccpy

Hi @cypypccpy , Thanks for your help. I wish you well in the project you are developing. :)

BunyaminEfe avatar Oct 30 '22 17:10 BunyaminEfe

Do we need to use cuda when configuring the environment for this project?

qinge0 avatar May 13 '23 08:05 qinge0

@qinge0 yes cuda is needed

BunyaminEfe avatar May 15 '23 16:05 BunyaminEfe