threestudio icon indicating copy to clipboard operation
threestudio copied to clipboard

CUDA Setup failed despite GPU being available

Open DuckersMcQuack opened this issue 1 year ago • 3 comments
trafficstars

Got this error. Using cuda 118 with torch 2.0.1

(treestudio) H:\Stable3D\threestudio-main>python launch.py --config configs/dreamfusion-if.yaml --train --gpu 0 system.prompt_processor.prompt="a zoomed out DSLR photo of a baby bunny sitting on top of a stack of pancakes"

===================================BUG REPORT=================================== Welcome to bitsandbytes. For bug reports, please run

python -m bitsandbytes

and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues

bin C:\Users\Duckers\AppData\Local\Programs\Python\Python310\lib\site-packages\bitsandbytes\libbitsandbytes_cpu.so C:\Users\Duckers\AppData\Local\Programs\Python\Python310\lib\site-packages\bitsandbytes\cuda_setup\main.py:145: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {WindowsPath('/Users/Duckers/miniconda3/envs/treestudio/lib'), WindowsPath('C')} warn(msg) C:\Users\Duckers\AppData\Local\Programs\Python\Python310\lib\site-packages\bitsandbytes\cuda_setup\main.py:145: UserWarning: C:\Users\Duckers\miniconda3\envs\treestudio did not contain ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] as expected! Searching further paths... warn(msg) CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths... C:\Users\Duckers\AppData\Local\Programs\Python\Python310\lib\site-packages\bitsandbytes\cuda_setup\main.py:145: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {WindowsPath('/usr/local/cuda/lib64')} warn(msg) CUDA SETUP: WARNING! libcuda.so not found! Do you have a CUDA driver installed? If you are on a cluster, make sure you are on a CUDA machine! C:\Users\Duckers\AppData\Local\Programs\Python\Python310\lib\site-packages\bitsandbytes\cuda_setup\main.py:145: UserWarning: WARNING: No libcudart.so found! Install CUDA or the cudatoolkit package (anaconda)! warn(msg) C:\Users\Duckers\AppData\Local\Programs\Python\Python310\lib\site-packages\bitsandbytes\cuda_setup\main.py:145: UserWarning: WARNING: No GPU detected! Check your CUDA paths. Proceeding to load CPU-only library... warn(msg) CUDA SETUP: Loading binary C:\Users\Duckers\AppData\Local\Programs\Python\Python310\lib\site-packages\bitsandbytes\libbitsandbytes_cpu.so... argument of type 'WindowsPath' is not iterable CUDA SETUP: Problem: The main issue seems to be that the main CUDA library was not detected. CUDA SETUP: Solution 1): Your paths are probably not up-to-date. You can update them via: sudo ldconfig. CUDA SETUP: Solution 2): If you do not have sudo rights, you can do the following: CUDA SETUP: Solution 2a): Find the cuda library via: find / -name libcuda.so 2>/dev/null CUDA SETUP: Solution 2b): Once the library is found add it to the LD_LIBRARY_PATH: export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:FOUND_PATH_FROM_2a CUDA SETUP: Solution 2c): For a permanent solution add the export from 2b into your .bashrc file, located at ~/.bashrc Traceback (most recent call last): File "C:\Users\Duckers\AppData\Local\Programs\Python\Python310\lib\site-packages\transformers\utils\import_utils.py", line 1146, in get_module return importlib.import_module("." + module_name, self.name) File "C:\Users\Duckers\AppData\Local\Programs\Python\Python310\lib\importlib_init.py", line 126, in import_module return _bootstrap.gcd_import(name[level:], package, level) File "", line 1050, in gcd_import File "", line 1027, in find_and_load File "", line 1006, in find_and_load_unlocked File "", line 688, in load_unlocked File "", line 883, in exec_module File "", line 241, in call_with_frames_removed File "C:\Users\Duckers\AppData\Local\Programs\Python\Python310\lib\site-packages\transformers\modeling_utils.py", line 83, in from accelerate import version as accelerate_version File "C:\Users\Duckers\AppData\Local\Programs\Python\Python310\lib\site-packages\accelerate_init.py", line 3, in from .accelerator import Accelerator File "C:\Users\Duckers\AppData\Local\Programs\Python\Python310\lib\site-packages\accelerate\accelerator.py", line 35, in from .checkpointing import load_accelerator_state, load_custom_state, save_accelerator_state, save_custom_state File "C:\Users\Duckers\AppData\Local\Programs\Python\Python310\lib\site-packages\accelerate\checkpointing.py", line 24, in from .utils import ( File "C:\Users\Duckers\AppData\Local\Programs\Python\Python310\lib\site-packages\accelerate\utils_init.py", line 131, in from .bnb import has_4bit_bnb_layers, load_and_quantize_model File "C:\Users\Duckers\AppData\Local\Programs\Python\Python310\lib\site-packages\accelerate\utils\bnb.py", line 42, in import bitsandbytes as bnb File "C:\Users\Duckers\AppData\Local\Programs\Python\Python310\lib\site-packages\bitsandbytes_init.py", line 6, in from . import cuda_setup, utils, research File "C:\Users\Duckers\AppData\Local\Programs\Python\Python310\lib\site-packages\bitsandbytes\research_init.py", line 1, in from . import nn File "C:\Users\Duckers\AppData\Local\Programs\Python\Python310\lib\site-packages\bitsandbytes\research\nn_init.py", line 1, in from .modules import LinearFP8Mixed, LinearFP8Global File "C:\Users\Duckers\AppData\Local\Programs\Python\Python310\lib\site-packages\bitsandbytes\research\nn\modules.py", line 8, in from bitsandbytes.optim import GlobalOptimManager File "C:\Users\Duckers\AppData\Local\Programs\Python\Python310\lib\site-packages\bitsandbytes\optim_init.py", line 6, in from bitsandbytes.cextension import COMPILED_WITH_CUDA File "C:\Users\Duckers\AppData\Local\Programs\Python\Python310\lib\site-packages\bitsandbytes\cextension.py", line 20, in raise RuntimeError(''' RuntimeError: CUDA Setup failed despite GPU being available. Please run the following command to get more information:

    python -m bitsandbytes

    Inspect the output of the command and see if you can locate CUDA libraries. You might need to add them
    to your LD_LIBRARY_PATH. If you suspect a bug, please take the information from python -m bitsandbytes
    and open an issue at: https://github.com/TimDettmers/bitsandbytes/issues

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "H:\Stable3D\threestudio-main\launch.py", line 301, in main(args, extras) File "H:\Stable3D\threestudio-main\launch.py", line 123, in main import pytorch_lightning as pl File "C:\Users\Duckers\AppData\Local\Programs\Python\Python310\lib\site-packages\pytorch_lightning_init_.py", line 26, in from pytorch_lightning.callbacks import Callback # noqa: E402 File "C:\Users\Duckers\AppData\Local\Programs\Python\Python310\lib\site-packages\pytorch_lightning\callbacks_init_.py", line 14, in from pytorch_lightning.callbacks.batch_size_finder import BatchSizeFinder File "C:\Users\Duckers\AppData\Local\Programs\Python\Python310\lib\site-packages\pytorch_lightning\callbacks\batch_size_finder.py", line 24, in from pytorch_lightning.callbacks.callback import Callback File "C:\Users\Duckers\AppData\Local\Programs\Python\Python310\lib\site-packages\pytorch_lightning\callbacks\callback.py", line 22, in from pytorch_lightning.utilities.types import STEP_OUTPUT File "C:\Users\Duckers\AppData\Local\Programs\Python\Python310\lib\site-packages\pytorch_lightning\utilities\types.py", line 40, in from torchmetrics import Metric File "C:\Users\Duckers\AppData\Local\Programs\Python\Python310\lib\site-packages\torchmetrics_init_.py", line 14, in from torchmetrics import functional # noqa: E402 File "C:\Users\Duckers\AppData\Local\Programs\Python\Python310\lib\site-packages\torchmetrics\functional_init_.py", line 120, in from torchmetrics.functional.text._deprecated import bleu_score as bleu_score File "C:\Users\Duckers\AppData\Local\Programs\Python\Python310\lib\site-packages\torchmetrics\functional\text_init.py", line 50, in from torchmetrics.functional.text.bert import bert_score # noqa: F401 File "C:\Users\Duckers\AppData\Local\Programs\Python\Python310\lib\site-packages\torchmetrics\functional\text\bert.py", line 23, in from torchmetrics.functional.text.helper_embedding_metric import ( File "C:\Users\Duckers\AppData\Local\Programs\Python\Python310\lib\site-packages\torchmetrics\functional\text\helper_embedding_metric.py", line 27, in from transformers import AutoModelForMaskedLM, AutoTokenizer, PreTrainedModel, PreTrainedTokenizerBase File "", line 1075, in _handle_fromlist File "C:\Users\Duckers\AppData\Local\Programs\Python\Python310\lib\site-packages\transformers\utils\import_utils.py", line 1136, in getattr module = self._get_module(self._class_to_module[name]) File "C:\Users\Duckers\AppData\Local\Programs\Python\Python310\lib\site-packages\transformers\utils\import_utils.py", line 1148, in _get_module raise RuntimeError( RuntimeError: Failed to import transformers.modeling_utils because of the following error (look up to see its traceback):

    CUDA Setup failed despite GPU being available. Please run the following command to get more information:

    python -m bitsandbytes

    Inspect the output of the command and see if you can locate CUDA libraries. You might need to add them
    to your LD_LIBRARY_PATH. If you suspect a bug, please take the information from python -m bitsandbytes
    and open an issue at: https://github.com/TimDettmers/bitsandbytes/issues

DuckersMcQuack avatar Dec 13 '23 23:12 DuckersMcQuack

Hi @DuckersMcQuack ,

You can reference to this issue.

DSaurus avatar Dec 14 '23 04:12 DSaurus

Thanks for responding :D But most of these answers there were for linux. Doesn't it work for straight in windows? Or do i need to run it in WSL "ubuntu on windows"?

Edit:

This is my path i've set for cuda.

SystemPropertiesAdvanced_3CqLLT8P1o

DuckersMcQuack avatar Dec 14 '23 14:12 DuckersMcQuack

I think the best way is to use WSL and you can refer to this issue. I have successfully run threestudio once in Windows using WSL.

DSaurus avatar Dec 14 '23 19:12 DSaurus