kohya_ss icon indicating copy to clipboard operation
kohya_ss copied to clipboard

CUDA_SETUP: WARNING! libcudart.so not found in any environmental path.

Open mrcjh079 opened this issue 2 years ago • 20 comments

Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues For effortless bug reporting copy-paste your error into this form: https://docs.google.com/forms/d/e/1FAIpQLScPB8emS3Thkp66nvqwmjTEgxp8Y9ufuWTzFyr9kJ5AoI47dQ/viewform?usp=sf_link

CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching /usr/local/cuda/lib64... C:\Users\VIP\AppData\Local\Programs\Python\Python310\lib\site-packages\bitsandbytes\cuda_setup\paths.py:27: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {WindowsPath('/usr/local/cuda/lib64')} warn( WARNING: No libcudart.so found! Install CUDA or the cudatoolkit package (anaconda)! CUDA SETUP: Loading binary C:\Users\VIP\AppData\Local\Programs\Python\Python310\lib\site-packages\bitsandbytes\libbitsandbytes_cpu.so... Traceback (most recent call last): File "C:\kohya_ss\train_network.py", line 548, in train(args) File "C:\kohya_ss\train_network.py", line 207, in train import bitsandbytes as bnb File "C:\Users\VIP\AppData\Local\Programs\Python\Python310\lib\site-packages\bitsandbytes_init_.py", line 6, in from .autograd._functions import ( File "C:\Users\VIP\AppData\Local\Programs\Python\Python310\lib\site-packages\bitsandbytes\autograd_functions.py", line 5, in import bitsandbytes.functional as F File "C:\Users\VIP\AppData\Local\Programs\Python\Python310\lib\site-packages\bitsandbytes\functional.py", line 13, in from .cextension import COMPILED_WITH_CUDA, lib File "C:\Users\VIP\AppData\Local\Programs\Python\Python310\lib\site-packages\bitsandbytes\cextension.py", line 41, in lib = CUDALibrary_Singleton.get_instance().lib File "C:\Users\VIP\AppData\Local\Programs\Python\Python310\lib\site-packages\bitsandbytes\cextension.py", line 37, in get_instance cls.instance.initialize() File "C:\Users\VIP\AppData\Local\Programs\Python\Python310\lib\site-packages\bitsandbytes\cextension.py", line 31, in initialize self.lib = ct.cdll.LoadLibrary(binary_path) File "C:\Users\VIP\AppData\Local\Programs\Python\Python310\lib\ctypes_init.py", line 452, in LoadLibrary return self.dlltype(name) File "C:\Users\VIP\AppData\Local\Programs\Python\Python310\lib\ctypes_init.py", line 364, in init if '/' in name or '\' in name: TypeError: argument of type 'WindowsPath' is not iterable Traceback (most recent call last): File "C:\Users\VIP\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "C:\Users\VIP\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in run_code exec(code, run_globals) File "C:\Users\VIP\AppData\Local\Programs\Python\Python310\Scripts\accelerate.exe_main.py", line 7, in File "C:\Users\VIP\AppData\Local\Programs\Python\Python310\lib\site-packages\accelerate\commands\accelerate_cli.py", line 45, in main args.func(args) File "C:\Users\VIP\AppData\Local\Programs\Python\Python310\lib\site-packages\accelerate\commands\launch.py", line 1104, in launch_command simple_launcher(args) File "C:\Users\VIP\AppData\Local\Programs\Python\Python310\lib\site-packages\accelerate\commands\launch.py", line 567, in simple_launcher raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd) subprocess.CalledProcessError: Command '['C:\Users\VIP\AppData\Local\Programs\Python\Python310\python.exe', 'train_network.py', '--pretrained_model_name_or_path=runwayml/stable-diffusion-v1-5', '--train_data_dir=C:/lora train/faces/proccessed/maryhib/image', '--resolution=512,512', '--output_dir=C:/lora train/faces/proccessed/maryhib/model', '--logging_dir=C:/lora train/faces/proccessed/maryhib/log', '--network_alpha=128', '--save_model_as=safetensors', '--network_module=networks.lora', '--text_encoder_lr=5e-5', '--unet_lr=0.0001', '--network_dim=128', '--output_name=Addams', '--lr_scheduler_num_cycles=1', '--learning_rate=0.0001', '--lr_scheduler=constant', '--train_batch_size=2', '--max_train_steps=750', '--save_every_n_epochs=1', '--mixed_precision=bf16', '--save_precision=bf16', '--seed=1234', '--caption_extension=.txt', '--cache_latents', '--max_data_loader_n_workers=1', '--clip_skip=2', '--bucket_reso_steps=64', '--xformers', '--use_8bit_adam', '--bucket_no_upscale']' returned non-zero exit status 1.

mrcjh079 avatar Feb 12 '23 00:02 mrcjh079

i installed the cuda drivers

mrcjh079 avatar Feb 12 '23 00:02 mrcjh079

Do you have CUDA Toolkit installed? I had a similar problem and was able to fix it by installing it from the official NVIDIA page.

Thund3rPat avatar Feb 12 '23 13:02 Thund3rPat

it is installed idk whats wrong

mrcjh079 avatar Feb 12 '23 14:02 mrcjh079

the same error before and after installing it

mrcjh079 avatar Feb 12 '23 14:02 mrcjh079

Same here, even after installing the newest CUDA drivers

ftenrique avatar Feb 12 '23 20:02 ftenrique

I have tried installing CUDA 11.0, 11.6, 11.7, 12.0, all still showing the same error

hbl917070 avatar Feb 13 '23 21:02 hbl917070

What else would come to my mind would be to disable 8bit adam. However, this would lead to a very high memory consumption...

Thund3rPat avatar Feb 13 '23 22:02 Thund3rPat

I am having this exact same issue.

Spaceisprettybig avatar Feb 17 '23 12:02 Spaceisprettybig

What else would come to my mind would be to disable 8bit adam. However, this would lead to a very high memory consumption...

This worked for me!

SaltyBarnacles avatar Feb 23 '23 07:02 SaltyBarnacles

Finally, I found the cause of this problem. It is caused by the bitsandbytes library not supporting Windows. kohya_ss use this library to support 8bit adam to reduce video mem usage. There is a windows compatible version in the bitsandbytes_windows folder. Execute below script in Readme Installation maybe fix this problem.

cp .\bitsandbytes_windows\*.dll .\venv\Lib\site-packages\bitsandbytes\
cp .\bitsandbytes_windows\cextension.py .\venv\Lib\site-packages\bitsandbytes\cextension.py
cp .\bitsandbytes_windows\main.py .\venv\Lib\site-packages\bitsandbytes\cuda_setup\main.py

SlimeYummy avatar Feb 23 '23 14:02 SlimeYummy

I got this error because I forgot to first activate the virtual environment.

.\venv\Scripts\activate
python .\lora_gui.py

jndietz avatar Mar 02 '23 02:03 jndietz

All the solutions above don't work. Solutions from other opened threads don't work either.

Why even make the main instruction for linux when 90% of sd users run windows?

When I run this code, it gives numerous errors and when one error is solved, another pops out. It is a nightmarish experience. Esp having a nice gpu, Jesus.

mockinbirdy avatar Mar 03 '23 20:03 mockinbirdy

Also, why even redirect us to bitsandbytes when all the recent opened issues are silently ignored?

mockinbirdy avatar Mar 03 '23 21:03 mockinbirdy

Having the same problem

filliptm avatar Mar 03 '23 21:03 filliptm

found my problem. If you installed the repo with CMD it didn't copy those bitsandbytes files into the directories. Powershell as admin will do the thing. or you can copy them in manually as I did and it fixed the problem

cp .\bitsandbytes_windows*.dll .\venv\Lib\site-packages\bitsandbytes
cp .\bitsandbytes_windows\cextension.py .\venv\Lib\site-packages\bitsandbytes\cextension.py cp .\bitsandbytes_windows\main.py .\venv\Lib\site-packages\bitsandbytes\cuda_setup\main.py

filliptm avatar Mar 03 '23 22:03 filliptm

发现我的问题。如果您使用 CMD 安装了 repo,它不会将那些 bitsandbytes 文件复制到目录中。作为管理员的 Powershell 会做这件事。或者你可以像我一样手动复制它们并解决问题

cp .\bitsandbytes_windows*.dll .\venv\Lib\site-packages\bitsandbytes cp .\bitsandbytes_windows\cextension.py .\venv\Lib\site-packages\bitsandbytes\cextension.py cp .\bitsandbytes_windows\main.py 。 \venv\Lib\site-packages\bitsandbytes\cuda_setup\main.py

I have tried manually copying all the files in the .\kohya_ss\bitsandbytes_windows folder to the ".\kohya_ss\venv\Lib\site-packages\bitsandbytes" folder, but it still doesn’t solve the problem. What should I do for Windows users?

suwadaimyojin avatar Mar 07 '23 08:03 suwadaimyojin

found my problem. If you installed the repo with CMD it didn't copy those bitsandbytes files into the directories. Powershell as admin will do the thing. or you can copy them in manually as I did and it fixed the problem

cp .\bitsandbytes_windows*.dll .\venv\Lib\site-packages\bitsandbytes cp .\bitsandbytes_windows\cextension.py .\venv\Lib\site-packages\bitsandbytes\cextension.py cp .\bitsandbytes_windows\main.py .\venv\Lib\site-packages\bitsandbytes\cuda_setup\main.py

My problem is solved, must use Powershell to execute these commands, manual can not fix this problem

suwadaimyojin avatar Mar 07 '23 14:03 suwadaimyojin

found my problem. If you installed the repo with CMD it didn't copy those bitsandbytes files into the directories. Powershell as admin will do the thing. or you can copy them in manually as I did and it fixed the problem cp .\bitsandbytes_windows*.dll .\venv\Lib\site-packages\bitsandbytes cp .\bitsandbytes_windows\cextension.py .\venv\Lib\site-packages\bitsandbytes\cextension.py cp .\bitsandbytes_windows\main.py .\venv\Lib\site-packages\bitsandbytes\cuda_setup\main.py

My problem is solved, must use Powershell to execute these commands, manual can not fix this problem

Still get the same error

Tongzhou-Yu avatar Mar 08 '23 02:03 Tongzhou-Yu

You can indeed use explorer to copy them manually. Here are the files you need to copy:

Copy to: venv\Lib\site-packages\bitsandbytes\

    bitsandbytes_windows\cextension.py
    bitsandbytes_windows\libbitsandbytes_cpu.dll
    bitsandbytes_windows\libbitsandbytes_cuda116.dll

Copy to: venv\Lib\site-packages\bitsandbytes\cuda_setup\

    bitsandbytes_windows\main.py

jamesWalker55 avatar Mar 11 '23 18:03 jamesWalker55

需要将这个设置下: "use_8bit_adam": false,

yujun2019 avatar Mar 12 '23 13:03 yujun2019

需要将这个设置下: “use_8bit_adam”: false,

解决了

yanlang0123 avatar Apr 25 '23 04:04 yanlang0123

Why don't i have bitsandbytes_windows folder?

Eleven1202 avatar Jun 04 '23 08:06 Eleven1202

I directly installed using pip install bitsandbytes-windows and it worked after that

fordeeplearning1 avatar Jul 29 '23 11:07 fordeeplearning1

Any instruction for linux or WSL? Thanks.

bruce2233 avatar Aug 04 '23 14:08 bruce2233