Running successfully but not on my 1090TI GPU

[cloudibis223]

I would like to express my appreciation for the excellent work you have done with this project. I admire your use of the Vicuna-7B model and InstructorEmbeddings to enhance performance and privacy.

While using your software, I have encountered an issue related to hardware compatibility. Specifically, my GPU, a 1080TI, appears to not be working with the project. While your project functions perfectly the CPU, I've been unsuccessful in making it work on my 1080TI hardware. I suspect this may be due to some nuances related to this specific GPU model or the CUDA libraries.

While the Python script does run and completes without any errors, I am able the expected results when querying documents. The exact symptoms are that my GPU use is zero, and generating answers takes more than an hour sometimes.

Here are my system specifications for reference:

Operating System: Windows 10 Python Version: Python 3.10.9 CUDA Version: nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2023 NVIDIA Corporation Built on Mon_Apr__3_17:36:15_Pacific_Daylight_Time_2023 Cuda compilation tools, release 12.1, V12.1.105 Build cuda_12.1.r12.1/compiler.32688072_0

NVIDIA Driver Version: 531.14 DCH running on an NVIDIA GeForce GTX 1080 TI (11GB VRAM)

I've tried the following troubleshooting action because I suspect this is where the issue is:

1. Uncommenting the following lines: model = LlamaForCausalLM.from_pretrained(model_id, load_in_8bit=True, # set these options if your GPU supports them! device_map=1, #'auto', torch_dtype=torch.float16, low_cpu_mem_usage=True )

Ingesting works well, however running the main program gives the following error:

python run_localGPT.py load INSTRUCTOR_Transformer max_seq_length 512 Using embedded DuckDB with persistence: data will be stored in: C:\Users\username\localGPT loading model setting tokenizer loading LlamaForCausalLM.from_pretrained

===================================BUG REPORT=================================== Welcome to bitsandbytes. For bug reports, please run

python -m bitsandbytes

and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues

bin C:\Users\username\AppData\Local\Programs\Python\Python310\lib\site-packages\bitsandbytes\libbitsandbytes_cpu.so False CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching in backup paths... C:\Users\username\AppData\Local\Programs\Python\Python310\lib\site-packages\bitsandbytes\cuda_setup\main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {WindowsPath('/usr/local/cuda/lib64')} warn(msg) CUDA SETUP: WARNING! libcuda.so not found! Do you have a CUDA driver installed? If you are on a cluster, make sure you are on a CUDA machine! C:\Users\username\AppData\Local\Programs\Python\Python310\lib\site-packages\bitsandbytes\cuda_setup\main.py:149: UserWarning: WARNING: No libcudart.so found! Install CUDA or the cudatoolkit package (anaconda)! warn(msg) C:\Users\username\AppData\Local\Programs\Python\Python310\lib\site-packages\bitsandbytes\cuda_setup\main.py:149: UserWarning: WARNING: No GPU detected! Check your CUDA paths. Proceeding to load CPU-only library... warn(msg) CUDA SETUP: Loading binary C:\Users\username\AppData\Local\Programs\Python\Python310\lib\site-packages\bitsandbytes\libbitsandbytes_cpu.so... argument of type 'WindowsPath' is not iterable CUDA SETUP: Problem: The main issue seems to be that the main CUDA library was not detected. CUDA SETUP: Solution 1): Your paths are probably not up-to-date. You can update them via: sudo ldconfig. CUDA SETUP: Solution 2): If you do not have sudo rights, you can do the following: CUDA SETUP: Solution 2a): Find the cuda library via: find / -name libcuda.so 2>/dev/null CUDA SETUP: Solution 2b): Once the library is found add it to the LD_LIBRARY_PATH: export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:FOUND_PATH_FROM_2a CUDA SETUP: Solution 2c): For a permanent solution add the export from 2b into your .bashrc file, located at ~/.bashrc Traceback (most recent call last): File "C:\Users\username\localGPT\run_localGPT.py", line 85, in main() File "C:\Users\username\localGPT\run_localGPT.py", line 58, in main llm = load_model() File "C:\Users\username\localGPT\run_localGPT.py", line 26, in load_model model = LlamaForCausalLM.from_pretrained(model_id, File "C:\Users\username\AppData\Local\Programs\Python\Python310\lib\site-packages\transformers\modeling_utils.py", line 2621, in from_pretrained from .utils.bitsandbytes import get_keys_to_not_convert, replace_8bit_linear File "C:\Users\username\AppData\Local\Programs\Python\Python310\lib\site-packages\transformers\utils\bitsandbytes.py", line 9, in import bitsandbytes as bnb File "C:\Users\username\AppData\Local\Programs\Python\Python310\lib\site-packages\bitsandbytes_init_.py", line 6, in from . import cuda_setup, utils, research File "C:\Users\username\AppData\Local\Programs\Python\Python310\lib\site-packages\bitsandbytes\research_init_.py", line 1, in from . import nn File "C:\Users\username\AppData\Local\Programs\Python\Python310\lib\site-packages\bitsandbytes\research\nn_init_.py", line 1, in from .modules import LinearFP8Mixed, LinearFP8Global File "C:\Users\username\AppData\Local\Programs\Python\Python310\lib\site-packages\bitsandbytes\research\nn\modules.py", line 8, in from bitsandbytes.optim import GlobalOptimManager File "C:\Users\username\AppData\Local\Programs\Python\Python310\lib\site-packages\bitsandbytes\optim_init_.py", line 6, in from bitsandbytes.cextension import COMPILED_WITH_CUDA File "C:\Users\username\AppData\Local\Programs\Python\Python310\lib\site-packages\bitsandbytes\cextension.py", line 20, in raise RuntimeError(''' RuntimeError: CUDA Setup failed despite GPU being available. Please run the following command to get more information:

    python -m bitsandbytes

    Inspect the output of the command and see if you can locate CUDA libraries. You might need to add them
    to your LD_LIBRARY_PATH. If you suspect a bug, please take the information from python -m bitsandbytes
    and open an issue at: https://github.com/TimDettmers/bitsandbytes/issues

Some further information:

nvcc --version

nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2023 NVIDIA Corporation Built on Mon_Apr__3_17:36:15_Pacific_Daylight_Time_2023 Cuda compilation tools, release 12.1, V12.1.105 Build cuda_12.1.r12.1/compiler.32688072_0

nvidia-smi

Mon May 29 12:21:08 2023 +---------------------------------------------------------------------------------------+ | NVIDIA-SMI 531.14 Driver Version: 531.14 CUDA Version: 12.1 | |-----------------------------------------+----------------------+----------------------+ | GPU Name TCC/WDDM | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+======================+======================| | 0 NVIDIA GeForce GTX 1080 Ti WDDM | 00000000:09:00.0 On | N/A | | 37% 63C P0 65W / 250W| 1585MiB / 11264MiB | 1% Default | | | | N/A | +-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=======================================================================================| | 0 N/A N/A 808 C+G ...1.0_x64__8wekyb3d8bbwe\Video.UI.exe N/A | | 0 N/A N/A 5188 C+G ...es\Contents\Windows\Illustrator.exe N/A | | 0 N/A N/A 6068 C+G ...s\mattermost-desktop\Mattermost.exe N/A | | 0 N/A N/A 9736 C+G ...ft Office\root\Office16\ONENOTE.EXE N/A | | 0 N/A N/A 10208 C+G C:\Windows\explorer.exe N/A | | 0 N/A N/A 11708 C+G ...2txyewy\StartMenuExperienceHost.exe N/A | | 0 N/A N/A 12364 C+G ....Search_cw5n1h2txyewy\SearchApp.exe N/A | | 0 N/A N/A 12428 C+G ...ows\CEPHtmlEngine\CEPHtmlEngine.exe N/A | | 0 N/A N/A 14004 C+G ...GeForce Experience\NVIDIA Share.exe N/A | | 0 N/A N/A 14012 C+G ...aam7r\AcrobatNotificationClient.exe N/A | | 0 N/A N/A 14204 C+G ...GeForce Experience\NVIDIA Share.exe N/A | | 0 N/A N/A 14976 C+G ...CBS_cw5n1h2txyewy\TextInputHost.exe N/A | | 0 N/A N/A 16428 C+G ...t5de4sc\Sepiro.WakeOnLan.WinApp.exe N/A | | 0 N/A N/A 19140 C+G ...l\Microsoft\Teams\current\Teams.exe N/A | | 0 N/A N/A 21224 C+G ...5n1h2txyewy\ShellExperienceHost.exe N/A | | 0 N/A N/A 29132 C+G ...ejd91yc\AdobeNotificationClient.exe N/A | | 0 N/A N/A 34480 C+G ...soft\EdgeWebView\msedgewebview2.exe N/A | | 0 N/A N/A 34948 C+G ....Search_cw5n1h2txyewy\SearchApp.exe N/A | | 0 N/A N/A 36888 C+G ...64__8wekyb3d8bbwe\CalculatorApp.exe N/A | | 0 N/A N/A 37084 C+G ...oogle\Chrome\Application\chrome.exe N/A | | 0 N/A N/A 40836 C+G ...soft Office\root\Office16\EXCEL.EXE N/A | | 0 N/A N/A 41816 C+G ...ekyb3d8bbwe\PhoneExperienceHost.exe N/A | | 0 N/A N/A 44436 C+G ...l\Microsoft\Teams\current\Teams.exe N/A | | 0 N/A N/A 44992 C+G ...t\SelfServicePlugin\SelfService.exe N/A | | 0 N/A N/A 47084 C+G ...61.0_x64__8wekyb3d8bbwe\GameBar.exe N/A | | 0 N/A N/A 47880 C+G ...crosoft\Edge\Application\msedge.exe N/A | | 0 N/A N/A 53188 C+G ...on\113.0.1774.57\msedgewebview2.exe N/A | | 0 N/A N/A 53440 C+G ...on\113.0.1774.50\msedgewebview2.exe N/A | | 0 N/A N/A 57444 C+G ....0_x64__8wekyb3d8bbwe\HxOutlook.exe N/A | | 0 N/A N/A 57592 C+G ...siveControlPanel\SystemSettings.exe N/A | | 0 N/A N/A 58236 C+G ...0_x64__8wekyb3d8bbwe\HxAccounts.exe N/A | | 0 N/A N/A 59788 C+G ...yb3d8bbwe\Microsoft.Msn.Weather.exe N/A | | 0 N/A N/A 59824 C+G ...ft Office\root\Office16\WINWORD.EXE N/A | +---------------------------------------------------------------------------------------+

Thank you in advance for your help and support!

[ksylvan]

@cloudibis223 Is this maybe an instance of this issue? https://github.com/TimDettmers/bitsandbytes/issues/336 - Are you trying to run in WSL-Ubuntu?