undefined symbol: cadam32bit_grad_fp32

Open robinsonmhj opened this issue 1 year ago • 3 comments

===================================BUG REPORT=================================== Welcome to bitsandbytes. For bug reports, please run

python -m bitsandbytes

and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues

bin /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cuda118_nocublaslt.so
CUDA SETUP: CUDA runtime path found: /opt/conda/envs/ray/lib/libcudart.so.11.0
CUDA SETUP: Highest compute capability among GPUs detected: 7.0
CUDA SETUP: Detected CUDA version 118
/opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: Compute capability < 7.5 detected! Only slow 8-bit matmul is supported for your GPU!
  warn(msg)
CUDA SETUP: Loading binary /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cuda118_nocublaslt.so...
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++ BUG REPORT INFORMATION ++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

+++++++++++++++++++ ANACONDA CUDA PATHS ++++++++++++++++++++
/opt/conda/envs/ray/lib/libicudata.so
/opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cuda110.so
/opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cuda110_nocublaslt.so
/opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cuda111.so
/opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cuda111_nocublaslt.so
/opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cuda114.so
/opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cuda114_nocublaslt.so
/opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cuda115.so
/opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cuda115_nocublaslt.so
/opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cuda117.so
/opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cuda117_nocublaslt.so
/opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cuda118.so
/opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cuda118_nocublaslt.so
/opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cuda120.so
/opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cuda120_nocublaslt.so
/opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cuda121.so
/opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cuda121_nocublaslt.so
/opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cuda122.so
/opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cuda122_nocublaslt.so
/opt/conda/envs/ray/lib/python3.10/site-packages/torch/lib/libc10_cuda.so
/opt/conda/envs/ray/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so
/opt/conda/envs/ray/lib/python3.10/site-packages/torch/lib/libtorch_cuda_linalg.so

++++++++++++++++++ /usr/local CUDA PATHS +++++++++++++++++++


+++++++++++++++ WORKING DIRECTORY CUDA PATHS +++++++++++++++


++++++++++++++++++ LD_LIBRARY CUDA PATHS +++++++++++++++++++
++++++++++++++++++ /usr/lib64 CUDA PATHS +++++++++++++++++++
/usr/lib64/libcuda.so
/usr/lib64/libicudata.so

++++++++++++++++++++++++++ OTHER +++++++++++++++++++++++++++
COMPILED_WITH_CUDA = True
COMPUTE_CAPABILITIES_PER_GPU = ['7.0']
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++++++++++++++++++++++ DEBUG INFO END ++++++++++++++++++++++
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Running a quick check that:
    + library is importable
    + CUDA function is callable


WARNING: Please be sure to sanitize sensible info from any such env vars!

SUCCESS!
Installation was successful!

stack trace

(TransformersTrainer pid=670) The `preprocessor` arg to Trainer is deprecated. Apply preprocessor transformations ahead of time by calling `preprocessor.transform(ds)`. Support for the preprocessor arg will be dropped in a future release.
(TransformersTrainer pid=670) Starting distributed worker processes: ['704 (10.68.161.60)']
(RayTrainWorker pid=704) Setting up process group for: env:// [rank=0, world_size=1]
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cextension.py:34: UserWarning: The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers, 8-bit multiplication, and GPU quantization are unavailable.
(RayTrainWorker pid=704)   warn("The installed version of bitsandbytes was compiled without GPU support. "
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: /opt/conda/envs/domino-jupyterlab did not contain ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] as expected! Searching further paths...
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: /usr/lib64: did not contain ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] as expected! Searching further paths...
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('80'), PosixPath('//172.20.235.50'), PosixPath('tcp')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('//172.20.236.248'), PosixPath('80'), PosixPath('tcp')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('mnt/llma_7b.ipynb')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('//172.20.207.154'), PosixPath('tcp'), PosixPath('8899')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('9000'), PosixPath('//172.20.155.140'), PosixPath('tcp')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('//172.20.215.206'), PosixPath('9000'), PosixPath('tcp')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('//172.20.136.68'), PosixPath('9000'), PosixPath('tcp')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('//localhost'), PosixPath('http'), PosixPath('8899')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('443'), PosixPath('//172.20.68.200'), PosixPath('tcp')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('//172.20.84.255'), PosixPath('7077'), PosixPath('tcp')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('//172.20.0.1'), PosixPath('443'), PosixPath('tcp')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('//172.20.101.218'), PosixPath('443'), PosixPath('tcp')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('//datasource-proxy.domino-platform.svc.cluster.local'), PosixPath('grpc'), PosixPath('8080')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('//172.20.226.99'), PosixPath('tcp'), PosixPath('8899')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('80'), PosixPath('//172.20.46.239'), PosixPath('tcp')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/opt/bitnami/spark/tmp/nss_group')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('//172.20.139.102'), PosixPath('tcp'), PosixPath('8899')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('7077'), PosixPath('//172.20.188.83'), PosixPath('tcp')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('9000'), PosixPath('//172.20.104.54'), PosixPath('tcp')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('//172.20.54.220'), PosixPath('7077'), PosixPath('tcp')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('80'), PosixPath('//172.20.221.53'), PosixPath('tcp')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('//172.20.200.30'), PosixPath('9000'), PosixPath('tcp')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('tcp'), PosixPath('7077'), PosixPath('//172.20.224.67')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('//172.20.212.43'), PosixPath('8080'), PosixPath('tcp')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('//172.20.212.43'), PosixPath('7077'), PosixPath('tcp')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('80'), PosixPath('//172.20.104.54'), PosixPath('tcp')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('//172.20.92.18'), PosixPath('8080'), PosixPath('tcp')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('//172.20.205.203'), PosixPath('tcp'), PosixPath('8899')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('80'), PosixPath('//172.20.89.153'), PosixPath('tcp')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('tcp'), PosixPath('//172.20.53.225'), PosixPath('8899')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('80'), PosixPath('//172.20.136.68'), PosixPath('tcp')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('//172.20.20.84'), PosixPath('8080'), PosixPath('tcp')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('9000'), PosixPath('//172.20.78.177'), PosixPath('tcp')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('80'), PosixPath('//172.20.128.155'), PosixPath('tcp')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('//localhost'), PosixPath('9000/executor/endSingleRun/656f32d00f7fa40071ce0659?exitCode=$code_at_exit'), PosixPath('${CURL} --silent --show-error -X POST http')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('//172.20.241.204'), PosixPath('7077'), PosixPath('tcp')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('80'), PosixPath('//172.20.72.220'), PosixPath('tcp')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('//172.20.89.153'), PosixPath('9000'), PosixPath('tcp')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('//172.20.5.170'), PosixPath('7077'), PosixPath('tcp')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('//172.20.136.89'), PosixPath('8080'), PosixPath('tcp')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('tcp'), PosixPath('//172.20.117.84'), PosixPath('8899')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('9000'), PosixPath('//172.20.82.45'), PosixPath('tcp')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('80'), PosixPath('//172.20.187.70'), PosixPath('tcp')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('//172.20.72.220'), PosixPath('9000'), PosixPath('tcp')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('80'), PosixPath('//172.20.123.123'), PosixPath('tcp')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('//172.20.86.14'), PosixPath('tcp'), PosixPath('8899')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('//172.20.117.174'), PosixPath('tcp'), PosixPath('8899')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('//172.20.92.18'), PosixPath('7077'), PosixPath('tcp')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('80'), PosixPath('//172.20.35.155'), PosixPath('tcp')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('tcp'), PosixPath('//172.20.93.107'), PosixPath('8899')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('80'), PosixPath('//172.20.97.208'), PosixPath('tcp')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('80'), PosixPath('//172.20.125.117'), PosixPath('tcp')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('//172.20.169.61'), PosixPath('9000'), PosixPath('tcp')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('80'), PosixPath('//nucleus-frontend.domino-platform.svc.cluster.local'), PosixPath('http')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('tcp'), PosixPath('8080'), PosixPath('//172.20.224.67')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('9000'), PosixPath('//172.20.46.239'), PosixPath('tcp')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('80'), PosixPath('//172.20.198.177'), PosixPath('tcp')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('80'), PosixPath('//172.20.109.119'), PosixPath('tcp')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('80'), PosixPath('//172.20.189.2'), PosixPath('tcp')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('//172.20.20.84'), PosixPath('7077'), PosixPath('tcp')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/repos')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('//172.20.171.222'), PosixPath('9000'), PosixPath('tcp')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('tcp'), PosixPath('//172.20.207.132'), PosixPath('8899')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('//172.20.104.181'), PosixPath('9000'), PosixPath('tcp')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('80'), PosixPath('//172.20.171.222'), PosixPath('tcp')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('module'), PosixPath('//matplotlib_inline.backend_inline')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('//172.20.241.204'), PosixPath('8080'), PosixPath('tcp')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('//172.20.54.220'), PosixPath('8080'), PosixPath('tcp')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('80'), PosixPath('//172.20.78.177'), PosixPath('tcp')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('//172.20.221.53'), PosixPath('9000'), PosixPath('tcp')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('America/New_York')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('7077'), PosixPath('//172.20.136.89'), PosixPath('tcp')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('80'), PosixPath('//172.20.104.181'), PosixPath('tcp')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('80'), PosixPath('//172.20.155.140'), PosixPath('tcp')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('80'), PosixPath('//172.20.215.206'), PosixPath('tcp')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('//172.20.84.255'), PosixPath('8080'), PosixPath('tcp')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('80'), PosixPath('//172.20.82.45'), PosixPath('tcp')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('//172.20.5.170'), PosixPath('8080'), PosixPath('tcp')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('tcp'), PosixPath('9000'), PosixPath('//172.20.53.28')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('9000'), PosixPath('//172.20.178.0'), PosixPath('tcp')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('9000'), PosixPath('//172.20.35.155'), PosixPath('tcp')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('//172.20.188.83'), PosixPath('8080'), PosixPath('tcp')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('80'), PosixPath('//172.20.169.61'), PosixPath('tcp')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('//172.20.128.155'), PosixPath('9000'), PosixPath('tcp')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('80'), PosixPath('tcp'), PosixPath('//172.20.53.28')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('80'), PosixPath('//172.20.178.0'), PosixPath('tcp')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('9000'), PosixPath('//172.20.235.50'), PosixPath('tcp')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('//172.20.236.248'), PosixPath('9000'), PosixPath('tcp')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('//localhost'), PosixPath('80'), PosixPath('http')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('80'), PosixPath('//datasource-proxy.domino-platform.svc.cluster.local'), PosixPath('http')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('//172.20.97.208'), PosixPath('9000'), PosixPath('tcp')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('9000'), PosixPath('//172.20.125.117'), PosixPath('tcp')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('//172.20.200.30'), PosixPath('80'), PosixPath('tcp')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('//172.20.187.70'), PosixPath('9000'), PosixPath('tcp')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('9000'), PosixPath('//172.20.123.123'), PosixPath('tcp')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/opt/bitnami/spark/tmp/nss_passwd')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('//172.20.198.177'), PosixPath('9000'), PosixPath('tcp')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('//172.20.109.119'), PosixPath('9000'), PosixPath('tcp')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('//172.20.189.2'), PosixPath('9000'), PosixPath('tcp')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/usr/local/cuda/lib64')}
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: No libcudart.so found! Install CUDA or the cudatoolkit package (anaconda)!
(RayTrainWorker pid=704)   warn(msg)
(RayTrainWorker pid=704) /opt/conda/envs/ray/lib/python3.10/site-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: Compute capability < 7.5 detected! Only slow 8-bit matmul is supported for your GPU!

Dec 05 '23 15:12 robinsonmhj

Hi @robinsonmhj Thanks for the issue, this is likely a CUDA setup issue but not sure, can you make sure the stack trace is properly formatted?

Dec 18 '23 13:12 younesbelkada

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Jan 11 '24 15:01 github-actions[bot]

Any update on this? Is there a restriction on torch version?

Getting the same with the cpu version :

libbitsandbytes_cpu.so: undefined symbol: cadam32bit_grad_fp32

EDIT: this seems to be unsignificant in the end, the cextension tries to load the symbol no matter the case (cpu or gpu): https://github.com/TimDettmers/bitsandbytes/blob/487632450733a1c638e690d54a36920a61918b49/bitsandbytes/cextension.py#L25

Mar 19 '24 18:03 ccoulombe

bitsandbytes bitsandbytes copied to clipboard

undefined symbol: cadam32bit_grad_fp32

and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues

bitsandbytes
bitsandbytes copied to clipboard