undefined symbol: _ZN3c104cuda9SetDeviceEi

[JoseGuilherme1904]

Sorry, please, I have this error: axolotl$ accelerate launch -m axolotl.cli.train examples/openllama-3b/lora.yml

... ... /.local/lib/python3.10/site-packages/flash_attn_2_cuda.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZN3c104cuda9SetDeviceEi

[Vector-Gaming]

I have the same error using auto-gptq (built from source)

Cuda 12.2 + python3.9

[zheyuye]

encouter the same error with

python 3.8 
transformers                 4.34.0
flash-attn                   2.3.2
torch                        2.1.0a0+fe05266
accelerate                   0.23.0

[tridao]

Probably an issue with pytorch version. Can you try pytorch 2.0.0 or 2.1.0?

[Kk1984up]

i have the same error, how to solving the error?

[lhl]

I ran into this issue and solved it by doing this:

pip install flash_attn -U --force-reinstall

For me this re-installed all the libraries with the versions that flash_attn wants...

[samikrc]

Reached here with the exact same error trying to use axolotl. I did a fresh virtual env install of axolotl, and still ended up with this error. Using the note from @lhl , this error went away, but I did end up getting a few installation errors:

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
datasets 2.15.0 requires fsspec[http]<=2023.10.0,>=2023.1.0, but you have fsspec 2023.12.2 which is incompatible.
gcsfs 2023.10.0 requires fsspec==2023.10.0, but you have fsspec 2023.12.2 which is incompatible.
s3fs 2023.10.0 requires fsspec==2023.10.0, but you have fsspec 2023.12.2 which is incompatible.
xformers 0.0.22 requires torch==2.0.1, but you have torch 2.1.1 which is incompatible.

But, FWIW, this particular error went away.

[linhx25]

I used the latest version: "pip install flash-attn --no-build-isolation" and it was solved.

[he-mark-qinglong]

I have this problem solved thoroughly. I know why this happened and have found the solutions. The reason it told us that "undefined symbol" of CUDA 9 is a compiller's linker error, which didn't found the approprate symbols from dynamic library(.so files) For this phenomenon, it is CUDA>12. didn't contain CUDA9 symbols(from source building, someone might abbandon the CUDA 9 sources to support the lower versions). To solve this, you need to find CUDA 11.8 python installer *whl. For example, if you found exllamav2_ext.cpython mismatch CUDA9 symbols, you need to install the newest one (normally could be enough). So, you need to change the requirements.txt, the counterpart of the exllamav2_ext.cpython: search this site https://github.com/turboderp/exllamav2/releases/download/ and through search cuda118, you can get this one to replace the old one in the requirments.txt (suppose you're using python==3.11)

https://github.com/turboderp/exllamav2/releases/download/v0.0.20/exllamav2-0.0.20+cu118-cp311-cp311-linux_x86_64.whl

then pip install -r requirments.txt

Also, for flash_rttn: you can replqace this as a example: https://github.com/Dao-AILab/flash-attention/releases/download/v2.5.8/flash_attn-2.5.8+cu118torch2.3cxx11abiFALSE-cp311-cp311-linux_x86_64.whl

[TJKlein]

I built the flash-attention from the source and then the problem was solved.

git clone https://github.com/Dao-AILab/flash-attention.git
cd flash-attention
MAX_JOBS=4 python setup.py install
pip install -e .

[songhat]

do not use version 2.5.9post1，use version 2.5.8, everything is fine

[1276440215]

I also used the latest version: "pip install flash-attn --no-build-isolation" and it was solved.

[rongkunxue]

I ran into this issue and solved it by doing this:

pip install flash_attn -U --force-reinstall

For me this re-installed all the libraries with the versions that flash_attn wants...

I also usde this ,to solve my problem