Flash Attention 2.0 doesn't work: undefined symbol: _ZNK3c1010TensorImpl27throw_data_ptr_access_errorEv

[FrancescoSaverioZuppichini]

trafficstars

Hi there,

cloing the repo and running pytest tests/test_flash_attn.py gives

ImportError while importing test module '/root/flash-attention/tests/test_flash_attn.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib/python3.10/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/test_flash_attn.py:10: in <module>
    from flash_attn import flash_attn_func, flash_attn_kvpacked_func, flash_attn_qkvpacked_func
/usr/local/lib/python3.10/dist-packages/flash_attn/__init__.py:3: in <module>
    from flash_attn.flash_attn_interface import flash_attn_func
/usr/local/lib/python3.10/dist-packages/flash_attn/flash_attn_interface.py:4: in <module>
    import flash_attn_2_cuda as flash_attn_cuda
E   ImportError: /usr/local/lib/python3.10/dist-packages/flash_attn_2_cuda.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZNK3c1010TensorImpl27throw_data_ptr_access_errorEv

maybe somebody else have encountered this

Thanks a lot,

Fra

[FrancescoSaverioZuppichini]

Error persist also on a small stupid tests

import torch 
from torch import nn
from flash_attn import flash_attn_qkvpacked_func

qvk = nn.Linear(1, 3, 196, 512).cuda()

res = flash_attn_qkvpacked_func(qvk=qvk)
print(res.shape)
print(res)

Traceback (most recent call last):
  File "/root/main.py", line 3, in <module>
    from flash_attn import flash_attn_qkvpacked_func
  File "/usr/local/lib/python3.10/dist-packages/flash_attn/__init__.py", line 3, in <module>
    from flash_attn.flash_attn_interface import flash_attn_func
  File "/usr/local/lib/python3.10/dist-packages/flash_attn/flash_attn_interface.py", line 4, in <module>
    import flash_attn_2_cuda as flash_attn_cuda
ImportError: /usr/local/lib/python3.10/dist-packages/flash_attn_2_cuda.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZNK3c1010TensorImpl27throw_data_ptr_access_errorEv

[FrancescoSaverioZuppichini]

I think you compiled with 2.0.7 and I am using torch 2.1.0

[FrancescoSaverioZuppichini]

using

FROM nvcr.io/nvidia/pytorch:23.06-py3

so python 3.10, cuda 12.1 and torch 2.1 (see nvidia doc) and the correct wheel (https://github.com/Dao-AILab/flash-attention/releases/download/v2.0.7/flash_attn-2.0.7+cu121torch2.1cxx11abiTRUE-cp310-cp310-linux_x86_64.whl) still result in the same error

import flash_attn_2_cuda as flash_attn_cuda

ImportError: /usr/local/lib/python3.10/dist-packages/flash_attn_2_cuda.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZNK3c1010TensorImpl27throw_data_ptr_access_errorEv

[tridao]

Thanks for the report. I saw just this error on nvcr 23.06 as well. nvcr 23.07 should work, can you try? The error is due to pytorch interface changing between the version used in 23.06 and 23.07.

[FrancescoSaverioZuppichini]

to

Thanks a lot, will try as soon as I go back to my pc, out of curiosity could to give me more details about due to pytorch interface changing between the version used in 23.06 and 23.07?

[tridao]

Oh it's a low-level change in error handling. Pytorch added this "throw_data_ptr_access_error" function in May 11. nvcr 23.06 uses pytorch version on May 2 and nvcr 23.07 uses pytorch version on June 7.

[janelu9]

Try a cxx11abiFALSE version

[junjun3518]

Hi @tridao! I have some troubles with python3.10, so I need to use nvcr 23.04. Is their any suggestion to handle this import error with lower version of images?

[tridao]

You can compile from source with FLASH_ATTENTION_FORCE_BUILD=TRUE: FLASH_ATTENTION_FORCE_BUILD=TRUE pip install flash-attn.

[junjun3518]

Thank you for fast response! I will try it

[junjun3518]

Thank you It seems that it is working now

[dtnghla]

Thanks for the report. I saw just this error on nvcr 23.06 as well. nvcr 23.07 should work, can you try? The error is due to pytorch interface changing between the version used in 23.06 and 23.07.

Hi @tridao what is nvcr and how to change its version?

[Niyathi3011]

import flash_attn_2_cuda as flash_attn_cuda ImportError: /home/nallu002/anaconda3/envs/myenv/lib/python3.9/site-packages/flash_attn_2_cuda.cpython-39-x86_64-linux-gnu.so: undefined symbol: _ZNK3c106SymIntltEl

I am still facing this error. Chnaged the torch version from 2.1.2 to 2.1.0. And it is still not working.

[achangtv]

Also facing this error, in Databricks. Torch Version: 2.1.0+cu121 torch.cuda.get_device_capability(): (7,0) Compiled from source but that did not work for me.

ImportError: /databricks/python/lib/python3.10/site-packages/flash_attn_2_cuda.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZNK3c106SymIntltEl

[sleeper1023]

Try a cxx11abiFALSE version 试用版本 cxx11abiFALSE You can use this to solve the problem. Use pip to install the version package that matches yours directly. I use python 10, cuda11.8 torch 2.0, cxx11abiFALSE

https://github.com/Dao-AILab/flash-attention/releases/download/v2.5.2/flash_attn-2.5.2+cu118torch2.0cxx11abiFALSE-cp310-cp310-linux_x86_64.whl

[jaanli]

Also facing this.

[LeonEricsson]

Also facing this issue with nvcr 24.01-py3

[tridao]

For nvcr 23.12 and 24.01 please use flash-attn 2.5.1.post1

[LeonEricsson]

For nvcr 23.12 and 24.01 please use flash-attn 2.5.1.post1

'preciate the immediate response 🙌🏼

[caoyang-sufe]

Try a cxx11abiFALSE version

Really helpful!

[kanseaveg]

Try a cxx11abiFALSE version 试用版本 cxx11abiFALSE You can use this to solve the problem. Use pip to install the version package that matches yours directly. I use python 10, cuda11.8 torch 2.0, cxx11abiFALSE

https://github.com/Dao-AILab/flash-attention/releases/download/v2.5.2/flash_attn-2.5.2+cu118torch2.0cxx11abiFALSE-cp310-cp310-linux_x86_64.whl

thank you. That's work for me.

[ponyzym]

Thanks for the report. I saw just this error on nvcr 23.06 as well. nvcr 23.07 should work, can you try? The error is due to pytorch interface changing between the version used in 23.06 and 23.07.

Hello, how can i use the version in nvcr 23.07, any details?

[yiyepiaoling0715]

You can compile from source with FLASH_ATTENTION_FORCE_BUILD=TRUE: FLASH_ATTENTION_FORCE_BUILD=TRUE pip install flash-attn.

is this genral useful for different nvcr versions? I a using "pytorch/pytorch:2.2.2-cuda12.1-cudnn8-devel" "# FROM nvcr.io/nvidia/pytorch:22.07-py3" "# FROM pytorch/pytorch:2.0.1-cuda11.7-cudnn8-devel "

[modelsplaid]

I got similar error. I think this error is caused by cuda version. I added: export CUDA_HOME=/usr/local/cuda export PATH="/usr/local/cuda/bin${PATH:+:${PATH}}" export LD_LIBRARY_PATH="/usr/local/cuda/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}"

to the end of ~/.bashrc and source ~/.bashrc then reinstall pytorch and flash attention. It works.

[Laz4rz]

Doing this:

Try a cxx11abiFALSE version 试用版本 cxx11abiFALSE You can use this to solve the problem. Use pip to install the version package that matches yours directly. I use python 10, cuda11.8 torch 2.0, cxx11abiFALSE

https://github.com/Dao-AILab/flash-attention/releases/download/v2.5.2/flash_attn-2.5.2+cu118torch2.0cxx11abiFALSE-cp310-cp310-linux_x86_64.whl

and also forcing:

pip install xformers==v0.0.22

Fixed the issue for me.

UPDATE: it didnt, more problems down the line with missing torch operations