undefined symbol: _ZN2at4_ops5zeros4callEN3c108ArrayRefINS2_6SymIntEEENS2_8optionalINS2_10ScalarTypeEEENS6_INS2_6LayoutEEENS6_INS2_6DeviceEEENS6_IbEE

[abhayjain0x]

[OriAlpha]

Oh yes i too have same issue

[BedirT]

same here

[gdippolito]

I got the same issue. In my case I'm using Pytorch 2.2.2 (compiled with ABI) and CUDA 11.8. I tried flash attention V2 from the wheel and compiling from source. I got this error in both cases. @tridao would you know by any chance if latest version of flash attention support pytorch 2.2.2 (note that at the moment this is the latest release)

[tridao]

I haven't tried with pytorch 2.2.2 but I don't see why compiling from source wouldn't work. The wheel may or may not be compatible.

[gdippolito]

Hi @tridao thanks for your response. I'm not too sure why it would fail. In my case I'm building flash-attention inside a docker container with the following commands:

ENV FLASH_ATTENTION_FORCE_BUILD=1
ENV FLASH_ATTENTION_FORCE_CXX11_ABI=1

RUN mkdir -p /build/flash_attention && cd /build/flash_attention \
  && git clone https://github.com/Dao-AILab/flash-attention -b 2.5.7 .

RUN cd /build/flash_attention \
  && python3 setup.py build bdist_wheel

The container where flash attention will be built has Pytorch 2.2.2 installed (With ABI). Pytorch is correctly detected during the setup.py`:

dist.fetch_build_eggs(dist.setup_requires)

torch.__version__  = 2.2.2

The build will complete fine:

running bdist_wheel
Guessing wheel URL:  https://github.com/Dao-AILab/flash-attention/releases/download/v2.5.7/flash_attn-2.5.7+cu118torch2.2cxx11abiTRUE-cp310-cp310-linux_x86_64.whl
Raw wheel path dist/flash_attn-2.5.7-cp310-cp310-linux_x86_64.whl

Let me know if I can help with other testing. It looks like there are a few people with the same issue recently (see https://github.com/Dao-AILab/flash-attention/issues/723, https://github.com/Dao-AILab/flash-attention/issues/836#issuecomment-1951450440, https://github.com/Dao-AILab/flash-attention/issues/854 and https://github.com/Dao-AILab/flash-attention/issues/821).

[tridao]

looks like it's still downloading the wheel? Can you try python3 setup.py install?

[gdippolito]

Thanks, I realised that ENV FLASH_ATTENTION_FORCE_BUILD=1 ENV FLASH_ATTENTION_FORCE_CXX11_ABI=1 were ignored because setup.py expect "TRUE" instead. This helped force rebuilding the wheel but the issue was something else.

In my case I had some python bindings that were using libtorch (the C++ pytorch library) but they were still using pointing to the older Pytorch version(2.1.2). Changing this to match the version used in Python did the trick.

Thanks for your help.

[chiragjn]

Might be unrelated but when using flash-attn 2.5.5 with torch 2.2.1 switching to py3.11 solved this issue for me.

[tridao]

We have new wheels (flash-attn 2.5.8) that should work with torch 2.2.2

[YikaPanic]

It used to work until I try to install vllm. I met the same issue.

[csukuangfj]

I just helped my colleague to fix this issue.

The reason is that flash attention 2.4.2 is using torch 2.2.0.dev20231106.

https://github.com/Dao-AILab/flash-attention/blob/1a2c3e8c25251fa30ebee074c27ecbf69c2bad2b/.github/workflows/publish.yml#L47

https://github.com/Dao-AILab/flash-attention/blob/1a2c3e8c25251fa30ebee074c27ecbf69c2bad2b/.github/workflows/publish.yml#L127

i.e., a nightly built version of pytorch.

So if you also use a nightly built version, e.g., https://download.pytorch.org/whl/nightly/cu121/torch-2.2.0.dev20231010%2Bcu121-cp39-cp39-linux_x86_64.whl

then you should get rid of the symbol not found issue.

---

torch was using c10::optional in its nightly version but the released version swithes to std::optional; that is why so many of you are having this issue.

[Bellocccc]

Thanks. Python 3.9.19，cuda12.2，torch2.3.0 flash_attn==2.5.8, it works.