ImportError causal_conv1d_cuda.cpython-310-x86_64-linux-gnu.so undefined symbol

[FloMru]

I am encountering a strange error, while using Mamba with torch 2.1.1+cu118 on Linux. In another environment with torch 1.13.1+cu116 the same code works fine.

ImportError: /home/user1/.conda/envs/2prod/lib/python3.10/site-packages/causal_conv1d_cuda.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZN3c104cuda20CUDACachingAllocator12recordStreamERKNS_7DataPtrENS0_10CUDAStreamE

Package Version

---

absl-py 1.4.0 aiohttp 3.9.1 aiosignal 1.3.1 annotated-types 0.6.0 antlr4-python3-runtime 4.9.3 anyio 3.7.1 appdirs 1.4.4 arrow 1.2.3 async-timeout 4.0.3 attrs 23.1.0 backoff 2.2.1 beautifulsoup4 4.12.2 binaryornot 0.4.4 biopython 1.81 blessed 1.19.1 boto3 1.26.106 botocore 1.29.106 Brotli 1.1.0 brotlipy 0.7.0 build 1.0.3 CacheControl 0.13.1 cachetools 5.3.0 causal-conv1d 1.0.0 certifi 2023.11.17 cffi 1.16.0 chardet 5.1.0 charset-normalizer 3.3.2 cleo 2.1.0 click 8.1.3 cmake 3.25.0 colorama 0.4.6 contourpy 1.0.7 cookiecutter 2.1.1 crashtest 0.4.1 croniter 1.4.1 cryptography 41.0.7 cycler 0.11.0 dataclasses 0.8 datasets 2.8.0 dateutils 0.6.12 deepdiff 6.7.1 deepspeed 0.12.4 dill 0.3.6 distlib 0.3.7 docker-pycreds 0.4.0 dulwich 0.21.7 einops 0.6.0 exceptiongroup 1.2.0 fastapi 0.104.1 fastjsonschema 2.19.0 filelock 3.13.1 fonttools 4.38.0 frozenlist 1.4.0 fsspec 2023.12.1 gitdb 4.0.10 GitPython 3.1.30 gmpy2 2.1.2 google-auth 2.17.1 google-auth-oauthlib 1.0.0 GPUtil 1.4.0 grpcio 1.53.0 h11 0.14.0 hjson 3.1.0 huggingface-hub 0.16.4 hydra-core 1.3.1 idna 3.6 importlib-metadata 7.0.0 inquirer 3.1.4 installer 0.7.0 itsdangerous 2.1.2 jaraco.classes 3.3.0 jeepney 0.8.0 Jinja2 3.1.2 jinja2-time 0.2.0 jmespath 1.0.1 joblib 1.2.0 keopscore 2.1.1 keyring 24.3.0 kiwisolver 1.4.4 lightning 2.1.2 lightning-cloud 0.5.57 lightning-utilities 0.10.0 mamba-ssm 1.0.1 Markdown 3.4.3 markdown-it-py 2.1.0 MarkupSafe 2.1.3 matplotlib 3.6.3 mdurl 0.1.2 more-itertools 10.1.0 mpmath 1.3.0 msgpack 1.0.7 multidict 6.0.4 multiprocess 0.70.14 munkres 1.1.4 networkx 3.2.1 ninja 1.11.1.1 numpy 1.26.2 oauthlib 3.2.2 omegaconf 2.3.0 opt-einsum 3.3.0 ordered-set 4.1.0 orjson 3.9.10 packaging 23.2 pandas 1.5.3 pathtools 0.1.2 patsy 0.5.3 pexpect 4.8.0 Pillow 9.4.0 pip 23.3.1 pkginfo 1.9.6 platformdirs 3.11.0 poetry 1.7.1 poetry-core 1.8.1 poetry-plugin-export 1.6.0 prettytable 3.9.0 prettyTables 1.1.5 protobuf 3.20.3 psutil 5.9.4 ptyprocess 0.7.0 py-cpuinfo 9.0.0 pyahocorasick 2.0.0 pyarrow 10.0.1 pyasn1 0.4.8 pyasn1-modules 0.2.8 pybedtools 0.9.0 pybind11 2.10.3 pycparser 2.21 pydantic 2.1.1 pydantic_core 2.4.0 Pygments 2.14.0 PyJWT 2.8.0 pykeops 2.1.1 pynvml 11.5.0 pyOpenSSL 23.3.0 pyparsing 3.0.9 pyproject_hooks 1.0.0 pysam 0.20.0 PySocks 1.7.1 python-dateutil 2.8.2 python-editor 1.0.4 python-multipart 0.0.6 python-slugify 8.0.1 pytorch-lightning 2.1.1 pytz 2022.7.1 PyYAML 6.0.1 rapidfuzz 3.5.2 readchar 4.0.5.dev0 regex 2022.10.31 requests 2.31.0 requests-oauthlib 1.3.1 requests-toolbelt 1.0.0 responses 0.18.0 rich 13.2.0 rsa 4.9 s3transfer 0.6.0 sacremoses 0.0.53 safetensors 0.3.3 scikit-learn 1.2.2 scipy 1.10.0 seaborn 0.13.0 SecretStorage 3.3.3 sentencepiece 0.1.97 sentry-sdk 1.14.0 seqeval 1.2.2 setproctitle 1.3.2 setuptools 68.2.2 shellingham 1.5.4 six 1.16.0 sklearn 0.0.post2 smmap 5.0.0 sniffio 1.3.0 soupsieve 2.5 starlette 0.27.0 starsessions 1.3.0 statsmodels 0.13.5 sympy 1.12 tensorboard 2.12.1 tensorboard-data-server 0.7.0 tensorboard-plugin-wit 1.8.1 tensorboardX 2.6 text-unidecode 1.3 threadpoolctl 3.1.0 tokenizers 0.13.3 tomli 2.0.1 tomlkit 0.12.3 torch 2.1.1+cu118 torchaudio 2.1.1+cu118 torchmetrics 1.2.1 torchvision 0.16.1+cu118 tqdm 4.66.1 traitlets 5.14.0 transformers 4.33.3 triton 2.1.0 trove-classifiers 2023.11.29 types-python-dateutil 2.8.19.14 typing_extensions 4.8.0 tzdata 2023.3 unicodedata2 15.1.0 urllib3 1.26.0 uvicorn 0.24.0.post1 virtualenv 20.25.0 wandb 0.13.9 wcwidth 0.2.12 websocket-client 1.7.0 websockets 12.0 Werkzeug 2.2.3 wheel 0.42.0 xxhash 3.2.0 yarl 1.9.3 zipp 3.17.0

I tried setting up the environment several times, and it would be great if I could use torch > 2.0.

Thanks for having a look!

[GeneZC]

Any suggestions on this till now?

[hrbigelow]

Hi @FloMru,

Would this issue help you?

From what I understand, it's required that both torch and causal_conv1d use the same version of CUDA.

[pkpro]

I had the same issue, recompiled both causal-conv1d and mamba from source, then cp -r build/lib.linux-x86_64-cpython-310/* ../venv/lib/python3.10/site-packages/ for both packages. Mind the ../venv - replace with the path to your venv.

[sux1ngyu]

Any solution on this now?

[w32zhong]

any update?

[MarioPaps]

I think the issue has to do with cuda. Generally, Pytorch and causal_conv1d need to be compiled with the same version of cuda.

Something that worked for me on a Linux cluster is Pytorch 2.0.0 with cuda 11.8 (later versions should also work), causal conv1d 1.1.0 (not earlier versions) and mamba ssm 1.1.0.

If you check out the pip pages for the packages, make sure whatever verrsion you select for causalconv1d works for mamba-ssm

[jorenretel]

Hi, this seems very related to this issue: https://github.com/state-spaces/mamba/issues/169

Maybe also here, try to pip uninstall mamba-ssm and/or causal-conv1d and pip install them again making sure to prevent using the pip cache by adding the

--no-cache-dir

argument.

[heitorrapela]

Hi, this seems very related to this issue: #169

Maybe also here, try to pip uninstall mamba-ssm and/or causal-conv1d and pip install them again making sure to prevent using the pip cache by adding the

--no-cache-dir

argument.

Hi, I was having same problem with another repo, and followed what you said. Now it is working, thanks :)

[sux1ngyu]

Any solution on this now?

I find torch 2.2 will have this problem, after switch it to torch 2.1, I solve it

[uxhao-o]

I have been successfully run. Environment follows: cuda 11.8 python 3.10.13 pytorch 2.1.1 causal_conv1d 1.1.1 mamba-ssm 1.2.0.post1

pip install torch==2.1.1 torchvision==0.16.1 torchaudio==2.1.1 --index-url https://download.pytorch.org/whl/cu118
pip install causal_conv1d==1.1.1
pip install mamba-ssm==1.2.0.post1

[poult-lab]

Any solution on this now?

I find torch 2.2 will have this problem, after switch it to torch 2.1, I solve it

I was using the PyTorch 2.3 and it was not working. On the contrary, I downgrade to PyTorch 2.2. it is working currently. and below are the configurations of 'causal-conv1d' and 'mamba-ssm ' causal-conv1d 1.1.3.post1 pypi_0 pypi mamba-ssm 1.2.0.post1 pypi_0 pypi

[zhouziyu02]

Well, all you need to do is download the corresponding .whl from https://github.com/Dao-AILab/causal-conv1d/releases/tag/v1.0.0 and https://github.com/state-spaces/mamba/releases/tag/v1.0.1 based on your cuda, python as well as torch version and pip install the two .whl file.