segmentation fault when running python -m train pipeline=mnist model=s4

[liluo2]

When I was running with the command python -X faulthandler -m train pipeline=mnist model=s4, I found that there's a segmentation fault. I located the problematic code with:

File "/home/liluo/code/s4/src/models/hippo/hippo.py", line 78 in transition                                            
File "/home/liluo/code/s4/src/models/hippo/hippo.py", line 220 in nplr                                                 
File "/home/liluo/code/s4/src/models/sequence/kernels/dplr.py", line 157 in ssm                                        
File "/home/liluo/code/s4/src/models/sequence/kernels/dplr.py", line 176 in <listcomp>                                 
File "/home/liluo/code/s4/src/models/sequence/kernels/dplr.py", line 176 in combination
  File "/home/liluo/code/s4/src/models/sequence/kernels/ssm.py", line 173 in init_ssm_dplr
  File "/home/liluo/code/s4/src/models/sequence/kernels/ssm.py", line 479 in __init__
  File "/home/liluo/code/s4/src/models/sequence/kernels/fftconv.py", line 71 in __init__
  File "/home/liluo/code/s4/src/utils/config.py", line 101 in instantiate
  File "/home/liluo/code/s4/src/models/sequence/modules/s4block.py", line 109 in __init__
  File "/home/liluo/code/s4/src/utils/config.py", line 101 in instantiate
  File "/home/liluo/code/s4/src/models/sequence/backbones/block.py", line 48 in __init__
  File "/home/liluo/code/s4/src/models/sequence/backbones/model.py", line 84 in __init__
  File "/home/liluo/code/s4/src/utils/config.py", line 101 in instantiate
  File "/home/liluo/code/s4/train.py", line 171 in setup
  File "/home/liluo/code/s4/train.py", line 147 in __init__
  File "/home/liluo/code/s4/train.py", line 701 in train
  File "/home/liluo/code/s4/train.py", line 830 in main

It seems that A = T @ M @ np.linalg.inv(T) causes segment fault. I don't know how to solve this problem... I set ulimit -s 65536 but it's not working.

My environments:

#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main    defaults
_openmp_mutex             5.1                       1_gnu    defaults
_sysroot_linux-64_curr_repodata_hack 3                   haa98f57_10    defaults
absl-py                   2.1.0                    pypi_0    pypi
aiohttp                   3.9.3                    pypi_0    pypi
aiosignal                 1.3.1                    pypi_0    pypi
annotated-types           0.6.0                    pypi_0    pypi
antlr4-python3-runtime    4.9.3                    pypi_0    pypi
appdirs                   1.4.4                    pypi_0    pypi
asttokens                 2.4.1                    pypi_0    pypi
async-timeout             4.0.3                    pypi_0    pypi
attrs                     23.2.0                   pypi_0    pypi
binutils_impl_linux-64    2.35.1               h27ae35d_9    defaults
binutils_linux-64         2.35.1              h454624a_30    defaults
black                     23.7.0                   pypi_0    pypi
blas                      1.0                         mkl    defaults
brotli                    1.0.9                h5eee18b_7    defaults
brotli-bin                1.0.9                h5eee18b_7    defaults
bzip2                     1.0.8                h5eee18b_5    defaults
c-ares                    1.19.1               h5eee18b_0    anaconda
ca-certificates           2023.12.12           h06a4308_0    defaults
certifi                   2024.2.2                 pypi_0    pypi
cfgv                      3.4.0                    pypi_0    pypi
charset-normalizer        3.3.2                    pypi_0    pypi
click                     8.1.7                    pypi_0    pypi
cmake                     3.28.3                   pypi_0    pypi
comm                      0.2.2                    pypi_0    pypi
contourpy                 1.2.0            py39hdb19cb5_0    defaults
coverage                  7.4.3                    pypi_0    pypi
cycler                    0.11.0             pyhd3eb1b0_0    defaults
cyrus-sasl                2.1.28               h52b45da_1    defaults
datasets                  2.18.0                   pypi_0    pypi
dbus                      1.13.18              hb2f20db_0    defaults
debugpy                   1.8.1                    pypi_0    pypi
decorator                 5.1.1                    pypi_0    pypi
dill                      0.3.8                    pypi_0    pypi
distlib                   0.3.8                    pypi_0    pypi
docformatter              1.7.5                    pypi_0    pypi
docker-pycreds            0.4.0                    pypi_0    pypi
einops                    0.7.0                    pypi_0    pypi
exceptiongroup            1.2.0                    pypi_0    pypi
executing                 2.0.1                    pypi_0    pypi
expat                     2.5.0                h6a678d5_0    defaults
filelock                  3.13.1                   pypi_0    pypi
flake8                    7.0.0                    pypi_0    pypi
flake8-bugbear            24.2.6                   pypi_0    pypi
flake8-comprehensions     3.14.0                   pypi_0    pypi
fontconfig                2.14.1               h4c34cd2_2    defaults
fonttools                 4.25.0             pyhd3eb1b0_0    defaults
freetype                  2.12.1               h4a9f257_0    defaults
frozenlist                1.4.1                    pypi_0    pypi
fsspec                    2024.2.0                 pypi_0    pypi
gcc_impl_linux-64         9.3.0               h6df7d76_17    defaults
gcc_linux-64              9.3.0               h1ee779e_30    defaults
gitdb                     4.0.11                   pypi_0    pypi
gitpython                 3.1.42                   pypi_0    pypi
glib                      2.78.4               h6a678d5_0    defaults
glib-tools                2.78.4               h6a678d5_0    defaults
gluonts                   0.14.4                   pypi_0    pypi
grpcio                    1.62.1                   pypi_0    pypi
gst-plugins-base          1.14.1               h6a678d5_1    defaults
gstreamer                 1.14.1               h5eee18b_1    defaults
gxx_impl_linux-64         9.3.0               hbdd7822_17    defaults
gxx_linux-64              9.3.0               h7e70986_30    defaults
h5py                      3.9.0            py39he06866b_0    anaconda
hdf5                      1.12.1               h2b7332f_3    anaconda
huggingface-hub           0.21.4                   pypi_0    pypi
hydra-core                1.3.2                    pypi_0    pypi
icu                       73.1                 h6a678d5_0    defaults
identify                  2.5.35                   pypi_0    pypi
idna                      3.6                      pypi_0    pypi
importlib-metadata        7.0.2                    pypi_0    pypi
importlib_resources       6.1.1            py39h06a4308_1    defaults
intel-openmp              2023.1.0         hdb19cb5_46306    defaults
ipython                   8.18.1                   pypi_0    pypi
ipywidgets                8.1.2                    pypi_0    pypi
isort                     5.13.2                   pypi_0    pypi
jedi                      0.19.1                   pypi_0    pypi
jinja2                    3.1.3                    pypi_0    pypi
joblib                    1.2.0            py39h06a4308_0    defaults
jpeg                      9e                   h5eee18b_1    defaults
jupyterlab-widgets        3.0.10                   pypi_0    pypi
keopscore                 2.2.2                    pypi_0    pypi
kernel-headers_linux-64   3.10.0              h57e8cba_10    defaults
kiwisolver                1.4.4            py39h6a678d5_0    defaults
krb5                      1.20.1               h143b758_1    defaults
lcms2                     2.12                 h3be6417_0    defaults
ld_impl_linux-64          2.35.1               h7274673_9    defaults
lerc                      3.0                  h295c915_0    defaults
libbrotlicommon           1.0.9                h5eee18b_7    defaults
libbrotlidec              1.0.9                h5eee18b_7    defaults
libbrotlienc              1.0.9                h5eee18b_7    defaults
libclang                  14.0.6          default_hc6dbbc7_1    defaults
libclang13                14.0.6          default_he11475f_1    defaults
libcups                   2.4.2                h2d74bed_1    defaults
libcurl                   7.88.1               h251f7ec_2    anaconda
libdeflate                1.17                 h5eee18b_1    defaults
libedit                   3.1.20230828         h5eee18b_0    defaults
libev                     4.33                 h7f8727e_1    anaconda
libffi                    3.4.4                h6a678d5_0    defaults
libgcc-devel_linux-64     9.3.0               hb95220a_17    defaults
libgcc-ng                 11.2.0               h1234567_1    defaults
libgfortran-ng            11.2.0               h00389a5_1    defaults
libgfortran5              11.2.0               h1234567_1    defaults
libglib                   2.78.4               hdc74915_0    defaults
libgomp                   11.2.0               h1234567_1    defaults
libiconv                  1.16                 h7f8727e_2    defaults
libllvm14                 14.0.6               hdb19cb5_3    defaults
libnghttp2                1.57.0               h2d74bed_0    anaconda
libpng                    1.6.39               h5eee18b_0    defaults
libpq                     12.17                hdbd6064_0    defaults
libssh2                   1.10.0               hdbd6064_2    anaconda
libstdcxx-devel_linux-64  9.3.0               hf0c5c8d_17    defaults
libstdcxx-ng              11.2.0               h1234567_1    defaults
libtiff                   4.5.1                h6a678d5_0    defaults
libuuid                   1.41.5               h5eee18b_0    defaults
libwebp-base              1.3.2                h5eee18b_0    defaults
libxcb                    1.15                 h7f8727e_0    defaults
libxkbcommon              1.0.1                h5eee18b_1    defaults
libxml2                   2.10.4               hf1b16e4_1    defaults
lightning-utilities       0.10.1                   pypi_0    pypi
lz4-c                     1.9.4                h6a678d5_0    defaults
markdown                  3.5.2                    pypi_0    pypi
markdown-it-py            3.0.0                    pypi_0    pypi
markupsafe                2.1.5                    pypi_0    pypi
matplotlib                3.8.0            py39h06a4308_0    defaults
matplotlib-base           3.8.0            py39h1128e8f_0    defaults
matplotlib-inline         0.1.6                    pypi_0    pypi
mccabe                    0.7.0                    pypi_0    pypi
mdurl                     0.1.2                    pypi_0    pypi
mkl                       2023.1.0         h213fc3f_46344    defaults
mkl-service               2.4.0            py39h5eee18b_1    defaults
mkl_fft                   1.3.8            py39h5eee18b_0    defaults
mkl_random                1.2.4            py39hdb19cb5_0    defaults
mp-pde-solvers            1.0                       dev_0    <develop>
multidict                 6.0.5                    pypi_0    pypi
multiprocess              0.70.16                  pypi_0    pypi
munch                     4.0.0                    pypi_0    pypi
munkres                   1.1.4                      py_0    defaults
mypy-extensions           1.0.0                    pypi_0    pypi
mysql                     5.7.24               h721c034_2    defaults
ncurses                   6.4                  h6a678d5_0    defaults
nengo                     4.0.0                    pypi_0    pypi
nodeenv                   1.8.0                    pypi_0    pypi
numpy                     1.26.4           py39h5f9d8c6_0    defaults
numpy-base                1.26.4           py39hb5e798b_0    defaults
omegaconf                 2.3.0                    pypi_0    pypi
openjpeg                  2.4.0                h3ad879b_0    defaults
openssl                   3.0.13               h7f8727e_0    defaults
packaging                 23.1             py39h06a4308_0    defaults
pandas                    2.1.4                    pypi_0    pypi
parso                     0.8.3                    pypi_0    pypi
pathspec                  0.12.1                   pypi_0    pypi
pcre2                     10.42                hebb0a14_0    defaults
pexpect                   4.9.0                    pypi_0    pypi
pillow                    10.2.0           py39h5eee18b_0    defaults
pip                       23.3.1           py39h06a4308_0    defaults
platformdirs              4.2.0                    pypi_0    pypi
ply                       3.11             py39h06a4308_0    defaults
pre-commit                3.6.2                    pypi_0    pypi
prompt-toolkit            3.0.43                   pypi_0    pypi
protobuf                  4.25.3                   pypi_0    pypi
psutil                    5.9.8                    pypi_0    pypi
ptyprocess                0.7.0                    pypi_0    pypi
pure-eval                 0.2.2                    pypi_0    pypi
pyarrow                   15.0.1                   pypi_0    pypi
pyarrow-hotfix            0.6                      pypi_0    pypi
pybind11                  2.11.1                   pypi_0    pypi
pycodestyle               2.11.1                   pypi_0    pypi
pydantic                  2.6.4                    pypi_0    pypi
pydantic-core             2.16.3                   pypi_0    pypi
pyflakes                  3.2.0                    pypi_0    pypi
pyg-lib                   0.4.0+pt113cu116          pypi_0    pypi
pygments                  2.17.2                   pypi_0    pypi
pykeops                   2.2.2                    pypi_0    pypi
pyparsing                 3.0.9            py39h06a4308_0    defaults
pyqt                      5.15.10          py39h6a678d5_0    defaults
pyqt5-sip                 12.13.0          py39h5eee18b_0    defaults
python                    3.9.18               h955ad1f_0    defaults
python-dateutil           2.8.2              pyhd3eb1b0_0    defaults
pytorch-lightning         2.0.4                    pypi_0    pypi
pytz                      2024.1                   pypi_0    pypi
pyyaml                    6.0.1                    pypi_0    pypi
qt-main                   5.15.2              h53bd1ea_10    defaults
readline                  8.2                  h5eee18b_0    defaults
regex                     2023.12.25               pypi_0    pypi
requests                  2.31.0                   pypi_0    pypi
rich                      13.7.1                   pypi_0    pypi
safetensors               0.4.2                    pypi_0    pypi
scikit-learn              1.3.0            py39h1128e8f_1    defaults
scipy                     1.11.4           py39h5f9d8c6_0    defaults
seaborn                   0.13.2                   pypi_0    pypi
sentry-sdk                1.41.0                   pypi_0    pypi
setproctitle              1.3.3                    pypi_0    pypi
setuptools                68.2.2           py39h06a4308_0    defaults
sip                       6.7.12           py39h6a678d5_0    defaults
six                       1.16.0             pyhd3eb1b0_1    defaults
smmap                     5.0.1                    pypi_0    pypi
sqlite                    3.41.2               h5eee18b_0    defaults
stack-data                0.6.3                    pypi_0    pypi
structured-kernels        0.1.0                    pypi_0    pypi
sysroot_linux-64          2.17                h57e8cba_10    defaults
tbb                       2021.8.0             hdb19cb5_0    defaults
tensorboard               2.16.2                   pypi_0    pypi
tensorboard-data-server   0.7.2                    pypi_0    pypi
threadpoolctl             2.2.0              pyh0d69192_0    defaults
timm                      0.5.4                    pypi_0    pypi
tk                        8.6.12               h1ccaba5_0    defaults
tokenizers                0.15.2                   pypi_0    pypi
tomli                     2.0.1            py39h06a4308_0    defaults
toolz                     0.12.1                   pypi_0    pypi
torch                     1.13.1+cu116             pypi_0    pypi
torch-cluster             1.6.1+pt113cu116          pypi_0    pypi
torch-geometric           2.3.0                    pypi_0    pypi
torch-scatter             2.1.1+pt113cu116          pypi_0    pypi
torch-sparse              0.6.17+pt113cu116          pypi_0    pypi
torch-spline-conv         1.2.2+pt113cu116          pypi_0    pypi
torchaudio                0.13.1+cu116             pypi_0    pypi
torchmetrics              1.3.1                    pypi_0    pypi
torchtext                 0.14.1                   pypi_0    pypi
torchvision               0.14.1+cu116             pypi_0    pypi
tornado                   6.3.3            py39h5eee18b_0    defaults
tqdm                      4.66.2                   pypi_0    pypi
traitlets                 5.14.2                   pypi_0    pypi
transformers              4.38.2                   pypi_0    pypi
typing-extensions         4.10.0                   pypi_0    pypi
tzdata                    2024.1                   pypi_0    pypi
untokenize                0.1.1                    pypi_0    pypi
urllib3                   2.2.1                    pypi_0    pypi
virtualenv                20.25.1                  pypi_0    pypi
wandb                     0.16.4                   pypi_0    pypi
wcwidth                   0.2.13                   pypi_0    pypi
werkzeug                  3.0.1                    pypi_0    pypi
wheel                     0.41.2           py39h06a4308_0    defaults
widgetsnbextension        4.0.10                   pypi_0    pypi
xxhash                    3.4.1                    pypi_0    pypi
xz                        5.4.6                h5eee18b_0    defaults
yarl                      1.9.4                    pypi_0    pypi
zipp                      3.17.0           py39h06a4308_0    defaults
zlib                      1.2.13               h5eee18b_0    defaults
zstd                      1.5.5                hc292b87_0    defaults

With Ubuntu 22.04.3 LTS (GNU/Linux 6.2.0-33-generic x86_64). Also, cuda_extension is installed successfully. So I don't know if you guys have some ideas. Looking forward to your reply. Thank you!