sporco-cuda icon indicating copy to clipboard operation
sporco-cuda copied to clipboard

Reopen the issue: sporco_cuda/tests/test_cbpdn.py *** stack smashing detected ***: python terminated

Open sekigh opened this issue 3 years ago • 18 comments

I have a problem similar to https://github.com/bwohlberg/sporco-cuda/issues/3#issue-470636815. Due to this I can not use cuda for sporco example script. How do I fix this ptoblem?

The error message is;

sporco_cuda/tests/test_cbpdn.py *** stack smashing detected ***: python terminated Fatal Python error: Aborted

This happens when I ran

python setup.py test

during a process to build/test/install sporco-cuda by following installation instruction described in sporco-cuda 0.0.6b1 documentation,

git clone https://github.com/bwohlberg/sporco-cuda.git cd sporco-cuda export CUDAHOME=/usr/local/cuda-9.2 python setup.py build python setup.py test python setup.py install

Before running the above, I properly installed sporco and set up a symlink to sporco directory from the root directory of sporco-cuda.

See full error message attached below:

(anaconda3-2019.03) (base) sekigh@04f0dd606f61:~/Windowsfolder/sporco-cuda$ python setup.py test running pytest running egg_info writing sporco_cuda.egg-info/PKG-INFO writing dependency_links to sporco_cuda.egg-info/dependency_links.txt writing requirements to sporco_cuda.egg-info/requires.txt writing top-level names to sporco_cuda.egg-info/top_level.txt reading manifest file 'sporco_cuda.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' no previously-included directories found matching 'docs/build' writing manifest file 'sporco_cuda.egg-info/SOURCES.txt' running build_ext skipping 'sporco_cuda/util.c' Cython extension (up-to-date) skipping 'sporco_cuda/cbpdn.c' Cython extension (up-to-date) ====================================== test session starts ====================================== platform linux -- Python 3.7.7, pytest-6.2.4, py-1.10.0, pluggy-0.13.1 rootdir: /home/sekigh/Windowsfolder/sporco-cuda, configfile: pytest.ini, testpaths: sporco_cuda collected 18 items

sporco_cuda/tests/test_cbpdn.py *** stack smashing detected ***: python terminated Fatal Python error: Aborted

Thread 0x00007f3bc6d8e700 (most recent call first): File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/pyfftw/interfaces/cache.py", line 224 in _run File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/threading.py", line 870 in run File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/threading.py", line 926 in _bootstrap_inner File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/threading.py", line 890 in _bootstrap

Current thread 0x00007f3bd9c5f700 (most recent call first): File "/home/sekigh/Windowsfolder/sporco-cuda/sporco_cuda/tests/test_cbpdn.py", line 30 in test_01 File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/_pytest/python.py", line 183 in pytest_pyfunc_call File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/pluggy/manager.py", line 87 in File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in call File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/_pytest/python.py", line 1641 in runtest File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/_pytest/runner.py", line 162 in pytest_runtest_call File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/pluggy/manager.py", line 87 in File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in call File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/_pytest/runner.py", line 255 in File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/_pytest/runner.py", line 311 in from_call File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/_pytest/runner.py", line 255 in call_runtest_hook File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/_pytest/runner.py", line 215 in call_and_report File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/_pytest/runner.py", line 126 in runtestprotocol File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/_pytest/runner.py", line 109 in pytest_runtest_protocol File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/pluggy/manager.py", line 87 in File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in call File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/_pytest/main.py", line 348 in pytest_runtestloop File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/pluggy/manager.py", line 87 in File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in call File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/_pytest/main.py", line 323 in _main File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/_pytest/main.py", line 269 in wrap_session File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/_pytest/main.py", line 316 in pytest_cmdline_main File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/pluggy/manager.py", line 87 in File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in call File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/_pytest/config/init.py", line 163 in main File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/ptr.py", line 220 in run_tests File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/ptr.py", line 209 in run File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/distutils/dist.py", line 985 in run_command File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/distutils/dist.py", line 966 in run_commands File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/distutils/core.py", line 148 in setup File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/setuptools/init.py", line 153 in setup File "setup.py", line 290 in Aborted (core dumped) (anaconda3-2019.03) (base) sekigh@04f0dd606f61:~/Windowsfolder/sporco-cuda$

The end of sentences

sekigh avatar Jun 29 '21 02:06 sekigh

My environment is;

ubuntu 16.04 python 3.3.7 cuda-9.2 docker 19.03.5

Thank you.

sekigh avatar Jun 29 '21 03:06 sekigh

Could you try running the example scripts and report back on which ones work and which don't?

bwohlberg avatar Jun 30 '21 03:06 bwohlberg

I tried out your updated setup.py for #8 to the first machine showing on this #7 but still see the error with the same "stack smashing detected" in python setup.py test:

(root) sekigh@04f0dd606f61:~/Windowsfolder/sporco-cuda$ python setup.py test running pytest running egg_info writing sporco_cuda.egg-info/PKG-INFO writing dependency_links to sporco_cuda.egg-info/dependency_links.txt writing requirements to sporco_cuda.egg-info/requires.txt writing top-level names to sporco_cuda.egg-info/top_level.txt reading manifest file 'sporco_cuda.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' no previously-included directories found matching 'docs/build' writing manifest file 'sporco_cuda.egg-info/SOURCES.txt' running build_ext skipping 'sporco_cuda/util.c' Cython extension (up-to-date) skipping 'sporco_cuda/cbpdn.c' Cython extension (up-to-date) ============================================================================================= test session starts ============================================================================================== platform linux -- Python 3.7.7, pytest-6.2.4, py-1.10.0, pluggy-0.13.1 rootdir: /home/sekigh/Windowsfolder/sporco-cuda, configfile: pytest.ini, testpaths: sporco_cuda collected 18 items

sporco_cuda/tests/test_cbpdn.py *** stack smashing detected ***: python terminated Fatal Python error: Aborted

Thread 0x00007f358a2d4700 (most recent call first): File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/pyfftw/interfaces/cache.py", line 224 in _run File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/threading.py", line 870 in run File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/threading.py", line 926 in _bootstrap_inner File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/threading.py", line 890 in _bootstrap

Current thread 0x00007f359d225700 (most recent call first): File "/home/sekigh/Windowsfolder/sporco-cuda/sporco_cuda/tests/test_cbpdn.py", line 30 in test_01 File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/_pytest/python.py", line 183 in pytest_pyfunc_call File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/pluggy/manager.py", line 87 in File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in call File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/_pytest/python.py", line 1641 in runtest File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/_pytest/runner.py", line 162 in pytest_runtest_call File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/pluggy/manager.py", line 87 in File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in call File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/_pytest/runner.py", line 255 in File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/_pytest/runner.py", line 311 in from_call File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/_pytest/runner.py", line 255 in call_runtest_hook File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/_pytest/runner.py", line 215 in call_and_report File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/_pytest/runner.py", line 126 in runtestprotocol File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/_pytest/runner.py", line 109 in pytest_runtest_protocol File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/pluggy/manager.py", line 87 in File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in call File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/_pytest/main.py", line 348 in pytest_runtestloop File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/pluggy/manager.py", line 87 in File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in call File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/_pytest/main.py", line 323 in _main File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/_pytest/main.py", line 269 in wrap_session File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/_pytest/main.py", line 316 in pytest_cmdline_main File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/pluggy/manager.py", line 87 in File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in call File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/_pytest/config/init.py", line 163 in main File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/ptr.py", line 220 in run_tests File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/ptr.py", line 209 in run File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/distutils/dist.py", line 985 in run_command File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/distutils/dist.py", line 966 in run_commands File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/distutils/core.py", line 148 in setup File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/setuptools/init.py", line 153 in setup File "setup.py", line 287 in Aborted (core dumped)

end of text

sekigh avatar Jun 30 '21 05:06 sekigh

The error log makes me suspect that this might actually be a problem in pyfftw, which is used by the sporco solvers, which are called by the sporco-cuda tests to compare results. Could you try running the example scripts, some of which do not call any sporco solvers? For example

python examples/big_cbpdn.py

bwohlberg avatar Jun 30 '21 12:06 bwohlberg

I just read your response on #8. Could you please also summarize the differences between the environment in which sporco-cuda works with no problems, and the one that results in the stack smashing error?

bwohlberg avatar Jun 30 '21 12:06 bwohlberg

Hi,

As for the result to python examples/big_cbpdn.py on the machine #7, unfortunately I have "stack smash error" again as follows:

(root) sekigh@04f0dd606f61: ~/Windowsfolder/sporco-cuda$ python ./examples/big_cbpdn.py ./examples/big_cbpdn.py:48: DeprecationWarning: Function sporco.util.tikhonov_filter is deprecated; please use function sporco.signal.tikhonov_filter instead. sl, sh = util.tikhonov_filter(img, fltlmbd, npd) *** stack smashing detected ***: python terminated Aborted (core dumped) (root) sekigh@04f0dd606f61:~/Windowsfolder/sporco-cuda$

The differences between installation on two machines include:

Machine 1 configuration for #7 issue:

ubuntu 16.04 python 3.3.7 cuda-9.2 gpu GTX1080ti (architecture docker 19.03.5

sporco installation by source build/test/install sporco-cuda installation by source build/test/install

Machine 2 configuration for #8 issue:

ubuntu 18.04 python 3.7.4 cuda-11.2 gpu RTX3090ti (architecture 8.6) docker 20.10.5

sporco installation by conda install sporco-cuda installation by source build/test/install

end of text

sekigh avatar Jun 30 '21 16:06 sekigh

Just to confirm, sporco-cuda is working fine on Machine 2, but gives the stack smashing error on Machine 1?

Could you try using python 3.8 via conda (see the script tools/install_conda.sh in the main sporco repo) on Machine 1?

bwohlberg avatar Jun 30 '21 16:06 bwohlberg

That is right! Machine 2 works but machine 1 does not.

My understanding is tools/install_conda.sh tries to install python3.8 based miniconda to the environment. Currently, I use pyenv/virtualenv. So I moved to sporco/tools directory and type:

./install_conda.sh /home/sekigh/.pyenv/versions

where the new python 3.8 based environment successfully looks installed under /home/sekigh/.pyenv/versions. I see 3 new miniconda environments by pyenv versions;

miniconda3 miniconda3/envs/py38 miniconda3/envs/py38cu

however, these three are all based on python 3.7.7 instead of 3.8. Is it OK ? python 3.8 may not be available for ubuntu16.04, because ubuntu 16.04 may be too old. What are difference among three and which one should I use for test? I decided to use miniconda3/envs/py38cu.

Under miniconda3/envs/py38cu, I build/test/install sporco and sporco-cuda with source codes.

Build works fine but test did not, showing *** stack smashing detected *** again.

Full messages follow:

(miniconda3/envs/py38cu) (base) sekigh@04f0dd606f61:~/Windowsfolder/sporco-cuda$ python setup.py test running pytest running egg_info writing sporco_cuda.egg-info/PKG-INFO writing dependency_links to sporco_cuda.egg-info/dependency_links.txt writing requirements to sporco_cuda.egg-info/requires.txt writing top-level names to sporco_cuda.egg-info/top_level.txt reading manifest file 'sporco_cuda.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' no previously-included directories found matching 'docs/build' writing manifest file 'sporco_cuda.egg-info/SOURCES.txt' running build_ext skipping 'sporco_cuda/util.c' Cython extension (up-to-date) skipping 'sporco_cuda/cbpdn.c' Cython extension (up-to-date) ============================================================================================= test session starts ============================================================================================== platform linux -- Python 3.7.7, pytest-6.2.4, py-1.10.0, pluggy-0.13.1 rootdir: /home/sekigh/Windowsfolder/sporco-cuda, configfile: pytest.ini, testpaths: sporco_cuda collected 18 items

sporco_cuda/tests/test_cbpdn.py *** stack smashing detected ***: python terminated Fatal Python error: Aborted

Thread 0x00007f6ce74c0700 (most recent call first): File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/pyfftw/interfaces/cache.py", line 224 in _run File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/threading.py", line 870 in run File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/threading.py", line 926 in _bootstrap_inner File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/threading.py", line 890 in _bootstrap

Current thread 0x00007f6cfa411700 (most recent call first): File "/home/sekigh/Windowsfolder/sporco-cuda/sporco_cuda/tests/test_cbpdn.py", line 30 in test_01 File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/_pytest/python.py", line 183 in pytest_pyfunc_call File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/pluggy/manager.py", line 87 in File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in call File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/_pytest/python.py", line 1641 in runtest File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/_pytest/runner.py", line 162 in pytest_runtest_call File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/pluggy/manager.py", line 87 in File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in call File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/_pytest/runner.py", line 255 in File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/_pytest/runner.py", line 311 in from_call File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/_pytest/runner.py", line 255 in call_runtest_hook File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/_pytest/runner.py", line 215 in call_and_report File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/_pytest/runner.py", line 126 in runtestprotocol File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/_pytest/runner.py", line 109 in pytest_runtest_protocol File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/pluggy/manager.py", line 87 in File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in call File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/_pytest/main.py", line 348 in pytest_runtestloop File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/pluggy/manager.py", line 87 in File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in call File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/_pytest/main.py", line 323 in _main File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/_pytest/main.py", line 269 in wrap_session File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/_pytest/main.py", line 316 in pytest_cmdline_main File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/pluggy/callers.py", line 187 in _multicall File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/pluggy/manager.py", line 87 in File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/pluggy/manager.py", line 93 in _hookexec File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/pluggy/hooks.py", line 286 in call File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/_pytest/config/init.py", line 163 in main File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/ptr.py", line 220 in run_tests File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/ptr.py", line 209 in run File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/distutils/dist.py", line 985 in run_command File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/distutils/dist.py", line 966 in run_commands File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/distutils/core.py", line 148 in setup File "/home/sekigh/.pyenv/versions/anaconda3-2019.03/lib/python3.7/site-packages/setuptools/init.py", line 153 in setup File "setup.py", line 287 in Aborted (core dumped) (miniconda3/envs/py38cu) (base) sekigh@04f0dd606f61:~/Windowsfolder/sporco-cuda$

Moreover, python setup.py install is executed and try to run big_cbpdn.py again but failed again.

(miniconda3/envs/py38cu) (base) sekigh@04f0dd606f61:~/Windowsfolder/sporco-cuda$ python ./examples/big_cbpdn.py ./examples/big_cbpdn.py:48: DeprecationWarning: Function sporco.util.tikhonov_filter is deprecated; please use function sporco.signal.tikhonov_filter instead. sl, sh = util.tikhonov_filter(img, fltlmbd, npd) *** stack smashing detected ***: python terminated Aborted (core dumped) (miniconda3/envs/py38cu) (base) sekigh@04f0dd606f61:~/Windowsfolder/sporco-cuda$

end of text

sekigh avatar Jun 30 '21 19:06 sekigh

Strange that you end up with Python 3.7 instead of 3.8; I suppose it's related to your use of a virtualenv. I don't think it matters, though, since 3.7 is not very old. The py38 environment is intended as the standard environment for sporco, and py38cu is an alternative that includes GPU support via cupy.

Since the sporco-cuda tests and examples are both failing, let's approach this from a different direction. My hypothesis is still that the problem is actually in running components of sporco, and not from sporco-cuda. Could you please clone the main sporco repo and run its tests?

bwohlberg avatar Jun 30 '21 20:06 bwohlberg

I double-checked python version used in py38cu and found it is 3.8.10 instead of 3.7.7. Therefore, it is reasonable. I delete previous sparco local repo and clone sporco rep again and run build/install from sporco source in environment py38cu. The results are not good; There two test results, one is cmp_cbpdn.py, and the other is big_cbpdn.py.

The result of python ./examples/cmp_cbpdn.py shows cpu solver works fine but gpu solver fails. That is the opposite to your hypothesis. The result of python ./examples/big_cbpdn.py fails also. So gpu solver apparently fails.

Find attached two execution logs below:

(miniconda3/envs/py38cu) (base) sekigh@04f0dd606f61:~/Windowsfolder/sporco-cuda$ python ./examples/cmp_cbpdn.py ./examples/cmp_cbpdn.py:32: DeprecationWarning: Function sporco.util.tikhonov_filter is deprecated; please use function sporco.signal.tikhonov_filter instead. sl, sh = util.tikhonov_filter(img, fltlmbd, npd) Itn Fnc DFid Regℓ1 r s

0 1.80e+02 9.84e+01 8.16e+03 8.71e-01 2.65e-01 1 1.75e+02 1.05e+02 7.03e+03 5.56e-01 3.62e-01 2 1.79e+02 1.06e+02 7.27e+03 3.70e-01 2.59e-01 3 1.71e+02 1.06e+02 6.51e+03 2.95e-01 2.48e-01 4 1.70e+02 1.06e+02 6.45e+03 2.51e-01 1.86e-01 5 1.66e+02 1.06e+02 5.98e+03 2.05e-01 1.58e-01 6 1.61e+02 1.06e+02 5.48e+03 1.71e-01 1.35e-01 7 1.57e+02 1.06e+02 5.08e+03 1.46e-01 1.11e-01 8 1.54e+02 1.06e+02 4.81e+03 1.26e-01 9.23e-02 9 1.52e+02 1.06e+02 4.58e+03 1.08e-01 8.33e-02 10 1.51e+02 1.06e+02 4.45e+03 9.41e-02 7.43e-02 11 1.50e+02 1.06e+02 4.39e+03 8.32e-02 6.52e-02 12 1.49e+02 1.06e+02 4.27e+03 7.39e-02 6.06e-02 13 1.48e+02 1.06e+02 4.13e+03 6.65e-02 5.64e-02 14 1.47e+02 1.06e+02 4.04e+03 6.07e-02 5.07e-02 15 1.46e+02 1.06e+02 3.95e+03 5.56e-02 4.61e-02 16 1.45e+02 1.06e+02 3.85e+03 5.09e-02 4.33e-02 17 1.44e+02 1.06e+02 3.78e+03 4.69e-02 4.06e-02 18 1.44e+02 1.06e+02 3.73e+03 4.35e-02 3.79e-02 19 1.43e+02 1.06e+02 3.68e+03 4.05e-02 3.55e-02

ConvBPDN solve time: 26.04s *** stack smashing detected ***: python terminated Aborted (core dumped) (miniconda3/envs/py38cu) (base) sekigh@04f0dd606f61:~/Windowsfolder/sporco-cuda$ python ./examples/big_cbpdn.py ./examples/big_cbpdn.py:48: DeprecationWarning: Function sporco.util.tikhonov_filter is deprecated; please use function sporco.signal.tikhonov_filter instead. sl, sh = util.tikhonov_filter(img, fltlmbd, npd) *** stack smashing detected ***: python terminated Aborted (core dumped) (miniconda3/envs/py38cu) (base) sekigh@04f0dd606f61:~/Windowsfolder/sporco-cuda$

(miniconda3/envs/py38cu) (base) sekigh@04f0dd606f61:~/Windowsfolder/sporco-cuda$ python ./examples/big_cbpdn.py ./examples/big_cbpdn.py:48: DeprecationWarning: Function sporco.util.tikhonov_filter is deprecated; please use function sporco.signal.tikhonov_filter instead. sl, sh = util.tikhonov_filter(img, fltlmbd, npd) *** stack smashing detected ***: python terminated Aborted (core dumped) (miniconda3/envs/py38cu) (base) sekigh@04f0dd606f61:~/Windowsfolder/sporco-cuda$

End of text

sekigh avatar Jul 01 '21 07:07 sekigh

I have motivation to make sporco and sporco-cuda work properly on machine 1. That machine has two gpus for higher computation power compared with machine2, which has only one gpu.

After solving #7 properly, I must consider how to combine gpu-based sporco capability with existing scripts based on tensorflow-gpu or pytorch-gpu to solve my problem efficiently.

sekigh avatar Jul 01 '21 07:07 sekigh

I found the way to work around *** stack smashing detected *** problem by changing installation method with trial and error. The workaround is the exact same way as one done for machine2, illustrating as;

  1. install sporco by conda, i.e. conda install -c conda-forge sporco, producing sporco 0.2.0 py38h578d9bd_1 conda-forge. (Note that when build/install sporco from sources produces sporco 0.2.1 observed by command, conda list | grep sporco in somehow)
  2. build/test/install sporco-cuda from source and all three procedures finish without errors. No *** stack smashing detected *** messages happens on python setup.py test at this time.
  3. Test cases, such as python ./examples/cmp_cbpdn.py, and python ./examples/big_cbpdn.py, work properly without any errors.

Another weird phenomenon is when launching tests in the above 3) by eclipse debugger, you need to include as python library path /home/sekigh/.pyenv/versions/miniconda3/envs/py38cu/lib/python3.8/site-packages/sporco_cuda-0.0.6b1-py3.8-linux-x86_64.egg.

Please make the installation procedure more robust and announce robust installation method on github pages.

sekigh avatar Jul 01 '21 10:07 sekigh

Thanks for your debugging efforts. I will certainly update the installation instructions if we can figure out which of the different steps you took made a critical difference. For example, is step 1. critical to getting it to work? How did you install sporco previously? From pypi, using pip?

bwohlberg avatar Jul 01 '21 13:07 bwohlberg

Now, "step 1 with conda install sporco" seems the most critical.

The previous method replacing step 1 is;

git clone https://github.com/bwohlberg/sporco.git cd sporco python setup.py build python setup.py install

After step 1, step 2 should be conducted. It seems that the step 2 with;

git clone https://github.com/bwohlberg/sporco-cuda.git cd sporco-cuda export CUDAHOME=/usr/local/cuda-9.2 python setup.py build python setup.py test python setup.py install

is successful without *** stack smashing detected *** errors, only after step 1 but not after the previous method replacing step 1.

Interestingly, the major critical difference is that sporco_cuda-0.0.6b1-py3.8-linux-x86_64.egg is properly extracted to /home/sekigh/.pyenv/versions/miniconda3/envs/py38cu/lib/python3.8/site-packages in the former case but not in the second case. sporco_cuda-0.0.6b1-py3.8-linux-x86_64.egg has critical function to properly work. It is a critical clue for the reason why step1 and step2 works fine but the previous method not.

The end of text

sekigh avatar Jul 01 '21 14:07 sekigh

So the only difference is whether you install sporco from the cloned repo (which leads to failure) or from conda (which leads to success)? I'm at a loss to explain this.

bwohlberg avatar Jul 02 '21 03:07 bwohlberg

Yes, that is right in short. My guess is that they are different somehow.

sekigh avatar Jul 02 '21 10:07 sekigh

What's really strange is that it's the installation method for sporco, and not sporco-cuda, that makes the difference. It's going to be difficult to add any guidance for the user to the documentation without understanding the underlying cause.

bwohlberg avatar Jul 05 '21 14:07 bwohlberg

My intuition to the cause to this problem is difference of sporco source scripts. I have no experience in making conda package from sources so I may be wrong. The source script produces sporco binary devian package version, sporco 0.2.0 py38h578d9bd_1 conda-forge, whereas, the source scripts shown on the sporco github produce different version, sporco 0.2.1. I observed these difference by conda list. Another point is that the former case lets sporco-cuda make sporco_cuda-0.0.6b1-py3.8-linux-x86_64.egg properly, whereas the latter case fails that egg. So *** stack smashing detected *** may happen.

sekigh avatar Jul 05 '21 17:07 sekigh