elephant Fix/add option: do not compile fim c-extension

This PR adds an extra to elephant, which allows installation of Elephant without compiling the c++ extension fim.cpp used in SPADE.

Do not compile any extension:

development pip install -e . --install-option='--no-compile'
install pip install elephant --install-option='--no-compile'

Do not compile spade extension:

development pip install -e . --install-option='--no-compile-spade'
install pip install elephant --install-option='--no-compile-spade'

This option allows to have a workaround in cases were there is no suitable c-compiler available on the target system.

Additionally extensions are now optional, according to setuptools documentation:

 class setuptools.Extension(name, sources, *args, **kw)

optional (bool) – specifies that a build failure in the extension should not abort the build process, but simply not install the failing extension.

See elephant documentation:

https://elephant--494.org.readthedocs.build/en/494/install.html

Jun 10 '22 15:06 Moritz-Alexander-Kern

I can confirm that the behavior is improved. When installing from this branch, in a configuration that is known for build failure, the optional feature works. Apparently, the build fails, but the installation proceeds.

Collecting git+https://github.com/INM-6/elephant.git@fix/do_not_compile_extension
  Cloning https://github.com/INM-6/elephant.git (to revision fix/do_not_compile_extension) to /tmp/pip-req-build-bxf4m2qf
  Running command git clone --filter=blob:none --quiet https://github.com/INM-6/elephant.git /tmp/pip-req-build-bxf4m2qf
  Running command git checkout -b fix/do_not_compile_extension --track origin/fix/do_not_compile_extension
  Switched to a new branch 'fix/do_not_compile_extension'
  Branch 'fix/do_not_compile_extension' set up to track remote branch 'fix/do_not_compile_extension' from 'origin'.
  Resolved https://github.com/INM-6/elephant.git to commit 47a62e97a71f1cbcd12a54c9bb67cde20e2a31a0
  Preparing metadata (setup.py) ... done
Requirement already satisfied: neo>=0.10.0 in /p/home/jusers/koehler4/jureca/envs/elephant-test/lib/python3.9/site-packages (from elephant==0.12.0b1) (0.10.2)
Requirement already satisfied: numpy>=1.18.1 in /p/home/jusers/koehler4/jureca/envs/elephant-test/lib/python3.9/site-packages (from elephant==0.12.0b1) (1.23.1)
Requirement already satisfied: quantities>=0.12.1 in /p/home/jusers/koehler4/jureca/envs/elephant-test/lib/python3.9/site-packages (from elephant==0.12.0b1) (0.13.0)
Requirement already satisfied: scipy>=1.5.4 in /p/home/jusers/koehler4/jureca/envs/elephant-test/lib/python3.9/site-packages (from elephant==0.12.0b1) (1.8.1)
Requirement already satisfied: six>=1.10.0 in /p/home/jusers/koehler4/jureca/envs/elephant-test/lib/python3.9/site-packages (from elephant==0.12.0b1) (1.16.0)
Requirement already satisfied: tqdm in /p/home/jusers/koehler4/jureca/envs/elephant-test/lib/python3.9/site-packages (from elephant==0.12.0b1) (4.64.0)
Building wheels for collected packages: elephant
  Building wheel for elephant (setup.py) ... done
  Created wheel for elephant: filename=elephant-0.12.0b1-cp39-cp39-linux_x86_64.whl size=511992 sha256=c7991819fdd9a75b90bc8eb3fd15c44de881e3e2fd8d17e492f332a2e55f968c
  Stored in directory: /tmp/pip-ephem-wheel-cache-xf7hejos/wheels/1f/3d/36/fb5e82c1bd6d7bd77d68df2aaa6aa211c5f2f28cf8c23531ad
Successfully built elephant
Installing collected packages: elephant
Successfully installed elephant-0.12.0b1

Jul 13 '22 12:07 kohlerca

However, when using the `--install-option='--nofim', build is not attempted.

WARNING: Disabling all use of wheels due to the use of --build-option / --global-option / --install-option.
Collecting git+https://github.com/INM-6/elephant.git@fix/do_not_compile_extension
  Cloning https://github.com/INM-6/elephant.git (to revision fix/do_not_compile_extension) to /tmp/pip-req-build-0tk4j5lt
  Running command git clone --filter=blob:none --quiet https://github.com/INM-6/elephant.git /tmp/pip-req-build-0tk4j5lt
  Running command git checkout -b fix/do_not_compile_extension --track origin/fix/do_not_compile_extension
  Switched to a new branch 'fix/do_not_compile_extension'
  Branch 'fix/do_not_compile_extension' set up to track remote branch 'fix/do_not_compile_extension' from 'origin'.
  Resolved https://github.com/INM-6/elephant.git to commit 47a62e97a71f1cbcd12a54c9bb67cde20e2a31a0
  Preparing metadata (setup.py) ... done
Requirement already satisfied: neo>=0.10.0 in /p/home/jusers/koehler4/jureca/envs/elephant-test/lib/python3.9/site-packages (from elephant==0.12.0b1) (0.10.2)
Requirement already satisfied: numpy>=1.18.1 in /p/home/jusers/koehler4/jureca/envs/elephant-test/lib/python3.9/site-packages (from elephant==0.12.0b1) (1.23.1)
Requirement already satisfied: quantities>=0.12.1 in /p/home/jusers/koehler4/jureca/envs/elephant-test/lib/python3.9/site-packages (from elephant==0.12.0b1) (0.13.0)
Requirement already satisfied: scipy>=1.5.4 in /p/home/jusers/koehler4/jureca/envs/elephant-test/lib/python3.9/site-packages (from elephant==0.12.0b1) (1.8.1)
Requirement already satisfied: six>=1.10.0 in /p/home/jusers/koehler4/jureca/envs/elephant-test/lib/python3.9/site-packages (from elephant==0.12.0b1) (1.16.0)
Requirement already satisfied: tqdm in /p/home/jusers/koehler4/jureca/envs/elephant-test/lib/python3.9/site-packages (from elephant==0.12.0b1) (4.64.0)
Skipping wheel build for elephant, due to binaries being disabled for it.
Installing collected packages: elephant
  Running setup.py install for elephant ... done
Successfully installed elephant-0.12.0b1

Jul 13 '22 12:07 kohlerca

It works for the develop option (pip install -e . --install-option='--nofim') from the repo cloned locally.

Trying to use the optional by just passing pip install -e . fails:

  Preparing metadata (setup.py) ... done
Requirement already satisfied: neo>=0.10.0 in /p/home/jusers/koehler4/jureca/envs/elephant-test/lib/python3.9/site-packages (from elephant==0.12.0b1) (0.10.2)
Requirement already satisfied: numpy>=1.18.1 in /p/home/jusers/koehler4/jureca/envs/elephant-test/lib/python3.9/site-packages (from elephant==0.12.0b1) (1.23.1)
Requirement already satisfied: quantities>=0.12.1 in /p/home/jusers/koehler4/jureca/envs/elephant-test/lib/python3.9/site-packages (from elephant==0.12.0b1) (0.13.0)
Requirement already satisfied: scipy>=1.5.4 in /p/home/jusers/koehler4/jureca/envs/elephant-test/lib/python3.9/site-packages (from elephant==0.12.0b1) (1.8.1)
Requirement already satisfied: six>=1.10.0 in /p/home/jusers/koehler4/jureca/envs/elephant-test/lib/python3.9/site-packages (from elephant==0.12.0b1) (1.16.0)
Requirement already satisfied: tqdm in /p/home/jusers/koehler4/jureca/envs/elephant-test/lib/python3.9/site-packages (from elephant==0.12.0b1) (4.64.0)
Installing collected packages: elephant
  Running setup.py develop for elephant
    error: subprocess-exited-with-error
    
    × python setup.py develop did not run successfully.
    │ exit code: 1
    ╰─> [33 lines of output]
        running develop
        /p/home/jusers/koehler4/jureca/envs/elephant-test/lib/python3.9/site-packages/setuptools/command/easy_install.py:144: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
          warnings.warn(
        /p/home/jusers/koehler4/jureca/envs/elephant-test/lib/python3.9/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
          warnings.warn(
        running egg_info
        writing elephant.egg-info/PKG-INFO
        writing dependency_links to elephant.egg-info/dependency_links.txt
        writing requirements to elephant.egg-info/requires.txt
        writing top-level names to elephant.egg-info/top_level.txt
        reading manifest file 'elephant.egg-info/SOURCES.txt'
        reading manifest template 'MANIFEST.in'
        no previously-included directories found matching 'doc/_build'
        no previously-included directories found matching 'doc/tutorials/.ipynb_checkpoints'
        no previously-included directories found matching 'doc/reference/toctree'
        warning: no files found matching 'doc/reference/toctree/kernels/*'
        warning: no previously-included files matching '*.h5' found under directory '*'
        warning: no previously-included files matching '*~' found under directory '*'
        adding license file 'LICENSE.txt'
        writing manifest file 'elephant.egg-info/SOURCES.txt'
        running build_ext
        building 'elephant.spade_src.fim' extension
        gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -O2 -ftree-vectorize -march=native -fno-math-errno -fPIC -O2 -ftree-vectorize -march=native -fno-math-errno -fPIC -fPIC -Ielephant/spade_src/include -I/p/home/jusers/koehler4/jureca/envs/elephant-test/include -I/p/software/jurecadc/stages/2022/software/Python/3.9.6-GCCcore-11.2.0/include/python3.9 -c elephant/spade_src/src/fim.cpp -o build/temp.linux-x86_64-cpython-39/elephant/spade_src/src/fim.o -DMODULE_NAME=fim -DUSE_OPENMP -DWITH_SIG_TERM -Dfim_EXPORTS -O3 -pedantic -Wextra -Weffc++ -Wunused-result -Werror -fopenmp -std=gnu++17
        In file included from elephant/spade_src/src/fim.cpp:41:
        elephant/spade_src/include/FPGrowth.h: In constructor ‘FPGrowth::FPGrowth(Transactions&, Support, uint32_t, uint32_t, ItemC, uint32_t, uint32_t, int32_t)’:
        elephant/spade_src/include/FPGrowth.h:198:37: error: loop variable ‘p’ of type ‘const RefPair&’ {aka ‘const std::pair<unsigned int, std::shared_ptr<FrequencyRef> >&’} binds to a temporary constructed from type ‘std::pair<const unsigned int, std::shared_ptr<FrequencyRef> >’ [-Werror=range-loop-construct]
          198 |                 for (const RefPair& p : F)
              |                                     ^
        elephant/spade_src/include/FPGrowth.h:198:37: note: use non-reference type ‘const RefPair’ {aka ‘const std::pair<unsigned int, std::shared_ptr<FrequencyRef> >’} to make the copy explicit or ‘const std::pair<const unsigned int, std::shared_ptr<FrequencyRef> >&’ to prevent copying
        cc1plus: all warnings being treated as errors
        warning: build_ext: building extension "elephant.spade_src.fim" failed: command '/p/software/jurecadc/stages/2022/software/GCCcore/11.2.0/bin/gcc' failed with exit code 1
        
        error: can't copy 'build/lib.linux-x86_64-cpython-39/elephant/spade_src/fim.cpython-39-x86_64-linux-gnu.so': doesn't exist or not a regular file
        [end of output]
    
    note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× python setup.py develop did not run successfully.
│ exit code: 1
╰─> [33 lines of output]
    running develop
    /p/home/jusers/koehler4/jureca/envs/elephant-test/lib/python3.9/site-packages/setuptools/command/easy_install.py:144: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
      warnings.warn(
    /p/home/jusers/koehler4/jureca/envs/elephant-test/lib/python3.9/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
      warnings.warn(
    running egg_info
    writing elephant.egg-info/PKG-INFO
    writing dependency_links to elephant.egg-info/dependency_links.txt
    writing requirements to elephant.egg-info/requires.txt
    writing top-level names to elephant.egg-info/top_level.txt
    reading manifest file 'elephant.egg-info/SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    no previously-included directories found matching 'doc/_build'
    no previously-included directories found matching 'doc/tutorials/.ipynb_checkpoints'
    no previously-included directories found matching 'doc/reference/toctree'
    warning: no files found matching 'doc/reference/toctree/kernels/*'
    warning: no previously-included files matching '*.h5' found under directory '*'
    warning: no previously-included files matching '*~' found under directory '*'
    adding license file 'LICENSE.txt'
    writing manifest file 'elephant.egg-info/SOURCES.txt'
    running build_ext
    building 'elephant.spade_src.fim' extension
    gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -O2 -ftree-vectorize -march=native -fno-math-errno -fPIC -O2 -ftree-vectorize -march=native -fno-math-errno -fPIC -fPIC -Ielephant/spade_src/include -I/p/home/jusers/koehler4/jureca/envs/elephant-test/include -I/p/software/jurecadc/stages/2022/software/Python/3.9.6-GCCcore-11.2.0/include/python3.9 -c elephant/spade_src/src/fim.cpp -o build/temp.linux-x86_64-cpython-39/elephant/spade_src/src/fim.o -DMODULE_NAME=fim -DUSE_OPENMP -DWITH_SIG_TERM -Dfim_EXPORTS -O3 -pedantic -Wextra -Weffc++ -Wunused-result -Werror -fopenmp -std=gnu++17
    In file included from elephant/spade_src/src/fim.cpp:41:
    elephant/spade_src/include/FPGrowth.h: In constructor ‘FPGrowth::FPGrowth(Transactions&, Support, uint32_t, uint32_t, ItemC, uint32_t, uint32_t, int32_t)’:
    elephant/spade_src/include/FPGrowth.h:198:37: error: loop variable ‘p’ of type ‘const RefPair&’ {aka ‘const std::pair<unsigned int, std::shared_ptr<FrequencyRef> >&’} binds to a temporary constructed from type ‘std::pair<const unsigned int, std::shared_ptr<FrequencyRef> >’ [-Werror=range-loop-construct]
      198 |                 for (const RefPair& p : F)
          |                                     ^
    elephant/spade_src/include/FPGrowth.h:198:37: note: use non-reference type ‘const RefPair’ {aka ‘const std::pair<unsigned int, std::shared_ptr<FrequencyRef> >’} to make the copy explicit or ‘const std::pair<const unsigned int, std::shared_ptr<FrequencyRef> >&’ to prevent copying
    cc1plus: all warnings being treated as errors
    warning: build_ext: building extension "elephant.spade_src.fim" failed: command '/p/software/jurecadc/stages/2022/software/GCCcore/11.2.0/bin/gcc' failed with exit code 1
    
    error: can't copy 'build/lib.linux-x86_64-cpython-39/elephant/spade_src/fim.cpython-39-x86_64-linux-gnu.so': doesn't exist or not a regular file
    [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.

Jul 13 '22 12:07 kohlerca

So, unless the develop install failure when not passing the --nofim flag is expected, this is the only current problem. Apart from that, as raised in the comments below, the naming of the flag could be revised before merge.

Jul 13 '22 12:07 kohlerca

Hello @Moritz-Alexander-Kern! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers:

Comment last updated at 2022-08-09 12:51:10 UTC

Jul 22 '22 12:07 pep8speaks

Coverage increased (+0.009%) to 87.21% when pulling 1ab415bc16b4ab1c9992bccdb4bc6fb2f6664500 on INM-6:fix/do_not_compile_extension into 646cbd444367f2b8918415536d7fe87948d745bb on NeuralEnsemble:master.

Aug 09 '22 12:08 coveralls

The problem highlighted by @kohlerca still persists: Development mode install fails without any options specified if the c++ file does not compile well. This happened to me with various versions of setuptools, but with different errors. This tells me it's a problem on the setuptools side. For regular installs, the module is skipped if there are errors -- this is probably the most important point. Thus, I think merging this is fine as is.

Oct 13 '22 14:10 mdenker

elephant elephant copied to clipboard

Fix/add option: do not compile fim c-extension

Comment last updated at 2022-08-09 12:51:10 UTC

elephant
elephant copied to clipboard