pytorch_scatter
pytorch_scatter copied to clipboard
rusty1s
Silent dependency expectations of pytorch-scatter and dependency management in setup.cfg
Hello!
We wish to use the torch-scatter package in our project as we find it very useful. Unfortunately, the dependency management of this package is not transparent enough to allow for a seamless install in production-level project. I think that for good production use the dependencies should be stated correctly in setup.py or setup.cfg to allow the dependency resolver to install them.
Hidden dependencies are:
- wheel
- numpy
State at the beginning:
# python3 --version
Python 3.9.12
# pip3 list
Package Version
----------------- -------
pip 22.0.4
setuptools 58.1.0
torch 1.10.1
typing_extensions 4.2.0
Error due to the dependency on wheel:
# pip3 install torch-scatter
Looking in indexes: https://artifactory.ida.avast.com/artifactory/api/pypi/pypi-remote/simple, https://artifactory.ida.avast.com/artifactory/api/pypi/pypi-local/simple
Collecting torch-scatter
Using cached https://artifactory.ida.avast.com/artifactory/api/pypi/pypi-remote/packages/packages/1b/a0/6e44e887eb7fff78a9642035fe9662fc22c850a377369a52f308dd553104/torch_scatter-2.0.9.tar.gz (21 kB)
Preparing metadata (setup.py) ... done
Using legacy 'setup.py install' for torch-scatter, since package 'wheel' is not installed.
Error due to the dependency on numpy:
# pip3 install torch-scatter
Looking in indexes: https://artifactory.ida.avast.com/artifactory/api/pypi/pypi-remote/simple, https://artifactory.ida.avast.com/artifactory/api/pypi/pypi-local/simple
Collecting torch-scatter
Using cached https://artifactory.ida.avast.com/artifactory/api/pypi/pypi-remote/packages/packages/1b/a0/6e44e887eb7fff78a9642035fe9662fc22c850a377369a52f308dd553104/torch_scatter-2.0.9.tar.gz (21 kB)
Preparing metadata (setup.py) ... done
Building wheels for collected packages: torch-scatter
Building wheel for torch-scatter (setup.py) ... error
error: subprocess-exited-with-error
× python setup.py bdist_wheel did not run successfully.
│ exit code: 1
╰─> [23 lines of output]
/usr/local/lib/python3.9/site-packages/torch/package/_directory_reader.py:17: UserWarning: Failed to initialize NumPy: No module named 'numpy' (Triggered internally at ../torch/csrc/utils/tensor_numpy.cpp:68.)
These issues look a bit strange to me and seem to be not really related to torch-scatter. The wheel dependency warning is coming from Python directly, while the numpy dependency is coming from PyTorch. Neither wheel nor numpy are used in torch-scatter anywhere.
I think that
pip install torch-scatter
should be a seamless command that works generally and resolves any of its direct or indirect dependency issues. If it does not do that then IMHO the torch-scatter dependency is a risky dependency in setup.cfg or pyproject.toml of any serious project because it propagates these hidden&tricky dependencies further.
Another cryptic and yet unresolved error we keep getting during the installation of our project which uses torch-scatter is the following one. Somehow it fails during the compilation without actually saying which library is missing. In comparison, installation of torch was successful.
• Updating torch-scatter (2.0.9 /home/sopikb/.local/lib/python3.8/site-packages -> 2.0.9): Failed
EnvCommandError
Command ['/home/sopikb/conda/envs/hmic-py38/bin/pip', 'install', '--no-deps', '-U', 'file:///home/sopikb/.cache/pypoetry/artifacts/14/02/05/effd284f789af1e9cc6640605668b2756da552521f85399c96bcdd9294/torch_scatter-2.0.9.tar.gz'] errored with the following return code 1, and output:
Processing /home/sopikb/.cache/pypoetry/artifacts/14/02/05/effd284f789af1e9cc6640605668b2756da552521f85399c96bcdd9294/torch_scatter-2.0.9.tar.gz
Preparing metadata (setup.py): started
Preparing metadata (setup.py): finished with status 'done'
Building wheels for collected packages: torch-scatter
Building wheel for torch-scatter (setup.py): started
Building wheel for torch-scatter (setup.py): finished with status 'error'
error: subprocess-exited-with-error
× python setup.py bdist_wheel did not run successfully.
│ exit code: 1
╰─> [33 lines of output]
/home/sopikb/conda/envs/hmic-py38/lib/python3.8/site-packages/setuptools/dist.py:772: UserWarning: Usage of dash-separated 'description-file' will not be supported in future versions. Please use the underscore name 'description_file' instead
warnings.warn(
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-x86_64-cpython-38
creating build/lib.linux-x86_64-cpython-38/torch_scatter
copying torch_scatter/segment_csr.py -> build/lib.linux-x86_64-cpython-38/torch_scatter
copying torch_scatter/utils.py -> build/lib.linux-x86_64-cpython-38/torch_scatter
copying torch_scatter/placeholder.py -> build/lib.linux-x86_64-cpython-38/torch_scatter
copying torch_scatter/__init__.py -> build/lib.linux-x86_64-cpython-38/torch_scatter
copying torch_scatter/scatter.py -> build/lib.linux-x86_64-cpython-38/torch_scatter
copying torch_scatter/segment_coo.py -> build/lib.linux-x86_64-cpython-38/torch_scatter
creating build/lib.linux-x86_64-cpython-38/torch_scatter/composite
copying torch_scatter/composite/std.py -> build/lib.linux-x86_64-cpython-38/torch_scatter/composite
copying torch_scatter/composite/__init__.py -> build/lib.linux-x86_64-cpython-38/torch_scatter/composite
copying torch_scatter/composite/logsumexp.py -> build/lib.linux-x86_64-cpython-38/torch_scatter/composite
copying torch_scatter/composite/softmax.py -> build/lib.linux-x86_64-cpython-38/torch_scatter/composite
running build_ext
building 'torch_scatter._scatter_cpu' extension
creating build/temp.linux-x86_64-cpython-38
creating build/temp.linux-x86_64-cpython-38/csrc
creating build/temp.linux-x86_64-cpython-38/csrc/cpu
gcc -pthread -B /home/sopikb/conda/envs/hmic-py38/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Icsrc -I/home/sopikb/conda/envs/hmic-py38/lib/python3.8/site-packages/torch/include -I/home/sopikb/conda/envs/hmic-py38/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home/sopikb/conda/envs/hmic-py38/lib/python3.8/site-packages/torch/include/TH -I/home/sopikb/conda/envs/hmic-py38/lib/python3.8/site-packages/torch/include/THC -I/home/sopikb/conda/envs/hmic-py38/include/python3.8 -c csrc/cpu/scatter_cpu.cpp -o build/temp.linux-x86_64-cpython-38/csrc/cpu/scatter_cpu.o -O2 -DAT_PARALLEL_OPENMP -fopenmp -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -DTORCH_EXTENSION_NAME=_scatter_cpu -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
cc1plus: warning: command-line option '-Wstrict-prototypes' is valid for C/ObjC but not for C++
gcc -pthread -B /home/sopikb/conda/envs/hmic-py38/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Icsrc -I/home/sopikb/conda/envs/hmic-py38/lib/python3.8/site-packages/torch/include -I/home/sopikb/conda/envs/hmic-py38/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home/sopikb/conda/envs/hmic-py38/lib/python3.8/site-packages/torch/include/TH -I/home/sopikb/conda/envs/hmic-py38/lib/python3.8/site-packages/torch/include/THC -I/home/sopikb/conda/envs/hmic-py38/include/python3.8 -c csrc/scatter.cpp -o build/temp.linux-x86_64-cpython-38/csrc/scatter.o -O2 -DAT_PARALLEL_OPENMP -fopenmp -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -DTORCH_EXTENSION_NAME=_scatter_cpu -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
cc1plus: warning: command-line option '-Wstrict-prototypes' is valid for C/ObjC but not for C++
g++ -pthread -B /home/sopikb/conda/envs/hmic-py38/compiler_compat -Wl,--sysroot=/ -pthread -shared -B /home/sopikb/conda/envs/hmic-py38/compiler_compat -L/home/sopikb/conda/envs/hmic-py38/lib -Wl,-rpath=/home/sopikb/conda/envs/hmic-py38/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-cpython-38/csrc/cpu/scatter_cpu.o build/temp.linux-x86_64-cpython-38/csrc/scatter.o -L/home/sopikb/conda/envs/hmic-py38/lib/python3.8/site-packages/torch/lib -lc10 -ltorch -ltorch_cpu -ltorch_python -o build/lib.linux-x86_64-cpython-38/torch_scatter/_scatter_cpu.so -s
/home/sopikb/conda/envs/hmic-py38/bin/../lib/gcc/x86_64-conda-linux-gnu/11.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find /lib64/libpthread.so.0: No such file or directory
/home/sopikb/conda/envs/hmic-py38/bin/../lib/gcc/x86_64-conda-linux-gnu/11.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find /usr/lib64/libpthread_nonshared.a: No such file or directory
collect2: error: ld returned 1 exit status
error: command '/home/sopikb/conda/envs/hmic-py38/bin/g++' failed with exit code 1
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for torch-scatter
Running setup.py clean for torch-scatter
Failed to build torch-scatter
Installing collected packages: torch-scatter
Attempting uninstall: torch-scatter
Found existing installation: torch-scatter 2.0.9
Uninstalling torch-scatter-2.0.9:
Successfully uninstalled torch-scatter-2.0.9
Running setup.py install for torch-scatter: started
Running setup.py install for torch-scatter: finished with status 'error'
error: subprocess-exited-with-error
× Running setup.py install for torch-scatter did not run successfully.
│ exit code: 1
╰─> [35 lines of output]
/home/sopikb/conda/envs/hmic-py38/lib/python3.8/site-packages/setuptools/dist.py:772: UserWarning: Usage of dash-separated 'description-file' will not be supported in future versions. Please use the underscore name 'description_file' instead
warnings.warn(
running install
/home/sopikb/conda/envs/hmic-py38/lib/python3.8/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
warnings.warn(
running build
running build_py
creating build
creating build/lib.linux-x86_64-cpython-38
creating build/lib.linux-x86_64-cpython-38/torch_scatter
copying torch_scatter/segment_csr.py -> build/lib.linux-x86_64-cpython-38/torch_scatter
copying torch_scatter/utils.py -> build/lib.linux-x86_64-cpython-38/torch_scatter
copying torch_scatter/placeholder.py -> build/lib.linux-x86_64-cpython-38/torch_scatter
copying torch_scatter/__init__.py -> build/lib.linux-x86_64-cpython-38/torch_scatter
copying torch_scatter/scatter.py -> build/lib.linux-x86_64-cpython-38/torch_scatter
copying torch_scatter/segment_coo.py -> build/lib.linux-x86_64-cpython-38/torch_scatter
creating build/lib.linux-x86_64-cpython-38/torch_scatter/composite
copying torch_scatter/composite/std.py -> build/lib.linux-x86_64-cpython-38/torch_scatter/composite
copying torch_scatter/composite/__init__.py -> build/lib.linux-x86_64-cpython-38/torch_scatter/composite
copying torch_scatter/composite/logsumexp.py -> build/lib.linux-x86_64-cpython-38/torch_scatter/composite
copying torch_scatter/composite/softmax.py -> build/lib.linux-x86_64-cpython-38/torch_scatter/composite
running build_ext
building 'torch_scatter._scatter_cpu' extension
creating build/temp.linux-x86_64-cpython-38
creating build/temp.linux-x86_64-cpython-38/csrc
creating build/temp.linux-x86_64-cpython-38/csrc/cpu
gcc -pthread -B /home/sopikb/conda/envs/hmic-py38/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Icsrc -I/home/sopikb/conda/envs/hmic-py38/lib/python3.8/site-packages/torch/include -I/home/sopikb/conda/envs/hmic-py38/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home/sopikb/conda/envs/hmic-py38/lib/python3.8/site-packages/torch/include/TH -I/home/sopikb/conda/envs/hmic-py38/lib/python3.8/site-packages/torch/include/THC -I/home/sopikb/conda/envs/hmic-py38/include/python3.8 -c csrc/cpu/scatter_cpu.cpp -o build/temp.linux-x86_64-cpython-38/csrc/cpu/scatter_cpu.o -O2 -DAT_PARALLEL_OPENMP -fopenmp -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -DTORCH_EXTENSION_NAME=_scatter_cpu -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
cc1plus: warning: command-line option '-Wstrict-prototypes' is valid for C/ObjC but not for C++
gcc -pthread -B /home/sopikb/conda/envs/hmic-py38/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Icsrc -I/home/sopikb/conda/envs/hmic-py38/lib/python3.8/site-packages/torch/include -I/home/sopikb/conda/envs/hmic-py38/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -I/home/sopikb/conda/envs/hmic-py38/lib/python3.8/site-packages/torch/include/TH -I/home/sopikb/conda/envs/hmic-py38/lib/python3.8/site-packages/torch/include/THC -I/home/sopikb/conda/envs/hmic-py38/include/python3.8 -c csrc/scatter.cpp -o build/temp.linux-x86_64-cpython-38/csrc/scatter.o -O2 -DAT_PARALLEL_OPENMP -fopenmp -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -DTORCH_EXTENSION_NAME=_scatter_cpu -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14
cc1plus: warning: command-line option '-Wstrict-prototypes' is valid for C/ObjC but not for C++
g++ -pthread -B /home/sopikb/conda/envs/hmic-py38/compiler_compat -Wl,--sysroot=/ -pthread -shared -B /home/sopikb/conda/envs/hmic-py38/compiler_compat -L/home/sopikb/conda/envs/hmic-py38/lib -Wl,-rpath=/home/sopikb/conda/envs/hmic-py38/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-cpython-38/csrc/cpu/scatter_cpu.o build/temp.linux-x86_64-cpython-38/csrc/scatter.o -L/home/sopikb/conda/envs/hmic-py38/lib/python3.8/site-packages/torch/lib -lc10 -ltorch -ltorch_cpu -ltorch_python -o build/lib.linux-x86_64-cpython-38/torch_scatter/_scatter_cpu.so -s
/home/sopikb/conda/envs/hmic-py38/bin/../lib/gcc/x86_64-conda-linux-gnu/11.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find /lib64/libpthread.so.0: No such file or directory
/home/sopikb/conda/envs/hmic-py38/bin/../lib/gcc/x86_64-conda-linux-gnu/11.2.0/../../../../x86_64-conda-linux-gnu/bin/ld: cannot find /usr/lib64/libpthread_nonshared.a: No such file or directory
collect2: error: ld returned 1 exit status
error: command '/home/sopikb/conda/envs/hmic-py38/bin/g++' failed with exit code 1
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
WARNING: No metadata found in /home/sopikb/.local/lib/python3.8/site-packages
Rolling back uninstall of torch-scatter
Moving to /home/sopikb/.local/lib/python3.8/site-packages/test/__init__.py
from /tmp/pip-uninstall-_x4_ler4/__init__.py
Moving to /home/sopikb/.local/lib/python3.8/site-packages/test/__pycache__/__init__.cpython-38.pyc
from /tmp/pip-uninstall-_x4_ler4/__pycache__/__init__.cpython-38.pyc
Moving to /home/sopikb/.local/lib/python3.8/site-packages/test/__pycache__/test_broadcasting.cpython-38.pyc
from /tmp/pip-uninstall-_x4_ler4/__pycache__/test_broadcasting.cpython-38.pyc
Moving to /home/sopikb/.local/lib/python3.8/site-packages/test/__pycache__/test_gather.cpython-38.pyc
from /tmp/pip-uninstall-_x4_ler4/__pycache__/test_gather.cpython-38.pyc
Moving to /home/sopikb/.local/lib/python3.8/site-packages/test/__pycache__/test_multi_gpu.cpython-38.pyc
from /tmp/pip-uninstall-_x4_ler4/__pycache__/test_multi_gpu.cpython-38.pyc
Moving to /home/sopikb/.local/lib/python3.8/site-packages/test/__pycache__/test_scatter.cpython-38.pyc
from /tmp/pip-uninstall-_x4_ler4/__pycache__/test_scatter.cpython-38.pyc
Moving to /home/sopikb/.local/lib/python3.8/site-packages/test/__pycache__/test_segment.cpython-38.pyc
from /tmp/pip-uninstall-_x4_ler4/__pycache__/test_segment.cpython-38.pyc
Moving to /home/sopikb/.local/lib/python3.8/site-packages/test/__pycache__/test_zero_tensors.cpython-38.pyc
from /tmp/pip-uninstall-_x4_ler4/__pycache__/test_zero_tensors.cpython-38.pyc
Moving to /home/sopikb/.local/lib/python3.8/site-packages/test/__pycache__/utils.cpython-38.pyc
from /tmp/pip-uninstall-_x4_ler4/__pycache__/utils.cpython-38.pyc
Moving to /home/sopikb/.local/lib/python3.8/site-packages/test/test_broadcasting.py
from /tmp/pip-uninstall-_x4_ler4/test_broadcasting.py
Moving to /home/sopikb/.local/lib/python3.8/site-packages/test/test_gather.py
from /tmp/pip-uninstall-_x4_ler4/test_gather.py
Moving to /home/sopikb/.local/lib/python3.8/site-packages/test/test_multi_gpu.py
from /tmp/pip-uninstall-_x4_ler4/test_multi_gpu.py
Moving to /home/sopikb/.local/lib/python3.8/site-packages/test/test_scatter.py
from /tmp/pip-uninstall-_x4_ler4/test_scatter.py
Moving to /home/sopikb/.local/lib/python3.8/site-packages/test/test_segment.py
from /tmp/pip-uninstall-_x4_ler4/test_segment.py
Moving to /home/sopikb/.local/lib/python3.8/site-packages/test/test_zero_tensors.py
from /tmp/pip-uninstall-_x4_ler4/test_zero_tensors.py
Moving to /home/sopikb/.local/lib/python3.8/site-packages/test/utils.py
from /tmp/pip-uninstall-_x4_ler4/utils.py
Moving to /home/sopikb/.local/lib/python3.8/site-packages/torch_scatter-2.0.9.dist-info/
from /home/sopikb/.local/lib/python3.8/site-packages/~orch_scatter-2.0.9.dist-info
Moving to /home/sopikb/.local/lib/python3.8/site-packages/torch_scatter/
from /home/sopikb/.local/lib/python3.8/site-packages/~orch_scatter
error: legacy-install-failure
× Encountered error while trying to install package.
╰─> torch-scatter
note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.
at ~/conda/envs/hmic-py38/lib/python3.8/site-packages/poetry/utils/env.py:1195 in _run
1191│ output = subprocess.check_output(
1192│ cmd, stderr=subprocess.STDOUT, **kwargs
1193│ )
1194│ except CalledProcessError as e:
→ 1195│ raise EnvCommandError(e, input=input_)
1196│
1197│ return decode(output)
1198│
1199│ def execute(self, bin, *args, **kwargs):
I think the first error can be fixed by adding wheel to install_requires, right? Can you see if that fixes issues on your end? The cannot find /lib64/libpthread.so.0 error also might indicate that you are running this with an incompatible version of gcc?
This issue had no activity for 6 months. It will be closed in 2 weeks unless there is some new activity. Is this issue already resolved?
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}