Surprise
Surprise copied to clipboard
Published
20 hours ago •
NicolasHug
NicolasHug
Error installing package using poetry or using pip with --use-pep517 flag. PEP 517, 518, 621 and 631 compliance
Description
Error installing scikit-surprise using poetry . Poetry is a python packaging and dependency management tool that I intended to use for my project.
Upon further investigation I discovered that the error is triggered by the --use-pep517 flag that poetry passes to pip during installation. The full comand that poetry executes is:
/home/mihaiblidaru/TFG/.venv/bin/python -m pip install --use-pep517 --disable-pip-version-check --prefix /home/mihaiblidaru/TFG/.venv --upgrade --no-deps /home/mihaiblidaru/.cache/pypoetry/artifacts/fa/12/78/2049d9ec612179e7df284f30f551e89918461601e3d0b91d5a060a12fb/scikit-surprise-1.1.3.tar.gz
I found no workaround, there is no way to configure if poetry executes pip with or without the --use-pep517 flag.
The package installation fails because the package is not PEP 517 compliant. PEP 517 (and extended in PEP 518, 621, 631) requires a project to have a pyproject.toml file which contains package metadata (that right now is mainly defined in setup.py) that includes a mandatory section where you can define build dependencies. Another change added in PEP 517 is that packages are build in isolation, that is, pip(or other front-end) builds the package in a isolated environment that only has the project build dependencies installed).
The installation fails because right now pip has no idea that numpy(or cython) has to be installed in the isolated environment before building, because that requirement it's not specified in any file.
While I understand that it's would be imposible to acomodate for all dependency managers in the world, PEP 517 seems to be pretty standard nowadays. It seem python packaging is moving away from the old setup.py file as it is a not very standardized, uniform way to manage project dependencies or build packages.
Steps/Code to Reproduce
Adding the package to a poetry managed project fails.
# Poetry
poetry add scikit-surprise
Also using pip with --use-pep517 flag directly from online pypi repo.
# Or using pip
pip3 install --use-pep517 --no-cache-dir scikit-surprise
Expected Results
Package is installed successfully.
Actual Results
The error trace from poetry
Very long error trace
$ poetry add scikit-surprise
Using version ^1.1.3 for scikit-surprise
Updating dependencies
Resolving dependencies... (1.5s)
Writing lock file
Package operations: 0 installs, 2 updates, 0 removals
• Updating pandas (1.3.5 -> 1.5.1): Installing...
• Updating scikit-surprise (1.1.3 /home/mihaiblidaru/TFG/surprise -> 1.1.3): Failed
CalledProcessError
Command '['/home/mihaiblidaru/TFG/.venv/bin/python', '-m', 'pip', 'install', '--use-pep517', '--disable-pip-version-check', '--prefix', '/home/mihaiblidaru/TFG/.venv', '--upgrade', '--no-deps', '/home/mihaiblidaru/.cache/pypoetry/artifacts/fa/12/78/2049d9ec612179e7df284f30f551e89918461601e3d0b91d5a060a12fb/scikit-surprise-1.1.3.tar.gz']' returned non-zero exit status 1.
at /usr/lib/python3.10/subprocess.py:524 in run
520│ # We don't call process.wait() as .__exit__ does that for us.
521│ raise
522│ retcode = process.poll()
523│ if check and retcode:
→ 524│ raise CalledProcessError(retcode, process.args,
525│ output=stdout, stderr=stderr)
526│ return CompletedProcess(process.args, retcode, stdout, stderr)
527│
528│
The following error occurred when trying to handle this error:
EnvCommandError
Command ['/home/mihaiblidaru/TFG/.venv/bin/python', '-m', 'pip', 'install', '--use-pep517', '--disable-pip-version-check', '--prefix', '/home/mihaiblidaru/TFG/.venv', '--upgrade', '--no-deps', '/home/mihaiblidaru/.cache/pypoetry/artifacts/fa/12/78/2049d9ec612179e7df284f30f551e89918461601e3d0b91d5a060a12fb/scikit-surprise-1.1.3.tar.gz'] errored with the following return code 1, and output:
Processing /home/mihaiblidaru/.cache/pypoetry/artifacts/fa/12/78/2049d9ec612179e7df284f30f551e89918461601e3d0b91d5a060a12fb/scikit-surprise-1.1.3.tar.gz
Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'error'
error: subprocess-exited-with-error
× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> [41 lines of output]
/tmp/pip-build-env-87wgugxs/overlay/lib/python3.10/site-packages/setuptools/installer.py:27: SetuptoolsDeprecationWarning: setuptools.installer is deprecated. Requirements should be satisfied by a PEP 517 installer.
warnings.warn(
/home/mihaiblidaru/TFG/.venv/bin/python: No module named pip
Traceback (most recent call last):
File "/tmp/pip-build-env-87wgugxs/overlay/lib/python3.10/site-packages/setuptools/installer.py", line 82, in fetch_build_egg
subprocess.check_call(cmd)
File "/usr/lib/python3.10/subprocess.py", line 369, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/home/mihaiblidaru/TFG/.venv/bin/python', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/tmp/tmp27vejxzu', '--quiet', 'numpy>=1.17.3']' returned non-zero exit status 1.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/mihaiblidaru/TFG/.venv/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 351, in <module>
main()
File "/home/mihaiblidaru/TFG/.venv/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 333, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
File "/home/mihaiblidaru/TFG/.venv/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 118, in get_requires_for_build_wheel
return hook(config_settings)
File "/tmp/pip-build-env-87wgugxs/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 338, in get_requires_for_build_wheel
return self._get_build_requires(config_settings, requirements=['wheel'])
File "/tmp/pip-build-env-87wgugxs/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 320, in _get_build_requires
self.run_setup()
File "/tmp/pip-build-env-87wgugxs/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 483, in run_setup
super(_BuildMetaLegacyBackend,
File "/tmp/pip-build-env-87wgugxs/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 335, in run_setup
exec(code, locals())
File "<string>", line 65, in <module>
File "/tmp/pip-build-env-87wgugxs/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 875, in fetch_build_eggs
resolved_dists = pkg_resources.working_set.resolve(
File "/tmp/pip-build-env-87wgugxs/overlay/lib/python3.10/site-packages/pkg_resources/__init__.py", line 789, in resolve
dist = best[req.key] = env.best_match(
File "/tmp/pip-build-env-87wgugxs/overlay/lib/python3.10/site-packages/pkg_resources/__init__.py", line 1075, in best_match
return self.obtain(req, installer)
File "/tmp/pip-build-env-87wgugxs/overlay/lib/python3.10/site-packages/pkg_resources/__init__.py", line 1087, in obtain
return installer(requirement)
File "/tmp/pip-build-env-87wgugxs/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 945, in fetch_build_egg
return fetch_build_egg(self, req)
File "/tmp/pip-build-env-87wgugxs/overlay/lib/python3.10/site-packages/setuptools/installer.py", line 84, in fetch_build_egg
raise DistutilsError(str(e)) from e
distutils.errors.DistutilsError: Command '['/home/mihaiblidaru/TFG/.venv/bin/python', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/tmp/tmp27vejxzu', '--quiet', 'numpy>=1.17.3']' returned non-zero exit status 1.
[end of output]
• Updating scikit-surprise (1.1.3 /home/mihaiblidaru/TFG/surprise -> 1.1.3): Failed
CalledProcessError
Command '['/home/mihaiblidaru/TFG/.venv/bin/python', '-m', 'pip', 'install', '--use-pep517', '--disable-pip-version-check', '--prefix', '/home/mihaiblidaru/TFG/.venv', '--upgrade', '--no-deps', '/home/mihaiblidaru/.cache/pypoetry/artifacts/fa/12/78/2049d9ec612179e7df284f30f551e89918461601e3d0b91d5a060a12fb/scikit-surprise-1.1.3.tar.gz']' returned non-zero exit status 1.
at /usr/lib/python3.10/subprocess.py:524 in run
520│ # We don't call process.wait() as .__exit__ does that for us.
521│ raise
522│ retcode = process.poll()
523│ if check and retcode:
→ 524│ raise CalledProcessError(retcode, process.args,
525│ output=stdout, stderr=stderr)
526│ return CompletedProcess(process.args, retcode, stdout, stderr)
527│
528│
The following error occurred when trying to handle this error:
EnvCommandError
Command ['/home/mihaiblidaru/TFG/.venv/bin/python', '-m', 'pip', 'install', '--use-pep517', '--disable-pip-version-check', '--prefix', '/home/mihaiblidaru/TFG/.venv', '--upgrade', '--no-deps', '/home/mihaiblidaru/.cache/pypoetry/artifacts/fa/12/78/2049d9ec612179e7df284f30f551e89918461601e3d0b91d5a060a12fb/scikit-surprise-1.1.3.tar.gz'] errored with the following return code 1, and output:
Processing /home/mihaiblidaru/.cache/pypoetry/artifacts/fa/12/78/2049d9ec612179e7df284f30f551e89918461601e3d0b91d5a060a12fb/scikit-surprise-1.1.3.tar.gz
Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'error'
error: subprocess-exited-with-error
× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> [41 lines of output]
/tmp/pip-build-env-87wgugxs/overlay/lib/python3.10/site-packages/setuptools/installer.py:27: SetuptoolsDeprecationWarning: setuptools.installer is deprecated. Requirements should be satisfied by a PEP 517 installer.
warnings.warn(
/home/mihaiblidaru/TFG/.venv/bin/python: No module named pip
Traceback (most recent call last):
File "/tmp/pip-build-env-87wgugxs/overlay/lib/python3.10/site-packages/setuptools/installer.py", line 82, in fetch_build_egg
subprocess.check_call(cmd)
File "/usr/lib/python3.10/subprocess.py", line 369, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/home/mihaiblidaru/TFG/.venv/bin/python', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/tmp/tmp27vejxzu', '--quiet', 'numpy>=1.17.3']' returned non-zero exit status 1.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/mihaiblidaru/TFG/.venv/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 351, in <module>
main()
File "/home/mihaiblidaru/TFG/.venv/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 333, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
File "/home/mihaiblidaru/TFG/.venv/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 118, in get_requires_for_build_wheel
return hook(config_settings)
File "/tmp/pip-build-env-87wgugxs/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 338, in get_requires_for_build_wheel
return self._get_build_requires(config_settings, requirements=['wheel'])
File "/tmp/pip-build-env-87wgugxs/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 320, in _get_build_requires
self.run_setup()
File "/tmp/pip-build-env-87wgugxs/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 483, in run_setup
super(_BuildMetaLegacyBackend,
File "/tmp/pip-build-env-87wgugxs/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 335, in run_setup
exec(code, locals())
File "<string>", line 65, in <module>
File "/tmp/pip-build-env-87wgugxs/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 875, in fetch_build_eggs
resolved_dists = pkg_resources.working_set.resolve(
File "/tmp/pip-build-env-87wgugxs/overlay/lib/python3.10/site-packages/pkg_resources/__init__.py", line 789, in resolve
dist = best[req.key] = env.best_match(
File "/tmp/pip-build-env-87wgugxs/overlay/lib/python3.10/site-packages/pkg_resources/__init__.py", line 1075, in best_match
return self.obtain(req, installer)
File "/tmp/pip-build-env-87wgugxs/overlay/lib/python3.10/site-packages/pkg_resources/__init__.py", line 1087, in obtain
return installer(requirement)
File "/tmp/pip-build-env-87wgugxs/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 945, in fetch_build_egg
return fetch_build_egg(self, req)
File "/tmp/pip-build-env-87wgugxs/overlay/lib/python3.10/site-packages/setuptools/installer.py", line 84, in fetch_build_egg
raise DistutilsError(str(e)) from e
distutils.errors.DistutilsError: Command '['/home/mihaiblidaru/TFG/.venv/bin/python', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/tmp/tmp27vejxzu', '--quiet', 'numpy>=1.17.3']' returned non-zero exit status 1.
[end of output]
• Updating pandas (1.3.5 -> 1.5.1)
• Updating scikit-surprise (1.1.3 /home/mihaiblidaru/TFG/surprise -> 1.1.3): Failed
CalledProcessError
Command '['/home/mihaiblidaru/TFG/.venv/bin/python', '-m', 'pip', 'install', '--use-pep517', '--disable-pip-version-check', '--prefix', '/home/mihaiblidaru/TFG/.venv', '--upgrade', '--no-deps', '/home/mihaiblidaru/.cache/pypoetry/artifacts/fa/12/78/2049d9ec612179e7df284f30f551e89918461601e3d0b91d5a060a12fb/scikit-surprise-1.1.3.tar.gz']' returned non-zero exit status 1.
at /usr/lib/python3.10/subprocess.py:524 in run
520│ # We don't call process.wait() as .__exit__ does that for us.
521│ raise
522│ retcode = process.poll()
523│ if check and retcode:
→ 524│ raise CalledProcessError(retcode, process.args,
525│ output=stdout, stderr=stderr)
526│ return CompletedProcess(process.args, retcode, stdout, stderr)
527│
528│
The following error occurred when trying to handle this error:
EnvCommandError
Command ['/home/mihaiblidaru/TFG/.venv/bin/python', '-m', 'pip', 'install', '--use-pep517', '--disable-pip-version-check', '--prefix', '/home/mihaiblidaru/TFG/.venv', '--upgrade', '--no-deps', '/home/mihaiblidaru/.cache/pypoetry/artifacts/fa/12/78/2049d9ec612179e7df284f30f551e89918461601e3d0b91d5a060a12fb/scikit-surprise-1.1.3.tar.gz'] errored with the following return code 1, and output:
Processing /home/mihaiblidaru/.cache/pypoetry/artifacts/fa/12/78/2049d9ec612179e7df284f30f551e89918461601e3d0b91d5a060a12fb/scikit-surprise-1.1.3.tar.gz
Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'error'
error: subprocess-exited-with-error
× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> [41 lines of output]
/tmp/pip-build-env-87wgugxs/overlay/lib/python3.10/site-packages/setuptools/installer.py:27: SetuptoolsDeprecationWarning: setuptools.installer is deprecated. Requirements should be satisfied by a PEP 517 installer.
warnings.warn(
/home/mihaiblidaru/TFG/.venv/bin/python: No module named pip
Traceback (most recent call last):
File "/tmp/pip-build-env-87wgugxs/overlay/lib/python3.10/site-packages/setuptools/installer.py", line 82, in fetch_build_egg
subprocess.check_call(cmd)
File "/usr/lib/python3.10/subprocess.py", line 369, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/home/mihaiblidaru/TFG/.venv/bin/python', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/tmp/tmp27vejxzu', '--quiet', 'numpy>=1.17.3']' returned non-zero exit status 1.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/mihaiblidaru/TFG/.venv/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 351, in <module>
main()
File "/home/mihaiblidaru/TFG/.venv/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 333, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
File "/home/mihaiblidaru/TFG/.venv/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 118, in get_requires_for_build_wheel
return hook(config_settings)
File "/tmp/pip-build-env-87wgugxs/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 338, in get_requires_for_build_wheel
return self._get_build_requires(config_settings, requirements=['wheel'])
File "/tmp/pip-build-env-87wgugxs/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 320, in _get_build_requires
self.run_setup()
File "/tmp/pip-build-env-87wgugxs/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 483, in run_setup
super(_BuildMetaLegacyBackend,
File "/tmp/pip-build-env-87wgugxs/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 335, in run_setup
exec(code, locals())
File "<string>", line 65, in <module>
File "/tmp/pip-build-env-87wgugxs/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 875, in fetch_build_eggs
resolved_dists = pkg_resources.working_set.resolve(
File "/tmp/pip-build-env-87wgugxs/overlay/lib/python3.10/site-packages/pkg_resources/__init__.py", line 789, in resolve
dist = best[req.key] = env.best_match(
File "/tmp/pip-build-env-87wgugxs/overlay/lib/python3.10/site-packages/pkg_resources/__init__.py", line 1075, in best_match
return self.obtain(req, installer)
File "/tmp/pip-build-env-87wgugxs/overlay/lib/python3.10/site-packages/pkg_resources/__init__.py", line 1087, in obtain
return installer(requirement)
File "/tmp/pip-build-env-87wgugxs/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 945, in fetch_build_egg
return fetch_build_egg(self, req)
File "/tmp/pip-build-env-87wgugxs/overlay/lib/python3.10/site-packages/setuptools/installer.py", line 84, in fetch_build_egg
raise DistutilsError(str(e)) from e
distutils.errors.DistutilsError: Command '['/home/mihaiblidaru/TFG/.venv/bin/python', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/tmp/tmp27vejxzu', '--quiet', 'numpy>=1.17.3']' returned non-zero exit status 1.
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error
× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.
at ~/.local/share/pypoetry/venv/lib/python3.10/site-packages/poetry/utils/env.py:1476 in _run
1472│ output = subprocess.check_output(
1473│ command, stderr=subprocess.STDOUT, env=env, **kwargs
1474│ )
1475│ except CalledProcessError as e:
→ 1476│ raise EnvCommandError(e, input=input_)
1477│
1478│ return decode(output)
1479│
1480│ def execute(self, bin: str, *args: str, **kwargs: Any) -> int:
The following error occurred when trying to handle this error:
PoetryException
Failed to install /home/mihaiblidaru/.cache/pypoetry/artifacts/fa/12/78/2049d9ec612179e7df284f30f551e89918461601e3d0b91d5a060a12fb/scikit-surprise-1.1.3.tar.gz
at ~/.local/share/pypoetry/venv/lib/python3.10/site-packages/poetry/utils/pip.py:51 in pip_install
47│
48│ try:
49│ return environment.run_pip(*args)
50│ except EnvCommandError as e:
→ 51│ raise PoetryException(f"Failed to install {path.as_posix()}") from e
52│
Versions
The surprise version I tried to install is 1.1.3 as shown in the error trace above.
>>> import platform; print(platform.platform())
Linux-5.15.0-52-generic-x86_64-with-glibc2.35
>>> import sys; print("Python", sys.version)
Python 3.10.6 (main, Nov 2 2022, 18:53:38) [GCC 11.3.0]
>>> import surprise; print("surprise", surprise.__version__)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'surprise'
I tried to fix it myself https://github.com/mihaiblidaru/Surprise/tree/pep517_compliant_package but I am reluctant to open a pull-request yet because this slightly changes how the package is build and how the project dependencies are maintained.
Before
python setup.py sdist
Now
python3 -m build --sdist
The generated package is a (almost) byte level copy of the one that you can download from pypi today, but it can be installed with the --use-pep517 pip flag and also trough poetry
I'm also using poetry, but with docker using python:3.10-buster image, and I'm also unable to complete poetry add scikit-surprise. Temporary solution for me is to add RUN pip install scikit-surprise after installation dependencies with poetry. Would like to eventually add this package to poetry dependencies.
Another alternative would be to have pre-built python wheels on pypi that to not require building at installation which would also eliminate the requirement to have a C compiler installed
Encountered the same issue.
As I see it the problem is not in absence of pyproject.toml but in a manual installation of numpy package inside setup.py: https://github.com/NicolasHug/Surprise/blob/master/setup.py#L65-L78
I'm unaware why it's there, because numpy is already in requirements.txt
I have no idea how to fix it on my side, without forking the repo 🤔
