pip-compile fails on a read-only project

[IRDonch]

If pip-compile is given a setup.py file from a directory that the current user doesn't have write permissions for, it fails.

Environment Versions

1. OS: Ubuntu 18.04
2. Python version: 3.6.9
3. pip version: 21.0.1
4. pip-tools version: 6.1.0

Steps to replicate

$ mkdir test-proj
$ echo 'import setuptools; setuptools.setup()' > test-proj/setup.py
$ sudo chown -R root:root test-proj
$ sudo chmod -R o-w test-proj
$ /tmp/venv/bin/pip-compile -otest.txt test-proj/setup.py

Expected result

pip-compile should generate a (blank) requirements file. This used to work with pip-tools 5.5.0.

Actual result

Traceback (most recent call last):
  File "/tmp/venv/bin/pip-compile", line 8, in <module>
    sys.exit(cli())
  File "/tmp/venv/lib/python3.6/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/tmp/venv/lib/python3.6/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/tmp/venv/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/tmp/venv/lib/python3.6/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/tmp/venv/lib/python3.6/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/tmp/venv/lib/python3.6/site-packages/piptools/scripts/compile.py", line 379, in cli
    dist = meta.load(os.path.dirname(os.path.abspath(src_file)))
  File "/tmp/venv/lib/python3.6/site-packages/pep517/meta.py", line 71, in load
    path = Path(build_as_zip(builder))
  File "/tmp/venv/lib/python3.6/site-packages/pep517/meta.py", line 58, in build_as_zip
    builder(dest=out_dir)
  File "/tmp/venv/lib/python3.6/site-packages/pep517/meta.py", line 53, in build
    _prep_meta(hooks, env, dest)
  File "/tmp/venv/lib/python3.6/site-packages/pep517/meta.py", line 28, in _prep_meta
    reqs = hooks.get_requires_for_build_wheel({})
  File "/tmp/venv/lib/python3.6/site-packages/pep517/wrappers.py", line 169, in get_requires_for_build_wheel
    'config_settings': config_settings
  File "/tmp/venv/lib/python3.6/site-packages/pep517/wrappers.py", line 268, in _call_hook
    extra_environ=extra_environ
  File "/tmp/venv/lib/python3.6/site-packages/pep517/wrappers.py", line 75, in quiet_subprocess_runner
    check_output(cmd, cwd=cwd, env=env, stderr=STDOUT)
  File "/usr/lib/python3.6/subprocess.py", line 356, in check_output
    **kwargs).stdout
  File "/usr/lib/python3.6/subprocess.py", line 438, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['/tmp/venv/bin/python3', '/tmp/venv/lib/python3.6/site-packages/pep517/in_process/_in_process.py', 'get_requires_for_build_wheel', '/tmp/tmpau8kwkcj']' returned non-zero exit status 1.

[atugushev]

I've racked down the exception, here are details below.

Stacktrace:

# .venv/lib/python3.9/site-packages/pep517/wrappers.py(269)
 266                 # Run the hook in a subprocess
 267                 with _in_proc_script_path() as script:
 268                     python = self.python_executable
 269  ->                 self._subprocess_runner(
 270                         [python, abspath(str(script)), hook_name, td],
 271                         cwd=self.source_dir,
 272                         extra_environ=extra_environ
 273                     )

Exception:

running egg_info
creating UNKNOWN.egg-info
error: could not create 'UNKNOWN.egg-info': Permission denied

It tries to generate files in cwd=self.source_dir folder which is RO for the current user ATM. This could be a bug on pep517 package. Would you like to open an issue on pep517 tracker?

[IRDonch]

Would you like to open an issue on pep517 tracker?

I would, but I wouldn't know how to describe the issue in terms of pep517 - I've never used it myself, and I don't know how pip-compile uses it.

Moreover, I'm not sure pep517 is at fault here. pip itself uses (a vendored copy of) pep517, but you don't get this problem with pip wheel or pip install.

[atugushev]

Moreover, I'm not sure pep517 is at fault here. pip itself uses (a vendored copy of) pep517, but you don't get this problem with pip wheel or pip install.

That's interesting. I've tried pip install -e test-proj and it doesn't work either, see:

$ pip install -e test-proj
Obtaining file:///Users/albert/Projects/pip-tools/test-proj
Installing collected packages: UNKNOWN
  Running setup.py develop for UNKNOWN
    ERROR: Command errored out with exit status 1:
     command: /Users/albert/Projects/pip-tools/.venv/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/Users/albert/Projects/pip-tools/test-proj/setup.py'"'"'; __file__='"'"'/Users/albert/Projects/pip-tools/test-proj/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps
         cwd: /Users/albert/Projects/pip-tools/test-proj/
    Complete output (4 lines):
    running develop
    running egg_info
    creating UNKNOWN.egg-info
    error: could not create 'UNKNOWN.egg-info': Permission denied
    ----------------------------------------
ERROR: Command errored out with exit status 1: /Users/albert/Projects/pip-tools/.venv/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/Users/albert/Projects/pip-tools/test-proj/setup.py'"'"'; __file__='"'"'/Users/albert/Projects/pip-tools/test-proj/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps Check the logs for full command output.

But pip install test-proj works:

$ pip install test-proj
Processing ./test-proj
  DEPRECATION: A future pip version will change local packages to be built in-place without first copying to a temporary directory. We recommend you use --use-feature=in-tree-build to test your packages with this new behavior before it becomes the default.
   pip 21.3 will remove support for this functionality. You can find discussion regarding this at https://github.com/pypa/pip/issues/7555.
Building wheels for collected packages: UNKNOWN
  Building wheel for UNKNOWN (setup.py) ... done
  Created wheel for UNKNOWN: filename=UNKNOWN-0.0.0-py3-none-any.whl size=981 sha256=3ab8696e9ac9b281a7765319dabc20359782b89291a316dd48b4d3e21df2274f
  Stored in directory: /Users/albert/Library/Caches/pip/wheels/8d/d7/68/9d1515b4b2740fdc8c8ed59bb67df9d1d73dc03a05c5b94b9a
Successfully built UNKNOWN
Installing collected packages: UNKNOWN
  Attempting uninstall: UNKNOWN
    Found existing installation: UNKNOWN 0.0.0
    Uninstalling UNKNOWN-0.0.0:
      Successfully uninstalled UNKNOWN-0.0.0
Successfully installed UNKNOWN-0.0.0

[webknjaz]

@atugushev editable installs don't use PEP 517 because it's not specified in the said PEP. There are discussions about coming up with a standard for editable installs going on PyPA forums for the past few years but there's still no reference implementation example, nor a PEP proposal at this point.

pip falls back to invoking setup.py develop for editable installs (as visible in your first log) and so it has nothing to do with the issue described above.

Bear in mind that pip creates an isolated virtualenv layout for performing the builds under /tmp (it's not one folder but two or three, glued together via $PYTHONPATH) where it installs the deps and invokes the PEP 517 build backend. And since pip creates those dirs, the backend doesn't have any problem writing files there.

I haven't looked into the code of pip-tools but my educated guess would be that it doesn't copy the project to a temporary directory with proper privileges set. Don't bother creating an issue in the pypa/pep517 repo — it's out of the scope of the lib. Just follow examples of pypa/pip and pypa/build that do this properly (both have an opt-out flag to disable an isolated build env creation but the default is that they manage the build env).

[chrihartl]

I have the exact same issue when running pip-compile (pip-tools version 6.4.0) inside a Centos Docker container (as part of a Docker build or interactively in the container).

UPDATE 2021-11-15: I was able to fix my problem by setting the environment variable XDG_CACHE_HOME when running pip-compile in my Dockerfile, as follows.

RUN . venv/bin/activate && XDG_CACHE_HOME=./.cache pip-compile

The default value of this variable would have been /.cache, but this path is not accessible when running as a non-root user. Btw, it was NOT sufficient to set the cache-dir in the pip.conf.