Cannot install into user site directory with editable source.

[agoose77]

In build_env.py the environment variable PYTHONNOUSERSITE is set: https://github.com/pypa/pip/blob/7b02273f3e40aaab95027d5d560bd8e76ab623e4/src/pip/_internal/build_env.py#L127

This prevents editable installs with PYTHONUSERBASE=<some-user-base> pip3.8 install --user -e <some-file-path> from succeeding if the user does not have write access to the base-Python site-packages directory.

This is a snapshot of the error.

running develop
    WARNING: The user site-packages directory is disabled.
    error: can't create or remove files in install directory

It may be that it is a deliberate design choice to disable editable installs under a user site directory. I personally needed this feature, however, and I am therefore setting site.ENABLE_USER_SITE = True in setup.py as a workaround.

[ncoghlan]

This affects the pipenv local build instructions in https://pipenv.pypa.io/en/latest/dev/contributing/#development-setup:

• pip3 install -e . doesn't work as the global site-packages isn't writable by regular users
• pip3 install --user -e . doesn't work due to pip indicating the user site directory is disabled

I'm adapting @agoose77's workaround into a PIPENV_BOOTSTRAP=1 environment variable.

[tristanvb]

Ran into this today on debian 9, not sure what has changed but we have been using pip3 install --user -e . for a long time for development purposes, it is strange for this to suddenly stop working.

[lpsinger]

Seeing this on macOS as well.

[tgpfeiffer]

I have seen this failing for a project that uses pyproject.toml (PEP517 install flow), where it works fine for projects without pyproject.toml. @siddalmia has described the same behavior in https://github.com/pytorch/fairseq/issues/1977. I wonder if there is something in the PEP517 install flow that breaks this...?

[McSinyx]

@tgpfeiffer, please see the discussion on Discourse for more information. Aside from setup.py develop, another work around for packages with __main__.py is to run it directly, e.g. for pip: python src/pip.

[joshbode]

Following the tip from @agoose77, I'm putting this in my setup.py as a workaround:

import site
import sys
site.ENABLE_USER_SITE = "--user" in sys.argv[1:]
...

[ssbarnea]

Nobody was able to fix this in pip? The whole idea of pep517 was to reduce use of setup.py ending in its removal, not adding more junk into to it.

[pfmoore]

@ssbarnea PEP 517 doesn't support editable installs, so I don't see how PEP 517 is relevant here?

[maxnoe]

@pfmoore you can support an editable install in a project using PEP 517 by adding a rather minimal setup.py.

But then, it is impossible to have -e and --user

[akaihola]

@ssbarnea PEP 517 doesn't support editable installs, so I don't see how PEP 517 is relevant here?

I don't know, but doing a

pip install --user --no-use-pep517 -e .

succeeds, and after that a

pip install --user -e .

doesn't fail anymore. However, this still fails:

pip install --user --force-reinstall -e .

[jyn514]

I also ran into this today. Is this intentionally not supported? I'm not sure why, I'd find it useful.

[uranusjr]

I would encourage people affected by this to dig into the source. The code path described does not make much sense. The code path for an editable install should not go through build_env.py at all, so there is either a misidentification to the cause, or a more obscure bug than the sympton shows.

[agoose77]

@uranusjr if you check the code, the environment is used by the legacy builder: https://github.com/pypa/pip/blob/8a949a1c52ae47584242e5bb0155d1d1018275ae/src/pip/_internal/commands/install.py#L282 https://github.com/pypa/pip/blob/25114e1f9ed4c5fa2b0407a26266080a5ff1cb2c/src/pip/_internal/req/req_install.py#L751 https://github.com/pypa/pip/blob/25114e1f9ed4c5fa2b0407a26266080a5ff1cb2c/src/pip/_internal/operations/install/editable_legacy.py#L43

[kousu]

This is super bizarre. I can't have an editable install in a "modernized" pyproject.toml package? How am I supposed to develop it then?

I guess you're assuming that I'll just use virtualenvs all the time. But then why have a USER_SITE at all?

[kojiromike]

If I try --no-use-pep517 I get

ERROR: Disabling PEP 517 processing is invalid: project specifies a build backend of setuptools.build_meta in pyproject.toml

The only workaround I've found so far is to "hide" pyproject.toml temporarily, like

mv pyproject.toml pyproject.nope
pip install -e .  # etc
mv pyproject.nope pyproject.toml

While this feels a little gross, I guess this is "fine" in the sense that there isn't much use for me having a pyproject.toml if I'm going to do an editable install anyway. I also suppose this means PEP517 might still be used for other packages that are dependencies, which would be a good thing.

[ssbarnea]

I am wondering why this bug still has the needstriage label considering that is so easy to reproduce.

[BenLand100]

Hit this issue today trying out the currently-recommended way to package Python code for a new package. Everything works fine exclusively using the previously-recommended setup.py, but trying to use a setup.cfg or even including a pyproject.toml results in failing to install editable source. Unfortunate that this issue has been around for more than a year now...

[Stannislav]

Issue still present with pip version pip 21.1.2.

[pradyunsg]

Can someone provide a self-contained reproducer for this?

[RuRo]

@pradyunsg

# 0) Prepare a reproducible environment, you can probably skip this step, but just in case
docker run -it python:3.6 /bin/bash
useradd -m -s /bin/bash someone
su someone
cd $(mktemp -d)

# 1) Prepare setup.py
cat >setup.py <<EOF
from setuptools import setup

if __name__ == "__main__":
    setup()
EOF

# 2) Prepare setup.cfg
cat >setup.cfg <<EOF
[metadata]
name = thepackage
[options]
package_dir =
    =src
packages = find_namespace:
[options.packages.find]
where = src
EOF

# 3) Prepare pyproject.toml
cat >pyproject.toml <<EOF
[build-system]
build-backend = "setuptools.build_meta"
requires = ["setuptools", "wheel"]
EOF

# 4) Prepare dummy package data
mkdir src
mkdir src/thepackage
touch src/thepackage/__init__.py

# 5) Boom!
pip install --user --editable .

Not sure if this is a minimal example, probably not, but I don't have the time to simplify this any further.

[purplesyringa]

Has anyone started working on this yet?

[ssbarnea]

Go for it!

[RuRo]

Ran into this again today. Very surprised, that this issue is still not fixed.

[dschwoerer]

Another workaround is --prefix=~/.local rather than --user

[Arnie97]

On my Mac, the minimal reproducible example given by RuRo raised an exception, even if I did not use -e.

$ cd $(mktemp -d)

...  (omitted some lines in RuRo's example, from step 1 to 4)

$ touch src/thepackage/__init__.py

$ pip install --user .
/var/folders/pz/dgm9jlg975l25gs04djqhw1h0000gn/T/tmp.T8Y3Yn0p
Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
Processing /private/var/folders/pz/dgm9jlg975l25gs04djqhw1h0000gn/T/tmp.T8Y3Yn0p
  Installing build dependencies ... error
  ERROR: Command errored out with exit status 1:
   command: /Library/Developer/CommandLineTools/usr/bin/python3 /private/var/folders/pz/dgm9jlg975l25gs04djqhw1h0000gn/T/pip-standalone-pip-9yb73kin/__env_pip__.zip/pip install --ignore-installed --no-user --prefix /private/var/folders/pz/dgm9jlg975l25gs04djqhw1h0000gn/T/pip-build-env-9utn6q_t/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.tuna.tsinghua.edu.cn/simple -- setuptools wheel
       cwd: None
  Complete output (29 lines):
  Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
  Collecting setuptools
    Downloading https://pypi.tuna.tsinghua.edu.cn/packages/40/a9/7deac76c58fa47c95360116a06b53b9b62f6db11336fe61b6ab53784d98b/setuptools-59.5.0-py3-none-any.whl (952 kB)
  Collecting wheel
    Downloading https://pypi.tuna.tsinghua.edu.cn/packages/04/80/cad93b40262f5d09f6de82adbee452fd43cdff60830b56a74c5930f7e277/wheel-0.37.0-py2.py3-none-any.whl (35 kB)
  Installing collected packages: wheel, setuptools
    WARNING: Value for scheme.platlib does not match. Please report this to <https://github.com/pypa/pip/issues/10151>
    distutils: /Library/Python/3.7/site-packages
    sysconfig: /private/var/folders/pz/dgm9jlg975l25gs04djqhw1h0000gn/T/pip-build-env-9utn6q_t/overlay/lib/python3.7/site-packages
    WARNING: Value for scheme.purelib does not match. Please report this to <https://github.com/pypa/pip/issues/10151>
    distutils: /Library/Python/3.7/site-packages
    sysconfig: /private/var/folders/pz/dgm9jlg975l25gs04djqhw1h0000gn/T/pip-build-env-9utn6q_t/overlay/lib/python3.7/site-packages
    WARNING: Value for scheme.headers does not match. Please report this to <https://github.com/pypa/pip/issues/10151>
    distutils: /Library/Python/3.7m/include/wheel
    sysconfig: /private/var/folders/pz/dgm9jlg975l25gs04djqhw1h0000gn/T/pip-build-env-9utn6q_t/overlay/include/python3.7m/wheel
    WARNING: Value for scheme.scripts does not match. Please report this to <https://github.com/pypa/pip/issues/10151>
    distutils: /usr/local/bin
    sysconfig: /private/var/folders/pz/dgm9jlg975l25gs04djqhw1h0000gn/T/pip-build-env-9utn6q_t/overlay/bin
    WARNING: Value for scheme.data does not match. Please report this to <https://github.com/pypa/pip/issues/10151>
    distutils: /Library/Python/3.7
    sysconfig: /private/var/folders/pz/dgm9jlg975l25gs04djqhw1h0000gn/T/pip-build-env-9utn6q_t/overlay
    WARNING: Additional context:
    user = False
    home = None
    root = None
    prefix = '/private/var/folders/pz/dgm9jlg975l25gs04djqhw1h0000gn/T/pip-build-env-9utn6q_t/overlay'
  ERROR: Could not install packages due to an OSError: [Errno 13] Permission denied: '/Library/Python/3.7'
  Consider using the `--user` option or check the permissions.

  ----------------------------------------
WARNING: Discarding file:///private/var/folders/pz/dgm9jlg975l25gs04djqhw1h0000gn/T/tmp.T8Y3Yn0p. Command errored out with exit status 1: /Library/Developer/CommandLineTools/usr/bin/python3 /private/var/folders/pz/dgm9jlg975l25gs04djqhw1h0000gn/T/pip-standalone-pip-9yb73kin/__env_pip__.zip/pip install --ignore-installed --no-user --prefix /private/var/folders/pz/dgm9jlg975l25gs04djqhw1h0000gn/T/pip-build-env-9utn6q_t/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.tuna.tsinghua.edu.cn/simple -- setuptools wheel Check the logs for full command output.
ERROR: Command errored out with exit status 1: /Library/Developer/CommandLineTools/usr/bin/python3 /private/var/folders/pz/dgm9jlg975l25gs04djqhw1h0000gn/T/pip-standalone-pip-9yb73kin/__env_pip__.zip/pip install --ignore-installed --no-user --prefix /private/var/folders/pz/dgm9jlg975l25gs04djqhw1h0000gn/T/pip-build-env-9utn6q_t/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.tuna.tsinghua.edu.cn/simple -- setuptools wheel Check the logs for full command output.
/var/folders/pz/dgm9jlg975l25gs04djqhw1h0000gn/T/tmp.T8Y3Yn0p

I don't understand why it was trying to install setuptools into the global prefix /Library/Python/3.7, since I already had latest setuptools in the user prefix /Users/Arnie97/Library/Python/3.7, as shown below:

$ sw_vers
ProductName:	Mac OS X
ProductVersion:	10.15.7
BuildVersion:	19H1419

$ which python3
/usr/bin/python3

$ python3 --version
Python 3.7.3
 
$ pip install -U --user pip setuptools wheel
Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
Requirement already satisfied: pip in /Users/Arnie97/Library/Python/3.7/lib/python/site-packages (21.3.1)
Requirement already satisfied: setuptools in /Users/Arnie97/Library/Python/3.7/lib/python/site-packages (59.5.0)
Requirement already satisfied: wheel in /Users/Arnie97/Library/Python/3.7/lib/python/site-packages (0.37.0)

[liudonghua123]

I have the same issue. My project is generated via putup pyoxidizer_demo (https://github.com/pyscaffold/pyscaffold), and my python is installed via ms store. I have tried --prefix using pip install --prefix="C:/Users/Liu.D.H/AppData/Local/Packages/PythonSoftwareFoundation.Python.3.10_qbz5n2kfra8p0/LocalCache/local-packages" -e -v . But it failed with ERROR: Can not combine '--user' and '--prefix' as they imply different installation locations.

Finally, I tried this (https://github.com/fgnt/paderbox/pull/58/files) solution and it worked. The changed setup.py is:

"""
    Setup file for pyoxidizer_demo.
    Use setup.cfg to configure your project.

    This file was generated with PyScaffold 4.1.1.
    PyScaffold helps you to put up the scaffold of your new Python project.
    Learn more under: https://pyscaffold.org/
"""
from setuptools import setup
import site
import sys
site.ENABLE_USER_SITE = '--user' in sys.argv[1:]

if __name__ == "__main__":
    try:
        setup(use_scm_version={"version_scheme": "no-guess-dev"})
    except:  # noqa
        print(
            "\n\nAn error occurred while building the project, "
            "please ensure you have the most updated version of setuptools, "
            "setuptools_scm and wheel with:\n"
            "   pip install -U setuptools setuptools_scm wheel\n\n"
        )
        raise

[boeddeker]

Are there any plans to fix this? It is annoying, that pip ignores the command line arguments, when a pyproject.toml exists. It can mess up an installation.

It is a pitty that so many projects have to add a workaround to their projects.

[pfmoore]

Are there any plans to fix this?

Not in the sense that any of the pip maintainers is currently expecting to work on it, no. It's simply a matter of too much work and not enough people, though - there's no objection in principle to a fix.

We're really relying on the possibility of someone who is actually affected by the issue having the time and willingness to contribute a fix, and we can't really plan for when or if that might happen. That's just the nature of open source, I'm afraid.

[pradyunsg]

You’re welcome to step up and fix this, because you clearly care enough to step up and complain that this hasn’t been fixed.

[boeddeker]

I will take a look, but I don't know, how pip internally works (e.g. why is that class only entered when pyproject.toml exists, why disabling the user install with an envvariable, why does the reset not work or is the actual installation done in that context, ...).

Not sure, if I am able to understand enough of pip in a reasonable time. If I get an idea how to fix it, I will open a PR.