Cannot use pipenv --system and --python together

[benney-au-le]

Issue description

I want to use pipenv to install dependencies in a python container without creating new virtual environments. Using pipenv install --system --python /usr/local/bin/python --site-packages creates a new virtual environment.

Expected result

Running pipenv install --system --python /usr/local/bin/python --site-packages should not create a new virtual environment.

Actual result

Running pipenv install --system --python /usr/local/bin/python --site-packages creates a new virtual environment.

Steps to replicate

Create a dockerfile

FROM mcr.microsoft.com/vscode/devcontainers/python:0-3.9-bullseye
USER vscode
COPY Pipfile .
COPY Pipfile.lock .
RUN pipenv install --deploy --verbose --system --python /usr/local/bin/python 

Build this dockerfile. Since this is a container, we don't need a separate virtual env inside a virtual env.

The base image has a separate virtual environments py-util. For instance there are separate python interpreters as follows:

/usr/local/py-utils/venvs/pipenv/bin/python
/usr/local/py-utils/venvs/pylint/bin/python
/usr/local/py-utils/venvs/bandit/bin/python

I would like to install the pipenv inside the system wide python interpreter /usr/local/bin/python and not pipenv's one or its nested one.

$ pipenv --support

Pipenv version: '2022.1.8'

Pipenv location: '/usr/local/py-utils/venvs/pipenv/lib/python3.9/site-packages/pipenv'

Python location: '/usr/local/py-utils/venvs/pipenv/bin/python'

Python installations found:

• 3.9.10: /usr/local/bin/python
• 3.9.10: /usr/local/bin/python3.9
• 3.9.10: /usr/local/bin/python3
• 3.9.10: /usr/local/py-utils/venvs/pipenv/bin/python3.9
• 3.9.10: /usr/local/py-utils/venvs/pipenv/bin/python3
• 3.9.10: /usr/local/py-utils/venvs/pipenv/bin/python
• 3.9.2: /usr/bin/python3.9
• 3.9.2: /usr/bin/python3

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.9.10',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '5.10.60.1-microsoft-standard-WSL2',
 'platform_system': 'Linux',
 'platform_version': '#1 SMP Wed Aug 25 23:20:18 UTC 2021',
 'python_full_version': '3.9.10',
 'python_version': '3.9',
 'sys_platform': 'linux'}

System environment variables:

• SHELL
• COLORTERM
• NVM_INC
• TERM_PROGRAM_VERSION
• HOSTNAME
• PYTHON_VERSION
• GOOGLE_CLOUD_PROJECT
• SSH_AUTH_SOCK
• REMOTE_CONTAINERS_IPC
• PWD
• PIPX_BIN_DIR
• GOOGLE_APPLICATION_CREDENTIALS
• PYTHON_SETUPTOOLS_VERSION
• VSCODE_GIT_ASKPASS_NODE
• HOME
• LANG
• LS_COLORS
• NVM_SYMLINK_CURRENT
• REMOTE_CONTAINERS
• CLOUDSDK_CONFIG
• GPG_KEY
• GIT_ASKPASS
• NVM_DIR
• VSCODE_GIT_ASKPASS_EXTRA_ARGS
• TERM
• REMOTE_CONTAINERS_SOCKETS
• USER
• VSCODE_GIT_IPC_HANDLE
• PIPX_HOME
• SHLVL
• NVM_CD_FLAGS
• PROMPT_DIRTRIM
• PYTHON_PIP_VERSION
• PYTHON_GET_PIP_SHA256
• VSCODE_GIT_ASKPASS_MAIN
• PYTHON_GET_PIP_URL
• BROWSER
• PATH
• NVM_BIN
• TERM_PROGRAM
• VSCODE_IPC_HOOK_CLI
• _
• PIP_SHIMS_BASE_MODULE
• PIP_DISABLE_PIP_VERSION_CHECK
• PYTHONDONTWRITEBYTECODE
• PIP_PYTHON_PATH
• PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:

Debug–specific environment variables:

• PATH: /usr/local/share/nvm/versions/node/v16.14.0/bin:/vscode/vscode-server/bin/linux-x64/f80445acd5a3dadef24aa209168452a3d97cc326/bin/remote-cli:/usr/local/share/nvm/current/bin:/usr/local/bin:/usr/local/py-utils/venvs/pipenv/bin:/usr/local/share/nvm/versions/node/v16.14.0/bin:/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/py-utils/bin:/home/vscode/.local/bin:/usr/local/py-utils/bin
• SHELL: /bin/bash
• LANG: C.UTF-8
• PWD: /workspaces/similarity-graph

---

Contents of Pipfile ('/workspaces/similarity-graph/Pipfile'):

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
pandas = "==1.3.5"
google-cloud-bigquery = "==2.32.0"
networkx = "==2.6.3"
leidenalg = "==0.8.8"
matplotlib = "==3.5.1"
python-igraph = "==0.9.9"
pandas-gbq = "==0.17.0"
aiohttp = "~=3.8.1"
igraph = "~=0.9.6"
requests = "~=2.27.1"

[dev-packages]

[requires]
python_version = "3.9"

[matteius]

I am using the master branch, actually my branch that upgrades pip to 22.x, but I am getting that is not a valid command:

matteius@matteius-VirtualBox:~/shared-projects/pipenv-triage/pipenv-4973$ pipenv install --system --python /usr/bin/python --site-packages
Usage: pipenv install [OPTIONS] [PACKAGES]...

ERROR:: --system is intended to be used for Pipfile installation, not installation of specific packages. Aborting.
See also: --deploy flag.

Noting that unfortunately neither command --deploy or --system aren;t in the help text. I just checked the source and yeah its this block:

    # Write out the lockfile if it doesn't exist.
    if not project.lockfile_exists and not skip_lock:
        # Unless we're in a virtualenv not managed by pipenv, abort if we're
        # using the system's python.
        if (system or allow_global) and not (project.s.PIPENV_VIRTUALENV):
            raise exceptions.PipenvOptionsError(
                "--system",
                "--system is intended to be used for Pipfile installation, "
                "not installation of specific packages. Aborting.\n"
                "See also: --deploy flag."
            )
        else:
            click.echo(
                crayons.normal(fix_utf8("Pipfile.lock not found, creating..."), bold=True),
                err=True,
            )
            do_lock(
                project,
                system=system,
                pre=pre,
                keep_outdated=keep_outdated,
                write=True,
                pypi_mirror=pypi_mirror,

I found that I could do what you want I think by doing: pipenv lock --python /usr/bin/python This creates the lockfile and a new virtualenv. pipenv --rm This removed the new virtualenv pipenv sync --python /usr/bin/python --system This installs the dependencies to the system level.

I hope that helps, there is probably a chance to clean up the documentation here and also we can consider your use case further. `

[benney-au-le]

Hey thanks for your quick reply! I tried the command on pipenv version 2022.1.8 (prob not master) and I couldn't get the desired outcome. I ran command pipenv sync --verbose --system --python /usr/local/bin/python

This resulted in the libraries being installed pipenv's venv rather than the system python's site packages.

/usr/local/py-utils/venvs/pipenv/bin/python -m pip install --verbose --upgrade --require-hashes --no-deps --exists-action=i -r /tmp/pipenv-e_7h_48q-requirements/pipenv-zfuoymjp-requirement.txt -i https://pypi.org/simple

You can see from the output above, even after specifying python interpreter, a different interpreter is being used to run pip.

Am I using pipenv in the incorrect way? My goal is:

• to take advantage of pipenv's lock file, scripts function and deterministic docker builds
• while keeping the docker image smallish by not creating new virtual environments

[matteius]

@benney-au-le I will have to think more about it, but I still think that recommended best practice is to have the virtualenv in the Docker container anyway and not run the application as root. @oz123 recently added to the documentation this section with an example Dockerfile: https://pipenv.pypa.io/en/latest/basics/#pipenv-and-docker-containers

[rbuckland]

@matteius That is good documentation by @oz123 on docker containers (referenced above). The challenge we face is the containers used are for Development, and therefore is the virtual environment, encapsulating more than just the python eco-system (other languages also)

recommended best practice is to have the virtualenv in the Docker container anyway and not run the application as root.

Over on the vscode-devcontainer github issue, a year or so back, there is a good thread on the challenges they were facing https://github.com/microsoft/vscode-dev-containers/pull/415

[OlmoBarberis]

Hi @benney-au-le ! I think that the problem is that pipenv is installed using pipx in this particular docker image. Pipx install it to an isolated virtual environment (/usr/local/py-utils/venvs/pipenv/bin/python) and pipenv uses it as its system python. You should uninstall pipenv using pipx uninstall pipenv

Then you can re-install it using pip install pipenv. This pipenv version will then see the correct system python interpreter and pipenv install --system works as expected. You should also update the PATH by adding the system python path (export PATH="$PATH:/usr/local/bin/python").

Your Dockerfile could look like this:

FROM mcr.microsoft.com/vscode/devcontainers/python:0-3.9-bullseye RUN pipx uninstall pipenv ENV PATH=$PATH:/usr/local/bin/python RUN pip install pipenv

USER vscode COPY Pipfile . COPY Pipfile.lock . RUN pipenv install --deploy --verbose --system --python /usr/local/bin/python

It would maybe be better to start from the official python image and install the pipenv in the Dockerfile, instead of starting from this image.