pipenv VCS install using https credentials in env variables fails for Pipfile

Issue description

Trying to install a package over https, from a private AWS CodeCommit repo, on an Ubuntu system.

Instead of using git credential.helper, we would like to inject credentials into our Pipfile using env variables.

So, in a bash terminal, we define USERNAME and PASSWORD (url-encoded).

Installing into an empty dir, directly from the command line, using these credentials, appears to work without issue:

pipenv install -e "git+https://${USERNAME}:${PASSWORD}@git-codecommit.eu-west-3.amazonaws.com/v1/repos/mypackage@main#egg=mypackage"

This implies that the credentials and user permissions are correct.

However, it turns out the username and password end up in both Pipfile and Pipfile.lock...

That does not sound like a good idea.

Luckily, the docs for Injecting credentials into Pipfile via environment variables say:

Keep in mind that environment variables are expanded in runtime, leaving the entries in Pipfile or Pipfile.lock untouched. This is to avoid the accidental leakage of credentials in the source code.

However, if we try to install from a Pipfile into an otherwise empty dir, using the exact same url with same USERNAME and PASSWORD, we get a status 403 (Forbidden):

# ...
[packages]
mypackage = {editable = true, ref = "main", git = "git+https://${USERNAME}:${PASSWORD}@git-codecommit.eu-west-3.amazonaws.com/v1/repos/mypackage"}
# ...

yields

...
INFO:pip.subprocessor:fatal: unable to access 'https://git-codecommit.eu-west-3.amazonaws.com/v1/repos/mypackage/': 
The requested URL returned error: 403
ERROR:pip.subprocessor:git clone --filter=blob:none --quiet 'https://myusername:****@git-codecommit.eu-west-3.amazonaws.com/v1/repos/mypackage' /tmp/pip-temp-5vfi10zn/mypackage_c61911349e2c4296b14b1168cf062f21
exited with 128
...

Expected result

I expect installation from Pipfile to work without issue, just like installation from the command line.

Actual result

command line output

/tmp/pipenvtest $ pipenv install --verbose
Pipfile.lock not found, creating...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
INFO:pipenv.patched.pip._internal.operations.prepare:Collecting mypackage@ 
git+https://myusername:****@git-codecommit.eu-west-3.a
mazonaws.com/v1/repos/mypackage@main (from -r 
/tmp/pipenv-2_t8zu99-requirements/pipenv-3cdgwj18-constraints.txt (line 2))
INFO:pipenv.patched.pip._internal.vcs.git:Cloning 
https://myusername:****@git-codecommit.eu-west-3.amazo
naws.com/v1/repos/mypackage (to revision main) to 
/tmp/pip-temp-5vfi10zn/mypackage_c61911349e2c4296b14b1168cf062f21
INFO:pip.subprocessor:Running command git clone --filter=blob:none --quiet 
'https://myusername:****@git-codecommit.eu-west-3.amaz
onaws.com/v1/repos/mypackage' 
/tmp/pip-temp-5vfi10zn/mypackage_c61911349e2c4296b14b1168cf062f21
INFO:pip.subprocessor:fatal: unable to access 
'https://git-codecommit.eu-west-3.amazonaws.com/v1/repos/mypackage/': The 
requested URL returned error: 403
ERROR:pip.subprocessor:git clone --filter=blob:none --quiet 
'https://myusername:****@git-codecommit.eu-west-3.amaz
onaws.com/v1/repos/mypackage' 
/tmp/pip-temp-5vfi10zn/mypackage_c61911349e2c4296b14b1168cf062f21 exited with 
128
Traceback (most recent call last):
  File "/home/me/.local/lib/python3.11/site-packages/pipenv/utils/resolver.py",
line 455, in resolve
    results = resolver.resolve(constraints, check_supported_wheels=False)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File 
"/home/me/.local/lib/python3.11/site-packages/pipenv/patched/pip/_internal/reso
lution/resolvelib/resolver.py", line 76, in resolve
    collected = self.factory.collect_root_requirements(root_reqs)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File 
"/home/me/.local/lib/python3.11/site-packages/pipenv/patched/pip/_internal/reso
lution/resolvelib/factory.py", line 534, in collect_root_requirements
    reqs = list(
           ^^^^^
  File 
"/home/me/.local/lib/python3.11/site-packages/pipenv/patched/pip/_internal/reso
lution/resolvelib/factory.py", line 490, in _make_requirements_from_install_req
    cand = self._make_base_candidate_from_link(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File 
"/home/me/.local/lib/python3.11/site-packages/pipenv/patched/pip/_internal/reso
lution/resolvelib/factory.py", line 228, in _make_base_candidate_from_link
    self._link_candidate_cache = LinkCandidate(
                                       ^^^^^^^^^^^^^^
  File 
"/home/me/.local/lib/python3.11/site-packages/pipenv/patched/pip/_internal/reso
lution/resolvelib/candidates.py", line 294, in __init__
    super().__init__(
  File 
"/home/me/.local/lib/python3.11/site-packages/pipenv/patched/pip/_internal/reso
lution/resolvelib/candidates.py", line 157, in __init__
    self.dist = self._prepare()
                ^^^^^^^^^^^^^^^
  File 
"/home/me/.local/lib/python3.11/site-packages/pipenv/patched/pip/_internal/reso
lution/resolvelib/candidates.py", line 223, in _prepare
    dist = self._prepare_distribution()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File 
"/home/me/.local/lib/python3.11/site-packages/pipenv/patched/pip/_internal/reso
lution/resolvelib/candidates.py", line 305, in _prepare_distribution
    return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File 
"/home/me/.local/lib/python3.11/site-packages/pipenv/patched/pip/_internal/oper
ations/prepare.py", line 525, in prepare_linked_requirement
    return self._prepare_linked_requirement(req, parallel_builds)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File 
"/home/me/.local/lib/python3.11/site-packages/pipenv/patched/pip/_internal/oper
ations/prepare.py", line 596, in _prepare_linked_requirement
    local_file = unpack_url(
                 ^^^^^^^^^^^
  File 
"/home/me/.local/lib/python3.11/site-packages/pipenv/patched/pip/_internal/oper
ations/prepare.py", line 157, in unpack_url
    unpack_vcs_link(link, location, verbosity=verbosity)
  File 
"/home/me/.local/lib/python3.11/site-packages/pipenv/patched/pip/_internal/oper
ations/prepare.py", line 80, in unpack_vcs_link
    vcs_backend.unpack(location, url=hide_url(link.url), verbosity=verbosity)
  File 
"/home/me/.local/lib/python3.11/site-packages/pipenv/patched/pip/_internal/vcs/
versioncontrol.py", line 608, in unpack
    self.obtain(location, url=url, verbosity=verbosity)
  File 
"/home/me/.local/lib/python3.11/site-packages/pipenv/patched/pip/_internal/vcs/
versioncontrol.py", line 521, in obtain
    self.fetch_new(dest, url, rev_options, verbosity=verbosity)
  File 
"/home/me/.local/lib/python3.11/site-packages/pipenv/patched/pip/_internal/vcs/
git.py", line 276, in fetch_new
    self.run_command(
  File 
"/home/me/.local/lib/python3.11/site-packages/pipenv/patched/pip/_internal/vcs/
versioncontrol.py", line 650, in run_command
    return call_subprocess(
           ^^^^^^^^^^^^^^^^
  File 
"/home/me/.local/lib/python3.11/site-packages/pipenv/patched/pip/_internal/util
s/subprocess.py", line 224, in call_subprocess
    raise error
pipenv.patched.pip._internal.exceptions.InstallationSubprocessError: git clone 
--filter=blob:none --quiet 
'https://myusername:****@git-codecommit.eu-west-3.amaz
onaws.com/v1/repos/mypackage' 
/tmp/pip-temp-5vfi10zn/mypackage_c61911349e2c4296b14b1168cf062f21 exited with 
128
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/me/.local/lib/python3.11/site-packages/pipenv/resolver.py", line 
675, in <module>
    main()
  File "/home/me/.local/lib/python3.11/site-packages/pipenv/resolver.py", line 
661, in main
    _main(
  File "/home/me/.local/lib/python3.11/site-packages/pipenv/resolver.py", line 
645, in _main
    resolve_packages(
  File "/home/me/.local/lib/python3.11/site-packages/pipenv/resolver.py", line 
612, in resolve_packages
    results, resolver = resolve(
                        ^^^^^^^^
  File "/home/me/.local/lib/python3.11/site-packages/pipenv/resolver.py", line 
592, in resolve
    return resolve_deps(
           ^^^^^^^^^^^^^
  File "/home/me/.local/lib/python3.11/site-packages/pipenv/utils/resolver.py",
line 932, in resolve_deps
    results, hashes, internal_resolver = actually_resolve_deps(
                                         ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/me/.local/lib/python3.11/site-packages/pipenv/utils/resolver.py",
line 700, in actually_resolve_deps
    resolver.resolve()
  File "/home/me/.local/lib/python3.11/site-packages/pipenv/utils/resolver.py",
line 457, in resolve
    raise ResolutionFailure(message=str(e))
pipenv.exceptions.ResolutionFailure: [31m[1mERROR[0m: [33mgit clone 
--filter=blob:none --quiet 
'https://myusername:****@git-codecommit.eu-west-3.amaz
onaws.com/v1/repos/mypackage' 
/tmp/pip-temp-5vfi10zn/mypackage_c61911349e2c4296b14b1168cf062f21 exited with 
128[0m
✘ Locking Failed!
⠋ Locking packages...
Traceback (most recent call last):
  File "/home/me/.local/bin/pipenv", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/home/me/.local/lib/python3.11/site-packages/pipenv/vendor/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/me/.local/lib/python3.11/site-packages/pipenv/cli/options.py", line 58, in main
    return super().main(*args, **kwargs, windows_expand_args=False)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/me/.local/lib/python3.11/site-packages/pipenv/vendor/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/me/.local/lib/python3.11/site-packages/pipenv/vendor/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/me/.local/lib/python3.11/site-packages/pipenv/vendor/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/me/.local/lib/python3.11/site-packages/pipenv/vendor/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/me/.local/lib/python3.11/site-packages/pipenv/vendor/click/decorators.py", line 92, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/me/.local/lib/python3.11/site-packages/pipenv/vendor/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/me/.local/lib/python3.11/site-packages/pipenv/cli/command.py", line 209, in install
    do_install(
  File "/home/me/.local/lib/python3.11/site-packages/pipenv/routines/install.py", line 93, in do_install
    do_init(
  File "/home/me/.local/lib/python3.11/site-packages/pipenv/routines/install.py", line 653, in do_init
    do_lock(
  File "/home/me/.local/lib/python3.11/site-packages/pipenv/routines/lock.py", line 66, in do_lock
    venv_resolve_deps(
  File "/home/me/.local/lib/python3.11/site-packages/pipenv/utils/resolver.py", line 873, in venv_resolve_deps
    c = resolve(cmd, st, project=project)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/me/.local/lib/python3.11/site-packages/pipenv/utils/resolver.py", line 737, in resolve
    raise RuntimeError("Failed to lock Pipfile.lock!")
RuntimeError: Failed to lock Pipfile.lock!

Steps to replicate

first

create a private aws codecommit repo with a minimal pyproject.toml, as follows, and setup a user with git credentials (perhaps a private github repo would also work, but haven't tried that yet)
```
[build-system]
requires = ["setuptools >= 61.0"]
build-backend = "setuptools.build_meta"

[project]
name = "mypackage"
version = "0.0.1"
```
in a bash terminal on the local system, create env variables USERNAME and PASSWORD (e.g. using read -s) with the corresponding values (url-encoded)

create an empty dir, locally, and use pipenv to install the repo, in editable mode (not sure if that matters), as in:

pipenv install -e "git+https://${USERNAME}:${PASSWORD}@git-codecommit.eu-west-3.amazonaws.com/v1/repos/mypackage@main#egg=mypackage"

observe that this installation succeeds

then

in the same terminal, create another empty dir

copy the Pipfile, generated above, into the new dir, and replace the actual username and password in the file by the corresponding env variables, as in:

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
mypackage = {editable = true, ref = "main", git = "git+https://${USERNAME}:${PASSWORD}@git-codecommit.eu-west-3.amazonaws.com/v1/repos/mypackage"}

[dev-packages]

[requires]
python_version = "3.11"

run pipenv install
observe the status 403 error

$ pipenv --support

Pipenv version: '2024.0.1'

Pipenv location: '/home/me/.local/lib/python3.11/site-packages/pipenv'

Python location: '/home/me/.pyenv/versions/3.11.6/bin/python3.11'

OS Name: 'posix'

User pip version: '24.0'

user Python installations found:

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.11.6',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '6.5.0-41-generic',
 'platform_system': 'Linux',
 'platform_version': '#41~22.04.2-Ubuntu SMP PREEMPT_DYNAMIC Mon Jun  3 '
                     '11:32:55 UTC 2',
 'python_full_version': '3.11.6',
 'python_version': '3.11',
 'sys_platform': 'linux'}

System environment variables:

SHELL
SESSION_MANAGER
QT_ACCESSIBILITY
PIPENV_VENV_IN_PROJECT
COLORTERM
PYENV_SHELL
XDG_CONFIG_DIRS
SSH_AGENT_LAUNCHER
XDG_MENU_PREFIX
GNOME_DESKTOP_SESSION_ID
LANGUAGE
LC_ADDRESS
GNOME_SHELL_SESSION_MODE
LC_NAME
SSH_AUTH_SOCK
GIT_PS1_SHOWDIRTYSTATE
XMODIFIERS
DESKTOP_SESSION
LC_MONETARY
GTK_MODULES
PWD
LOGNAME
XDG_SESSION_DESKTOP
XDG_SESSION_TYPE
SYSTEMD_EXEC_PID
XAUTHORITY
HOME
USERNAME
IM_CONFIG_PHASE
LC_PAPER
LANG
LS_COLORS
XDG_CURRENT_DESKTOP
VTE_VERSION
WAYLAND_DISPLAY
GNOME_TERMINAL_SCREEN
GNOME_SETUP_DISPLAY
LESSCLOSE
XDG_SESSION_CLASS
TERM
LC_IDENTIFICATION
LESSOPEN
USER
GNOME_TERMINAL_SERVICE
DISPLAY
SHLVL
LC_TELEPHONE
QT_IM_MODULE
LC_MEASUREMENT
PAPERSIZE
XDG_RUNTIME_DIR
PYENV_ROOT
LC_TIME
XDG_DATA_DIRS
PATH
GDMSESSION
DBUS_SESSION_BUS_ADDRESS
LC_NUMERIC
_
PIP_DISABLE_PIP_VERSION_CHECK
PYTHONDONTWRITEBYTECODE
PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:

PIPENV_VENV_IN_PROJECT: 1

Debug–specific environment variables:

PATH: /home/me/.pyenv/versions/3.9.18/bin:/home/me/.pyenv/versions/3.8.18/bin:/home/me/.pyenv/versions/3.11.6/bin:/home/me/.pyenv/versions/3.8.13/bin:/home/me/.pyenv/shims:/home/me/.npm-global/bin:/home/me/.local/bin:/home/me/.pyenv/bin:/home/me/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin
SHELL: /bin/bash
LANG: en_CA.UTF-8
PWD: /home/me

Jul 09 '24 14:07 dennisvang

I am wondering if anyone can setup a sample private repo for testing this case -- I'd like to help improve the code around it.

Jul 29 '24 01:07 matteius

@matteius I've tried to reproduce this using one of my private github repos, because those are easier to set up. However, in this case pipenv already fails at replication step 3. mentioned above (non-editable).

For example, pip succeeds (with github username and fine-grained personal access token as credentials.)

pip install "git+https://${USERNAME}:${PASSWORD}@github.com/myusername/my-private-repo.git"

but pipenv, with the same command, same terminal, same credentials

pipenv install "git+https://${USERNAME}:${PASSWORD}@github.com/myusername/my-private-repo.git"

fails with

...
Added my-private-repo to Pipfile's [packages] ...
✔ Installation Succeeded
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.
Installing dependencies from Pipfile.lock (13a959)...
[pipenv.exceptions.InstallError]: Collecting my-private-repo@ git+https://myusername:****@0888...b155 (from -r /tmp/pipenv-hhg4plkm-requirements/pipenv-vmlg4kcl-reqs.txt (line 1))
[pipenv.exceptions.InstallError]:   Cloning https://myusername:****@0888...b155 to /tmp/pip-install-_1xf_5i1/my-private-repo_3cef3c3357fd4617av89cf07dea69a8e
[pipenv.exceptions.InstallError]: Running command git clone --filter=blob:none --quiet 'https://myusername:****@0888...b155' /tmp/pip-install-_1xf_5i1/my-private-repo_3cef3c3357fd4617av89cf07dea69a8e
[pipenv.exceptions.InstallError]:   fatal: unable to access 'https://0888...b155/': Could not resolve host: 0888...b155
...
ERROR: Couldn't install package: {}
Package installation failed...

It looks unrelated to the present issue. Should I open a new issue for this?

Jul 29 '24 10:07 dennisvang

@dennisvang Could you test this report against this branch with I believe does better with the VCS env variables: https://github.com/pypa/pipenv/pull/6242

Sep 14 '24 03:09 matteius

Analysis of Issue #6195:

1. Problem Summary

The issue reports a discrepancy in Pipenv's handling of HTTPS credentials for VCS installations when specified via environment variables. While direct command line installation with embedded credentials works as expected, using the same URL format with environment variables in a Pipfile leads to a 403 Forbidden error during installation.

2. Comment Discussion Analysis

matteius seeks help setting up a private repo for testing, indicating an interest in addressing the issue.
dennisvang provides valuable insights by attempting to reproduce the issue with a private Github repository. They note that even direct command-line installation with embedded credentials fails in their case, suggesting a potential broader issue with Pipenv's VCS credential handling.
matteius suggests testing against a specific branch (PR #6242), hinting at potential improvements in VCS environment variable handling.

3. Proposed Resolution

The issue likely stems from Pipenv not correctly substituting environment variables within the Pipfile before passing the URL to the underlying VCS command. The solution involves ensuring proper variable expansion and possibly refining how credentials are handled during VCS installations.

Here's a detailed breakdown:

Variable Expansion: Pipenv should expand environment variables within the Pipfile's VCS URL before invoking the installation command. This ensures the actual username and password are passed to the VCS tool.
Credential Handling: Investigate if the current approach of directly embedding credentials in the URL is the most secure and robust method. Consider alternative approaches, like using a credential helper or dedicated environment variables for VCS credentials.
Regression Testing: Thoroughly test various VCS installation scenarios, including different VCS providers (GitHub, CodeCommit, etc.) and credential formats, to ensure the fix effectively addresses the issue and doesn't introduce regressions.

4. Potential Code Changes

Enhance pipenv/utils/project.py:

from pipenv.utils.shell import safe_expandvars

class Project:
	  # ... existing code ...

	  def get_vcs_deps(self, dev=False):
	      # ... existing code ...

	      # Expand environment variables in the URL.
	      for k, v in packages.items():
	          if is_vcs(v) or is_vcs(k):
	              if isinstance(v, dict):
	                  for key, value in v.items():
	                      if key in VCS_LIST:
	                          v[key] = safe_expandvars(value)

	      return packages or {}
	  # ... existing code ...

This modification expands environment variables within the VCS URL retrieved from the Pipfile before further processing.

5. Additional Steps/Investigations

Review PR #6242: Analyze the changes in PR #6242 to understand its impact on VCS credential handling and whether it addresses this specific issue.
Explore Credential Helpers: Investigate if leveraging a VCS credential helper (like the Git Credential Manager) provides a more secure and flexible alternative to embedding credentials in URLs.
Security Considerations: Thoroughly assess the security implications of the chosen credential handling approach, ensuring no sensitive information is inadvertently exposed or stored insecurely.

By addressing the variable expansion issue and potentially refining VCS credential management, Pipenv can ensure consistent and secure handling of private repositories, enhancing its usability and reliability for diverse projects and environments.

Oct 18 '24 21:10 matteius

Could you check @dennisvang if https://github.com/pypa/pipenv/pull/6276 solves for this (I think it should).

Oct 22 '24 03:10 matteius

pipenv pipenv copied to clipboard

VCS install using https credentials in env variables fails for Pipfile

Issue description

Expected result

Actual result

Steps to replicate

Analysis of Issue #6195:

1. Problem Summary

2. Comment Discussion Analysis

3. Proposed Resolution

4. Potential Code Changes

5. Additional Steps/Investigations

pipenv
pipenv copied to clipboard