pipenv Locking fails when using `--extra-pip-args="--proxy=`

Issue description

Can't install a package behind a proxy using the --extra-pip-args flag. The locking fails with pipenv.exceptions.ResolutionFailure.

following pip command works: pip install --proxy 127.0.0.1:3128 numpy following pipenv command doesn't: pipenv install --extra-pip-args="--proxy=127.0.0.1:3128" numpy

Important detail: I'm utilizing a Px proxy to bypass the actual NTLM proxy. The 407 errors listed below are the results I encounter when attempting a basic pipenv install numpy. That's why I suspect that the Px proxy I provided isn't being utilized.

Expected result

Package gets added to pipfile and the pipfile.lock is updated.

Actual result

C:\Users\helyux\Desktop\Aiden>pipenv install --verbose --extra-pip-args="--proxy=127.0.0.1:3128" numpy
Installing numpy...
Resolving numpy...
Added numpy to Pipfile's [packages] ...
Installation Succeeded
Pipfile.lock (e4eef2) out of date, updating to (a6e810)...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
INFO:pipenv.patched.pip._internal.resolution.resolvelib.reporter:Reporter.starting()
INFO:pipenv.patched.pip._internal.resolution.resolvelib.reporter:Reporter.adding_requirement(SpecifierRequirement('numpy'), None)
WARNING:pipenv.patched.pip._vendor.urllib3.connectionpool:Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by
'ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 407 Proxy Authentication Required'))': /simple/numpy/
WARNING:pipenv.patched.pip._vendor.urllib3.connectionpool:Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by
'ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 407 Proxy Authentication Required'))': /simple/numpy/
WARNING:pipenv.patched.pip._vendor.urllib3.connectionpool:Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by
'ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 407 Proxy Authentication Required'))': /simple/numpy/
WARNING:pipenv.patched.pip._vendor.urllib3.connectionpool:Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by
'ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 407 Proxy Authentication Required'))': /simple/numpy/
WARNING:pipenv.patched.pip._vendor.urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by
'ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 407 Proxy Authentication Required'))': /simple/numpy/
CRITICAL:pipenv.patched.pip._internal.resolution.resolvelib.factory:Could not find a version that satisfies the requirement numpy (from versions: none)
Traceback (most recent call last):
  File "C:\Users\helyux\AppData\Roaming\Python\Python310\site-packages\pipenv\patched\pip\_vendor\resolvelib\resolvers.py", line 397, in resolve
    self._add_to_criteria(self.state.criteria, r, parent=None)
  File "C:\Users\helyux\AppData\Roaming\Python\Python310\site-packages\pipenv\patched\pip\_vendor\resolvelib\resolvers.py", line 174, in _add_to_criteria
    raise RequirementsConflicted(criterion)
pipenv.patched.pip._vendor.resolvelib.resolvers.RequirementsConflicted: Requirements conflict: SpecifierRequirement('numpy')
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "C:\Users\helyux\AppData\Roaming\Python\Python310\site-packages\pipenv\patched\pip\_internal\resolution\resolvelib\resolver.py", line 95, in resolve
    result = self._result = resolver.resolve(
  File "C:\Users\helyux\AppData\Roaming\Python\Python310\site-packages\pipenv\patched\pip\_vendor\resolvelib\resolvers.py", line 546, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "C:\Users\helyux\AppData\Roaming\Python\Python310\site-packages\pipenv\patched\pip\_vendor\resolvelib\resolvers.py", line 399, in resolve
    raise ResolutionImpossible(e.criterion.information)
pipenv.patched.pip._vendor.resolvelib.resolvers.ResolutionImpossible: [RequirementInformation(requirement=SpecifierRequirement('numpy'), parent=None)]
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "C:\Users\helyux\AppData\Roaming\Python\Python310\site-packages\pipenv\utils\resolver.py", line 440, in resolve
    results = resolver.resolve(constraints, check_supported_wheels=False)
  File "C:\Users\helyux\AppData\Roaming\Python\Python310\site-packages\pipenv\patched\pip\_internal\resolution\resolvelib\resolver.py", line 104, in resolve
    raise error from e
pipenv.patched.pip._internal.exceptions.DistributionNotFound: No matching distribution found for numpy
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "C:\Users\helyux\AppData\Roaming\Python\Python310\site-packages\pipenv\resolver.py", line 675, in <module>
    main()
  File "C:\Users\helyux\AppData\Roaming\Python\Python310\site-packages\pipenv\resolver.py", line 661, in main
    _main(
  File "C:\Users\helyux\AppData\Roaming\Python\Python310\site-packages\pipenv\resolver.py", line 645, in _main
    resolve_packages(
  File "C:\Users\helyux\AppData\Roaming\Python\Python310\site-packages\pipenv\resolver.py", line 612, in resolve_packages
    results, resolver = resolve(
  File "C:\Users\helyux\AppData\Roaming\Python\Python310\site-packages\pipenv\resolver.py", line 592, in resolve
    return resolve_deps(
  File "C:\Users\helyux\AppData\Roaming\Python\Python310\site-packages\pipenv\utils\resolver.py", line 908, in resolve_deps
    results, hashes, internal_resolver = actually_resolve_deps(
  File "C:\Users\helyux\AppData\Roaming\Python\Python310\site-packages\pipenv\utils\resolver.py", line 681, in actually_resolve_deps
    resolver.resolve()
  File "C:\Users\helyux\AppData\Roaming\Python\Python310\site-packages\pipenv\utils\resolver.py", line 442, in resolve
    raise ResolutionFailure(message=str(e))
pipenv.exceptions.ResolutionFailure: ERROR: No matching distribution found for numpy
Locking Failed!
[   =] Locking...
Traceback (most recent call last):
  File "C:\Program Files\Python310\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Program Files\Python310\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Program Files\Python310\Scripts\pipenv.exe\__main__.py", line 7, in <module>
    # when invoked as python -m pip <command>
  File "C:\Users\helyux\AppData\Roaming\Python\Python310\site-packages\pipenv\vendor\click\core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "C:\Users\helyux\AppData\Roaming\Python\Python310\site-packages\pipenv\cli\options.py", line 58, in main
    return super().main(*args, **kwargs, windows_expand_args=False)
  File "C:\Users\helyux\AppData\Roaming\Python\Python310\site-packages\pipenv\vendor\click\core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "C:\Users\helyux\AppData\Roaming\Python\Python310\site-packages\pipenv\vendor\click\core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\Users\helyux\AppData\Roaming\Python\Python310\site-packages\pipenv\vendor\click\core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Users\helyux\AppData\Roaming\Python\Python310\site-packages\pipenv\vendor\click\core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "C:\Users\helyux\AppData\Roaming\Python\Python310\site-packages\pipenv\vendor\click\decorators.py", line 92, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "C:\Users\helyux\AppData\Roaming\Python\Python310\site-packages\pipenv\vendor\click\core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "C:\Users\helyux\AppData\Roaming\Python\Python310\site-packages\pipenv\cli\command.py", line 209, in install
    do_install(
  File "C:\Users\helyux\AppData\Roaming\Python\Python310\site-packages\pipenv\routines\install.py", line 297, in do_install
    raise e
  File "C:\Users\helyux\AppData\Roaming\Python\Python310\site-packages\pipenv\routines\install.py", line 281, in do_install
    do_init(
  File "C:\Users\helyux\AppData\Roaming\Python\Python310\site-packages\pipenv\routines\install.py", line 648, in do_init
    do_lock(
  File "C:\Users\helyux\AppData\Roaming\Python\Python310\site-packages\pipenv\routines\lock.py", line 65, in do_lock
    venv_resolve_deps(
  File "C:\Users\helyux\AppData\Roaming\Python\Python310\site-packages\pipenv\utils\resolver.py", line 849, in venv_resolve_deps
    c = resolve(cmd, st, project=project)
  File "C:\Users\helyux\AppData\Roaming\Python\Python310\site-packages\pipenv\utils\resolver.py", line 718, in resolve
    raise RuntimeError("Failed to lock Pipfile.lock!")
RuntimeError: Failed to lock Pipfile.lock!

Steps to replicate

Create a new project pipenv install Install a package passing a proxy pipenv install --extra-pip-args="--proxy=127.0.0.1:3128" numpy

$ pipenv --support

Pipenv version: '2023.10.24'

Pipenv location: 'C:\\Users\\helyux\\AppData\\Roaming\\Python\\Python310\\site-packages\\pipenv'

Python location: 'C:\\Program Files\\Python310\\python.exe'

OS Name: 'nt'

User pip version: '23.3.1'

user Python installations found:

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.10.6',
 'os_name': 'nt',
 'platform_machine': 'AMD64',
 'platform_python_implementation': 'CPython',
 'platform_release': '10',
 'platform_system': 'Windows',
 'platform_version': '10.0.22000',
 'python_full_version': '3.10.6',
 'python_version': '3.10',
 'sys_platform': 'win32'}

System environment variables:

ALLUSERSPROFILE
APPDATA
COMMONPROGRAMFILES
COMMONPROGRAMFILES(X86)
COMMONPROGRAMW6432
COMPUTERNAME
COMSPEC
DRIVERDATA
FPS_BROWSER_APP_PROFILE_STRING
FPS_BROWSER_USER_PROFILE_STRING
GOPATH
HOME
HOMEDRIVE
HOMEPATH
HOMESHARE
JAVA_HOME
LOCALAPPDATA
LOGONSERVER
MAGICK_HOME
NUMBER_OF_PROCESSORS
ONEDRIVE
OS
PATH
PATHEXT
PROCESSOR_ARCHITECTURE
PROCESSOR_IDENTIFIER
PROCESSOR_LEVEL
PROCESSOR_REVISION
PROGRAMDATA
PROGRAMFILES
PROGRAMFILES(X86)
PROGRAMW6432
PROMPT
PSMODULEPATH
PUBLIC
PYCHARM COMMUNITY EDITION
SESSIONNAME
SYSTEMDRIVE
SYSTEMROOT
TEMP
TMP
USERDNSDOMAIN
USERDOMAIN
USERDOMAIN_ROAMINGPROFILE
USERNAME
USERPROFILE
WINDIR
PIP_DISABLE_PIP_VERSION_CHECK
PYTHONDONTWRITEBYTECODE
PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenvûspecific environment variables:

Debugûspecific environment variables:

Contents of Pipfile ('C:\Users\helyux\Desktop\Aiden\Pipfile'):

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]

[dev-packages]

[requires]
python_version = "3.10"

Contents of Pipfile.lock ('C:\Users\helyux\Desktop\Aiden\Pipfile.lock'):

{
    "_meta": {
        "hash": {
            "sha256": "fedbd2ab7afd84cf16f128af0619749267b62277b4cb6989ef16d4bef6e4eef2"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.10"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {},
    "develop": {}
}

Nov 01 '23 14:11 Helyux

Have you tried passing this just as a regular --index url option or --pypi-mirror url? I think with the way we have customized the resolver, what you are trying to do wouldn't work with extra pip args, as you point out.

Nov 01 '23 23:11 matteius

Hello Matteius, I'll give it a try on Monday. Thank you for the suggestion. I'm curious about why it's included as an example in the documentation then. Am I perhaps misunderstanding something? advanced/

Nov 03 '23 18:11 Helyux

The doc example is for sync which is skipping a lock resolution phase. It turns out extra pip args were never passed to the resolver, but I have a PR out that does this and I am looking for feedback on the various use cases. https://github.com/pypa/pipenv/pull/6006

Nov 05 '23 14:11 matteius

Passing the proxy address to --index (pipenv install --index 127.0.0.1:3128 numpy) resulted in the same 407's as without any args. Passing to --pypi-mirror resulted in Error: Invalid value for '--pypi-mirror': Invalid PyPI mirror URL: 127.0.0.1:3128

Sadly i don't know how to update my pipenv.exe to your patched version, if you can give me a pointer on that i'm more then happy to test.

Nov 06 '23 11:11 Helyux

@Helyux Does your proxy require authentication parameters? https://stackoverflow.com/questions/46466241/pip-install-failing-with-407-proxy-authentication-required

Nov 08 '23 02:11 matteius

It does but that's what the px proxy is for, it automatically handles the authentication, see my initial comment.

Nov 08 '23 10:11 Helyux

@Helyux to try the branch: pip uninstall pipenv --yes pip install git+https://github.com/pypa/pipenv.git@issue-6001-extra-pip-args

Nov 14 '23 23:11 matteius

pipenv pipenv copied to clipboard

Locking fails when using `--extra-pip-args="--proxy=`

Issue description

Expected result

Actual result

Steps to replicate

pipenv
pipenv copied to clipboard