pipenv icon indicating copy to clipboard operation
pipenv copied to clipboard

Published 20 hours ago • verified publisher icon pypa

pipenv fails during dependency-locking for certain private repositories on systems without internet access

Open fstraetz opened this issue 4 years ago • 1 comments

Issue description

When configured to use certain private repositories, pipenv will fail during the dependency-locking step on systems without internet access, reporting it couldn't reach pypi.org (instead of the private repository). The reason for this is that utils.collect_hashes() will silently (even on verbose mode) revert the source URL to https://pypi.org if the keywords python.org or pypi.org appear anywhere (!) in the private repository URL. Therefore private repository URLs like https://artifactory.fancy.company.com/artifactory/api/pypi/pypi.org/simple - a sensible choice for a PyPI remote repository - will not work.

Expected result

pipenv should use the provided private repository for all steps or at least fall back to using it, no matter which keywords are present in the private repository URL.

Actual result

pipenv fails, shows that it's configured to use the private repository, yet still refers to pypi.org.

Installing structlog...
Installing package: structlog
Writing supplied requirement line to temporary file: 'structlog'
⠹ Installing structlog...$ ['/tmp/pipenv_test/.venv/bin/pip', 'install', '--verbose', '--upgrade', '--exists-action=i', '-r', '/tmp/pipenv-r3hbdbfp-requirements/pipenv-nilscc9b-requirement.txt', '-i', 'https://artifactory.fancy.company.de/artifactory/api/pypi/pypi.org/simple']
✔ Installation Succeeded
Pipfile.lock (ca72e7) out of date, updating to (6d3f58)...
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
Building requirements...
✘ Locking Failed!
                          ROUND 1
Current constraints:
  structlog (from -r /tmp/pipenv5_g2zgrnrequirements/pipenv-7enmllwe-constraints.txt (line 2))

Finding the best candidates:
  found candidate structlog==21.1.0 (constraint was <any>)

Finding secondary dependencies:
  structlog==21.1.0         requires typing-extensions; python_version < "3.8"

New dependencies found in this round:
  adding ['typing-extensions', '', '[]']
Removed dependencies in this round:
------------------------------------------------------------
Result of round 1: not stable

                          ROUND 2
Current constraints:
  structlog (from -r /tmp/pipenv5_g2zgrnrequirements/pipenv-7enmllwe-constraints.txt (line 2))
  typing-extensions (from structlog==21.1.0->-r /tmp/pipenv5_g2zgrnrequirements/pipenv-7enmllwe-constraints.txt (line 2))

Finding the best candidates:
  found candidate structlog==21.1.0 (constraint was <any>)
  found candidate typing-extensions==3.10.0.0 (constraint was <any>)

Finding secondary dependencies:
  typing-extensions==3.10.0.0 requires -
  structlog==21.1.0         requires typing-extensions; python_version < "3.8"
------------------------------------------------------------
Result of round 2: stable, done

Generating hashes:
  typing-extensions
  structlog
Traceback (most recent call last):
  File "/home/fstraetz/.local/lib/python3.6/site-packages/pipenv/vendor/urllib3/connection.py", line 170, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw
  File "/home/fstraetz/.local/lib/python3.6/site-packages/pipenv/vendor/urllib3/util/connection.py", line 73, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/usr/lib/python3.6/socket.py", line 745, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Der Name oder der Dienst ist nicht bekannt

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/fstraetz/.local/lib/python3.6/site-packages/pipenv/vendor/urllib3/connectionpool.py", line 706, in urlopen
    chunked=chunked,
  File "/home/fstraetz/.local/lib/python3.6/site-packages/pipenv/vendor/urllib3/connectionpool.py", line 382, in _make_request
    self._validate_conn(conn)
  File "/home/fstraetz/.local/lib/python3.6/site-packages/pipenv/vendor/urllib3/connectionpool.py", line 1010, in _validate_conn
    conn.connect()
  File "/home/fstraetz/.local/lib/python3.6/site-packages/pipenv/vendor/urllib3/connection.py", line 353, in connect
    conn = self._new_conn()
  File "/home/fstraetz/.local/lib/python3.6/site-packages/pipenv/vendor/urllib3/connection.py", line 182, in _new_conn
    self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7f5614ed2470>: Failed to establish a new connection: [Errno -2] Der Name oder der Dienst ist nicht bekannt

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/fstraetz/.local/lib/python3.6/site-packages/pipenv/vendor/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/home/fstraetz/.local/lib/python3.6/site-packages/pipenv/vendor/urllib3/connectionpool.py", line 756, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/home/fstraetz/.local/lib/python3.6/site-packages/pipenv/vendor/urllib3/util/retry.py", line 573, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /pypi/typing-extensions/json (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f5614ed2470>: Failed to establish a new connection: [Errno -2] Der Name oder der Dienst ist nicht bekannt',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/fstraetz/.local/lib/python3.6/site-packages/pipenv/resolver.py", line 764, in <module>
    main()
  File "/home/fstraetz/.local/lib/python3.6/site-packages/pipenv/resolver.py", line 760, in main
    dev=parsed.dev)
  File "/home/fstraetz/.local/lib/python3.6/site-packages/pipenv/resolver.py", line 741, in _main
    resolve_packages(pre, clear, verbose, system, write, requirements_dir, packages, dev)
  File "/home/fstraetz/.local/lib/python3.6/site-packages/pipenv/resolver.py", line 709, in resolve_packages
    requirements_dir=requirements_dir,
  File "/home/fstraetz/.local/lib/python3.6/site-packages/pipenv/resolver.py", line 692, in resolve
    req_dir=requirements_dir
  File "/home/fstraetz/.local/lib/python3.6/site-packages/pipenv/utils.py", line 1405, in resolve_deps
    req_dir=req_dir,
  File "/home/fstraetz/.local/lib/python3.6/site-packages/pipenv/utils.py", line 1113, in actually_resolve_deps
    results = resolver.clean_results()
  File "/home/fstraetz/.local/lib/python3.6/site-packages/pipenv/utils.py", line 1019, in clean_results
    collected_hashes = self.collect_hashes(ireq)
  File "/home/fstraetz/.local/lib/python3.6/site-packages/pipenv/utils.py", line 902, in collect_hashes
    r = session.get(pkg_url, timeout=10)
  File "/home/fstraetz/.local/lib/python3.6/site-packages/pipenv/vendor/requests/sessions.py", line 555, in get
    return self.request('GET', url, **kwargs)
  File "/home/fstraetz/.local/lib/python3.6/site-packages/pipenv/vendor/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/fstraetz/.local/lib/python3.6/site-packages/pipenv/vendor/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/home/fstraetz/.local/lib/python3.6/site-packages/pipenv/vendor/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /pypi/typing-extensions/json (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f5614ed2470>: Failed to establish a new connection: [Errno -2] Der Name oder der Dienst ist nicht bekannt',))
ROUND 1
Current constraints:
  structlog (from -r /tmp/pipenv5_g2zgrnrequirements/pipenv-7enmllwe-constraints.txt (line 2))

Finding the best candidates:
  found candidate structlog==21.1.0 (constraint was <any>)

Finding secondary dependencies:
  structlog==21.1.0         requires typing-extensions; python_version < "3.8"

New dependencies found in this round:
  adding ['typing-extensions', '', '[]']
Removed dependencies in this round:
------------------------------------------------------------
Result of round 1: not stable

                          ROUND 2
Current constraints:
  structlog (from -r /tmp/pipenv5_g2zgrnrequirements/pipenv-7enmllwe-constraints.txt (line 2))
  typing-extensions (from structlog==21.1.0->-r /tmp/pipenv5_g2zgrnrequirements/pipenv-7enmllwe-constraints.txt (line 2))

Finding the best candidates:
  found candidate structlog==21.1.0 (constraint was <any>)
  found candidate typing-extensions==3.10.0.0 (constraint was <any>)

Finding secondary dependencies:
  typing-extensions==3.10.0.0 requires -
  structlog==21.1.0         requires typing-extensions; python_version < "3.8"
------------------------------------------------------------
Result of round 2: stable, done

Generating hashes:
  typing-extensions
  structlog
Traceback (most recent call last):
  File "/home/fstraetz/.local/lib/python3.6/site-packages/pipenv/vendor/urllib3/connection.py", line 170, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw
  File "/home/fstraetz/.local/lib/python3.6/site-packages/pipenv/vendor/urllib3/util/connection.py", line 73, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/usr/lib/python3.6/socket.py", line 745, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Der Name oder der Dienst ist nicht bekannt

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/fstraetz/.local/lib/python3.6/site-packages/pipenv/vendor/urllib3/connectionpool.py", line 706, in urlopen
    chunked=chunked,
  File "/home/fstraetz/.local/lib/python3.6/site-packages/pipenv/vendor/urllib3/connectionpool.py", line 382, in _make_request
    self._validate_conn(conn)
  File "/home/fstraetz/.local/lib/python3.6/site-packages/pipenv/vendor/urllib3/connectionpool.py", line 1010, in _validate_conn
    conn.connect()
  File "/home/fstraetz/.local/lib/python3.6/site-packages/pipenv/vendor/urllib3/connection.py", line 353, in connect
    conn = self._new_conn()
  File "/home/fstraetz/.local/lib/python3.6/site-packages/pipenv/vendor/urllib3/connection.py", line 182, in _new_conn
    self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7f5614ed2470>: Failed to establish a new connection: [Errno -2] Der Name oder der Dienst ist nicht bekannt

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/fstraetz/.local/lib/python3.6/site-packages/pipenv/vendor/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/home/fstraetz/.local/lib/python3.6/site-packages/pipenv/vendor/urllib3/connectionpool.py", line 756, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/home/fstraetz/.local/lib/python3.6/site-packages/pipenv/vendor/urllib3/util/retry.py", line 573, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /pypi/typing-extensions/json (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f5614ed2470>: Failed to establish a new connection: [Errno -2] Der Name oder der Dienst ist nicht bekannt',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/fstraetz/.local/lib/python3.6/site-packages/pipenv/resolver.py", line 764, in <module>
    main()
  File "/home/fstraetz/.local/lib/python3.6/site-packages/pipenv/resolver.py", line 760, in main
    dev=parsed.dev)
  File "/home/fstraetz/.local/lib/python3.6/site-packages/pipenv/resolver.py", line 741, in _main
    resolve_packages(pre, clear, verbose, system, write, requirements_dir, packages, dev)
  File "/home/fstraetz/.local/lib/python3.6/site-packages/pipenv/resolver.py", line 709, in resolve_packages
    requirements_dir=requirements_dir,
  File "/home/fstraetz/.local/lib/python3.6/site-packages/pipenv/resolver.py", line 692, in resolve
    req_dir=requirements_dir
  File "/home/fstraetz/.local/lib/python3.6/site-packages/pipenv/utils.py", line 1405, in resolve_deps
    req_dir=req_dir,
  File "/home/fstraetz/.local/lib/python3.6/site-packages/pipenv/utils.py", line 1113, in actually_resolve_deps
    results = resolver.clean_results()
  File "/home/fstraetz/.local/lib/python3.6/site-packages/pipenv/utils.py", line 1019, in clean_results
    collected_hashes = self.collect_hashes(ireq)
  File "/home/fstraetz/.local/lib/python3.6/site-packages/pipenv/utils.py", line 902, in collect_hashes
    r = session.get(pkg_url, timeout=10)
  File "/home/fstraetz/.local/lib/python3.6/site-packages/pipenv/vendor/requests/sessions.py", line 555, in get
    return self.request('GET', url, **kwargs)
  File "/home/fstraetz/.local/lib/python3.6/site-packages/pipenv/vendor/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/fstraetz/.local/lib/python3.6/site-packages/pipenv/vendor/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/home/fstraetz/.local/lib/python3.6/site-packages/pipenv/vendor/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /pypi/typing-extensions/json (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f5614ed2470>: Failed to establish a new connection: [Errno -2] Der Name oder der Dienst ist nicht bekannt',))

Steps to replicate

  1. Use a Pipfile with a private repository URL containing python.org or pypi.org:
[[source]]
url = "https://artifactory.fancy.company.de/artifactory/api/pypi/pypi.org/simple"
verify_ssl = true
name = "artifactory"
  1. Disconnect the system from the internet
  2. Try to install one of the packages found in the private repository

Pipenv version: '2021.5.29'

Pipenv location: '/home/fstraetz/.local/lib/python3.6/site-packages/pipenv'

Python location: '/usr/bin/python3'

Python installations found:

  • 3.6.9: /usr/bin/python3
  • 3.6.9: /usr/bin/python3.6m
  • 3.6.9: /usr/bin/python3.6
  • 2.7.17: /usr/bin/python2
  • 2.7.17: /usr/bin/python2.7

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.6.9',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '4.15.0-126-generic',
 'platform_system': 'Linux',
 'platform_version': '#129-Ubuntu SMP Mon Nov 23 18:53:38 UTC 2020',
 'python_full_version': '3.6.9',
 'python_version': '3.6',
 'sys_platform': 'linux'}

System environment variables:

  • NVM_DIR
  • LS_COLORS
  • SSH_CONNECTION
  • LESSCLOSE
  • LANG
  • OLDPWD
  • NVM_CD_FLAGS
  • S_COLORS
  • XDG_SESSION_ID
  • USER
  • PWD
  • HOME
  • SSH_CLIENT
  • XDG_DATA_DIRS
  • SSH_TTY
  • MAIL
  • TERM
  • SHELL
  • NVM_BIN
  • SHLVL
  • MANPATH
  • LOGNAME
  • DBUS_SESSION_BUS_ADDRESS
  • XDG_RUNTIME_DIR
  • PATH
  • LESSOPEN
  • _
  • PIP_DISABLE_PIP_VERSION_CHECK
  • PYTHONDONTWRITEBYTECODE
  • PIP_SHIMS_BASE_MODULE
  • PIP_PYTHON_PATH
  • PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:

Debug–specific environment variables:

  • PATH: /home/fstraetz/.cargo/bin:/home/fstraetz/.local/bin:/home/fstraetz/bin:/home/fstraetz/.nvm/versions/node/v10.15.3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/opt/puppetlabs/bin
  • SHELL: /bin/bash
  • LANG: de_DE.UTF-8
  • PWD: /tmp/pipenv_test

Contents of Pipfile ('/tmp/pipenv_test/Pipfile'):

[[source]]
url = "https://artifactory.fancy.company.de/artifactory/api/pypi/pypi.org/simple"
verify_ssl = true
name = "artifactory"

[packages]
structlog = "*"

[dev-packages]

[requires]
python_version = "3.6"

Contents of Pipfile.lock ('/tmp/pipenv_test/Pipfile.lock'):

{
    "_meta": {
        "hash": {
            "sha256": "415dfdcb118dd9bdfef17671cb7dcd78dbd69b6ae7d4f39e8b44e71d60ca72e7"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.6"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {},
    "develop": {}
}

fstraetz avatar Jul 09 '21 11:07 fstraetz

I am guessing it is as you say because the URL has pypi.org in the URL, its matching this logic: https://github.com/pypa/pipenv/blob/main/pipenv/utils/resolver.py#L747-L753

Will have to consider how we can further refine that.

matteius avatar Apr 01 '22 13:04 matteius