pipenv
pipenv copied to clipboard
pypa
SSLCertVerificationError with pipenv install, but not with pip in same docker image
Issue description
I am behind a proxy and would like to use pipenv to build my docker images locally. Within my existing setup (using conda and pip) I have a build-argument in my Dockerfile indicating whether I want to disable the SSL certificate checking for conda and pip. This way, when building locally I choose to disable ssl verifications and while building on the CI/CD pipeline I don't provide this build-arg because the CI/CD workers have direct internet.
Using this approach, I can use the same Dockerfile both locally and on the CI/CD system by modifying just the build-arg and I don't have to try and conditionally include PEM certificate chains
I have tried things like:
- Using
--extra-pip-argslikepipenv -vvv install --extra-pip-args '--trusted-host "pypi.org files.pythonhosted.org pypi.python.org"' -r requirements.txt - Setting
PIP_TRUSTED_HOSTenvironment variable - After the initial fail, modify the generated Pipfile to have verify_ssl=False instead of true for the source
I can't seem to be able to instruct pipenv to NOT check the certificates. From the documentation it is not clear how this should be possible
Expected result
I expect pipenv to be able to install packages without checking the SSL certificates if I instruct it to do so. I do know that this negates the purpose of SSL certificates, but this way I am able to not have to manually copy additional items (pem files) into my docker image during local builds
Actual result
(base) root@0c252e9e93cc:/foo# pipenv -v install -r requirements.txt
Using python: None
Path to python: /opt/conda/bin/python3
Creating a virtualenv for this project...
Pipfile: /foo/Pipfile
Using /opt/conda/bin/python3 (3.10.8) to create virtualenv...
⠹ Creating virtual environment...created virtual environment CPython3.10.8.final.0-64 in 864ms
creator CPython3Posix(dest=/root/.local/share/virtualenvs/foo-YJbvrhmB, clear=False, no_vcs_ignore=False, global=False)
seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/root/.local/share/virtualenv)
added seed packages: pip==23.1, setuptools==67.6.1, wheel==0.40.0
activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
✔ Successfully created virtual environment!
Virtualenv location: /root/.local/share/virtualenvs/foo-YJbvrhmB
Creating a Pipfile for this project...
Requirements file provided! Importing into Pipfile...
Pipfile.lock not found, creating...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
Reporter.starting()
INFO:pipenv.patched.pip._internal.resolution.resolvelib.reporter:Reporter.starting()
Reporter.adding_requirement(SpecifierRequirement('six'), None)
INFO:pipenv.patched.pip._internal.resolution.resolvelib.reporter:Reporter.adding_requirement(SpecifierRequirement('six'), None)
Reporter.starting_round(0)
INFO:pipenv.patched.pip._internal.resolution.resolvelib.reporter:Reporter.starting_round(0)
Reporter.pinning(LinkCandidate('https://files.pythonhosted.org/packages/d9/5a/e7c31adbe875f2abbb91bd84cf2dc52d792b5a01506781dbcf25c91daf11/six-1.16.0-py2.py3-none-any.whl (from
https://pypi.org/simple/six/) (requires-python:>=2.7, !=3.0.*, !=3.1.*, !=3.2.*)'))
INFO:pipenv.patched.pip._internal.resolution.resolvelib.reporter:Reporter.pinning(LinkCandidate('https://files.pythonhosted.org/packages/d9/5a/e7c31adbe875f2abbb91bd84cf2dc52d792b5a01506781dbcf25c91da
f11/six-1.16.0-py2.py3-none-any.whl (from https://pypi.org/simple/six/) (requires-python:>=2.7, !=3.0.*, !=3.1.*, !=3.2.*)'))
Reporter.ending_round(0, state)
INFO:pipenv.patched.pip._internal.resolution.resolvelib.reporter:Reporter.ending_round(0, state)
Reporter.starting_round(1)
INFO:pipenv.patched.pip._internal.resolution.resolvelib.reporter:Reporter.starting_round(1)
Reporter.ending(State(mapping=OrderedDict([('six',
LinkCandidate('https://files.pythonhosted.org/packages/d9/5a/e7c31adbe875f2abbb91bd84cf2dc52d792b5a01506781dbcf25c91daf11/six-1.16.0-py2.py3-none-any.whl (from https://pypi.org/simple/six/)
(requires-python:>=2.7, !=3.0.*, !=3.1.*, !=3.2.*)'))]), criteria={'six': Criterion((SpecifierRequirement('six'), via=None))}, backtrack_causes=[]))
INFO:pipenv.patched.pip._internal.resolution.resolvelib.reporter:Reporter.ending(State(mapping=OrderedDict([('six',
LinkCandidate('https://files.pythonhosted.org/packages/d9/5a/e7c31adbe875f2abbb91bd84cf2dc52d792b5a01506781dbcf25c91daf11/six-1.16.0-py2.py3-none-any.whl (from https://pypi.org/simple/six/)
(requires-python:>=2.7, !=3.0.*, !=3.1.*, !=3.2.*)'))]), criteria={'six': Criterion((SpecifierRequirement('six'), via=None))}, backtrack_causes=[]))
Traceback (most recent call last):
File "/opt/conda/lib/python3.10/site-packages/pipenv/patched/pip/_vendor/urllib3/connectionpool.py", line 700, in urlopen
self._prepare_proxy(conn)
File "/opt/conda/lib/python3.10/site-packages/pipenv/patched/pip/_vendor/urllib3/connectionpool.py", line 996, in _prepare_proxy
conn.connect()
File "/opt/conda/lib/python3.10/site-packages/pipenv/patched/pip/_vendor/urllib3/connection.py", line 419, in connect
self.sock = ssl_wrap_socket(
File "/opt/conda/lib/python3.10/site-packages/pipenv/patched/pip/_vendor/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
ssl_sock = _ssl_wrap_socket_impl(
File "/opt/conda/lib/python3.10/site-packages/pipenv/patched/pip/_vendor/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
File "/opt/conda/lib/python3.10/ssl.py", line 513, in wrap_socket
return self.sslsocket_class._create(
File "/opt/conda/lib/python3.10/ssl.py", line 1071, in _create
self.do_handshake()
File "/opt/conda/lib/python3.10/ssl.py", line 1342, in do_handshake
self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/conda/lib/python3.10/site-packages/pipenv/patched/pip/_vendor/requests/adapters.py", line 489, in send
resp = conn.urlopen(
File "/opt/conda/lib/python3.10/site-packages/pipenv/patched/pip/_vendor/urllib3/connectionpool.py", line 787, in urlopen
retries = retries.increment(
File "/opt/conda/lib/python3.10/site-packages/pipenv/patched/pip/_vendor/urllib3/util/retry.py", line 592, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
pipenv.patched.pip._vendor.urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /pypi/six/json (Caused by
SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)')))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/conda/lib/python3.10/site-packages/pipenv/resolver.py", line 845, in <module>
main()
File "/opt/conda/lib/python3.10/site-packages/pipenv/resolver.py", line 831, in main
_main(
File "/opt/conda/lib/python3.10/site-packages/pipenv/resolver.py", line 811, in _main
resolve_packages(
File "/opt/conda/lib/python3.10/site-packages/pipenv/resolver.py", line 759, in resolve_packages
results, resolver = resolve(
File "/opt/conda/lib/python3.10/site-packages/pipenv/resolver.py", line 738, in resolve
return resolve_deps(
File "/opt/conda/lib/python3.10/site-packages/pipenv/utils/resolver.py", line 1097, in resolve_deps
results, hashes, markers_lookup, resolver, skipped = actually_resolve_deps(
File "/opt/conda/lib/python3.10/site-packages/pipenv/utils/resolver.py", line 897, in actually_resolve_deps
hashes = resolver.resolve_hashes()
File "/opt/conda/lib/python3.10/site-packages/pipenv/utils/resolver.py", line 798, in resolve_hashes
self.hashes = self.collect_hashes(ireq)
File "/opt/conda/lib/python3.10/site-packages/pipenv/utils/resolver.py", line 778, in collect_hashes
hashes = self._get_hashes_from_pypi(ireq)
File "/opt/conda/lib/python3.10/site-packages/pipenv/utils/resolver.py", line 738, in _get_hashes_from_pypi
r = session.get(pkg_url, timeout=10)
File "/opt/conda/lib/python3.10/site-packages/pipenv/patched/pip/_vendor/requests/sessions.py", line 600, in get
return self.request("GET", url, **kwargs)
File "/opt/conda/lib/python3.10/site-packages/pipenv/patched/pip/_vendor/requests/sessions.py", line 587, in request
resp = self.send(prep, **send_kwargs)
File "/opt/conda/lib/python3.10/site-packages/pipenv/patched/pip/_vendor/requests/sessions.py", line 701, in send
r = adapter.send(request, **kwargs)
File "/opt/conda/lib/python3.10/site-packages/pipenv/patched/pip/_vendor/requests/adapters.py", line 563, in send
raise SSLError(e, request=request)
pipenv.patched.pip._vendor.requests.exceptions.SSLError: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /pypi/six/json (Caused by SSLError(SSLCertVerificationError(1,
'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)')))
✘ Locking Failed!
⠇ Locking...
Steps to replicate
I am running all the commands below in a clean ubuntu base image that has miniconda3 installed.
requirements.txt file
six
Pipfile
After running the command pipenv -v install -r requirements.txt, automatically a Pipfile is generated with the following contents:
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"
[packages]
six = "*"
[dev-packages]
[requires]
python_version = "3.10"
Even after I change the verify_ssl in this file to false and run the pipenv install again (or pipenv lock), I still get the same CERTIFICATE_VERIFY_FAILED error.
Installing with pip itself works
When running the following command:
(base) root@0c252e9e93cc:/foo# pip -v install six
Using pip 22.2.2 from /opt/conda/lib/python3.10/site-packages/pip (python 3.10)
Collecting six
Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Installing collected packages: six
Successfully installed six-1.16.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
the package can be installed without a problem (i.e. no checking of the certificates).
pip config
(base) root@0c252e9e93cc:/foo# pip config list
global.timeout='900'
global.trusted-host='pypi.org files.pythonhosted.org pypi.python.org'
$ pipenv --support
Pipenv version: '2023.4.20'
Pipenv location: '/opt/conda/lib/python3.10/site-packages/pipenv'
Python location: '/opt/conda/bin/python'
OS Name: 'posix'
User pip version: '23.1'
user Python installations found:
-
3.10.8:/opt/conda/bin/python3 -
3.10.8:/opt/conda/bin/python -
3.10.8:/opt/conda/bin/python3.1
PEP 508 Information:
{'implementation_name': 'cpython',
'implementation_version': '3.10.8',
'os_name': 'posix',
'platform_machine': 'x86_64',
'platform_python_implementation': 'CPython',
'platform_release': '4.19.128-microsoft-standard',
'platform_system': 'Linux',
'platform_version': '#1 SMP Tue Jun 23 12:58:10 UTC 2020',
'python_full_version': '3.10.8',
'python_version': '3.10',
'sys_platform': 'linux'}
System environment variables:
-
CONDA_EXE -
_CE_M -
HOSTNAME -
PWD -
CONDA_PREFIX -
HOME -
LANG -
LS_COLORS -
CONDA_PROMPT_MODIFIER -
https_proxy -
TERM -
_CE_CONDA -
CONDA_SHLVL -
SHLVL -
HTTPS_PROXY -
HTTP_PROXY -
http_proxy -
CONDA_PYTHON_EXE -
CONDA_DEFAULT_ENV -
LC_ALL -
PATH -
DEBIAN_FRONTEND -
OLDPWD -
_ -
PIP_DISABLE_PIP_VERSION_CHECK -
PIP_PYTHON_PATH -
PYTHONDONTWRITEBYTECODE -
PYTHONFINDER_IGNORE_UNSUPPORTED
Pipenv–specific environment variables:
Debug–specific environment variables:
-
PATH:/opt/conda/bin:/opt/conda/condabin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin -
LANG:C.UTF-8 -
PWD:/foo
Contents of Pipfile ('/foo/Pipfile'):
[[source]]
url = "https://pypi.org/simple"
verify_ssl = false
name = "pypi"
[packages]
six = "*"
[dev-packages]
[requires]
python_version = "3.10"
Reviewing your description, I believe I know why it's failing. We (@matteius and myself) added a few additional features related to pip indexes, but you seemed to have found the one hole in the logic. Heh. Easy to fix though.
- If you add the
index-urlto yourpip.conffile, the logic that checks thepip.conffile will work properly. - If you separate the
--trusted-hostentries in yourextra-pip-args(at the command line, you have to specify only one hostname per--trusted-hostargument but you can repeat the argument, e.g.
python -m pip install pypkg --trusted-host=pypi.org --trusted-host=pypi.python.org
that will work also.
- If you specify the index inside the
requirements.txtfile, which would be unusual, but that would also work. - If you don't use a
requirements.txtfile and just install usingpipenv install pypkg, the install code will also do this correctly.
I plan to consolidate the trusted-host / verify_ssl logic into one source of truth and fix this hole in the logic in an upcoming version.
Thanks for the reply.
I tried your suggestions, but unfortunately, I still am getting the same errors.
Output of pip config list:
global.index-url='https://pypi.org/simple'
global.timeout='900'
global.trusted-host='pypi.org files.pythonhosted.org pypi.python.org'
http.sslverify='false'
Based on your suggestion, tried to install directly a package instead of using requirements.txt with the following command (where I also split the trusted host information and even provided an explicit proxy):
pipenv -v install --extra-pip-args "--trusted-host=pypi.org --trusted-host=files.pythonhosted.org --trusted-host=pypi.python.org --proxy=$https_proxy" six
This results in the following output:
(base) root@d941cd147a69:/foo# pipenv -v install --extra-pip-args "--trusted-host=pypi.org --trusted-host=files.pythonhosted.org --trusted-host=pypi.python.org --proxy=$https_proxy" six
Installing six...
Resolving six...
Installing...
Installing package: six
⠋ Installing six...Writing supplied requirement line to temporary file: 'six'
Installing 'six'
$ /root/.local/share/virtualenvs/foo-YJbvrhmB/bin/python /opt/conda/lib/python3.10/site-packages/pipenv/patched/pip/__pip-runner__.py install --no-input --verbose --upgrade --exists-action=i --trusted-host=pypi.org --trusted-host=files.pythonhosted.org --trusted-host=pypi.python.org --proxy=http://gateway.docker.internal:7070 -r /tmp/pipenv-uqxjtngs-requirements/pipenv-3dmw3xf7-requirement.txt -i https://pypi.org/simple --trusted-host pypi.org
Using source directory: '/root/.local/share/virtualenvs/foo-YJbvrhmB/src'
Adding six to Pipfile's [packages] ...
✔ Installation Succeeded
Pipfile.lock not found, creating...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
Reporter.starting()
INFO:pipenv.patched.pip._internal.resolution.resolvelib.reporter:Reporter.starting()
Reporter.adding_requirement(SpecifierRequirement('six'), None)
INFO:pipenv.patched.pip._internal.resolution.resolvelib.reporter:Reporter.adding_requirement(SpecifierRequirement('six'), None)
Reporter.starting_round(0)
INFO:pipenv.patched.pip._internal.resolution.resolvelib.reporter:Reporter.starting_round(0)
Reporter.pinning(LinkCandidate('https://files.pythonhosted.org/packages/d9/5a/e7c31adbe875f2abbb91bd84cf2dc52d792b5a01506781dbcf25c91daf11/six-1.16.0-py2.py3-none-any.whl (from
https://pypi.org/simple/six/) (requires-python:>=2.7, !=3.0.*, !=3.1.*, !=3.2.*)'))
INFO:pipenv.patched.pip._internal.resolution.resolvelib.reporter:Reporter.pinning(LinkCandidate('https://files.pythonhosted.org/packages/d9/5a/e7c31adbe875f2abbb91bd84cf2dc52d792b5a01506781dbcf25c91da
f11/six-1.16.0-py2.py3-none-any.whl (from https://pypi.org/simple/six/) (requires-python:>=2.7, !=3.0.*, !=3.1.*, !=3.2.*)'))
Reporter.ending_round(0, state)
INFO:pipenv.patched.pip._internal.resolution.resolvelib.reporter:Reporter.ending_round(0, state)
Reporter.starting_round(1)
INFO:pipenv.patched.pip._internal.resolution.resolvelib.reporter:Reporter.starting_round(1)
Reporter.ending(State(mapping=OrderedDict([('six',
LinkCandidate('https://files.pythonhosted.org/packages/d9/5a/e7c31adbe875f2abbb91bd84cf2dc52d792b5a01506781dbcf25c91daf11/six-1.16.0-py2.py3-none-any.whl (from https://pypi.org/simple/six/)
(requires-python:>=2.7, !=3.0.*, !=3.1.*, !=3.2.*)'))]), criteria={'six': Criterion((SpecifierRequirement('six'), via=None))}, backtrack_causes=[]))
INFO:pipenv.patched.pip._internal.resolution.resolvelib.reporter:Reporter.ending(State(mapping=OrderedDict([('six',
LinkCandidate('https://files.pythonhosted.org/packages/d9/5a/e7c31adbe875f2abbb91bd84cf2dc52d792b5a01506781dbcf25c91daf11/six-1.16.0-py2.py3-none-any.whl (from https://pypi.org/simple/six/)
(requires-python:>=2.7, !=3.0.*, !=3.1.*, !=3.2.*)'))]), criteria={'six': Criterion((SpecifierRequirement('six'), via=None))}, backtrack_causes=[]))
Traceback (most recent call last):
File "/opt/conda/lib/python3.10/site-packages/pipenv/patched/pip/_vendor/urllib3/connectionpool.py", line 700, in urlopen
self._prepare_proxy(conn)
File "/opt/conda/lib/python3.10/site-packages/pipenv/patched/pip/_vendor/urllib3/connectionpool.py", line 996, in _prepare_proxy
conn.connect()
File "/opt/conda/lib/python3.10/site-packages/pipenv/patched/pip/_vendor/urllib3/connection.py", line 419, in connect
self.sock = ssl_wrap_socket(
File "/opt/conda/lib/python3.10/site-packages/pipenv/patched/pip/_vendor/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
ssl_sock = _ssl_wrap_socket_impl(
File "/opt/conda/lib/python3.10/site-packages/pipenv/patched/pip/_vendor/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
File "/opt/conda/lib/python3.10/ssl.py", line 513, in wrap_socket
return self.sslsocket_class._create(
File "/opt/conda/lib/python3.10/ssl.py", line 1071, in _create
self.do_handshake()
File "/opt/conda/lib/python3.10/ssl.py", line 1342, in do_handshake
self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/conda/lib/python3.10/site-packages/pipenv/patched/pip/_vendor/requests/adapters.py", line 489, in send
resp = conn.urlopen(
File "/opt/conda/lib/python3.10/site-packages/pipenv/patched/pip/_vendor/urllib3/connectionpool.py", line 787, in urlopen
retries = retries.increment(
File "/opt/conda/lib/python3.10/site-packages/pipenv/patched/pip/_vendor/urllib3/util/retry.py", line 592, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
pipenv.patched.pip._vendor.urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /pypi/six/json (Caused by
SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)')))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/conda/lib/python3.10/site-packages/pipenv/resolver.py", line 845, in <module>
main()
File "/opt/conda/lib/python3.10/site-packages/pipenv/resolver.py", line 831, in main
_main(
File "/opt/conda/lib/python3.10/site-packages/pipenv/resolver.py", line 811, in _main
resolve_packages(
File "/opt/conda/lib/python3.10/site-packages/pipenv/resolver.py", line 759, in resolve_packages
results, resolver = resolve(
File "/opt/conda/lib/python3.10/site-packages/pipenv/resolver.py", line 738, in resolve
return resolve_deps(
File "/opt/conda/lib/python3.10/site-packages/pipenv/utils/resolver.py", line 1097, in resolve_deps
results, hashes, markers_lookup, resolver, skipped = actually_resolve_deps(
File "/opt/conda/lib/python3.10/site-packages/pipenv/utils/resolver.py", line 897, in actually_resolve_deps
hashes = resolver.resolve_hashes()
File "/opt/conda/lib/python3.10/site-packages/pipenv/utils/resolver.py", line 798, in resolve_hashes
self.hashes = self.collect_hashes(ireq)
File "/opt/conda/lib/python3.10/site-packages/pipenv/utils/resolver.py", line 778, in collect_hashes
hashes = self._get_hashes_from_pypi(ireq)
File "/opt/conda/lib/python3.10/site-packages/pipenv/utils/resolver.py", line 738, in _get_hashes_from_pypi
r = session.get(pkg_url, timeout=10)
File "/opt/conda/lib/python3.10/site-packages/pipenv/patched/pip/_vendor/requests/sessions.py", line 600, in get
return self.request("GET", url, **kwargs)
File "/opt/conda/lib/python3.10/site-packages/pipenv/patched/pip/_vendor/requests/sessions.py", line 587, in request
resp = self.send(prep, **send_kwargs)
File "/opt/conda/lib/python3.10/site-packages/pipenv/patched/pip/_vendor/requests/sessions.py", line 701, in send
r = adapter.send(request, **kwargs)
File "/opt/conda/lib/python3.10/site-packages/pipenv/patched/pip/_vendor/requests/adapters.py", line 563, in send
raise SSLError(e, request=request)
pipenv.patched.pip._vendor.requests.exceptions.SSLError: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /pypi/six/json (Caused by SSLError(SSLCertVerificationError(1,
'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)')))
✘ Locking Failed!
⠹ Locking...
Also wanted to check if there is a difference between using an = sign in the trusted-host arguments (e.g. provide it like --trusted-host=pypi.org or --trusted-host pypi.org (both work when using this within a pip install) I do see a difference in the output.
If I run the command without the = signs, so the following:
(base) root@56ff58188d78:/# pipenv install --extra-pip-args="--trusted-host pypi.org --trusted-host files.pythonhosted.org --trusted-host pypi.python.org" six
Installing six...
Resolving six...
Installing...
⠙ Installing six...[31m[1mError: [0m An error occurred while installing [32msix[0m!
Error text:
[36mERROR: Could not find a version that satisfies the requirement files.pythonhosted.org (from versions: none)
ERROR: No matching distribution found for files.pythonhosted.org
[0m
✘ Installation Failed
Seems here it tries to install the 'package' files.pythonhosted.org. Look at the documentation of pip, it is actually shown that you should provide the argument as --trusted-host <hostname>, so without the = sign.
In all of the runs btw, I start out with the Pipfile file missing. It initially will create this with a verify_ssl=true for the pypi.org source, which I then change into verify_ssl=false.
The weird thing to me is that when using pip directly, all the above things work, only using pipenv it is not working.
Are there any other things I can do to bypass this ssl_verify check?
To answer the latter inquiry first, pip uses a Python argument parser that will allow either a space or = between the argument and value. Pipenv is parsing the string you submit as a string to create the arguments for pip.
I have an experiment to try. I noticed the version of pip Conda uses is different from the one inside the virtual environment. Please run the following line with an existing pipenv virtual environment and let me know the output:
pipenv run pip install six
P.S. I noticed this difference in the output above: --proxy=$https_proxy. I'm not sure if this environment variable is being expanded or if it needs to be passed into the virtual environment via .env file. I cannot find references right now, but I seem to recall braces being needed in some contexts, e.g. ${https_proxy}.
Not 100% sure what you mean with the part with an existing pipenv virtual environment.
What I did is the following (in an empty directory):
Execute pipenv --python=3.10.8
This creates the following output:
(base) root@0040736c8498:/foo# pipenv --python=3.10.8
Creating a virtualenv for this project...
Pipfile: /foo/Pipfile
Using /opt/conda/bin/python3.1 (3.10.8) to create virtualenv...
⠴ Creating virtual environment...created virtual environment CPython3.10.8.final.0-64 in 4080ms
creator CPython3Posix(dest=/root/.local/share/virtualenvs/foo-YJbvrhmB, clear=False, no_vcs_ignore=False, global=False)
seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/root/.local/share/virtualenv)
added seed packages: pip==23.1, setuptools==67.6.1, wheel==0.40.0
activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
✔ Successfully created virtual environment!
Virtualenv location: /root/.local/share/virtualenvs/foo-YJbvrhmB
requirements.txt found in /foo instead of Pipfile! Converting...
✔ Success!
Warning: Your Pipfile now contains pinned versions, if your requirements.txt did.
We recommend updating your Pipfile to specify the "*" version, instead.
After that, with the pipfile created, I ran the command you provided: pipenv run pip install six
(base) root@0040736c8498:/foo# pipenv run pip install six
Collecting six
Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Installing collected packages: six
Successfully installed six-1.16.0
Regarding your PS question, yes, I am behind a proxy. Due to some specifics, not always possible to copy/include the PEM file for pip / pipenv to use, that is why I want to disable ssl_verify for this (it is a conscious decision)
Not 100% sure what you mean with the part with an existing pipenv virtual environment.
Exactly as you did. 🙂
Can you now rerun pipenv --support in this directory?
Also, after that, can you run pipenv install six --skip-lock in a fresh environment (or run pipenv --rm to remove the virtual environment)?
I just want to verify if this SSL fails on just locking or both locking and installing. If it does what I think it will do, I have a vague recollection of seeing something like this myself and have to find my notes to see how I addressed it.
Digging through the code, it appears that during locking, a requests.Session is created to get hashes from pypi.org. This GET request is independent of the settings for pip and its default in the requests library is verify=True. Hence, locking currently requires creating a valid SSL Context. I'm not certain, but I don't think this can be changed for security reasons. (@matteius can verify this.) The lock file is supposed to contain the correct hashes from PyPI to avoid malicious intent, and without the valid SSL Context, there's no way to verify the hashes come from PyPI and avoid a MitM attack. You can use --skip-lock to test without creating a lock file, if that works for you.
Your last message makes sense, because I can see in the proxy log that there are successful calls to the pypi servers being done and things appear to be downloaded, but after that I get the error about the SSL.
I completely understand that when we do not have a valid SSL context, we are vulnerable to a MitM attack. However, this is only for building things locally (especially inside docker containers locally) and not for building things inside our CI/CD pipeline. Also without pipenv, but just using pip I would be vulnerable to this if I make use of trusted-host. For sure the default behavior should always be to require a valid SSL context. However, would like to at least have the option to overrule it (potentially with all kinds of big warning messages being shown).
Thinking of the particular use-case, it might also be an option to use the --skip-lock when building things locally in a docker container. Will give that a try tomorrow (not behind my other computer today).
It’s not really my call, but also (and I need to verify this later when I have more time), I don’t believe you need a client cert when connecting to the PyPI public API. I believe the verify=True is verifying PyPI’s server cert in this case. You only need to (1) have a standard certificate authority bundle installed and (2) set REQUESTS_CA_BUNDLE to its location.
The problem is the bit 'You only need to have bundle installed'. When building Docker images locally, there is no generic way for me to ensure this bundle is available. The nice advantage of the disabling of SSL is that I don't have to provide any other information.
However, as mentioned, it might be that I can just skip the whole lock-file generation anyways when building Docker images locally, since these images are not going to be published and used anywhere else, except on the local computer.
However, in general I think it still would be nice if for whatever reason it is needed, you are at least able to overrule this SSL certificate checking for retrieving the hashes (especially because there is the possibility already to set config option ssl_verify = false for specific sources in the Pipfile).
It seems strange to me that the base docker image doesn’t contain the CA cents bundle or there wouldn’t be one available. shrugs
Nevertheless, if this bug is solved; I suggest submitting a feature request for this specific option so it doesn’t get lost in the shuffle, and those who have decision-making abilities can review it.
Side note: after some quick searches, I discovered that conda can install the ca certificates:
conda install -c anaconda ca-certificates
@matteius Just gave it a try and with this version, if I ensure the Pipfile contains verify_ssl = false, everything is working without a problem!
AFAIK there is no way to instruct pipenv to generate this line automatically when no Pipfile exists yet and you use pipenv install -r requirements.txt? The only way I was able to do this is to generate a new Pipfile by running pipenv install in empty folder and then modify the = true to = false
Hey @gdiepen! This is actually related to one of the updates I made (see #5572 / #5615). With environment variable PIP_TRUSTED_HOSTS, you can usually set this variable equal to false upon creation of Pipfile. Additionally, having a pip.conf file and setting it there can also help to do this.
AFAIK there is no way to instruct pipenv to generate this line automatically when no Pipfile exists yet and you use pipenv install -r requirements.txt
Not currently, but always open to improvement PRs!
@matteius @gdiepen As soon as I am able, I hope to consolidate all the index SSL logic so it’s applied consistently and uniformly across all options.
