cli icon indicating copy to clipboard operation
cli copied to clipboard
verified publisher icon httpie

HTTPS connections failing since requests version 2.32.3

Open timsutton opened this issue 1 year ago • 2 comments

👋 thanks for taking a look :)

Checklist

  • [x] I've searched for similar issues.
  • [x] I'm using the latest version of HTTPie.

Minimal reproduction code and steps

  1. (create new virtualenv)
  2. pip install httpie
  3. https https://raw.githubusercontent.com/Homebrew/homebrew-core/HEAD/Formula/h/httpie.rb

Current result

https: error: SSLError: HTTPSConnectionPool(host='raw.githubusercontent.com', port=443): Max retries exceeded with url: /Homebrew/homebrew-core/HEAD/Formula/h/httpie.rb (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1000)'))) while doing a GET request to URL: https://raw.githubusercontent.com/Homebrew/homebrew-core/HEAD/Formula/h/httpie.rb

Expected result

Request would succeed.

Debug output

Please re-run the command with --debug, then copy the entire command & output and paste both below:

$ https --debug https://raw.githubusercontent.com/Homebrew/homebrew-core/HEAD/Formula/h/httpie.rb
https --debug https://raw.githubusercontent.com/Homebrew/homebrew-core/HEAD/Formula/h/httpie.rb
HTTPie 3.2.2
Requests 2.32.3
Pygments 2.18.0
Python 3.12.3 (main, Jun  8 2024, 18:03:49) [Clang 15.0.0 (clang-1500.3.9.4)]
/private/tmp/.venv-pyenv-py3/bin/python3
Darwin 23.5.0

<Environment {'apply_warnings_filter': <function Environment.apply_warnings_filter at 0x10219a160>,
 'args': Namespace(),
 'as_silent': <function Environment.as_silent at 0x10219a020>,
 'colors': 256,
 'config': {'default_options': []},
 'config_dir': PosixPath('/Users/tsutton/.config/httpie'),
 'devnull': <property object at 0x10217f560>,
 'is_windows': False,
 'log_error': <function Environment.log_error at 0x10219a0c0>,
 'program_name': 'https',
 'quiet': 0,
 'rich_console': <functools.cached_property object at 0x102114ec0>,
 'rich_error_console': <functools.cached_property object at 0x10109d580>,
 'show_displays': True,
 'stderr': <_io.TextIOWrapper name='<stderr>' mode='w' encoding='utf-8'>,
 'stderr_isatty': True,
 'stdin': <_io.TextIOWrapper name='<stdin>' mode='r' encoding='utf-8'>,
 'stdin_encoding': 'utf-8',
 'stdin_isatty': True,
 'stdout': <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>,
 'stdout_encoding': 'utf-8',
 'stdout_isatty': True}>

<PluginManager {'adapters': [],
 'auth': [<class 'httpie.plugins.builtin.BasicAuthPlugin'>,
          <class 'httpie.plugins.builtin.DigestAuthPlugin'>,
          <class 'httpie.plugins.builtin.BearerAuthPlugin'>],
 'converters': [],
 'formatters': [<class 'httpie.output.formatters.headers.HeadersFormatter'>,
                <class 'httpie.output.formatters.json.JSONFormatter'>,
                <class 'httpie.output.formatters.xml.XMLFormatter'>,
                <class 'httpie.output.formatters.colors.ColorFormatter'>]}>

>>> requests.request(**{'auth': None,
 'data': RequestJSONDataDict(),
 'headers': <HTTPHeadersDict('User-Agent': b'HTTPie/3.2.2')>,
 'method': 'get',
 'params': <generator object MultiValueOrderedDict.items at 0x10236e980>,
 'url': 'https://raw.githubusercontent.com/Homebrew/homebrew-core/HEAD/Formula/h/httpie.rb'})


https: error: SSLError: HTTPSConnectionPool(host='raw.githubusercontent.com', port=443): Max retries exceeded with url: /Homebrew/homebrew-core/HEAD/Formula/h/httpie.rb (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1000)'))) while doing a GET request to URL: https://raw.githubusercontent.com/Homebrew/homebrew-core/HEAD/Formula/h/httpie.rb


Traceback (most recent call last):
  File "/private/tmp/.venv-pyenv-py3/lib/python3.12/site-packages/urllib3/connectionpool.py", line 466, in _make_request
    self._validate_conn(conn)
  File "/private/tmp/.venv-pyenv-py3/lib/python3.12/site-packages/urllib3/connectionpool.py", line 1095, in _validate_conn
    conn.connect()
  File "/private/tmp/.venv-pyenv-py3/lib/python3.12/site-packages/urllib3/connection.py", line 652, in connect
    sock_and_verified = _ssl_wrap_socket_and_match_hostname(
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/private/tmp/.venv-pyenv-py3/lib/python3.12/site-packages/urllib3/connection.py", line 805, in _ssl_wrap_socket_and_match_hostname
    ssl_sock = ssl_wrap_socket(
               ^^^^^^^^^^^^^^^^
  File "/private/tmp/.venv-pyenv-py3/lib/python3.12/site-packages/urllib3/util/ssl_.py", line 465, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls, server_hostname)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/private/tmp/.venv-pyenv-py3/lib/python3.12/site-packages/urllib3/util/ssl_.py", line 509, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/tsutton/.pyenv/versions/3.12.3/lib/python3.12/ssl.py", line 455, in wrap_socket
    return self.sslsocket_class._create(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/tsutton/.pyenv/versions/3.12.3/lib/python3.12/ssl.py", line 1042, in _create
    self.do_handshake()
  File "/Users/tsutton/.pyenv/versions/3.12.3/lib/python3.12/ssl.py", line 1320, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1000)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/private/tmp/.venv-pyenv-py3/lib/python3.12/site-packages/urllib3/connectionpool.py", line 789, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "/private/tmp/.venv-pyenv-py3/lib/python3.12/site-packages/urllib3/connectionpool.py", line 490, in _make_request
    raise new_e
urllib3.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1000)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/private/tmp/.venv-pyenv-py3/lib/python3.12/site-packages/requests/adapters.py", line 667, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/private/tmp/.venv-pyenv-py3/lib/python3.12/site-packages/urllib3/connectionpool.py", line 843, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/private/tmp/.venv-pyenv-py3/lib/python3.12/site-packages/urllib3/util/retry.py", line 519, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='raw.githubusercontent.com', port=443): Max retries exceeded with url: /Homebrew/homebrew-core/HEAD/Formula/h/httpie.rb (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1000)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/private/tmp/.venv-pyenv-py3/bin/https", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/private/tmp/.venv-pyenv-py3/lib/python3.12/site-packages/httpie/__main__.py", line 9, in main
    exit_status = main()
                  ^^^^^^
  File "/private/tmp/.venv-pyenv-py3/lib/python3.12/site-packages/httpie/core.py", line 162, in main
    return raw_main(
           ^^^^^^^^^
  File "/private/tmp/.venv-pyenv-py3/lib/python3.12/site-packages/httpie/core.py", line 136, in raw_main
    handle_generic_error(propagated_exc, annotation=annotation)
  File "/private/tmp/.venv-pyenv-py3/lib/python3.12/site-packages/httpie/core.py", line 100, in raw_main
    exit_status = main_program(
                  ^^^^^^^^^^^^^
  File "/private/tmp/.venv-pyenv-py3/lib/python3.12/site-packages/httpie/core.py", line 213, in program
    for message in messages:
  File "/private/tmp/.venv-pyenv-py3/lib/python3.12/site-packages/httpie/client.py", line 113, in collect_messages
    response = requests_session.send(
               ^^^^^^^^^^^^^^^^^^^^^^
  File "/private/tmp/.venv-pyenv-py3/lib/python3.12/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/private/tmp/.venv-pyenv-py3/lib/python3.12/site-packages/requests/adapters.py", line 698, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='raw.githubusercontent.com', port=443): Max retries exceeded with url: /Homebrew/homebrew-core/HEAD/Formula/h/httpie.rb (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1000)')))

Additional information, screenshots, or code examples

Noticed this issue when homebrew-core attempted to bump the httpie formula's requests pypi dependency: https://github.com/Homebrew/homebrew-core/pull/174896#issuecomment-2175968950

In the above example I used a pyenv-installed Python for the virtualenv just to rule out whether there could have been an issue with using homebrew's python3 for some reason.

timsutton avatar Jun 20 '24 17:06 timsutton

I can confirm that downgrading to requests==2.32.2 resolves the issue. I haven't dug any further into it though.

➜  httpie-test poetry show
certifi            2024.6.2 Python package for providing Mozilla's CA Bundle.
charset-normalizer 3.3.2    The Real First Universal Charset Detector. Open, modern and actively maintained alternative to Chardet.
defusedxml         0.7.1    XML bomb protection for Python stdlib modules
httpie             3.2.2    HTTPie: modern, user-friendly command-line HTTP client for the API era.
idna               3.7      Internationalized Domain Names in Applications (IDNA)
markdown-it-py     3.0.0    Python port of markdown-it. Markdown parsing, done right!
mdurl              0.1.2    Markdown URL utilities
multidict          6.0.5    multidict implementation
pip                24.1.1   The PyPA recommended tool for installing Python packages.
pygments           2.18.0   Pygments is a syntax highlighting package written in Python.
pysocks            1.7.1    A Python SOCKS client module. See https://github.com/Anorov/PySocks for more information.
requests           2.32.3   Python HTTP for Humans.
requests-toolbelt  1.0.0    A utility belt for advanced users of python-requests
rich               13.7.1   Render rich text, tables, progress bars, syntax highlighting, markdown and more to the terminal
setuptools         70.1.1   Easily download, build, install, upgrade, and uninstall Python packages
urllib3            2.2.2    HTTP library with thread-safe connection pooling, file post, and more.
➜  httpie-test poetry run https github.com

https: error: SSLError: HTTPSConnectionPool(host='github.com', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1000)'))) while doing a GET request to URL: https://github.com/


➜  httpie-test poetry add requests==2.32.2

Updating dependencies
Resolving dependencies... (0.1s)

Package operations: 0 installs, 1 update, 0 removals

  - Downgrading requests (2.32.3 -> 2.32.2)

Writing lock file
➜  httpie-test poetry run https api.github.com
HTTP/1.1 200 OK
Accept-Ranges: bytes
Access-Control-Allow-Origin: *
Access-Control-Expose-Headers: ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Resource, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, X-GitHub-SSO, X-GitHub-Request-Id, Deprecation, Sunset
Cache-Control: public, max-age=60, s-maxage=60
Content-Encoding: gzip
Content-Length: 510
Content-Security-Policy: default-src 'none'
Content-Type: application/json; charset=utf-8
Date: Thu, 27 Jun 2024 15:06:43 GMT
ETag: W/"4f825cc84e1c733059d46e76e6df9db557ae5254f9625dfe8e1b09499c449438"
Referrer-Policy: origin-when-cross-origin, strict-origin-when-cross-origin
Server: GitHub.com
Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
Vary: Accept, Accept-Encoding, Accept, X-Requested-With
X-Content-Type-Options: nosniff
X-Frame-Options: deny
X-GitHub-Media-Type: github.v3; format=json
X-GitHub-Request-Id: D858:142C:20F1CC:3F070E:667D8003
X-RateLimit-Limit: 60
X-RateLimit-Remaining: 59
X-RateLimit-Reset: 1719504404
X-RateLimit-Resource: core
X-RateLimit-Used: 1
X-XSS-Protection: 0
x-github-api-version-selected: 2022-11-28

{
    "authorizations_url": "https://api.github.com/authorizations",
    "code_search_url": "https://api.github.com/search/code?q={query}{&page,per_page,sort,order}",
    "commit_search_url": "https://api.github.com/search/commits?q={query}{&page,per_page,sort,order}",
    "current_user_authorizations_html_url": "https://github.com/settings/connections/applications{/client_id}",
    "current_user_repositories_url": "https://api.github.com/user/repos{?type,page,per_page,sort}",
    "current_user_url": "https://api.github.com/user",
    "emails_url": "https://api.github.com/user/emails",
    "emojis_url": "https://api.github.com/emojis",
    "events_url": "https://api.github.com/events",
    "feeds_url": "https://api.github.com/feeds",
    "followers_url": "https://api.github.com/user/followers",
    "following_url": "https://api.github.com/user/following{/target}",
    "gists_url": "https://api.github.com/gists{/gist_id}",
    "hub_url": "https://api.github.com/hub",
    "issue_search_url": "https://api.github.com/search/issues?q={query}{&page,per_page,sort,order}",
    "issues_url": "https://api.github.com/issues",
    "keys_url": "https://api.github.com/user/keys",
    "label_search_url": "https://api.github.com/search/labels?q={query}&repository_id={repository_id}{&page,per_page}",
    "notifications_url": "https://api.github.com/notifications",
    "organization_repositories_url": "https://api.github.com/orgs/{org}/repos{?type,page,per_page,sort}",
    "organization_teams_url": "https://api.github.com/orgs/{org}/teams",
    "organization_url": "https://api.github.com/orgs/{org}",
    "public_gists_url": "https://api.github.com/gists/public",
    "rate_limit_url": "https://api.github.com/rate_limit",
    "repository_search_url": "https://api.github.com/search/repositories?q={query}{&page,per_page,sort,order}",
    "repository_url": "https://api.github.com/repos/{owner}/{repo}",
    "starred_gists_url": "https://api.github.com/gists/starred",
    "starred_url": "https://api.github.com/user/starred{/owner}{/repo}",
    "topic_search_url": "https://api.github.com/search/topics?q={query}{&page,per_page}",
    "user_organizations_url": "https://api.github.com/user/orgs",
    "user_repositories_url": "https://api.github.com/users/{user}/repos{?type,page,per_page,sort}",
    "user_search_url": "https://api.github.com/search/users?q={query}{&page,per_page,sort,order}",
    "user_url": "https://api.github.com/users/{user}"
}

sbchisholm avatar Jun 27 '24 15:06 sbchisholm

Could this be related to https://github.com/psf/requests/issues/6730? This seems like the same regression, if downgrading the package fixes the problem.

archer-321 avatar Jun 27 '24 15:06 archer-321

Quick workaround is to append --verify=no although in most scenarios it's not recommended. Use this only if you really need to bypass SSL Verification.

RVFET avatar Jul 07 '24 19:07 RVFET

The --help documentation for the --verify option is

Set to "no" (or "false") to skip checking the host's SSL certificate. Defaults to "yes" ("true"). You can also pass the path to a CA_BUNDLE file for private certs. (Or you can set the REQUESTS_CA_BUNDLE environment variable instead.)

At least for me, the issue is that HTTPie doesn't find the correct bundle of CA certificates. Instead of opting out of TLS certificate verification, I'd recommend pointing HTTPie to the correct bundle path in this case. On Arch, a bundle is available at /etc/ssl/certs/ca-bundle.crt. On Debian and Alpine, a bundle is in /etc/ssl/certs/ca-certificates.crt.

$ http --verify=/etc/ssl/certs/ca-bundle.crt https://github.com/

HTTP/1.1 200 OK
...

If you don't want to set the option for every call to http, you can set the environment variable mentioned in the documentation:

$ export REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-bundle.crt
$ http https://github.com/

HTTP/1.1 200 OK
...

Setting the environment variable at the start of every terminal session is my current workaround for this bug.

archer-321 avatar Jul 07 '24 19:07 archer-321

Please stop posting for this issue.

  • Yes, HTTPie mainstream version is most likely at least partially broken with HTTPS using Requests 2.32+
  • Yes, the author is aware.
  • Yes, Requests is at fault here.

We cannot do anything until the author (aka. maintainer) either publish a workaround or merge #1531 [...]

Two possibles resolutions

Plan A

Downgrade Requests to 2.31

Plan B

Try HTTPie v4 (unofficial) which detach itself from Requests. If you rely on pip and not OS packages.

pip install "git+https://github.com/Ousret/HTTPie.git@feature-tryout-niquests" -U

regards,

Ousret avatar Jul 07 '24 20:07 Ousret

Duplicate of https://github.com/httpie/cli/issues/1581

veilm avatar Jul 07 '24 20:07 veilm

✅ Fixed in v3.2.3

jkbrzt avatar Jul 10 '24 14:07 jkbrzt

Downgrading Requests to 2.31 causes conflicts with other pip packages that require Requests >2.31. So this is not a good solution IMO.

abdgad avatar Jul 11 '24 07:07 abdgad

My fix to make it work with requests 2.32.3: Just add to http and https:

import os
os.environ['REQUESTS_CA_BUNDLE'] = "/etc/ssl/certs/ca-bundle.crt"

Example:

#!/usr/bin/python3
# -*- coding: utf-8 -*-
import re
import sys
import os
os.environ['REQUESTS_CA_BUNDLE'] = "/etc/ssl/certs/ca-bundle.crt"
from httpie.__main__ import main
if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
    sys.exit(main())

darkshram avatar Aug 04 '24 00:08 darkshram

Sent a fix that actually works with requests 2.32.3 as https://github.com/httpie/cli/pull/1596 .

AdamWill avatar Sep 04 '24 00:09 AdamWill

How long it this going to take to get into production?

Neutrino-Sunset avatar Feb 20 '25 11:02 Neutrino-Sunset