Proposal 4.0 (HTTP/2+3, OS Trust Store, Custom DNS, OCSP, ...)

[Ousret]

This PR showcases how HTTPie could evolve outside of Requests.

Niquests is supposed to be a (mostly) compatible fork of Requests.

Try this preview:

$ pip install "git+https://github.com/Ousret/HTTPie.git@feature-tryout-niquests" -U

Here are the biggest pros of this:

• OS truststore by default, no more certifi!
• Object-oriented headers. Could bring additional features!
• Fully type-annotated!
• HTTP/3 over QUIC.
• HTTP/2 by default.
• Exit Python http.client! in favor of h11.
• Timeout by default.
• Inspect peer certificate, HTTP version, TLS version, cipher, and so on via hook/callback before a request is sent.
• All the features you expose are available in all three protocols.
• Python 3.7+, no sacrifice needed.
• Encrypted DNS support. w/ DNS-over-HTTPS, DNS-over-QUIC, DNS-over-TLS and plain DNS-over-UDP.

Obviously, cons:

• Stricter on emitted requests per RFCs
• Young project but based on solid bases, knowledge, and experiences.
• Need to publish new packages to distro. Easy but time-consuming
• Exit pyopenssl, not supported. more of a pro to me
• Require major bump? Could be. Should be.

Complete list of changes in the fork: https://github.com/jawah/niquests/blob/main/HISTORY.md

• Close #576
• Close #480
• Close #692
• Close #523
• Close #549
• Close #1023
• Close #826
• Close #800
• Close #1495
• Close #1202
• Close #99
• Close #1422
• Close #94
• Close #1480
• Close #1522
• Close #1413
• Close #1446
• Close #414
• Close #1551
• Close #1456
• Close #1338
• Close #1401
• Close #1527
• Close #602
• Close #962
• Close #1554
• Close #423
• Close #1400
• Close #1458
• Close #722
• Close #579
• Close #1580
• Close #1581
• Close #752
• Close #1599
• Close #1530

---

4.0.0 (unreleased)

• Switched from the requests library to the compatible niquests. (#1531)
• Added support for HTTP/2, and HTTP/3 protocols. (#523, #692, #1531)
• Added support for early (informational) responses. (#752) (#1531)
• Added support for Happy Eyeballs algorithm via --heb flag (disabled by default). #1599 #1531
• Added support for IPv4/IPv6 enforcement with -6 and -4. (#94, #1531)
• Added support for alternative DNS resolvers via --resolver. DNS over HTTPS, DNS over TLS, DNS over QUIC, and DNS over UDP are accepted. (#99, #1531)
• Added support for binding to a specific network adapter with --interface. (#1422, #1531)
• Added support for specifying the local port with --local-port. (#1456, #1531)
• Added request metadata for the TLS certificate, negotiated version with cipher, the revocation status and the remote peer IP address. (#1495, #1023, #826, #1531)
• Added support to load the operating system trust store for the peer certificate validation. (#480, #1531)
• Added support for using the system trust store to retrieve root CAs for verifying TLS certificates. (#1531)
• Added detailed timings in response metadata with DNS resolution, established, TLS handshake, and request sending delays. (#1023, #1531)
• Added automated resolution of hosts ending with .localhost to the default loopback address. (#1458, #1527)
• Fixed the case when multiple headers where concatenated in the response output. (#1413, #1531)
• Fixed an edge case where HTTPie could be lead to believe data was passed in stdin, thus sending a POST by default. (#1551, #1531) This fix has the particularity to consider 0 byte long stdin buffer as absent stdin. Empty stdin buffer will be ignored.
• Improved performance while downloading by setting chunk size to -1 to retrieve packets as they arrive. (#1531)
• Fixed multipart form data having filename not rfc2231 compliant when name contain non-ascii characters. (#1401)
• Fixed issue where the configuration directory was not created at runtime that made the update fetcher run everytime. (#1527)
• Fixed cookie persistence in HTTPie session when targeting localhost. They were dropped due to the standard library. (#1527)
• Fixed downloader when trying to fetch compressed content. The process will no longer exit with the "Incomplete download" error. (#1554, #423, #1527)
• Fixed downloader yielding an incorrect speed when the remote is using Content-Encoding aka. compressed body. (#1554, #423, #1527)
• Removed support for preserving the original casing of HTTP headers. This comes as a constraint of newer protocols, namely HTTP/2+ that normalize header keys by default. From the HTTPie user perspective, they are "prettified" in the output by default. e.g. x-hello-world is displayed as X-Hello-World.
• Removed support for pyopenssl. (#1531)
• Removed support for dead SSL protocols < TLS 1.0 (e.g. sslv3) as per pyopenssl removal. (#1531)
• Removed dependency on requests_toolbelt in favor of directly including MultipartEncoder into HTTPie due to its direct dependency to requests. (#1531)
• Removed dependency on multidict in favor of an internal one due to often missing pre-built wheels. (#1522, #1531)

Existing plugins are expected to work without any changes. The only caveat would be that certain plugin explicitly require requests. Future contributions may be made in order to relax the constraints where applicable.

[Ousret]

update:

debug + verbose output

• Niquests support universal revocation verification in addition to the native OS root CAs.
• Reviewed output format for debug info to match HTTPie's format.
• Down to 9 failures, which seems trivial enough to fix.

[Ousret]

The remaining 9 cases run fine if standalone. something tampers with the exception management. thus making assertions fail consistently. need investigation.

[codecov-commenter]

:warning: Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 0.00%. Comparing base (4d7d6b6) to head (7cd6579). Report is 370 commits behind head on master.

:exclamation: Current head 7cd6579 differs from pull request most recent head 55aea9c

Please upload reports for the commit 55aea9c to get more accurate results.

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #1531       +/-   ##
==========================================
- Coverage   97.28%       0   -97.29%     
==========================================
  Files          67       0       -67     
  Lines        4235       0     -4235     
==========================================
- Hits         4120       0     -4120     
+ Misses        115       0      -115     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[Ousret]

Update:

Now everything has passed. I identified some parts of the code that could be removed.

• https GET pie.dev/get

• https -v GET pie.dev/get

• https -v --debug GET pie.dev/get

• https --ssl=tls1.2 -v --debug GET pie.dev/get

• http -v --debug GET pie.dev/get

[Ousret]

Tests / test (macos-latest, 3.11) (pull_request) Failing after 2m Details @github-actions Tests / test (macos-latest, 3.12) (pull_request) Failing after 4m

it is about CI flakyness, a restart will fix that.

[Ousret]

So far, I identified this:

• Close #576
• Close #480
• Close #692
• Close #523
• Close #549

[Ousret]

We can add the following arguments and document them:

• --disable-http2
• --disable-http3
• --http3, e.g. force trying it.

[Ousret]

I've added and documented it.

• --disable-http2
• --disable-http3
• --http3, e.g. force trying it.

Ideally, the connection information (TLS, Cipher, Peer/Issuer Certificate, and OCSP) should be in the request metadata and with a lexer instruction.

[Ousret]

Now, the result is pleasant to the eye. I have implemented the metadata for Request and the result is as follows:

Niquests now support tracking upload progress, this can also be done here. It should be straightforward enough.

[dwt]

Damn this looks so tantalizing. I'd love for this to get packaged as a tryout / beta. Something installable via e.g. pip install httpie[niquest] perhaps?

[Ousret]

Hello there,

---

@dwt

Damn this looks so tantalizing.

Couldn't agree more, wait until you see what just landed in this PR.

I'd love for this to get packaged as a tryout / beta.

That's the main idea I had in mind. If you or anyone you know is willing to give this a try, then feel free to share and ask for feedbacks.

---

@jkbrzt

I, again, am still hopeful you could spare a bit of time to steer this proposal. With the most recent push, there's some interesting news.

Starting now, HTTPie PR/1531 can:

• Use custom DNS resolver, with but not limited to: #99
  • DNS over HTTPS
  • DNS over QUIC
  • DNS over TLS
  • DNS over UDP

$ https --resolver "doh+cloudflare://" pie.dev/get

or..

$ https --resolver "in-memory://default/?hosts=pie.dev:10.11.11.1" pie.dev/get

or..using the shortcut:

$ https --resolver "pie.dev:10.11.11.1" pie.dev/get

• Bind to a specific network interface. #1422

$ https --interface 172.17.0.1 pie.dev/get

• Enforce IPv4, or IPv6. #94

$ https -4 pie.dev/get

• Detailed timings in response metadata with DNS resolution, established, TLS handshake, and request sending delays. #1023

This is by far, according to me, the most exiting upgrade HTTPie can land and justify the major bump. I took the liberty to draft a changelog for a "potential" 4.0.0 beta 1.

Regards,

[dwt]

Well, that would be the most interesting update to httpie in some time according to me too. :)

[dwt]

@Ousret I'm trying out the http3 support and wasn't able to get it working. Could you perhaps provide some guidance on how to best debug this?

❯ https --http3 -vvv pie.dev/get
Connected to: 188.114.96.4 port 443
Connection secured using: TLSv1.3 with AES-256-GCM-SHA384
Server certificate: commonName="pie.dev"; DNS="*.pie.dev"; DNS="pie.dev"
Certificate validity: "Nov 11 01:14:24 2023 GMT" to "Feb  9 01:14:23 2024 GMT"
Issuer: countryName="US"; organizationName="Let's Encrypt"; commonName="E1"
Revocation status: Good

GET /get HTTP/2
Accept: */*
Accept-Encoding: gzip, deflate
Connection: keep-alive
Host: pie.dev
User-Agent: HTTPie/4.0.0.b1

Details

```shell ❯ https --http3 -vvv pie.dev/get Connected to: 188.114.96.4 port 443 Connection secured using: TLSv1.3 with AES-256-GCM-SHA384 Server certificate: commonName="pie.dev"; DNS="*.pie.dev"; DNS="pie.dev" Certificate validity: "Nov 11 01:14:24 2023 GMT" to "Feb 9 01:14:23 2024 GMT" Issuer: countryName="US"; organizationName="Let's Encrypt"; commonName="E1" Revocation status: Good

GET /get HTTP/2 Accept: / Accept-Encoding: gzip, deflate Connection: keep-alive Host: pie.dev User-Agent: HTTPie/4.0.0.b1

HTTP/2 200 OK Access-Control-Allow-Credentials: true Access-Control-Allow-Origin: * Alt-Svc: h3=":443"; ma=86400 Cf-Cache-Status: DYNAMIC Cf-Ray: 83fa5064bee41d9a-FRA Content-Encoding: gzip Content-Type: application/json Date: Wed, 03 Jan 2024 09:46:20 GMT Nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800} Report-To: {"endpoints":[{"url":"https://a.nel.cloudflare.com/report/v3?s=Ca0pVbLbx9QmhpCG%2Fq03dTYDMeHurFMramSLFHceVhaq6dMGWBep340o4G9B3t%2FlHxfF5Agpp7UHgkxGa5AMFVyvov%2BhOGIsDikuFWJMVFWp4lN9Xnd%2FILz9"}],"group":"cf-nel","max_age":604800} Server: cloudflare

{ "args": {}, "headers": { "Accept": "/", "Accept-Encoding": "gzip", "Cdn-Loop": "cloudflare", "Cf-Connecting-Ip": "91.65.247.116", "Cf-Ipcountry": "DE", "Cf-Ray": "83fa5064bee41d9a-FRA", "Cf-Visitor": "{"scheme":"https"}", "Connection": "Keep-Alive", "Host": "pie.dev", "User-Agent": "HTTPie/4.0.0.b1" }, "origin": "91.65.247.116", "url": "https://pie.dev/get" }

Elapsed DNS: 0.002622s Elapsed established connection: 0.026514s Elapsed TLS handshake: 0.079068s Elapsed emitting request: 0.000366s Elapsed time: 0.254063667s⏎
~/C/P/httpie 🐍 tempenv-13fd235984bf4 🌱 feature-tryout-niquests
❯ macosver 10.16

~/C/P/httpie 🐍 tempenv-13fd235984bf4 🌱 feature-tryout-niquests
❯ python --version Python 3.12.1

~/C/P/httpie 🐍 tempenv-13fd235984bf4 🌱 feature-tryout-niquests
❯ pip list Package Version Editable project location

---

certifi 2023.11.17 cffi 1.16.0 charset-normalizer 3.3.2 cryptography 41.0.7 defusedxml 0.7.1 h11 0.14.0 h2 4.1.0 hpack 4.0.0 httpie 4.0.0b1 /Users/dwt/Code/Projekte/httpie hyperframe 6.0.1 idna 3.6 kiss-headers 2.4.3 markdown-it-py 3.0.0 mdurl 0.1.2 multidict 6.0.4 niquests 3.4.0 pip 23.3.2 pycparser 2.21 Pygments 2.17.2 PySocks 1.7.1 python-socks 2.4.4 qh3 0.14.0 requests 2.31.0 requests-toolbelt 1.0.0 rich 13.7.0 setuptools 69.0.3 urllib3 2.1.0 urllib3-future 2.4.902 wassima 1.0.3

</p>
</details> 

[Ousret]

OK, I can try. First, running the exact same command locally, work fine.

You don't have to specify --http3 as pie.dev is covered by Cloudflare preemptive QUIC declaration in DNS records.

According to your dependencies listening, you have qh3 installed, so it should work and everything is up to date.

Something is preventing it on your environment. Let's try to understand. Are you using conda? or poetry? or..?

Try to run the following, standalone:

from niquests import Session

with Session(resolver="doh+google://") as s:
    print(s.get("https://pie.dev/get"))

and then:

from urllib3.contrib.hface import HTTPProtocolFactory, HTTP3Protocol
# or
from urllib3_future.contrib.hface import HTTPProtocolFactory, HTTP3Protocol

print(HTTPProtocolFactory.new(HTTP3Protocol))

Create a specific issue at Niquests tracker, we shall continue there to avoid spamming this thread.

[Ousret]

Thanks to @dwt testing, we discovered an edge case where http3 would be silently disabled, it depended on the ssl ctx created with the system default ciphers list, which vary. With this, I discovered a unusual bug that is present in the 3.x major, see https://github.com/httpie/cli/issues/1551 for more details. It is fixed now (both cases).

[Ousret]

Now, this closes 20+ issues. Just added support for --local-port args. Accept single port or range. Also fixed a bunch of warnings from v3.x, from 120+ warnings to 24.

[Ousret]

gentle ping @jkbrzt

[dwt]

@jkbrzt hey, this code seems like a phenomenal update to the capabilities of httpie. Could you please take a look or recommend someone else you trust to take one?

[dwt]

@Ousret It seems to be hard to reach @jkbrzt - any Idea how to get him to notice? I didn't find any social addresses outside of twitter, and it seems they want me to pay now to send private messages there.

[Ousret]

I understand the frustration.

There's almost no chance that @jkbrzt didn't saw this (amongst other PR and issue piling up of course), so there's really no reason for us to ping him any more. This PR is 4 months old already, and we didn't get the slightest sign of whether it is a desirable PR or not.

According to stats at my disposal we have a significant amount of "git clone" that must originate from people that can't setup HTTPie with Python 3.12+ around 10 a week, and I expect that number to quickly raise, this is undesirable, maintenance wise.

I tend to wait 5 to 6 months before giving up on a PR, usually. If, HTTPie would be stuck this way, someone would have to assume a fork, and, well, it won't be me as I have no bandwidth to allocate for long term maintenance.

[dwt]

I have to admit that I'm already very impressed that you invested all of. this work without having any word from the project owner whether he likes this or not. I sure hope that we can get him to reply at some point, because this looks like a big present he got for free for this project.

[dwt]

@jkbrzt How can we reach you? It would be really nice (but after this long a time also neccessary) for you to chime into this pull request and give some information what you would require to get this merged.

[jkbrzt]

@Ousret Impressive stuff, and sorry for the lack of feedback. The initial effort looked like a speculative experiment, but it’s evolved quite a bit since then! I have some questions. Would you be up for connecting over Zoom/Meet to discuss the code, niquests, and the potential switch from requests? If it sounds good, please shoot me a message via jakub at httpie.io.

[Ousret]

Thank you for your answer. I would gladly answer any question you have.

Would you be up for connecting over Zoom/Meet

Excellent idea, let's do that. I will email you so that we can schedule a meet.

Regards,

[Ousret]

Made some general (minor) code cleanup and fixed almost all warnings.

Passed from (master) ===== 1019 passed, 5 skipped, 4 xfailed, 135 warnings in 71.36s (0:01:11) ======

to (this PR) ====== 1037 passed, 5 skipped, 1 xfailed, 5 warnings in 77.09s (0:01:17) =======

The 5 warnings remaining are going to be annoying to track down and fix for now.

[Ousret]

I've applied the initial batch of suggestions @jkbrzt if I missed anything, let me know. Will be waiting for further instructions.

[jkbrzt]

@Ousret thank you for the call yesterday.

I’ve started diving into the code and doing some manual QA.

Here are a few cosmetic issues I’ve noticed so far:

# Test command
$ https -vv pie.dev/get

• [ ] Request-Line and Status-Line are no longer syntax-highlighted in the output. The HTTP Pygments lexer probably needs to be updated for HTTP > 1.
• [ ] We need an extra blank line after the --vv output. Currently the output is Elapsed time: 0.118802041s$ where $ is my shell prompt.
• [ ] We need consistent and user-friendly rounding for various durations. The download progress probably has something usable we could apply everywhere. (The current output sometimes uses scientific notation: Elapsed established connection: 4.8e-05s)

[Ousret]

OK! The cosmetic issues are now fixed. I took the liberty to add a single assert that verify we've not missed the extraneous RC for the metadata.

For now, all clear.

[Ousret]

Did found another cosmetic issue (when follow redirects) the meta was glued to the next request line. Now fixed. And made the metadata look even nicer by replacing "0.0s" with "0s". (Found it in same scenario where redirect did not require DNS resolve and such..) Finally, I could resolve 4 warnings, remain 1! ======= 1039 passed, 5 skipped, 1 xfailed, 1 warning in 77.85s (0:01:17) =======

[Ousret]

So, I did some additional testing on my end and found nothing. Additionally, to mark this milestone, the very last warning is now fixed!

This branch now have 0 warning. ============ 1039 passed, 5 skipped, 1 xfailed in 81.02s (0:01:21) =============

For a side note, Niquests just landed Happy Eyeballs feature in the last version, if this is of any interest to you, let me know.

and that commit https://github.com/httpie/cli/pull/1531/commits/cce24fea7f933b5978461b31da5bdc5a93c590ae has nothing to do with Niquests or HTTPie really, you must have encountered this situation where a single MacOS test was flaky, this circumvent the random CI failures.