Support HTTP/2

[NghiaTranUIT]

🐶 Brief

At the moment, all intercepted HTTP/HTTPS through Proxyman will be downgraded to HTTP/1.1. We should support HTTP/2 too and implement the HTTP/2 parser correctly.

👑 Criteria

• [x] Support h2 ALPN in CocoaAsyncSocket
• [x] Determine whether or not the request is HTTP/2
• [x] Write HTTP/2 Parser. We could inspired from Apple-Server source
• [x] Work with current architecture. HTTP/1.x and HTTP/2

Update 1 (5 Feb 2023): Beta build for HTTP/2 is ready ✅

Comment: https://github.com/ProxymanApp/Proxyman/issues/416#issuecomment-1417959460

Update 2 (14 Feb 2023): New Update

Comment: https://github.com/ProxymanApp/Proxyman/issues/416#issuecomment-1429048662

Update 3 (22 Feb 2023):

https://github.com/ProxymanApp/Proxyman/issues/416#issuecomment-1439878308

[Saklad5]

Any progress on this?

[NghiaTranUIT]

@Saklad5 Sorry, but at the moment, I'm working on Protobuf feature.

When it's done, I will kick start the HTTP/2 since it's crucial to support gRPC feature too 🙌

[Saklad5]

Glad to see the Protobuf feature is done: it’s not something I’m interested in, but it means you might work on this soon. Once Proxyman can run without downgrading traffic, I think I’ll purchase it. Until then, I can’t really justify running it for long periods of time, which limits its utility.

[NghiaTranUIT]

I would like to say that HTTP/2 and HTTP/3 are must-have features of Proxyman, and the progress is really good at the moment 🙌

Protobuf, the first blocker is done, it's time for HTTP/2.

Stay tuned 🌮

[Saklad5]

HTTP/3 isn’t formally accepted yet, so I’d say that’s less of a must-have than HTTP/2. I generally don’t expect things to support draft standards, and I understand it is pretty different.

By the way, does Proxyman do any other downgrading at the moment? Besides forcing HTTP/1, does simply running it affect outgoing traffic at all?

[NghiaTranUIT]

At the moment, only HTTPS domains, which enable SSL-Proxying (To see the HTTP Response Content) is downgraded to HTTP/1.

Otherwise, HTTPS (no SSL-Proxying) and HTTP are not, since it's just a proxy server, no interception. Thus, it would keep HTTP/2 or other protocols without changing it.

As there are no reliable HTTP/2 servers on macOS (Swift/Objc) yet, so it's the reason why Proxyman doesn't support HTTP/2 at the first implementation. However, Swift-NIO from Apple is fully supported recently, so I'm planning to rewrite the Proxyman Core with NIO stack.

When 1.23.0 build is out this weekend, I will push HTTP/2 forward 👍

[wasder]

Looking forward for gRPC support. Could be a killer feature, as nothing yet so handy support it.👍

[bbuckland]

@NghiaTranUIT sounds like you've made a ton of progress towards this. Could you share where you're at with HTTP/2, specifically to enable gRPC?

[NghiaTranUIT]

@bbuckland We're migrating the codebase to HTTP/2 but it hasn't finished yet. I will push it forward soon after the Multiple Filters and Diffing is done 🙌

To workaround and see the gRPC data, you might consider using Atlantis from Proxyman (https://github.com/ProxymanApp/atlantis#2-my-app-use-grpc)

[inickt]

Would love to know if there are any updates on this, our backend microservices only talk over HTTP/2 and I would love to use Proxyman + the beta of the reverse proxy feature to improve my backend workflow.

That being said I switched over from Charles this week and it has been amazing for testing iOS things!

[NghiaTranUIT]

Thanks for the upvote @inickt. Since the Reverse Proxy is almost done (we're going to release it in this week), so our team is working on the HTTP/2 👍

We will send a Beta build here when it's available 😄

[inickt]

Hey, I was curious if there have been any updates on this? Still been loving the app so thanks for all of your work and the great updates!

[NghiaTranUIT]

Hey @inickt Thanks for your shoutout.

To be honest, I tried to support HTTP/2 months ago but it doesn't work well since it's a breaking change in our codebase 😿 . The problem is not just supporting HTTP/2, it also includes HTTP/1.1 as backward compatible.

At the moment, I'm focusing on the Folder Rule (where you can organize the tool rules in folders). When it's done, I will back to HTTP/2 👍

[NicolasCombe5555]

Hello @NghiaTranUIT 👍🏻; could we expect http2 support any time soon or not likely to happen soon? Thanks 🤝

[NghiaTranUIT]

Sorry @NicolasCombe5555 This ticket is implemented but it's delayed due to technical problems. Basically, in order to support HTTP/2, we have to redesign the ProxymanCore, which is primarily designed for HTTP/1.1. Hence, we decide to postpone this ticket until we found a better solution.

Can you elaborate on why you need HTTP/2? Does it important for your work? @NicolasCombe5555 🤔

[NicolasCombe5555]

@NghiaTranUIT Well part of my project has http/2 in our tech stack; so I cannot see that traffic when using Proxyman. But I understand if we are having issues with supporting it here; thanks for the quick response tho 👍🏻

[eseay]

I've read this thread and understand the complexity surrounding adding HTTP/2 support, but I just wanted to more or less throw a "+1" onto this issue as someone who would like to see it supported.

In the meantime, I am just using mitmproxy to view any traffic from my application where HTTP/2 is required.

@NghiaTranUIT to give you some context about the HTTP/2 need - we're starting to use more gRPC across my teams, and since that's HTTP/2, we're not able to proxy those requests using Proxyman. In fact, having Proxyman open when those requests go out essentially breaks them. (I work on the iOS apps, and am debugging that app's traffic to our servers)

Thanks for your work on such a great tool!

[NghiaTranUIT]

Even Proxyman can support HTTP/2 and Protobuf, but there is no guarantee that it can capture and properly decrypt the gRPC package because we don't support it yet.

To workaround, @eseay you can use Atlantis, which allows you to capture gPRC traffic 👍 https://github.com/ProxymanApp/atlantis#2-my-app-use-grpc

[eseay]

@NghiaTranUIT well I'll clarify that the back-end is serving a gRPC hybrid called Connect, so we're actually just using POST requests with normal request bodies, but those requests require HTTP/2.

In either case - I am curious what I may be doing wrong in Proxyman to not be seeing HTTP/2 traffic.

When I run mitmproxy, I see that all of my requests (both gRPC-Connect and classic REST) are being conducted via HTTP/2; I see the same thing when proxying through Charles.

In Proxyman, when I go to inspect my requests, they are all showing as using HTTP/1.1. Is there something I may have configured incorrectly in Proxyman that's preventing the HTTP/2 transmissions?

[j-j-m]

I see the same thing. Is there any way to prevent Proxyman from breaking my gRPC services in app? I have tried Atlantis with no luck.

[NghiaTranUIT]

@j-j-m can you use this Atlantis code to capture gRPC traffic? https://github.com/ProxymanApp/atlantis#2-my-app-use-grpc

and open Tools Menu -> SSL Proxying List -> Remove the gRPC domain in the Include List. It will fix the SSL Error, and allows the Atlantis to work properly 👍

[NghiaTranUIT]

Good news 🎉 🎉

I've supported the HTTP/2 protocol (Beta) for the Proxyman app 🎉

Beta build: https://proxyman.s3.us-east-2.amazonaws.com/beta/Proxyman_4.1.0_Support_HTTP_2.dmg

What's new

• Support HTTP/2 and HTTP/1.1 (can negotiate during the SSL Handshake)
• Tested and HTTP/2 work with the following debugging tools:

1. Breakpoint
2. Map Local
3. Scripting
4. Map Remote
5. Compose & Edit
6. Repeat
7. Block / Allow List
8. External Proxy
9. A lot of Unit Tests to cover all cases

Notes

• In the HTTP Message (Breakpoint or Map Local), you can use HTTP/1.1 or HTTP/2. It doesn't matter since Proxyman will use the correct HTTP Protocol behind the scenes => It means, you can use your current Map Local Rule/ Breakpoint Raw Message without any problems ✅

Screenshots

How to enable HTTP/2

• Open Preference -> Check "Use HTTP/2" -> Restart the app to take effect.

Known issues:

• WSS doesn't work over HTTP/2 (WS still works as usual)
• ???

If you guys find any bugs, please let me know, I'm all ears to fix it 👍

---

Just a friendly tag: @j-j-m @eseay @NicolasCombe5555 @inickt @bbuckland @Saklad5, @Ashraf-Ali-aa, @julasamer, @wasder, @ildar-gilfanov, @NSMyself, @heyzooi, @inickt, @nxtSwitch, @basecde, water-a

[derekmurawsky]

Tested it out, and it looks like it works to me! I was able to request an image where, previously, the connection was downgraded to 1.1. Thank you!

[brr53]

Thank you. There is a bug though. HTTP2 does not have the host header. So this RAW is incorrect. HTTP2 heading should look something like this.

:method: GET :authority: www.google.com :scheme: https :path: /

@NghiaTranUIT

[NghiaTranUIT]

Thanks @brr53. It's just a UI bug. Behind the sense, Proxyman automatically translates the HTTP/1.1 Raw format into the HTTP/2 format, so it's still working.

I'm going to fix the Raw Tab for HTTP/2 now 👍

[NghiaTranUIT]

Update HTTP/2 v2 🎉

• Beta: https://proxyman.s3.us-east-2.amazonaws.com/beta/Proxyman_4.2.0_HTTP_2_v3.dmg

Changelog

• Refactor how the Proxyman handles the HTTP/2
• Fix the bug when Proxyman could not load the websites (via HTTP/2) on Google Chrome
• Updated from Proxyman 4.2.0

Known issues:

• WS doesn't work with HTTP/2

[NghiaTranUIT]

Bug 2

A user reports that TLS/Extension for the HTTP/2 doesn't meet the standard from Google Chrome / Charles Proxy. We should fix it.

on https://tls.peet.ws/api/all
this is proxyman: 3:100,6:16384|00|0|p,m,s,a
this is chrome: 1:65536,2:0,3:1000,4:6291456,6:262144|15663105|0|m,a,s,p 
this is charles: "1:65536,2:0,3:1000,4:6291456,6:262144|15663105|0|m,a,s,p",

[brr53]

@NghiaTranUIT What you have listed aren't related to TLS by the way. I think these are HTTP2 components.

[NghiaTranUIT]

Update HTTP/2 v4 🎉

• Beta: https://proxyman.s3.us-east-2.amazonaws.com/beta/Proxyman_4.2.1_Update_HTTP_2_v4.dmg

Changelog

• Update the hotfix from Proxyman v4.2.1
• Fix the Raw Tab for HTTP/2

Known issues:

• WS doesn't work with HTTP/2

[NghiaTranUIT]

Update HTTP/2 v5

• Download: https://download.proxyman.io/beta/Proxyman_4.3.1_HTTP_2_v5.dmg

Changelogs

• Update from the latest Proxyman v4.3.1
• HTTP 2 Connection is now reused -> Faster and reliable.
• Fix the request/response timing issue

Known issues:

WS doesn't work with HTTP/2