badGateway errors when using AsyncHTTPClient version > 1.6.4 with Vapor 4 project

[pbodsk]

AsyncHTTPClient commit hash:

ec2e080d7011a81bd67f10bf41efe6104d7799d6

Swift Version:

swift-driver version: 1.26.21 Apple Swift version 5.5.2 (swiftlang-1300.0.47.5 clang-1300.0.29.30) Target: x86_64-apple-macosx12.0

Context

We have a Vapor 4 project that has been running in production close to a year now. This project communicates with 3rd party APIs using a Client which has worked fine so far. Vapors Client (EventLoopHTTPClient to be specific) wraps a HTTPClient from AsyncHTTPClient.

Recently I was tasked with implementing some new features and as part of this I updated all dependencies so I could take advantage of async/await.

Issue

After updating Vapor and other dependencies I noticed that the existing calls to 3rd party APIs stopped working. Calling the endpoints would return a .badGateway error instead. I could write this off as errors in the 3rd party API if it was one of them, but when all of them started to fail...that indicated that the error was on my side 😅

Workaround

I managed to get the endpoints working again by manually adding an .exact dependency to async-http-client in my Package.swift file like so:

.package(url: "https://github.com/swift-server/async-http-client.git", .exact("1.6.4"))

If I upgrade to anything above 1.6.4, I start seeing the .badGateway errors on my existing Client calls (which are using EventLoopFutures btw.)

I can see that AsyncHTTPClient 1.7.0 introduces automatic HTTP/2 support by default and I wonder if that has broken something in my setup.

Additional Workaround

As mentioned in this issue, setting the httpVersion manually also does the trick and means that you do not have to stay on 1.6.4.

So, somewhere in your config before you start using the client add this: app.http.client.configuration.httpVersion = .http1Only

Additional Observations

• the jobs I added are using async/await and that seems to work (they are speaking to a different 3rd party though so that may be the reason why)
• one of the existing jobs does an initial login to obtain an access token which is then used for the actual call immediately after. The login call works, but the following call to get relevant content fails
• I can call all of the "failing" endpoints using cURL from the same machine and get a successful response back

I know that this is probably a case of me doing something wrong somehow but I hope you can assist me in pinpointing where the issue lies 😄

Thank you for your time.

[Lukasa]

Thanks for filing this issue! I think to begin with we'd like to try to see packet captures using tools like Wireshark or tcpdump to compare what's happening in both cases.

[pbodsk]

Hey @Lukasa

First of all, sorry for the late reply! I managed to get something working (mentioned in the "Additional Workaround" section) so the initial fire was put out 😅

You asked for Wireshark files. I've tried...hope it is useful. Attached here you'll find two files (delete the .txt extension to get them as pcapng files 🤷 ):

• WorkingFiltered.pcapng: Contains a trace/dump from a working version (that is; this line added app.http.client.configuration.httpVersion = .http1Only, AsyncHttpVersion = 1.9.0)
• NotWorkingFiltered.pcapng: Contains a trace/dump from a not working version (that is; AsyncHttpVersion = 1.9.0)

in both cases I ran Wireshark to capture data and then filtered to only get packets send to the 3rd party API. I furthermore took the liberty of "anonymizing" both my own and the 3rd party API IP addresses (just to say you won't find the 3rd party API at 10.10.10.10 😄 )

I'm a noob when it comes to Wireshark files so I hope this is what you asked for...if not...just let me know and I'll happily make another attempt.

WorkingFiltered.pcapng.txt NotWorkingFiltered.pcapng.txt

[Lukasa]

Hmm, in both cases we're using TLS to connect so the Wireshark trace is not immediately useful to us. @fabianfett do we have a good way to grab the plaintext by inserting a pcap handler?

[pbodsk]

I initially tried using Charles for this but couldn't detect traffic. I suspect it is because the communication done using AsyncHTTPClient is on a lower level in the network stack...maybe...(I'm out of my league here 😉 )

[Lukasa]

I think in the case of Charles you need to manually configure the HTTP proxy. If you told async-http-client to use the Charles proxy directly (by setting the config appropriately) then it should work fine.

[pbodsk]

Hey @Lukasa

Good news and bad news.

That did indeed the trick! Adding these line to my config file

let proxy = AsyncHTTPClient.HTTPClient.Configuration.Proxy.server(host: "127.0.0.1", port: 8888)
app.http.client.configuration.proxy = proxy

Allowed me to see traffic from my Vapor app in Charles 🎉

As part of the setup I also installed a root certificate from Charles on my machine (following this description), allowing me to see SSL traffic.

However (uh oh!)

If I did that...and then removed this line:

app.http.client.configuration.httpVersion = .http1Only

In theory making my app go back to not working....it still worked!

So...going through a proxy and adding a root certificate so I could see the traffic also means that my app works as expected, even thought it shouldn't.

So to recap

Without Proxy

• Running normally with AsyncHTTPClient 1.9.0: I get a badGateway error
• Adding app.http.client.configuration.httpVersion = .http1Only with AsyncHTTPClient 1.9.0: successful communication with endpoints

With Proxy and SSL proxying

• Running normally with AsyncHTTPClient 1.9.0: successful communication with endpoints
• Adding app.http.client.configuration.httpVersion = .http1Only with AsyncHTTPClient 1.9.0: successful communication with endpoints

With Proxy and NO SSL proxying

• Running normally with AsyncHTTPClient 1.9.0: I get a badGateway error
• Adding app.http.client.configuration.httpVersion = .http1Only with AsyncHTTPClient 1.9.0: successful communication with endpoints

Observations

Both of the calls return 200 OK it seems in Charles...but I still see badGateway errors 🤔

I've added a couple of screenshots from Charles, don't know if they help you or not

Here's the traffic when running with app.http.client.configuration.httpVersion = .http1Only added

And here's how it looks when just running default (not working)

Hope it gives you a straw to grasp for 😄

[Lukasa]

The 200 there seems to be the Charles response, not the response from the server. Can we see an actual request running through that proxy?