High throughput via Proxy usage is strongly limited duo usage of default-dispatcher within HttpsProxyTransport

[TiDiCom]

https://github.com/akka/akka-http/blob/eaf87a3d80b7ac037f50e19e6220ca3e0531c41b/akka-http-core/src/main/scala/akka/http/scaladsl/ClientTransport.scala#L116

The HttpsProxyTransport seems to always use the system default-dispatcher, which I think is not an ideal solution. It would be good if the HttpsProxyTransport would (or optionally be possible to) use the same dispatcher as configured within the actor itself.

In our case, we would like to be able to handle more then 3.000 API requests per second (request-response) through proxy usage:

    val httpsProxyTransport = ClientTransport.httpsProxy(InetSocketAddress.createUnresolved(privateProxyIP, 8888))
    val settings: ConnectionPoolSettings = ConnectionPoolSettings(context.system)
      .withTransport(httpsProxyTransport)
      .withMaxRetries(0)
      .withMaxConnectionBackoff(2.seconds)

    val f: Future[(StatusCode, String)] = Http(context.system)
        .singleRequest(request, settings = settings) 
        .flatMap(_.toStrict(2.seconds)) 
        .flatMap { resp =>
          Unmarshal(resp.entity).to[String].map((resp.status, _))
        }

[jrudolph]

Hi @TiDiCom, thanks for the report. Do you have any indications why this would be the case? How did you find out that this would be a bottleneck? It seems quite unlikely.

[TiDiCom]

We figured out that there seems to be a bottleneck somewhere within the Proxy implementation, as our application without proxy works stable and scalable on constant performance, while just adding the proxy settings seems to limit us.

Like you said, maybe it's not necessarily the dispatcher but something else in the akka proxy implementation which seems to create a bottleneck for higher throughputs.

Important to mention, that for testing we used multiple proxies via private ips and they are all own hosted on different EC2 instances all within the same VPC. From a variety of tests there seems there to be no problem with the proxies, they could handle way more throughput.

Towards the API endpoint we addressed, it has an avg. send-response time of around 50MS.

application.conf:

akka {
    http {
      client {
        connecting-timeout = 5 s
        idle-timeout = 5 s

        parsing {
          illegal-header-warnings = off
        }
      }
      host-connection-pool {
        max-connections = 650
        max-retries = 0
        max-open-requests = 1024
        idle-timeout = 5 s
      }
    }
}

[jrudolph]

I would suggest checking response times of the proxy on the application server with tcpdump. This way you can rule out that it is the proxy itself which is slow.