Jean-Luc Deprez

Looks like we get to join the dance. (Akka HTTP 10.1.x). I don't have any more information to bring to the table other then @vilunov isn't the only one suffering...

Another thing I can add to is that I have code doing Akka HTTP to Akka HTTP calls, that don't seem to suffer while these calls have an ALB sitting...

Does that commit fix some slot release escapes somehow? (I've looked at the commit, but it doesn't seem to be apparent from there to me) Yes, high CPU load would...

Well the state is 32k slots in WaitingForResponse, with seemingly all SlotConnections to claim being connected while nothing is moving anymore (since last evening) and server side and ALB timeouts...

While studying on this code (v10.1.x; but still valid) https://github.com/akka/akka-http/blob/v10.2.7/akka-http-core/src/main/scala/akka/http/impl/engine/client/pool/NewHostConnectionPool.scala#L289 Isn't that a race condition? If lines 271 and 288 are reached at the same time two invocations of updateState()...

Further study: If a materialized `connectionFlow: Flow[HttpRequest, HttpResponse, Future[Http.OutgoingConnection]]` of `HostConnectionPoolStage` , once connected, never produces an error. The slot will get stuck in WaitingForResponse(. At that point, no timeout...

This fix seems to acknowledge such a possibility. https://github.com/akka/akka-http/blame/v10.1.14/akka-http-core/src/main/resources/reference.conf#L342 Any chance that a heavy CPU load could make the 100 ms fall short. And if so, what is the risk...

By now I have as much as a smoking gun. I'm able to reproduce by creating load on a local system and get some hanging slots. (was trying to validate...

By now I've figured that thing is sort of the end of the line. But more is going on before it ever gets there. But I'm working on making that...

Ok, I went through 7 hells to exert the right amount of stress to reproduce the problem and at the same time not loose the debug logging, but I think...