zio-http icon indicating copy to clipboard operation
zio-http copied to clipboard

Client does not handle broken servers

Open erikvanoosten opened this issue 1 year ago • 4 comments

When a HTTP server accepts the connection, streams back a response header (or at least some bytes) but then stops before sending a body, zio-http client (3.0.0-RC2+47-60c31e2b-SNAPSHOT) doesn't do anything any more (dead-lock?).

This server behavior was observed in production after replacing the zio-kafka HTTP client with the JVM's HTTP client. The server we need to talk to has the described behavior roughly every 10 minutes. Since this is only correlation, not causation, it is possible that behavior of the zio-kafka client has a different cause.

To reproduce one can use an intentionally evil HTTP server. For example https://github.com/kiewic/heyhttp. Such a server is also great for unit testing.

Instead of deadlocking, I expect the client to timeout after some time and replace the connection in the pool.

erikvanoosten avatar Aug 13 '23 07:08 erikvanoosten

If you use ZClientAspect.timeout, do you get the behavior you are looking for? It's also possible we could add a ZClientAspect.retryAfter(duration) to address this issue.

jdegoes avatar Aug 17 '23 08:08 jdegoes

I was not aware of that option. We did try ZIO.timeout, but that did not trigger. In addition, all other uses of the http-client were blocked as well, perhaps because the connection pool was exhausted (even though we did configure an idle timeout).

erikvanoosten avatar Aug 17 '23 08:08 erikvanoosten

Not too sure, but I think this might be related to #2297

FVelasquezM avatar Aug 18 '23 01:08 FVelasquezM

Not too sure, but I think this might be related to #2297

Not in my case. The http server I have to deal with is the GCP Cloud Run load balancer. Given its erratic behavior I am pretty sure they do not use zio-http 😄

erikvanoosten avatar Aug 18 '23 06:08 erikvanoosten