zio-http icon indicating copy to clipboard operation
zio-http copied to clipboard
verified publisher icon zio

ZClient cannot recover after a network failure or connection timeouts

Open mikail-khan opened this issue 1 year ago • 15 comments

Describe the bug When a network recovers after a temporary failure, a running ZClient program with a retry schedule policy specified fails to determine that the network connection has recovered and continues retrying with the following error message:

[2024-01-24T22:53:48,110]  [i.n.h.s.u.InsecureTrustManagerFactory] [DEBUG:] - Accepting a server certificate: CN=xxx.xxx.com
[2024-01-24T22:53:48,113]  [i.n.h.s.SslHandler] [DEBUG:] - [id: 0x6ce64463, L:/xxx.xxx.xxx.xxx - R:xxx.xxx.com] HANDSHAKEN: protocol:TLSv1.3 cipher suite:TLS_AES_128_GCM_SHA256
[2024-01-24T22:54:04,108]  [b.s.T.Live] [ERROR:] - : Could not get system status - null: xxx.xxx.com/xxx.xxx:443.
[2024-01-24T22:54:04,109]  [<.M.program] [ERROR:] - : null: xxx.xxx.com/xxx.xxx:443.
[2024-01-24T22:54:04,108]  [b.h.K.Live] [DEBUG:] - Fiber zio-fiber-250 did not handle an error
 io.netty.channel.AbstractChannel$AnnotatedNoRouteToHostException: null: xxx.xxx.com/xxx.xxx:443
Caused by: java.net.NoRouteToHostException
	at io.netty.channel.unix.Errors.newConnectException0(Errors.java:158) ~[netty-transport-native-unix-common-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:131) ~[netty-transport-native-unix-common-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.channel.unix.Socket.connect(Socket.java:351) ~[netty-transport-native-unix-common-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.channel.epoll.AbstractEpollChannel.doConnect0(AbstractEpollChannel.java:773) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.channel.epoll.EpollSocketChannel.doConnect0(EpollSocketChannel.java:144) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.channel.epoll.AbstractEpollChannel.doConnect(AbstractEpollChannel.java:758) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.connect(AbstractEpollChannel.java:600) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1342) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:653) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:632) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.handler.ssl.SslHandler.connect(SslHandler.java:737) ~[netty-handler-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:657) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:632) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.connect(CombinedChannelDuplexHandler.java:495) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.channel.ChannelOutboundHandlerAdapter.connect(ChannelOutboundHandlerAdapter.java:51) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.channel.CombinedChannelDuplexHandler.connect(CombinedChannelDuplexHandler.java:296) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:655) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:632) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:616) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.channel.DefaultChannelPipeline.connect(DefaultChannelPipeline.java:978) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.channel.AbstractChannel.connect(AbstractChannel.java:265) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.bootstrap.Bootstrap$3.run(Bootstrap.java:264) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:413) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
	at java.base/java.lang.Thread.run(Thread.java:840) ~[?:?]
[2024-01-24T22:54:04,128]  [i.n.h.s.u.InsecureTrustManagerFactory] [DEBUG:] - Accepting a server certificate: CN=xxx.xxx.com
[2024-01-24T22:54:04,130]  [i.n.h.s.SslHandler] [DEBUG:] - [id: 0x16d75050, L:/xxx.xxx.xxx.xxx - R:xxx.xxx.com/xxx.xxx:443] HANDSHAKEN: protocol:TLSv1.3 cipher suite:TLS_AES_128_GCM_SHA256

To Reproduce Steps to reproduce the behaviour:

  1. Create a simple ZClient program which makes repeat HTTP requests and has a schedule policy defined
  2. Run the program and observe that the HTTP requests are being made successfully
  3. Kill the network and observe in the logs the following exception: java.net.NoRouteToHostException
  4. Observe that the exponential back-off schedule policy kicks in and retries the request based on the schedule policy.
  5. Restore the network
  6. Observe that ZClient fails to determine the network is up and the request is continuously retried but fails.

Expected behaviour On network recovery, the request is retried and succeeds.

mikail-khan avatar Jan 25 '24 00:01 mikail-khan

@mikail-khan I believe this might have been resolved by https://github.com/zio/zio-http/pull/2610. Can you try with the latest snapshot (3.0.0-RC4+58-a70a964c-SNAPSHOT) and see if it's still an issue?

kyri-petrou avatar Jan 25 '24 01:01 kyri-petrou

3.0.0-RC4+58-a70a964c-SNAPSHOT

I created a small repo program below to demonstrate the problem and so you can debug this as well.

On a java.net.NoRouteToHostException the program fails to recover after the network recovers. With the snapshot version, I recovered the network at 2024-01-25T10:52.00, but the fiber does not detect it and continues retrying.

import zio._

import zio.http._
import zio.logging.backend.SLF4J
import java.util.concurrent.TimeUnit

object SimpleClient extends ZIOAppDefault {

  override val bootstrap: ZLayer[ZIOAppArgs, Any, Any] =
    Runtime.removeDefaultLoggers >>> SLF4J.slf4j

  val url = URL.decode("http://www.google.com").toOption.get

  val program = for {
    client <- ZIO.service[Client]
    res    <- client.url(url).get("/")
    data   <- res.body.asString
    _      <- ZIO.logInfo(s"Got data: ${data.take(30)}")
  } yield ()

  override val run = program
                      .tapError(_ => ZIO.logError(s"boom!"))
                      .repeat(Schedule.spaced(Duration(3, TimeUnit.SECONDS)))
                      .retry(Schedule.exponential(Duration(1, TimeUnit.SECONDS)))
                      .provide(Client.default, Scope.default)

}
3.0.0-RC4+58-a70a964c-SNAPSHOT log output

[2024-01-25T10:50:15,420] [i.n.u.Recycler] [DEBUG:] - -Dio.netty.recycler.chunkSize: 32 [2024-01-25T10:50:15,421] [i.n.u.Recycler] [DEBUG:] - -Dio.netty.recycler.blocking: false [2024-01-25T10:50:15,422] [i.n.u.Recycler] [DEBUG:] - -Dio.netty.recycler.batchFastThreadLocalOnly: true [2024-01-25T10:50:15,666] [<.SimpleClient] [INFO :] - Got data: <html itemscope="" itemtype="http:/ [2024-01-25T10:50:18,782] [<.SimpleClient] [INFO :] - Got data: <html itemscope="" itemtype="http:/ [2024-01-25T10:50:22,003] [<.SimpleClient] [INFO :] - Got data: <html itemscope="" itemtype="http:/ [2024-01-25T10:51:05,397] [zio-slf4j-logger] [DEBUG:] - ReadTimeoutException caught [2024-01-25T10:51:05,402] [<.SimpleClient] [ERROR:] - boom! [2024-01-25T10:51:05,447] [zio-slf4j-logger] [DEBUG:] - ReadTimeoutException caught [2024-01-25T10:51:05,462] [zio-slf4j-logger] [DEBUG:] - ReadTimeoutException caught [2024-01-25T10:51:05,474] [zio-slf4j-logger] [DEBUG:] - ReadTimeoutException caught [2024-01-25T10:51:05,483] [zio-slf4j-logger] [DEBUG:] - ReadTimeoutException caught [2024-01-25T10:51:05,489] [zio-slf4j-logger] [DEBUG:] - ReadTimeoutException caught [2024-01-25T10:51:05,507] [zio-slf4j-logger] [DEBUG:] - ReadTimeoutException caught [2024-01-25T10:51:05,619] [zio-slf4j-logger] [DEBUG:] - ReadTimeoutException caught [2024-01-25T10:51:08,783] [zio-slf4j-logger] [DEBUG:] - ReadTimeoutException caught [2024-01-25T10:51:08,786] [<.SimpleClient] [ERROR:] - boom! [2024-01-25T10:51:12,002] [zio-slf4j-logger] [DEBUG:] - ReadTimeoutException caught [2024-01-25T10:51:12,004] [<.SimpleClient] [ERROR:] - boom! [2024-01-25T10:51:16,025] [<.SimpleClient] [ERROR:] - boom! [2024-01-25T10:51:16,033] [<.SimpleClient] [DEBUG:] - Fiber zio-fiber-105 did not handle an error io.netty.channel.AbstractChannel$AnnotatedNoRouteToHostException: null: www.google.com/216.239.38.120:80 Caused by: java.net.NoRouteToHostException at io.netty.channel.unix.Errors.newConnectException0(Errors.java:158) ~[netty-transport-native-unix-common-4.1.101.Final.jar:4.1.101.Final] at io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:131) ~[netty-transport-native-unix-common-4.1.101.Final.jar:4.1.101.Final] at io.netty.channel.unix.Socket.connect(Socket.java:351) ~[netty-transport-native-unix-common-4.1.101.Final.jar:4.1.101.Final] at io.netty.channel.epoll.AbstractEpollChannel.doConnect0(AbstractEpollChannel.java:773) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final] at io.netty.channel.epoll.EpollSocketChannel.doConnect0(EpollSocketChannel.java:144) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final] at io.netty.channel.epoll.AbstractEpollChannel.doConnect(AbstractEpollChannel.java:758) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final] at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.connect(AbstractEpollChannel.java:600) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1342) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:653) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final] at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:632) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final] at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.connect(CombinedChannelDuplexHandler.java:495) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final] at io.netty.channel.ChannelOutboundHandlerAdapter.connect(ChannelOutboundHandlerAdapter.java:51) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final] at io.netty.channel.CombinedChannelDuplexHandler.connect(CombinedChannelDuplexHandler.java:296) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:655) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final] at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:632) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final] at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:616) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final] at io.netty.channel.DefaultChannelPipeline.connect(DefaultChannelPipeline.java:978) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final] at io.netty.channel.AbstractChannel.connect(AbstractChannel.java:265) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final] at io.netty.bootstrap.Bootstrap$3.run(Bootstrap.java:264) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final] at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) ~[netty-common-4.1.101.Final.jar:4.1.101.Final] at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) ~[netty-common-4.1.101.Final.jar:4.1.101.Final] at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) ~[netty-common-4.1.101.Final.jar:4.1.101.Final] at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:413) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.101.Final.jar:4.1.101.Final] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.101.Final.jar:4.1.101.Final] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.101.Final.jar:4.1.101.Final] at java.base/java.lang.Thread.run(Thread.java:840) ~[?:?] [2024-01-25T10:51:24,046] [<.SimpleClient] [ERROR:] - boom! [2024-01-25T10:51:24,045] [<.SimpleClient] [DEBUG:] - Fiber zio-fiber-107 did not handle an error io.netty.channel.AbstractChannel$AnnotatedNoRouteToHostException: null: www.google.com/216.239.38.120:80 Caused by: java.net.NoRouteToHostException at io.netty.channel.unix.Errors.newConnectException0(Errors.java:158) ~[netty-transport-native-unix-common-4.1.101.Final.jar:4.1.101.Final] at io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:131) ~[netty-transport-native-unix-common-4.1.101.Final.jar:4.1.101.Final] at io.netty.channel.unix.Socket.connect(Socket.java:351) ~[netty-transport-native-unix-common-4.1.101.Final.jar:4.1.101.Final] at io.netty.channel.epoll.AbstractEpollChannel.doConnect0(AbstractEpollChannel.java:773) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final] at io.netty.channel.epoll.EpollSocketChannel.doConnect0(EpollSocketChannel.java:144) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final] at io.netty.channel.epoll.AbstractEpollChannel.doConnect(AbstractEpollChannel.java:758) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final] at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.connect(AbstractEpollChannel.java:600) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1342) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:653) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final] at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:632) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final] at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.connect(CombinedChannelDuplexHandler.java:495) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final] at io.netty.channel.ChannelOutboundHandlerAdapter.connect(ChannelOutboundHandlerAdapter.java:51) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final] at io.netty.channel.CombinedChannelDuplexHandler.connect(CombinedChannelDuplexHandler.java:296) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:655) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final] at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:632) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final] at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:616) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final] at io.netty.channel.DefaultChannelPipeline.connect(DefaultChannelPipeline.java:978) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final] at io.netty.channel.AbstractChannel.connect(AbstractChannel.java:265) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final] at io.netty.bootstrap.Bootstrap$3.run(Bootstrap.java:264) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final] at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) ~[netty-common-4.1.101.Final.jar:4.1.101.Final] at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) ~[netty-common-4.1.101.Final.jar:4.1.101.Final] at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) ~[netty-common-4.1.101.Final.jar:4.1.101.Final] at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:413) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.101.Final.jar:4.1.101.Final] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.101.Final.jar:4.1.101.Final] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.101.Final.jar:4.1.101.Final] at java.base/java.lang.Thread.run(Thread.java:840) ~[?:?] [2024-01-25T10:51:40,064] [<.SimpleClient] [ERROR:] - boom! [2024-01-25T10:51:40,064] [<.SimpleClient] [DEBUG:] - Fiber zio-fiber-109 did not handle an error io.netty.channel.AbstractChannel$AnnotatedNoRouteToHostException: null: www.google.com/216.239.38.120:80 Caused by: java.net.NoRouteToHostException at io.netty.channel.unix.Errors.newConnectException0(Errors.java:158) ~[netty-transport-native-unix-common-4.1.101.Final.jar:4.1.101.Final] at io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:131) ~[netty-transport-native-unix-common-4.1.101.Final.jar:4.1.101.Final] at io.netty.channel.unix.Socket.connect(Socket.java:351) ~[netty-transport-native-unix-common-4.1.101.Final.jar:4.1.101.Final] at io.netty.channel.epoll.AbstractEpollChannel.doConnect0(AbstractEpollChannel.java:773) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final] at io.netty.channel.epoll.EpollSocketChannel.doConnect0(EpollSocketChannel.java:144) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final] at io.netty.channel.epoll.AbstractEpollChannel.doConnect(AbstractEpollChannel.java:758) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final] at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.connect(AbstractEpollChannel.java:600) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1342) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:653) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final] at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:632) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final] at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.connect(CombinedChannelDuplexHandler.java:495) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final] at io.netty.channel.ChannelOutboundHandlerAdapter.connect(ChannelOutboundHandlerAdapter.java:51) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final] at io.netty.channel.CombinedChannelDuplexHandler.connect(CombinedChannelDuplexHandler.java:296) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:655) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final] at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:632) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final] at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:616) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final] at io.netty.channel.DefaultChannelPipeline.connect(DefaultChannelPipeline.java:978) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final] at io.netty.channel.AbstractChannel.connect(AbstractChannel.java:265) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final] at io.netty.bootstrap.Bootstrap$3.run(Bootstrap.java:264) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final] at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) ~[netty-common-4.1.101.Final.jar:4.1.101.Final] at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) ~[netty-common-4.1.101.Final.jar:4.1.101.Final] at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) ~[netty-common-4.1.101.Final.jar:4.1.101.Final] at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:413) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.101.Final.jar:4.1.101.Final] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.101.Final.jar:4.1.101.Final] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.101.Final.jar:4.1.101.Final] at java.base/java.lang.Thread.run(Thread.java:840) ~[?:?] [2024-01-25T10:52:12,083] [<.SimpleClient] [DEBUG:] - Fiber zio-fiber-111 did not handle an error io.netty.channel.AbstractChannel$AnnotatedNoRouteToHostException: null: www.google.com/216.239.38.120:80 Caused by: java.net.NoRouteToHostException at io.netty.channel.unix.Errors.newConnectException0(Errors.java:158) ~[netty-transport-native-unix-common-4.1.101.Final.jar:4.1.101.Final] at io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:131) ~[netty-transport-native-unix-common-4.1.101.Final.jar:4.1.101.Final] at io.netty.channel.unix.Socket.connect(Socket.java:351) ~[netty-transport-native-unix-common-4.1.101.Final.jar:4.1.101.Final] at io.netty.channel.epoll.AbstractEpollChannel.doConnect0(AbstractEpollChannel.java:773) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final] at io.netty.channel.epoll.EpollSocketChannel.doConnect0(EpollSocketChannel.java:144) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final] at io.netty.channel.epoll.AbstractEpollChannel.doConnect(AbstractEpollChannel.java:758) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final] at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.connect(AbstractEpollChannel.java:600) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1342) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:653) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final] at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:632) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final] at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.connect(CombinedChannelDuplexHandler.java:495) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final] at io.netty.channel.ChannelOutboundHandlerAdapter.connect(ChannelOutboundHandlerAdapter.java:51) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final] at io.netty.channel.CombinedChannelDuplexHandler.connect(CombinedChannelDuplexHandler.java:296) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:655) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final] at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:632) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final] at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:616) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final] at io.netty.channel.DefaultChannelPipeline.connect(DefaultChannelPipeline.java:978) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final] at io.netty.channel.AbstractChannel.connect(AbstractChannel.java:265) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final] at io.netty.bootstrap.Bootstrap$3.run(Bootstrap.java:264) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final] at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) ~[netty-common-4.1.101.Final.jar:4.1.101.Final] at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) ~[netty-common-4.1.101.Final.jar:4.1.101.Final] at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) ~[netty-common-4.1.101.Final.jar:4.1.101.Final] at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:413) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.101.Final.jar:4.1.101.Final] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.101.Final.jar:4.1.101.Final] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.101.Final.jar:4.1.101.Final] at java.base/java.lang.Thread.run(Thread.java:840) ~[?:?] [2024-01-25T10:52:12,085] [<.SimpleClient] [ERROR:] - boom! [2024-01-25T10:52:14,093] [zio-slf4j-logger] [DEBUG:] - ReadTimeoutException caught [2024-01-25T10:52:30,113] [zio-slf4j-logger] [DEBUG:] - ReadTimeoutException caught [2024-01-25T10:53:02,129] [zio-slf4j-logger] [DEBUG:] - ReadTimeoutException caught

mikail-khan avatar Jan 25 '24 10:01 mikail-khan

same issue for connection timeouts, program fails to recover after network recovery

logs [2024-01-25T21:28:31,817] [b.h.K.Live] [DEBUG:] - Fiber zio-fiber-69763 did not handle an error io.netty.channel.ConnectTimeoutException: connection timed out: xxx.xxx.xxx:443 at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe$2.run(AbstractEpollChannel.java:613) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final] at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) ~[netty-common-4.1.101.Final.jar:4.1.101.Final] at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:153) ~[netty-common-4.1.101.Final.jar:4.1.101.Final] at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) ~[netty-common-4.1.101.Final.jar:4.1.101.Final] at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) ~[netty-common-4.1.101.Final.jar:4.1.101.Final] at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) ~[netty-common-4.1.101.Final.jar:4.1.101.Final] at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:416) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.101.Final.jar:4.1.101.Final] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.101.Final.jar:4.1.101.Final] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.101.Final.jar:4.1.101.Final] at java.base/java.lang.Thread.run(Thread.java:840) ~[?:?]

Cannot build a resilient service if retries don't succeed when they should.

mikail-khan avatar Jan 25 '24 21:01 mikail-khan

@mikail-khan I'm away for the weekend so I don't have access to my laptop so I can't test it, but could you try running it via ZIO.scoped(program), with the retries / error handling applied outside of it? I think the scope might need to be closed prior to retrying otherwise the client won't release the invalid connection.

Note, that means you should not need Scope.default as a layer to your program

kyri-petrou avatar Jan 25 '24 21:01 kyri-petrou

/bounty $150

jdegoes avatar Feb 08 '24 09:02 jdegoes

💎 $150 bounty • ZIO

Steps to solve:

  1. Start working: Comment /attempt #2653 with your implementation plan
  2. Submit work: Create a pull request including /claim #2653 in the PR body to claim the bounty
  3. Receive payment: 100% of the bounty is received 2-5 days post-reward. Make sure you are eligible for payouts

Thank you for contributing to zio/zio-http!

Add a bounty • Share on socials

Attempt Started (GMT+0) Solution
🔴 @TheVixhal Feb 10, 2024, 10:24:00 AM WIP
🔴 @asr2003 Jun 22, 2024, 6:45:18 PM WIP

algora-pbc[bot] avatar Feb 08 '24 09:02 algora-pbc[bot]

/attempt #2653

Options

TheVixhal avatar Feb 10 '24 10:02 TheVixhal

@mikail-khan I can't seem to be able to reproduce the issue you're facing. With the current main, and the code below, I get the following console output after I kill the network and enable it again:

Code:

object SimpleClient extends ZIOAppDefault {

  val url = URL.decode("http://www.google.com").toOption.get

  val program = for {
    client <- ZIO.service[Client]
    res    <- client.url(url).get("/")
    data   <- res.body.asString
    _      <- ZIO.debug(s"Got data: ${data.take(30)}")
  } yield ()

  override val run =
    ZIO
      .scoped(program)
      .tapError(_ => ZIO.debug(s"boom!"))
      .repeat(Schedule.spaced(Duration(3, TimeUnit.SECONDS)))
      .retry(Schedule.fixed(Duration(1, TimeUnit.SECONDS)))
      .provide(Client.default)
}

Output:

Got data: <!doctype html><html itemscope
Got data: <!doctype html><html itemscope
Got data: <!doctype html><html itemscope
boom!
boom!
boom!
boom!
boom!
boom!
boom!
boom!
boom!
boom!
boom!
boom!
boom!
boom!
boom!
boom!
boom!
boom!
boom!
boom!
boom!
boom!
Got data: <!doctype html><html itemscope
Got data: <!doctype html><html itemscope
Got data: <!doctype html><html itemscope
Got data: <!doctype html><html itemscope

kyri-petrou avatar Feb 13 '24 11:02 kyri-petrou

@mikail-khan I can't seem to be able to reproduce the issue you're facing.

See below, I've reproduced it again using the program I posted earlier. At 21:14 the network was recovered but the program does not detect it.

"dev.zio" %% "zio-http" % "3.0.0-RC4+58-a70a964c-SNAPSHOT",

[2024-02-13T21:12:14,559]  [i.n.u.i.l.InternalLoggerFactory] [DEBUG:] - Using SLF4J as the default logging framework
[2024-02-13T21:12:14,602]  [i.n.u.i.PlatformDependent0] [DEBUG:] - -Dio.netty.noUnsafe: false
[2024-02-13T21:12:14,604]  [i.n.u.i.PlatformDependent0] [DEBUG:] - Java version: 17
[2024-02-13T21:12:14,606]  [i.n.u.i.PlatformDependent0] [DEBUG:] - sun.misc.Unsafe.theUnsafe: available
[2024-02-13T21:12:14,607]  [i.n.u.i.PlatformDependent0] [DEBUG:] - sun.misc.Unsafe.copyMemory: available
[2024-02-13T21:12:14,608]  [i.n.u.i.PlatformDependent0] [DEBUG:] - sun.misc.Unsafe.storeFence: available
[2024-02-13T21:12:14,609]  [i.n.u.i.PlatformDependent0] [DEBUG:] - java.nio.Buffer.address: available
[2024-02-13T21:12:14,610]  [i.n.u.i.PlatformDependent0] [DEBUG:] - direct buffer constructor: unavailable: Reflective setAccessible(true) disabled
[2024-02-13T21:12:14,611]  [i.n.u.i.PlatformDependent0] [DEBUG:] - java.nio.Bits.unaligned: available, true
[2024-02-13T21:12:14,613]  [i.n.u.i.PlatformDependent0] [DEBUG:] - jdk.internal.misc.Unsafe.allocateUninitializedArray(int): unavailable: class io.netty.util.internal.PlatformDependent0$7 cannot access class jdk.internal.misc.Unsafe (in module java.base) because module java.base does not export jdk.internal.misc to unnamed module @768b0f44
[2024-02-13T21:12:14,615]  [i.n.u.i.PlatformDependent0] [DEBUG:] - java.nio.DirectByteBuffer.<init>(long, {int,long}): unavailable
[2024-02-13T21:12:14,615]  [i.n.u.i.PlatformDependent] [DEBUG:] - sun.misc.Unsafe: available
[2024-02-13T21:12:14,616]  [i.n.u.i.PlatformDependent] [DEBUG:] - -Dio.netty.tmpdir: /tmp (java.io.tmpdir)
[2024-02-13T21:12:14,616]  [i.n.u.i.PlatformDependent] [DEBUG:] - -Dio.netty.bitMode: 64 (sun.arch.data.model)
[2024-02-13T21:12:14,621]  [i.n.u.i.PlatformDependent] [DEBUG:] - -Dio.netty.maxDirectMemory: -1 bytes
[2024-02-13T21:12:14,623]  [i.n.u.i.PlatformDependent] [DEBUG:] - -Dio.netty.uninitializedArrayAllocationThreshold: -1
[2024-02-13T21:12:14,630]  [i.n.u.i.CleanerJava9] [DEBUG:] - java.nio.ByteBuffer.cleaner(): available
[2024-02-13T21:12:14,632]  [i.n.u.i.PlatformDependent] [DEBUG:] - -Dio.netty.noPreferDirect: false
[2024-02-13T21:12:14,647]  [i.n.u.i.NativeLibraryLoader] [DEBUG:] - -Dio.netty.native.workdir: /tmp (io.netty.tmpdir)
[2024-02-13T21:12:14,648]  [i.n.u.i.NativeLibraryLoader] [DEBUG:] - -Dio.netty.native.deleteLibAfterLoading: true
[2024-02-13T21:12:14,649]  [i.n.u.i.NativeLibraryLoader] [DEBUG:] - -Dio.netty.native.tryPatchShadedId: true
[2024-02-13T21:12:14,650]  [i.n.u.i.NativeLibraryLoader] [DEBUG:] - -Dio.netty.native.detectNativeLibraryDuplicates: true
[2024-02-13T21:12:14,665]  [i.n.u.i.NativeLibraryLoader] [DEBUG:] - Successfully loaded the library /tmp/libnetty_transport_native_epoll_x86_643484322894985639787.so
[2024-02-13T21:12:14,673]  [i.n.u.NetUtil] [DEBUG:] - -Djava.net.preferIPv4Stack: false
[2024-02-13T21:12:14,675]  [i.n.u.NetUtil] [DEBUG:] - -Djava.net.preferIPv6Addresses: false
[2024-02-13T21:12:14,682]  [i.n.u.NetUtilInitializations] [DEBUG:] - Loopback interface: lo (lo, 0:0:0:0:0:0:0:1%lo)
[2024-02-13T21:12:14,684]  [i.n.u.NetUtil] [DEBUG:] - /proc/sys/net/core/somaxconn: 4096
[2024-02-13T21:12:14,703]  [i.n.c.MultithreadEventLoopGroup] [DEBUG:] - -Dio.netty.eventLoopThreads: 8
[2024-02-13T21:12:14,726]  [i.n.u.c.GlobalEventExecutor] [DEBUG:] - -Dio.netty.globalEventExecutor.quietPeriodSeconds: 1
[2024-02-13T21:12:14,743]  [i.n.u.i.InternalThreadLocalMap] [DEBUG:] - -Dio.netty.threadLocalMap.stringBuilder.initialSize: 1024
[2024-02-13T21:12:14,745]  [i.n.u.i.InternalThreadLocalMap] [DEBUG:] - -Dio.netty.threadLocalMap.stringBuilder.maxSize: 4096
[2024-02-13T21:12:14,773]  [i.n.u.i.PlatformDependent] [DEBUG:] - org.jctools-core.MpscChunkedArrayQueue: available
[2024-02-13T21:12:15,019]  [i.n.c.DefaultChannelId] [DEBUG:] - -Dio.netty.processId: 134982 (auto-detected)
[2024-02-13T21:12:15,026]  [i.n.c.DefaultChannelId] [DEBUG:] - -Dio.netty.machineId: 5c:e0:c5:ff:fe:51:f5:ef (auto-detected)
[2024-02-13T21:12:15,048]  [i.n.u.ResourceLeakDetector] [DEBUG:] - -Dio.netty.leakDetection.level: simple
[2024-02-13T21:12:15,050]  [i.n.u.ResourceLeakDetector] [DEBUG:] - -Dio.netty.leakDetection.targetRecords: 4
[2024-02-13T21:12:15,098]  [i.n.b.PooledByteBufAllocator] [DEBUG:] - -Dio.netty.allocator.numHeapArenas: 8
[2024-02-13T21:12:15,098]  [i.n.b.PooledByteBufAllocator] [DEBUG:] - -Dio.netty.allocator.numDirectArenas: 8
[2024-02-13T21:12:15,099]  [i.n.b.PooledByteBufAllocator] [DEBUG:] - -Dio.netty.allocator.pageSize: 8192
[2024-02-13T21:12:15,099]  [i.n.b.PooledByteBufAllocator] [DEBUG:] - -Dio.netty.allocator.maxOrder: 9
[2024-02-13T21:12:15,100]  [i.n.b.PooledByteBufAllocator] [DEBUG:] - -Dio.netty.allocator.chunkSize: 4194304
[2024-02-13T21:12:15,100]  [i.n.b.PooledByteBufAllocator] [DEBUG:] - -Dio.netty.allocator.smallCacheSize: 256
[2024-02-13T21:12:15,101]  [i.n.b.PooledByteBufAllocator] [DEBUG:] - -Dio.netty.allocator.normalCacheSize: 64
[2024-02-13T21:12:15,102]  [i.n.b.PooledByteBufAllocator] [DEBUG:] - -Dio.netty.allocator.maxCachedBufferCapacity: 32768
[2024-02-13T21:12:15,103]  [i.n.b.PooledByteBufAllocator] [DEBUG:] - -Dio.netty.allocator.cacheTrimInterval: 8192
[2024-02-13T21:12:15,104]  [i.n.b.PooledByteBufAllocator] [DEBUG:] - -Dio.netty.allocator.cacheTrimIntervalMillis: 0
[2024-02-13T21:12:15,107]  [i.n.b.PooledByteBufAllocator] [DEBUG:] - -Dio.netty.allocator.useCacheForAllThreads: false
[2024-02-13T21:12:15,108]  [i.n.b.PooledByteBufAllocator] [DEBUG:] - -Dio.netty.allocator.maxCachedByteBuffersPerChunk: 1023
[2024-02-13T21:12:15,133]  [i.n.b.ByteBufUtil] [DEBUG:] - -Dio.netty.allocator.type: pooled
[2024-02-13T21:12:15,134]  [i.n.b.ByteBufUtil] [DEBUG:] - -Dio.netty.threadLocalDirectBufferSize: 0
[2024-02-13T21:12:15,134]  [i.n.b.ByteBufUtil] [DEBUG:] - -Dio.netty.maxThreadLocalCharBufferSize: 16384
[2024-02-13T21:12:15,165]  [i.n.b.ChannelInitializerExtensions] [DEBUG:] - -Dio.netty.bootstrap.extensions: null
[2024-02-13T21:12:15,224]  [i.n.b.AbstractByteBuf] [DEBUG:] - -Dio.netty.buffer.checkAccessible: true
[2024-02-13T21:12:15,226]  [i.n.b.AbstractByteBuf] [DEBUG:] - -Dio.netty.buffer.checkBounds: true
[2024-02-13T21:12:15,228]  [i.n.u.ResourceLeakDetectorFactory] [DEBUG:] - Loaded default ResourceLeakDetector: io.netty.util.ResourceLeakDetector@6c06e93
[2024-02-13T21:12:15,350]  [i.n.u.Recycler] [DEBUG:] - -Dio.netty.recycler.maxCapacityPerThread: 4096
[2024-02-13T21:12:15,351]  [i.n.u.Recycler] [DEBUG:] - -Dio.netty.recycler.ratio: 8
[2024-02-13T21:12:15,351]  [i.n.u.Recycler] [DEBUG:] - -Dio.netty.recycler.chunkSize: 32
[2024-02-13T21:12:15,351]  [i.n.u.Recycler] [DEBUG:] - -Dio.netty.recycler.blocking: false
[2024-02-13T21:12:15,352]  [i.n.u.Recycler] [DEBUG:] - -Dio.netty.recycler.batchFastThreadLocalOnly: true
[2024-02-13T21:12:15,602]  [<.SimpleClient] [INFO :] - Got data: <!doctype html><html itemscope="" itemtype="http:/
[2024-02-13T21:12:18,700]  [<.SimpleClient] [INFO :] - Got data: <!doctype html><html itemscope="" itemtype="http:/
[2024-02-13T21:12:22,165]  [<.SimpleClient] [INFO :] - Got data: <!doctype html><html itemscope="" itemtype="http:/
[2024-02-13T21:13:05,348]  [zio-slf4j-logger] [DEBUG:] - ReadTimeoutException caught
[2024-02-13T21:13:05,361]  [<.SimpleClient] [ERROR:] - boom!
[2024-02-13T21:13:05,374]  [zio-slf4j-logger] [DEBUG:] - ReadTimeoutException caught
[2024-02-13T21:13:05,389]  [zio-slf4j-logger] [DEBUG:] - ReadTimeoutException caught
[2024-02-13T21:13:05,394]  [zio-slf4j-logger] [DEBUG:] - ReadTimeoutException caught
[2024-02-13T21:13:05,406]  [zio-slf4j-logger] [DEBUG:] - ReadTimeoutException caught
[2024-02-13T21:13:05,409]  [zio-slf4j-logger] [DEBUG:] - ReadTimeoutException caught
[2024-02-13T21:13:05,425]  [zio-slf4j-logger] [DEBUG:] - ReadTimeoutException caught
[2024-02-13T21:13:05,570]  [zio-slf4j-logger] [DEBUG:] - ReadTimeoutException caught
[2024-02-13T21:13:08,698]  [zio-slf4j-logger] [DEBUG:] - ReadTimeoutException caught
[2024-02-13T21:13:08,700]  [<.SimpleClient] [ERROR:] - boom!
[2024-02-13T21:13:12,166]  [zio-slf4j-logger] [DEBUG:] - ReadTimeoutException caught
[2024-02-13T21:13:12,171]  [<.SimpleClient] [ERROR:] - boom!
[2024-02-13T21:13:16,184]  [<.SimpleClient] [ERROR:] - boom!
[2024-02-13T21:13:16,186]  [<.SimpleClient] [DEBUG:] - Fiber zio-fiber-108 did not handle an error
 io.netty.channel.AbstractChannel$AnnotatedNoRouteToHostException: null: www.google.com/216.239.38.120:80
Caused by: java.net.NoRouteToHostException
        at io.netty.channel.unix.Errors.newConnectException0(Errors.java:158) ~[netty-transport-native-unix-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:131) ~[netty-transport-native-unix-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.unix.Socket.connect(Socket.java:351) ~[netty-transport-native-unix-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.epoll.AbstractEpollChannel.doConnect0(AbstractEpollChannel.java:773) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.epoll.EpollSocketChannel.doConnect0(EpollSocketChannel.java:144) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.epoll.AbstractEpollChannel.doConnect(AbstractEpollChannel.java:758) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.connect(AbstractEpollChannel.java:600) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1342) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:653) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:632) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.connect(CombinedChannelDuplexHandler.java:495) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.ChannelOutboundHandlerAdapter.connect(ChannelOutboundHandlerAdapter.java:51) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.CombinedChannelDuplexHandler.connect(CombinedChannelDuplexHandler.java:296) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:655) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:632) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:616) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.DefaultChannelPipeline.connect(DefaultChannelPipeline.java:978) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.AbstractChannel.connect(AbstractChannel.java:265) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.bootstrap.Bootstrap$3.run(Bootstrap.java:264) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:413) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
        at java.base/java.lang.Thread.run(Thread.java:840) ~[?:?]
[2024-02-13T21:13:24,191]  [<.SimpleClient] [ERROR:] - boom!
[2024-02-13T21:13:24,191]  [<.SimpleClient] [DEBUG:] - Fiber zio-fiber-110 did not handle an error
 io.netty.channel.AbstractChannel$AnnotatedNoRouteToHostException: null: www.google.com/216.239.38.120:80
Caused by: java.net.NoRouteToHostException
        at io.netty.channel.unix.Errors.newConnectException0(Errors.java:158) ~[netty-transport-native-unix-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:131) ~[netty-transport-native-unix-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.unix.Socket.connect(Socket.java:351) ~[netty-transport-native-unix-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.epoll.AbstractEpollChannel.doConnect0(AbstractEpollChannel.java:773) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.epoll.EpollSocketChannel.doConnect0(EpollSocketChannel.java:144) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.epoll.AbstractEpollChannel.doConnect(AbstractEpollChannel.java:758) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.connect(AbstractEpollChannel.java:600) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1342) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:653) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:632) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.connect(CombinedChannelDuplexHandler.java:495) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.ChannelOutboundHandlerAdapter.connect(ChannelOutboundHandlerAdapter.java:51) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.CombinedChannelDuplexHandler.connect(CombinedChannelDuplexHandler.java:296) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:655) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:632) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:616) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.DefaultChannelPipeline.connect(DefaultChannelPipeline.java:978) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.AbstractChannel.connect(AbstractChannel.java:265) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.bootstrap.Bootstrap$3.run(Bootstrap.java:264) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:413) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
        at java.base/java.lang.Thread.run(Thread.java:840) ~[?:?]
[2024-02-13T21:13:40,196]  [<.SimpleClient] [ERROR:] - boom!
[2024-02-13T21:13:40,196]  [<.SimpleClient] [DEBUG:] - Fiber zio-fiber-112 did not handle an error
 io.netty.channel.AbstractChannel$AnnotatedNoRouteToHostException: null: www.google.com/216.239.38.120:80
Caused by: java.net.NoRouteToHostException
        at io.netty.channel.unix.Errors.newConnectException0(Errors.java:158) ~[netty-transport-native-unix-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:131) ~[netty-transport-native-unix-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.unix.Socket.connect(Socket.java:351) ~[netty-transport-native-unix-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.epoll.AbstractEpollChannel.doConnect0(AbstractEpollChannel.java:773) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.epoll.EpollSocketChannel.doConnect0(EpollSocketChannel.java:144) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.epoll.AbstractEpollChannel.doConnect(AbstractEpollChannel.java:758) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.connect(AbstractEpollChannel.java:600) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1342) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:653) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:632) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.connect(CombinedChannelDuplexHandler.java:495) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.ChannelOutboundHandlerAdapter.connect(ChannelOutboundHandlerAdapter.java:51) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.CombinedChannelDuplexHandler.connect(CombinedChannelDuplexHandler.java:296) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:655) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:632) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:616) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.DefaultChannelPipeline.connect(DefaultChannelPipeline.java:978) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.AbstractChannel.connect(AbstractChannel.java:265) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.bootstrap.Bootstrap$3.run(Bootstrap.java:264) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:413) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
        at java.base/java.lang.Thread.run(Thread.java:840) ~[?:?]
[2024-02-13T21:14:12,215]  [<.SimpleClient] [ERROR:] - boom!
[2024-02-13T21:14:12,215]  [<.SimpleClient] [DEBUG:] - Fiber zio-fiber-114 did not handle an error
 io.netty.channel.AbstractChannel$AnnotatedNoRouteToHostException: null: www.google.com/216.239.38.120:80
Caused by: java.net.NoRouteToHostException
        at io.netty.channel.unix.Errors.newConnectException0(Errors.java:158) ~[netty-transport-native-unix-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:131) ~[netty-transport-native-unix-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.unix.Socket.connect(Socket.java:351) ~[netty-transport-native-unix-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.epoll.AbstractEpollChannel.doConnect0(AbstractEpollChannel.java:773) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.epoll.EpollSocketChannel.doConnect0(EpollSocketChannel.java:144) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.epoll.AbstractEpollChannel.doConnect(AbstractEpollChannel.java:758) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.connect(AbstractEpollChannel.java:600) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1342) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:653) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:632) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.connect(CombinedChannelDuplexHandler.java:495) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.ChannelOutboundHandlerAdapter.connect(ChannelOutboundHandlerAdapter.java:51) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.CombinedChannelDuplexHandler.connect(CombinedChannelDuplexHandler.java:296) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:655) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:632) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:616) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.DefaultChannelPipeline.connect(DefaultChannelPipeline.java:978) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.AbstractChannel.connect(AbstractChannel.java:265) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.bootstrap.Bootstrap$3.run(Bootstrap.java:264) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:413) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
        at java.base/java.lang.Thread.run(Thread.java:840) ~[?:?]
[2024-02-13T21:15:02,262]  [zio-slf4j-logger] [DEBUG:] - ReadTimeoutException caught
[2024-02-13T21:15:16,296]  [<.SimpleClient] [ERROR:] - boom!
[2024-02-13T21:15:16,297]  [<.SimpleClient] [DEBUG:] - Fiber zio-fiber-117 did not handle an error
 io.netty.channel.AbstractChannel$AnnotatedNoRouteToHostException: null: www.google.com/216.239.38.120:80
Caused by: java.net.NoRouteToHostException
        at io.netty.channel.unix.Errors.newConnectException0(Errors.java:158) ~[netty-transport-native-unix-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:131) ~[netty-transport-native-unix-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.unix.Socket.connect(Socket.java:351) ~[netty-transport-native-unix-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.epoll.AbstractEpollChannel.doConnect0(AbstractEpollChannel.java:773) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.epoll.EpollSocketChannel.doConnect0(EpollSocketChannel.java:144) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.epoll.AbstractEpollChannel.doConnect(AbstractEpollChannel.java:758) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.connect(AbstractEpollChannel.java:600) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1342) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:653) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:632) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.connect(CombinedChannelDuplexHandler.java:495) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.ChannelOutboundHandlerAdapter.connect(ChannelOutboundHandlerAdapter.java:51) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.CombinedChannelDuplexHandler.connect(CombinedChannelDuplexHandler.java:296) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:655) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:632) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:616) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.DefaultChannelPipeline.connect(DefaultChannelPipeline.java:978) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.AbstractChannel.connect(AbstractChannel.java:265) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.bootstrap.Bootstrap$3.run(Bootstrap.java:264) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:413) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
        at java.base/java.lang.Thread.run(Thread.java:840) ~[?:?]
[2024-02-13T21:16:06,345]  [zio-slf4j-logger] [DEBUG:] - ReadTimeoutException caught
[2024-02-13T21:17:24,314]  [<.SimpleClient] [ERROR:] - boom!
[2024-02-13T21:17:24,314]  [<.SimpleClient] [DEBUG:] - Fiber zio-fiber-120 did not handle an error
 io.netty.channel.AbstractChannel$AnnotatedNoRouteToHostException: null: www.google.com/216.239.38.120:80
Caused by: java.net.NoRouteToHostException
        at io.netty.channel.unix.Errors.newConnectException0(Errors.java:158) ~[netty-transport-native-unix-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.unix.Errors.handleConnectErrno(Errors.java:131) ~[netty-transport-native-unix-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.unix.Socket.connect(Socket.java:351) ~[netty-transport-native-unix-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.epoll.AbstractEpollChannel.doConnect0(AbstractEpollChannel.java:773) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.epoll.EpollSocketChannel.doConnect0(EpollSocketChannel.java:144) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.epoll.AbstractEpollChannel.doConnect(AbstractEpollChannel.java:758) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.connect(AbstractEpollChannel.java:600) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.connect(DefaultChannelPipeline.java:1342) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:653) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:632) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.connect(CombinedChannelDuplexHandler.java:495) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.ChannelOutboundHandlerAdapter.connect(ChannelOutboundHandlerAdapter.java:51) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.CombinedChannelDuplexHandler.connect(CombinedChannelDuplexHandler.java:296) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeConnect(AbstractChannelHandlerContext.java:655) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:632) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.AbstractChannelHandlerContext.connect(AbstractChannelHandlerContext.java:616) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.DefaultChannelPipeline.connect(DefaultChannelPipeline.java:978) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.AbstractChannel.connect(AbstractChannel.java:265) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.bootstrap.Bootstrap$3.run(Bootstrap.java:264) ~[netty-transport-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:413) ~[netty-transport-classes-epoll-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.101.Final.jar:4.1.101.Final]
        at java.base/java.lang.Thread.run(Thread.java:840) ~[?:?]
[2024-02-13T21:12:14,559] [i.n.u.i.l.InternalLoggerFactory] [DEBUG:] - Using SLF4J as the default logging framework [2024-02-13T21:12:14,602] [i.n.u.i.PlatformDependent0] [DEBUG:] - -Dio.netty.noUnsafe: false [2024-02-13T21:12:14,604] [i.n.u.i.PlatformDependent0] [DEBUG:] - Java version: 17 [2024-02-13T21:12:14,606] [i.n.u.i.PlatformDependent0] [DEBUG:] - sun.misc.Unsafe.theUnsafe: available [2024-02-13T21:12:14,607] [i.n.u.i.PlatformDependent0] [DEBUG:] - sun.misc.Unsafe.copyMemory: available [2024-02-13T21:12:14,608] [i.n.u.i.PlatformDependent0] [DEBUG:] - sun.misc.Unsafe.storeFence: available [2024-02-13T21:12:14,609] [i.n.u.i.PlatformDependent0] [DEBUG:] - java.nio.Buffer.address: available [2024-02-13T21:12:14,610] [i.n.u.i.PlatformDependent0] [DEBUG:] - direct buffer constructor: unavailable: Reflective setAccessible(true) disabled [2024-02-13T21:12:14,611] [i.n.u.i.PlatformDependent0] [DEBUG:] - java.nio.Bits.unaligned: available, true [2024-02-13T21:12:14,613] [i.n.u.i.PlatformDependent0] [DEBUG:] - jdk.internal.misc.Unsafe.allocateUninitializedArray(int): unavailable: class io.netty.util.internal.PlatformDependent0$7 cannot access class jdk.internal.misc.Unsafe (in module java.base) because module java.base does not export jdk.internal.misc to unnamed module @768b0f44 [2024-02-13T21:12:14,615] [i.n.u.i.PlatformDependent0] [DEBUG:] - java.nio.DirectByteBuffer.(long, {int,long}): unavailable [2024-02-13T21:12:14,615] [i.n.u.i.PlatformDependent] [DEBUG:] - sun.misc.Unsafe: available [2024-02-13T21:12:14,616] [i.n.u.i.PlatformDependent] [DEBUG:] - -Dio.netty.tmpdir: /tmp (java.io.tmpdir) [2024-02-13T21:12:14,616] [i.n.u.i.PlatformDependent] [DEBUG:] - -Dio.netty.bitMode: 64 (sun.arch.data.model) [2024-02-13T21:12:14,621] [i.n.u.i.PlatformDependent] [DEBUG:] - -Dio.netty.maxDirectMemory: -1 bytes [2024-02-13T21:12:14,623] [i.n.u.i.PlatformDependent] [DEBUG:] - -Dio.netty.uninitializedArrayAllocationThreshold: -1 [2024-02-13T21:12:14,630] [i.n.u.i.CleanerJava9] [DEBUG:] - java.nio.ByteBuffer.cleaner(): available [2024-02-13T21:12:14,632] [i.n.u.i.PlatformDependent] [DEBUG:] - -Dio.netty.noPreferDirect: false [2024-02-13T21:12:14,647] [i.n.u.i.NativeLibraryLoader] [DEBUG:] - -Dio.netty.native.workdir: /tmp (io.netty.tmpdir) [2024-02-13T21:12:14,648] [i.n.u.i.NativeLibraryLoader] [DEBUG:] - -Dio.netty.native.deleteLibAfterLoading: true [2024-02-13T21:12:14,649] [i.n.u.i.NativeLibraryLoader] [DEBUG:] - -Dio.netty.native.tryPatchShadedId: true [2024-02-13T21:12:14,650] [i.n.u.i.NativeLibraryLoader] [DEBUG:] - -Dio.netty.native.detectNativeLibraryDuplicates: true [2024-02-13T21:12:14,665] [i.n.u.i.NativeLibraryLoader] [DEBUG:] - Successfully loaded the library /tmp/libnetty_transport_native_epoll_x86_643484322894985639787.so [2024-02-13T21:12:14,673] [i.n.u.NetUtil] [DEBUG:] - -Djava.net.preferIPv4Stack: false [2024-02-13T21:12:14,675] [i.n.u.NetUtil] [DEBUG:] - -Djava.net.preferIPv6Addresses: false [2024-02-13T21:12:14,682] [i.n.u.NetUtilInitializations] [DEBUG:] - Loopback interface: lo (lo, 0:0:0:0:0:0:0:1%lo) [2024-02-13T21:12:14,684] [i.n.u.NetUtil] [DEBUG:] - /proc/sys/net/core/somaxconn: 4096 [2024-02-13T21:12:14,703] [i.n.c.MultithreadEventLoopGroup] [DEBUG:] - -Dio.netty.eventLoopThreads: 8 [2024-02-13T21:12:14,726] [i.n.u.c.GlobalEventExecutor] [DEBUG:] - -Dio.netty.globalEventExecutor.quietPeriodSeconds: 1 [2024-02-13T21:12:14,743] [i.n.u.i.InternalThreadLocalMap] [DEBUG:] - -Dio.netty.threadLocalMap.stringBuilder.initialSize: 1024 [2024-02-13T21:12:14,745] [i.n.u.i.InternalThreadLocalMap] [DEBUG:] - -Dio.netty.threadLocalMap.stringBuilder.maxSize: 4096 [2024-02-13T21:12:14,773] [i.n.u.i.PlatformDependent] [DEBUG:] - org.jctools-core.MpscChunkedArrayQueue: available [2024-02-13T21:12:15,019] [i.n.c.DefaultChannelId] [DEBUG:] - -Dio.netty.processId: 134982 (auto-detected) [2024-02-13T21:12:15,026] [i.n.c.DefaultChannelId] [DEBUG:] - -Dio.netty.machineId: 5c:e0:c5:ff:fe:51:f5:ef (auto-detected) [2024-02-13T21:12:15,048] [i.n.u.ResourceLeakDetector] [DEBUG:] - -Dio.netty.leakDetection.level: simple [2024-02-13T21:12:15,050] [i.n.u.ResourceLeakDetector] [DEBUG:] - -Dio.netty.leakDetection.targetRecords: 4 [2024-02-13T21:12:15,098] [i.n.b.PooledByteBufAllocator] [DEBUG:] - -Dio.netty.allocator.numHeapArenas: 8 [2024-02-13T21:12:15,098] [i.n.b.PooledByteBufAllocator] [DEBUG:] - -Dio.netty.allocator.numDirectArenas: 8 [2024-02-13T21:12:15,099] [i.n.b.PooledByteBufAllocator] [DEBUG:] - -Dio.netty.allocator.pageSize: 8192 [2024-02-13T21:12:15,099] [i.n.b.PooledByteBufAllocator] [DEBUG:] - -Dio.netty.allocator.maxOrder: 9 [2024-02-13T21:12:15,100] [i.n.b.PooledByteBufAllocator] [DEBUG:] - -Dio.netty.allocator.chunkSize: 4194304 [2024-02-13T21:12:15,100] [i.n.b.PooledByteBufAllocator] [DEBUG:] - -Dio.netty.allocator.smallCacheSize: 256 [2024-02-13T21:12:15,101] [i.n.b.PooledByteBufAllocator] [DEBUG:] - -Dio.netty.allocator.normalCacheSize: 64 [2024-02-13T21:12:15,102] [i.n.b.PooledByteBufAllocator] [DEBUG:] - -Dio.netty.allocator.maxCachedBufferCapacity: 32768 [2024-02-13T21:12:15,103] [i.n.b.PooledByteBufAllocator] [DEBUG:] - -Dio.netty.allocator.cacheTrimInterval: 8192 [2024-02-13T21:12:15,104] [i.n.b.PooledByteBufAllocator] [DEBUG:] - -Dio.netty.allocator.cacheTrimIntervalMillis: 0 [2024-02-13T21:12:15,107] [i.n.b.PooledByteBufAllocator] [DEBUG:] - -Dio.netty.allocator.useCacheForAllThreads: false [2024-02-13T21:12:15,108] [i.n.b.PooledByteBufAllocator] [DEBUG:] - -Dio.netty.allocator.maxCachedByteBuffersPerChunk: 1023 [2024-02-13T21:12:15,133] [i.n.b.ByteBufUtil] [DEBUG:] - -Dio.netty.allocator.type: pooled [2024-02-13T21:12:15,134] [i.n.b.ByteBufUtil] [DEBUG:] - -Dio.netty.threadLocalDirectBufferSize: 0 [2024-02-13T21:12:15,134] [i.n.b.ByteBufUtil] [DEBUG:] - -Dio.netty.maxThreadLocalCharBufferSize: 16384 [2024-02-13T21:12:15,165] [i.n.b.ChannelInitializerExtensions] [DEBUG:] - -Dio.netty.bootstrap.extensions: null [2024-02-13T21:12:15,224] [i.n.b.AbstractByteBuf] [DEBUG:] - -Dio.netty.buffer.checkAccessible: true [2024-02-13T21:12:15,226] [i.n.b.AbstractByteBuf] [DEBUG:] - -Dio.netty.buffer.checkBounds: true [2024-02-13T21:12:15,228] [i.n.u.ResourceLeakDetectorFactory] [DEBUG:] - Loaded default ResourceLeakDetector: io.netty.util.ResourceLeakDetector@6c06e93 [2024-02-13T21:12:15,350] [i.n.u.Recycler] [DEBUG:] - -Dio.netty.recycler.maxCapacityPerThread: 4096 [2024-02-13T21:12:15,351] [i.n.u.Recycler] [DEBUG:] - -Dio.netty.recycler.ratio: 8 [2024-02-13T21:12:15,351] [i.n.u.Recycler] [DEBUG:] - -Dio.netty.recycler.chunkSize: 32 [2024-02-13T21:12:15,351] [i.n.u.Recycler] [DEBUG:] - -Dio.netty.recycler.blocking: false [2024-02-13T21:12:15,352] [i.n.u.Recycler] [DEBUG:] - -Dio.netty.recycler.batchFastThreadLocalOnly: true [2024-02-13T21:12:15,602] [<.simpleclient : got data: html>

mikail-khan avatar Feb 13 '24 21:02 mikail-khan

@mikail-khan can you try with the latest snapshot (3.0.0-RC4+71-b1da91b6-SNAPSHOT), and the program I provided?

The reason I'm asking to use the program I provided is because it uses ZIO.scoped which is how it's meant to be run (as opposed to providing the scope as a layer)

kyri-petrou avatar Feb 13 '24 22:02 kyri-petrou

@mikail-khan can you try with the latest snapshot (3.0.0-RC4+71-b1da91b6-SNAPSHOT), and the program I provided?

The reason I'm asking to use the program I provided is because it uses ZIO.scoped which is how it's meant to be run (as opposed to providing the scope as a layer)

where does it say ZIO.scoped is "meant" to be run how you say? The example I gave is from the ZIO-HTTP docs.... @jdegoes can you confirm @kyri-petrou 's assertion? IMO, @kyri-petrou 's program doesn't seem idiomatic ZIO.

mikail-khan avatar Feb 14 '24 08:02 mikail-khan

@mikail-khan I'm not sure whether it's idiomatic or not, but it's the correct way to run it. The Scope returned by the get method is responsible for returning the connection back to the connection pool, potentially discarding it if it's dead. Regarding the docs, they should probably be corrected.

Having said that, do we need to have a requirement on Scope? For that I'm not sure. AFAICT its main purpose is so that the caller handles streaming responses properly prior to returning the connection back to the pool, but there might be more UX-friendly (or idiomatic) ways of handling it. Or at the very least the purpose of the scope should be better documented

kyri-petrou avatar Feb 14 '24 10:02 kyri-petrou

I will address this by adding the documentation to explain the purpose and correct usage of scopes and why ZIO.scoped is crucial for proper resource management, particularly in network connection scenarios. /attempt #2653

Options

asr2003 avatar Jun 22 '24 18:06 asr2003

@asr2003: Reminder that in 7 days the bounty will become up for grabs, so please submit a pull request before then 🙏

algora-pbc[bot] avatar Jun 29 '24 18:06 algora-pbc[bot]

The bounty is up for grabs! Everyone is welcome to /attempt #2653 🙌

algora-pbc[bot] avatar Jul 06 '24 18:07 algora-pbc[bot]

Fixed by #3008

987Nabil avatar Aug 31 '24 06:08 987Nabil