seatunnel icon indicating copy to clipboard operation
seatunnel copied to clipboard

Published 20 hours ago • verified publisher icon apache

[Bug] [connector-redis] Failed to connect to any host resolved for DNS name.

Open matianhe3 opened this issue 11 months ago • 5 comments

Search before asking

  • [X] I had searched in the issues and found no similar issues.

What happened

redis.clients.jedis.exceptions.JedisConnectionException: Failed to connect to any host resolved for DNS name. Redis sink error in 2.3.4. work successfully in 2.3.3.

SeaTunnel Version

v2.3.4

SeaTunnel Config

sink {
  Redis {
    source_table_name = [source]
    host = 192.168.16.210
    port = 9999
    key = "gd:traveltime"
    mode = cluster
    nodes = ["192.168.16.210:9998", "192.168.16.213:9999", "192.168.16.213:9998",
    "192.168.17.86:9999", "192.168.17.86:9998"]
    data_type = list
  }
}

Running Command

seatunnel.sh -c test.conf

Error Exception

e
2024-03-11 11:09:22,931 INFO  [o.a.s.e.s.d.p.PhysicalPlan    ] [seatunnel-coordinator-service-1] - Job traveltime (819406615231856641) turned from state RUNNING to FAILING.
2024-03-11 11:09:22,932 INFO  [o.a.s.e.s.d.p.PhysicalPlan    ] [seatunnel-coordinator-service-1] - Job traveltime (819406615231856641) turned from state FAILING to FAILED.
2024-03-11 11:09:22,932 INFO  [o.a.s.e.s.d.p.PhysicalPlan    ] [seatunnel-coordinator-service-1] - Job traveltime (819406615231856641) state process is stop
2024-03-11 11:09:22,947 INFO  [o.a.s.e.c.j.ClientJobProxy    ] [main] - Job (819406615231856641) end with state FAILED
2024-03-11 11:09:22,947 INFO  [c.h.c.LifecycleService        ] [main] - hz.client_1 [seatunnel-648169] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTTING_DOWN
2024-03-11 11:09:22,949 INFO  [c.h.i.s.t.TcpServerConnection ] [hz.main.IO.thread-in-1] - [localhost]:5801 [seatunnel-648169] [5.1] Connection[id=1, /127.0.0.1:5801->/127.0.0.1:55557, qualifier=null, endpoint=[127.0.0.1]:55557, remoteUuid=81c64a1c-e850-4d88-86c3-a22593a5fb43, alive=false, connectionType=JVM, planeIndex=-1] closed. Reason: Connection closed by the other side
2024-03-11 11:09:22,950 INFO  [.c.i.c.ClientConnectionManager] [main] - hz.client_1 [seatunnel-648169] [5.1] Removed connection to endpoint: [localhost]:5801:d3830339-31de-482a-9e97-938886da2ad5, connection: ClientConnection{alive=false, connectionId=1, channel=NioChannel{/127.0.0.1:55557->localhost/127.0.0.1:5801}, remoteAddress=[localhost]:5801, lastReadTime=2024-03-11 11:09:22.946, lastWriteTime=2024-03-11 11:09:22.172, closedTime=2024-03-11 11:09:22.948, connected server version=5.1}
2024-03-11 11:09:22,950 INFO  [c.h.c.LifecycleService        ] [main] - hz.client_1 [seatunnel-648169] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is CLIENT_DISCONNECTED
2024-03-11 11:09:22,951 INFO  [c.h.c.i.ClientEndpointManager ] [hz.main.event-3] - [localhost]:5801 [seatunnel-648169] [5.1] Destroying ClientEndpoint{connection=Connection[id=1, /127.0.0.1:5801->/127.0.0.1:55557, qualifier=null, endpoint=[127.0.0.1]:55557, remoteUuid=81c64a1c-e850-4d88-86c3-a22593a5fb43, alive=false, connectionType=JVM, planeIndex=-1], clientUuid=81c64a1c-e850-4d88-86c3-a22593a5fb43, clientName=hz.client_1, authenticated=true, clientVersion=5.1, creationTime=1710126560835, latest clientAttributes=lastStatisticsCollectionTime=1710126560870,enterprise=false,clientType=JVM,clientVersion=5.1,clusterConnectionTimestamp=1710126560824,clientAddress=127.0.0.1,clientName=hz.client_1,credentials.principal=null,os.committedVirtualMemorySize=0,os.freePhysicalMemorySize=4762411008,os.freeSwapSpaceSize=0,os.maxFileDescriptorCount=0,os.openFileDescriptorCount=0,os.processCpuTime=0,os.systemLoadAverage=1.11474609375,os.totalPhysicalMemorySize=16665174016,os.totalSwapSpaceSize=0,runtime.availableProcessors=4,runtime.freeMemory=166343504,runtime.maxMemory=536870912,runtime.totalMemory=268435456,runtime.uptime=2411,runtime.usedMemory=102091952, labels=[]}
2024-03-11 11:09:22,952 INFO  [c.h.c.LifecycleService        ] [main] - hz.client_1 [seatunnel-648169] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTDOWN
2024-03-11 11:09:22,952 INFO  [s.c.s.s.c.ClientExecuteCommand] [main] - Closed SeaTunnel client......
2024-03-11 11:09:22,952 INFO  [c.h.c.LifecycleService        ] [main] - [localhost]:5801 [seatunnel-648169] [5.1] [localhost]:5801 is SHUTTING_DOWN
2024-03-11 11:09:22,954 INFO  [c.h.i.p.i.MigrationManager    ] [hz.main.cached.thread-1] - [localhost]:5801 [seatunnel-648169] [5.1] Shutdown request of Member [localhost]:5801 - d3830339-31de-482a-9e97-938886da2ad5 this is handled
2024-03-11 11:09:22,958 INFO  [c.h.i.i.Node                  ] [main] - [localhost]:5801 [seatunnel-648169] [5.1] Shutting down connection manager...
2024-03-11 11:09:22,960 INFO  [c.h.i.i.Node                  ] [main] - [localhost]:5801 [seatunnel-648169] [5.1] Shutting down node engine...
2024-03-11 11:09:22,966 INFO  [o.a.s.e.s.CoordinatorService  ] [pool-4-thread-1] - [localhost]:5801 [seatunnel-648169] [5.1] This node become leave active master node, begin clear coordinator service
2024-03-11 11:09:22,968 ERROR [o.a.s.e.s.CoordinatorService  ] [pool-4-thread-1] - [localhost]:5801 [seatunnel-648169] [5.1] org.apache.seatunnel.engine.common.exception.SeaTunnelEngineException: wait clean executor service error
        at org.apache.seatunnel.engine.server.CoordinatorService.clearCoordinatorService(CoordinatorService.java:360)
        at org.apache.seatunnel.engine.server.CoordinatorService.checkNewActiveMaster(CoordinatorService.java:342)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
        at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:842)
Caused by: java.lang.InterruptedException
        at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1679)
        at java.base/java.util.concurrent.ThreadPoolExecutor.awaitTermination(ThreadPoolExecutor.java:1464)
        at org.apache.seatunnel.engine.server.CoordinatorService.clearCoordinatorService(CoordinatorService.java:358)
        ... 7 more

2024-03-11 11:09:25,506 INFO  [c.h.i.i.NodeExtension         ] [main] - [localhost]:5801 [seatunnel-648169] [5.1] Destroying node NodeExtension.
2024-03-11 11:09:25,507 INFO  [c.h.i.i.Node                  ] [main] - [localhost]:5801 [seatunnel-648169] [5.1] Hazelcast Shutdown is completed in 2553 ms.
2024-03-11 11:09:25,508 INFO  [c.h.c.LifecycleService        ] [main] - [localhost]:5801 [seatunnel-648169] [5.1] [localhost]:5801 is SHUTDOWN
2024-03-11 11:09:25,508 INFO  [s.c.s.s.c.ClientExecuteCommand] [main] - Closed HazelcastInstance ......
2024-03-11 11:09:25,510 INFO  [s.c.s.s.c.ClientExecuteCommand] [main] - Closed metrics executor service ......
2024-03-11 11:09:25,511 ERROR [o.a.s.c.s.SeaTunnel           ] [main] -

===============================================================================


2024-03-11 11:09:25,511 ERROR [o.a.s.c.s.SeaTunnel           ] [main] - Fatal Error,

2024-03-11 11:09:25,511 ERROR [o.a.s.c.s.SeaTunnel           ] [main] - Please submit bug report in https://github.com/apache/seatunnel/issues

2024-03-11 11:09:25,511 ERROR [o.a.s.c.s.SeaTunnel           ] [main] - Reason:SeaTunnel job executed failed

2024-03-11 11:09:25,514 ERROR [o.a.s.c.s.SeaTunnel           ] [main] - Exception StackTrace:org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
        at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:199)
        at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
        at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: org.apache.seatunnel.engine.common.exception.SeaTunnelEngineException: org.apache.seatunnel.engine.server.checkpoint.CheckpointException: CheckpointCoordinator inside have error.
        at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.handleCoordinatorError(CheckpointCoordinator.java:270)
        at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.handleCoordinatorError(CheckpointCoordinator.java:266)
        at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.reportCheckpointErrorFromTask(CheckpointCoordinator.java:372)
        at org.apache.seatunnel.engine.server.checkpoint.CheckpointManager.reportCheckpointErrorFromTask(CheckpointManager.java:183)
        at org.apache.seatunnel.engine.server.checkpoint.operation.CheckpointErrorReportOperation.run(CheckpointErrorReportOperation.java:48)
        at com.hazelcast.spi.impl.operationservice.Operation.call(Operation.java:189)
        at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.call(OperationRunnerImpl.java:273)
        at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:248)
        at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:213)
        at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:175)
        at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:139)
        at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.executeRun(OperationThread.java:123)
        at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:102)
Caused by: org.apache.seatunnel.common.utils.SeaTunnelException: redis.clients.jedis.exceptions.JedisConnectionException: Failed to connect to any host resolved for DNS name.
        at redis.clients.jedis.DefaultJedisSocketFactory.connectToFirstSuccessfulHost(DefaultJedisSocketFactory.java:63)
        at redis.clients.jedis.DefaultJedisSocketFactory.createSocket(DefaultJedisSocketFactory.java:87)
        at redis.clients.jedis.Connection.connect(Connection.java:180)
        at redis.clients.jedis.Connection.sendCommand(Connection.java:152)
        at redis.clients.jedis.Connection.sendCommand(Connection.java:147)
        at redis.clients.jedis.Jedis.select(Jedis.java:800)
        at org.apache.seatunnel.connectors.seatunnel.redis.config.RedisParameters.buildJedis(RedisParameters.java:134)
        at org.apache.seatunnel.connectors.seatunnel.redis.sink.RedisSinkWriter.<init>(RedisSinkWriter.java:47)
        at org.apache.seatunnel.connectors.seatunnel.redis.sink.RedisSink.createWriter(RedisSink.java:53)
        at org.apache.seatunnel.connectors.seatunnel.redis.sink.RedisSink.createWriter(RedisSink.java:32)
        at org.apache.seatunnel.engine.server.task.flow.SinkFlowLifeCycle.restoreState(SinkFlowLifeCycle.java:297)
        at org.apache.seatunnel.engine.server.task.SeaTunnelTask.lambda$restoreState$16(SeaTunnelTask.java:425)
        at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
        at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
        at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179)
        at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
        at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
        at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
        at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
        at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
        at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
        at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)
        at org.apache.seatunnel.engine.server.task.SeaTunnelTask.restoreState(SeaTunnelTask.java:422)
        at org.apache.seatunnel.engine.server.checkpoint.operation.NotifyTaskRestoreOperation.lambda$null$0(NotifyTaskRestoreOperation.java:106)
        at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:842)
        Suppressed: java.net.ConnectException: Connection refused
                at java.base/sun.nio.ch.Net.pollConnect(Native Method)
                at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672)
                at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:547)
                at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:602)
                at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
                at java.base/java.net.Socket.connect(Socket.java:633)
                at redis.clients.jedis.DefaultJedisSocketFactory.connectToFirstSuccessfulHost(DefaultJedisSocketFactory.java:73)
                ... 27 more

        ... 11 more

        at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:191)
        ... 2 more

2024-03-11 11:09:25,515 ERROR [o.a.s.c.s.SeaTunnel           ] [main] -
===============================================================================



Exception in thread "main" org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
        at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:199)
        at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
        at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: org.apache.seatunnel.engine.common.exception.SeaTunnelEngineException: org.apache.seatunnel.engine.server.checkpoint.CheckpointException: CheckpointCoordinator inside have error.
        at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.handleCoordinatorError(CheckpointCoordinator.java:270)
        at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.handleCoordinatorError(CheckpointCoordinator.java:266)
        at org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator.reportCheckpointErrorFromTask(CheckpointCoordinator.java:372)
        at org.apache.seatunnel.engine.server.checkpoint.CheckpointManager.reportCheckpointErrorFromTask(CheckpointManager.java:183)
        at org.apache.seatunnel.engine.server.checkpoint.operation.CheckpointErrorReportOperation.run(CheckpointErrorReportOperation.java:48)
        at com.hazelcast.spi.impl.operationservice.Operation.call(Operation.java:189)
        at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.call(OperationRunnerImpl.java:273)
        at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:248)
        at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:213)
        at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:175)
        at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:139)
        at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.executeRun(OperationThread.java:123)
        at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:102)
Caused by: org.apache.seatunnel.common.utils.SeaTunnelException: redis.clients.jedis.exceptions.JedisConnectionException: Failed to connect to any host resolved for DNS name.
        at redis.clients.jedis.DefaultJedisSocketFactory.connectToFirstSuccessfulHost(DefaultJedisSocketFactory.java:63)
        at redis.clients.jedis.DefaultJedisSocketFactory.createSocket(DefaultJedisSocketFactory.java:87)
        at redis.clients.jedis.Connection.connect(Connection.java:180)
        at redis.clients.jedis.Connection.sendCommand(Connection.java:152)
        at redis.clients.jedis.Connection.sendCommand(Connection.java:147)
        at redis.clients.jedis.Jedis.select(Jedis.java:800)
        at org.apache.seatunnel.connectors.seatunnel.redis.config.RedisParameters.buildJedis(RedisParameters.java:134)
        at org.apache.seatunnel.connectors.seatunnel.redis.sink.RedisSinkWriter.<init>(RedisSinkWriter.java:47)
        at org.apache.seatunnel.connectors.seatunnel.redis.sink.RedisSink.createWriter(RedisSink.java:53)
        at org.apache.seatunnel.connectors.seatunnel.redis.sink.RedisSink.createWriter(RedisSink.java:32)
        at org.apache.seatunnel.engine.server.task.flow.SinkFlowLifeCycle.restoreState(SinkFlowLifeCycle.java:297)
        at org.apache.seatunnel.engine.server.task.SeaTunnelTask.lambda$restoreState$16(SeaTunnelTask.java:425)
        at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
        at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
        at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179)
        at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
        at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
        at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
        at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
        at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
        at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
        at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)
        at org.apache.seatunnel.engine.server.task.SeaTunnelTask.restoreState(SeaTunnelTask.java:422)
        at org.apache.seatunnel.engine.server.checkpoint.operation.NotifyTaskRestoreOperation.lambda$null$0(NotifyTaskRestoreOperation.java:106)
        at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:842)
        Suppressed: java.net.ConnectException: Connection refused
                at java.base/sun.nio.ch.Net.pollConnect(Native Method)
                at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672)
                at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:547)
                at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:602)
                at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
                at java.base/java.net.Socket.connect(Socket.java:633)
                at redis.clients.jedis.DefaultJedisSocketFactory.connectToFirstSuccessfulHost(DefaultJedisSocketFactory.java:73)
                ... 27 more

        ... 11 more

        at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:191)
        ... 2 more
2024-03-11 11:09:25,518 INFO  [s.c.s.s.c.ClientExecuteCommand] [ForkJoinPool.commonPool-worker-1] - run shutdown hook because get close signal

Zeta or Flink or Spark Version

No response

Java or Scala Version

No response

Screenshots

No response

Are you willing to submit PR?

  • [X] Yes I am willing to submit a PR!

Code of Conduct

matianhe3 avatar Mar 11 '24 03:03 matianhe3

version 2.3.4 I also had this problem

FengSq avatar Mar 12 '24 06:03 FengSq

https://docs.docker.com/engine/release-notes/26.0/#bug-fixes-and-enhancements

Have you tried with yesterday's release of Docker engine? The change log hints at some internal DNS resolution fixes.

dzoech avatar Mar 21 '24 09:03 dzoech

https://docs.docker.com/engine/release-notes/26.0/#bug-fixes-and-enhancements

Have you tried with yesterday's release of Docker engine? The change log hints at some internal DNS resolution fixes.

updated, but same error

matianhe3 avatar Mar 21 '24 10:03 matianhe3

same in 2.3.5

matianhe3 avatar May 11 '24 06:05 matianhe3

single mode can work, only cluster mode error.

matianhe3 avatar May 11 '24 06:05 matianhe3

I also encountered such a mistake. How did you solve it

aokunsang avatar Aug 21 '24 09:08 aokunsang

I also encountered such a mistake. How did you solve it

i don't solve it , now , i don't use cluster mode.

matianhe3 avatar Aug 21 '24 10:08 matianhe3