seatunnel icon indicating copy to clipboard operation
seatunnel copied to clipboard

Published 20 hours ago • verified publisher icon apache

[Bug] [Module Name] seatunnel-cluster.sh -d is run, can't seatunnel.sh -m local ?

Open matianhe3 opened this issue 1 year ago • 5 comments

Search before asking

  • [X] I had searched in the issues and found no similar issues.

What happened

when i run cluster mode , i can't run local mode together?

SeaTunnel Version

2.3.3

SeaTunnel Config

v2.batch.config.template

Running Command

seatunnel.sh -m local -c v2.batch.config.template

Error Exception

com.hazelcast.client.AuthenticationException: Authentication failed. The configured cluster name on the client (see ClientConfig.setClusterName()) does not match the one configured in the cluster or the credentials set in the Client security config could not be authenticated
	at com.hazelcast.client.impl.connection.tcp.TcpClientConnectionManager.checkAuthenticationResponse(TcpClientConnectionManager.java:995) ~[seatunnel-starter.jar:2.3.3]
	at com.hazelcast.client.impl.connection.tcp.TcpClientConnectionManager.onAuthenticated(TcpClientConnectionManager.java:898) ~[seatunnel-starter.jar:2.3.3]
	at com.hazelcast.client.impl.connection.tcp.TcpClientConnectionManager.getOrConnectToAddress(TcpClientConnectionManager.java:615) ~[seatunnel-starter.jar:2.3.3]
	at com.hazelcast.client.impl.connection.tcp.TcpClientConnectionManager.lambda$doConnectToCandidateCluster$3(TcpClientConnectionManager.java:505) ~[seatunnel-starter.jar:2.3.3]
	at com.hazelcast.client.impl.connection.tcp.TcpClientConnectionManager.connect(TcpClientConnectionManager.java:459) [seatunnel-starter.jar:2.3.3]
	at com.hazelcast.client.impl.connection.tcp.TcpClientConnectionManager.doConnectToCandidateCluster(TcpClientConnectionManager.java:505) [seatunnel-starter.jar:2.3.3]
	at com.hazelcast.client.impl.connection.tcp.TcpClientConnectionManager.doConnectToCluster(TcpClientConnectionManager.java:411) [seatunnel-starter.jar:2.3.3]
	at com.hazelcast.client.impl.connection.tcp.TcpClientConnectionManager.connectToCluster(TcpClientConnectionManager.java:372) [seatunnel-starter.jar:2.3.3]
	at com.hazelcast.client.impl.clientside.HazelcastClientInstanceImpl.start(HazelcastClientInstanceImpl.java:381) [seatunnel-starter.jar:2.3.3]
	at com.hazelcast.client.HazelcastClient.constructHazelcastClient(HazelcastClient.java:460) [seatunnel-starter.jar:2.3.3]
	at com.hazelcast.client.HazelcastClient.newHazelcastClientInternal(HazelcastClient.java:416) [seatunnel-starter.jar:2.3.3]
	at com.hazelcast.client.HazelcastClient.newHazelcastClient(HazelcastClient.java:136) [seatunnel-starter.jar:2.3.3]
	at org.apache.seatunnel.engine.client.SeaTunnelHazelcastClient.<init>(SeaTunnelHazelcastClient.java:53) [seatunnel-starter.jar:2.3.3]
	at org.apache.seatunnel.engine.client.SeaTunnelClient.<init>(SeaTunnelClient.java:46) [seatunnel-starter.jar:2.3.3]
	at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:96) [seatunnel-starter.jar:2.3.3]
	at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40) [seatunnel-starter.jar:2.3.3]
	at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34) [seatunnel-starter.jar:2.3.3]
2024-01-10 11:43:26,988 WARN  com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel-478597] [5.1] Exception during initial connection to [192.168.16.210]:5801: com.hazelcast.client.AuthenticationException: Authentication failed. The configured cluster name on the client (see ClientConfig.setClusterName()) does not match the one configured in the cluster or the credentials set in the Client security config could not be authenticated
2024-01-10 11:43:26,988 INFO  com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel-478597] [5.1] Trying to connect to [192.168.16.212]:5801
2024-01-10 11:43:26,988 WARN  com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel-478597] [5.1] Exception during initial connection to [192.168.16.212]:5801: com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused to address /192.168.16.212:5801
2024-01-10 11:43:26,989 INFO  com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel-478597] [5.1] Trying to connect to [192.168.16.213]:5801
2024-01-10 11:43:26,990 WARN  com.hazelcast.client.impl.connection.tcp.TcpClientConnection - hz.client_1 [seatunnel-478597] [5.1] ClientConnection{alive=false, connectionId=12, channel=NioChannel{/192.168.16.210:52863->/192.168.16.213:5801}, remoteAddress=null, lastReadTime=2024-01-10 11:43:26.990, lastWriteTime=2024-01-10 11:43:26.989, closedTime=2024-01-10 11:43:26.990, connected server version=null} closed. Reason: Failed to authenticate connection
com.hazelcast.client.AuthenticationException: Authentication failed. The configured cluster name on the client (see ClientConfig.setClusterName()) does not match the one configured in the cluster or the credentials set in the Client security config could not be authenticated
	at com.hazelcast.client.impl.connection.tcp.TcpClientConnectionManager.checkAuthenticationResponse(TcpClientConnectionManager.java:995) ~[seatunnel-starter.jar:2.3.3]
	at com.hazelcast.client.impl.connection.tcp.TcpClientConnectionManager.onAuthenticated(TcpClientConnectionManager.java:898) ~[seatunnel-starter.jar:2.3.3]
	at com.hazelcast.client.impl.connection.tcp.TcpClientConnectionManager.getOrConnectToAddress(TcpClientConnectionManager.java:615) ~[seatunnel-starter.jar:2.3.3]
	at com.hazelcast.client.impl.connection.tcp.TcpClientConnectionManager.lambda$doConnectToCandidateCluster$3(TcpClientConnectionManager.java:505) ~[seatunnel-starter.jar:2.3.3]
	at com.hazelcast.client.impl.connection.tcp.TcpClientConnectionManager.connect(TcpClientConnectionManager.java:459) [seatunnel-starter.jar:2.3.3]
	at com.hazelcast.client.impl.connection.tcp.TcpClientConnectionManager.doConnectToCandidateCluster(TcpClientConnectionManager.java:505) [seatunnel-starter.jar:2.3.3]
	at com.hazelcast.client.impl.connection.tcp.TcpClientConnectionManager.doConnectToCluster(TcpClientConnectionManager.java:411) [seatunnel-starter.jar:2.3.3]
	at com.hazelcast.client.impl.connection.tcp.TcpClientConnectionManager.connectToCluster(TcpClientConnectionManager.java:372) [seatunnel-starter.jar:2.3.3]
	at com.hazelcast.client.impl.clientside.HazelcastClientInstanceImpl.start(HazelcastClientInstanceImpl.java:381) [seatunnel-starter.jar:2.3.3]
	at com.hazelcast.client.HazelcastClient.constructHazelcastClient(HazelcastClient.java:460) [seatunnel-starter.jar:2.3.3]
	at com.hazelcast.client.HazelcastClient.newHazelcastClientInternal(HazelcastClient.java:416) [seatunnel-starter.jar:2.3.3]
	at com.hazelcast.client.HazelcastClient.newHazelcastClient(HazelcastClient.java:136) [seatunnel-starter.jar:2.3.3]
	at org.apache.seatunnel.engine.client.SeaTunnelHazelcastClient.<init>(SeaTunnelHazelcastClient.java:53) [seatunnel-starter.jar:2.3.3]
	at org.apache.seatunnel.engine.client.SeaTunnelClient.<init>(SeaTunnelClient.java:46) [seatunnel-starter.jar:2.3.3]
	at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:96) [seatunnel-starter.jar:2.3.3]
	at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40) [seatunnel-starter.jar:2.3.3]
	at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34) [seatunnel-starter.jar:2.3.3]
2024-01-10 11:43:26,991 WARN  com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel-478597] [5.1] Exception during initial connection to [192.168.16.213]:5801: com.hazelcast.client.AuthenticationException: Authentication failed. The configured cluster name on the client (see ClientConfig.setClusterName()) does not match the one configured in the cluster or the credentials set in the Client security config could not be authenticated
2024-01-10 11:43:26,991 WARN  com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel-478597] [5.1] Unable to get live cluster connection, cluster connect timeout (3000 ms) is reached. Attempt 4.
2024-01-10 11:43:26,991 INFO  com.hazelcast.client.impl.connection.ClientConnectionManager - hz.client_1 [seatunnel-478597] [5.1] Unable to connect to any address from the cluster with name: seatunnel-478597. The following addresses were tried: [[192.168.16.213]:5801, [192.168.16.212]:5801, [192.168.16.210]:5801, [192.168.16.209]:5801]
2024-01-10 11:43:26,992 INFO  com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel-478597] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTTING_DOWN
2024-01-10 11:43:26,994 INFO  com.hazelcast.core.LifecycleService - hz.client_1 [seatunnel-478597] [5.1] HazelcastClient 5.1 (20220228 - 21f20e7) is SHUTDOWN
2024-01-10 11:43:26,994 INFO  com.hazelcast.core.LifecycleService - [192.168.16.210]:5802 [seatunnel-478597] [5.1] [192.168.16.210]:5802 is SHUTTING_DOWN
2024-01-10 11:43:26,997 INFO  com.hazelcast.internal.partition.impl.MigrationManager - [192.168.16.210]:5802 [seatunnel-478597] [5.1] Shutdown request of Member [192.168.16.210]:5802 - 3d79a5e1-74c2-4210-9552-99d770ec0550 this is handled
2024-01-10 11:43:27,001 INFO  com.hazelcast.instance.impl.Node - [192.168.16.210]:5802 [seatunnel-478597] [5.1] Shutting down connection manager...
2024-01-10 11:43:27,002 INFO  com.hazelcast.instance.impl.Node - [192.168.16.210]:5802 [seatunnel-478597] [5.1] Shutting down node engine...
2024-01-10 11:43:30,015 INFO  com.hazelcast.instance.impl.NodeExtension - [192.168.16.210]:5802 [seatunnel-478597] [5.1] Destroying node NodeExtension.
2024-01-10 11:43:30,016 INFO  com.hazelcast.instance.impl.Node - [192.168.16.210]:5802 [seatunnel-478597] [5.1] Hazelcast Shutdown is completed in 3020 ms.
2024-01-10 11:43:30,016 INFO  com.hazelcast.core.LifecycleService - [192.168.16.210]:5802 [seatunnel-478597] [5.1] [192.168.16.210]:5802 is SHUTDOWN
2024-01-10 11:43:30,016 INFO  org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand - Closed HazelcastInstance ......
2024-01-10 11:43:30,016 ERROR org.apache.seatunnel.core.starter.SeaTunnel -

===============================================================================


2024-01-10 11:43:30,016 ERROR org.apache.seatunnel.core.starter.SeaTunnel - Fatal Error,

2024-01-10 11:43:30,016 ERROR org.apache.seatunnel.core.starter.SeaTunnel - Please submit bug report in https://github.com/apache/seatunnel/issues

2024-01-10 11:43:30,018 ERROR org.apache.seatunnel.core.starter.SeaTunnel - Reason:SeaTunnel job executed failed

2024-01-10 11:43:30,019 ERROR org.apache.seatunnel.core.starter.SeaTunnel - Exception StackTrace:org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
	at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:191)
	at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
	at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: java.lang.IllegalStateException: Unable to connect to any cluster.
	at com.hazelcast.client.impl.connection.tcp.TcpClientConnectionManager.doConnectToCluster(TcpClientConnectionManager.java:433)
	at com.hazelcast.client.impl.connection.tcp.TcpClientConnectionManager.connectToCluster(TcpClientConnectionManager.java:372)
	at com.hazelcast.client.impl.clientside.HazelcastClientInstanceImpl.start(HazelcastClientInstanceImpl.java:381)
	at com.hazelcast.client.HazelcastClient.constructHazelcastClient(HazelcastClient.java:460)
	at com.hazelcast.client.HazelcastClient.newHazelcastClientInternal(HazelcastClient.java:416)
	at com.hazelcast.client.HazelcastClient.newHazelcastClient(HazelcastClient.java:136)
	at org.apache.seatunnel.engine.client.SeaTunnelHazelcastClient.<init>(SeaTunnelHazelcastClient.java:53)
	at org.apache.seatunnel.engine.client.SeaTunnelClient.<init>(SeaTunnelClient.java:46)
	at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:96)
	... 2 more

2024-01-10 11:43:30,019 ERROR org.apache.seatunnel.core.starter.SeaTunnel -
===============================================================================



Exception in thread "main" org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
	at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:191)
	at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
	at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: java.lang.IllegalStateException: Unable to connect to any cluster.
	at com.hazelcast.client.impl.connection.tcp.TcpClientConnectionManager.doConnectToCluster(TcpClientConnectionManager.java:433)
	at com.hazelcast.client.impl.connection.tcp.TcpClientConnectionManager.connectToCluster(TcpClientConnectionManager.java:372)
	at com.hazelcast.client.impl.clientside.HazelcastClientInstanceImpl.start(HazelcastClientInstanceImpl.java:381)
	at com.hazelcast.client.HazelcastClient.constructHazelcastClient(HazelcastClient.java:460)
	at com.hazelcast.client.HazelcastClient.newHazelcastClientInternal(HazelcastClient.java:416)
	at com.hazelcast.client.HazelcastClient.newHazelcastClient(HazelcastClient.java:136)
	at org.apache.seatunnel.engine.client.SeaTunnelHazelcastClient.<init>(SeaTunnelHazelcastClient.java:53)
	at org.apache.seatunnel.engine.client.SeaTunnelClient.<init>(SeaTunnelClient.java:46)
	at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:96)
	... 2 more

Zeta or Flink or Spark Version

No response

Java or Scala Version

No response

Screenshots

No response

Are you willing to submit PR?

  • [ ] Yes I am willing to submit a PR!

Code of Conduct

matianhe3 avatar Jan 10 '24 03:01 matianhe3

same issue in SeaTunnel 2.3.3

inkkim avatar Jan 16 '24 08:01 inkkim

same issue in SeaTunnel 2.3.3

Aiden-Rose avatar Feb 02 '24 07:02 Aiden-Rose

Perhaps it has to do with the contents of the configuration files in the config folder. For example, port: 5801

hazelcast-client:
  cluster-name: seatunnel
  properties:
    hazelcast.logging.type: log4j2
  connection-strategy:
    connection-retry:
      cluster-connect-timeout-millis: 3000
  network:
    cluster-members:
      - localhost:5801

hazelcast:
  cluster-name: seatunnel
  network:
    rest-api:
      enabled: true
      endpoint-groups:
        CLUSTER_WRITE:
          enabled: true
        DATA:
          enabled: true
    join:
      tcp-ip:
        enabled: true
        member-list:
          - localhost
    port:
      auto-increment: false
      port: 5801
  properties:
    hazelcast.invocation.max.retry.count: 20
    hazelcast.tcp.join.port.try.count: 30
    hazelcast.logging.type: log4j2
    hazelcast.operation.generic.thread.count: 50

cqutwangyu avatar Feb 23 '24 05:02 cqutwangyu

Perhaps it has to do with the contents of the configuration files in the config folder. For example, port: 5801

hazelcast-client:
  cluster-name: seatunnel
  properties:
    hazelcast.logging.type: log4j2
  connection-strategy:
    connection-retry:
      cluster-connect-timeout-millis: 3000
  network:
    cluster-members:
      - localhost:5801

hazelcast:
  cluster-name: seatunnel
  network:
    rest-api:
      enabled: true
      endpoint-groups:
        CLUSTER_WRITE:
          enabled: true
        DATA:
          enabled: true
    join:
      tcp-ip:
        enabled: true
        member-list:
          - localhost
    port:
      auto-increment: false
      port: 5801
  properties:
    hazelcast.invocation.max.retry.count: 20
    hazelcast.tcp.join.port.try.count: 30
    hazelcast.logging.type: log4j2
    hazelcast.operation.generic.thread.count: 50

not this problem, same as you, when seatunnel-cluster.sh -d is running, seatunnel.sh -m local can not run.

matianhe3 avatar Feb 23 '24 06:02 matianhe3

Because local mode startup also requires port 5801

cqutwangyu avatar Feb 23 '24 06:02 cqutwangyu

yes, when you run seatunnel-cluster.sh -d command it will start the server with 5801 port. when you run -e local command again, it will start the server again. so you will got error

liunaijie avatar Mar 15 '24 08:03 liunaijie

yes, when you run seatunnel-cluster.sh -d command it will start the server with 5801 port. when you run -e local command again, it will start the server again. so you will got error

thanks, can i change the hazelcast-client.yaml or hazelcast.yaml config to comply run local and sever both?

matianhe3 avatar Mar 15 '24 08:03 matianhe3

yes, when you run seatunnel-cluster.sh -d command it will start the server with 5801 port. when you run -e local command again, it will start the server again. so you will got error

thanks, can i change the hazelcast-client.yaml or hazelcast.yaml config to comply run local and sever both?

yes, hazelcast.yaml is used on server side. hezalcast-client.yaml is used on client side. you need both update the files. also please use same cluster-name. it will verify the ip, port and cluster-name, all match can be join together.

By the way, if you has run ./bin/seatunnl-cluster.sh -d, then the server cluster is startup, you can use ./bin/seatunnel.sh -c <config_file> command without -e local parameter to submit the job, it will be ok. and if you run ./bin/seatunnel.sh -c <config file> -e local. it will start the server node in this process and the client join the server created in this process.

liunaijie avatar Mar 15 '24 08:03 liunaijie

create an pr to fix this issue https://github.com/apache/seatunnel/pull/6521. in your case, i think it's the server created in local mode port is bigger than 5815. so the client can't connect to the server.

liunaijie avatar Mar 15 '24 09:03 liunaijie