ignite icon indicating copy to clipboard operation
ignite copied to clipboard

Ignite client nodes are getting terminated when we stop the ignite server.

Open jinhwan116 opened this issue 1 year ago • 1 comments

ThickClient mode is in use. There is an issue where the client terminates when the ignite server is stopped. The error occurs after the join timeout time has expired. version : 2.16.0

Even if the server is stopped, can't the client continue to wait without being terminated? Similar symptoms were seen in the previous issue https://issues.apache.org/jira/browse/IGNITE-20690

[client node configuration]

igniteConfiguration.setClientMode(true);
igniteConfiguration.setIgniteInstanceName("server_instance");

TcpDiscoverySpi tcpDiscoverySpi = new TcpDiscoverySpi();
tcpDiscoverySpi.setSocketTimeout(10000)
                .setJoinTimeout(60000)
                .setNetworkTimeout(10000);
tcpDiscoverySpi.setMaxAckTimeout(10000);
tcpDiscoverySpi.setReconnectCount(1);

KubernetesConnectionConfiguration kcfg = new KubernetesConnectionConfiguration();
kcfg.setNamespace(namespace);
kcfg.setServiceName(serviceName);

TcpDiscoveryKubernetesIpFinder finder = new TcpDiscoveryKubernetesIpFinder(kcfg);
tcpDiscoverySpi.setIpFinder(finder);

igniteConfiguration.setDiscoverySpi(tcpDiscoverySpi);

TcpCommunicationSpi tcpCommunicationSpi = new TcpCommunicationSpi();
tcpCommunicationSpi.setMessageQueueLimit(1024);
igniteConfiguration.setCommunicationSpi(tcpCommunicationSpi);

CacheConfiguration cacheConfiguration = new CacheConfiguration("IgniteCache");
cacheConfiguration.setDataRegionName("DataStorageRegion");
cacheConfiguration.setAtomicityMode(CacheAtomicityMode.ATOMIC);
cacheConfiguration.setCacheMode(CacheMode.PARTITIONED);
cacheConfiguration.setBackups(1);
cacheConfiguration.setWriteSynchronizationMode(CacheWriteSynchronizationMode.FULL_ASYNC);
cacheConfiguration.setPartitionLossPolicy(PartitionLossPolicy.READ_ONLY_SAFE);
cacheConfiguration.setRebalanceMode(CacheRebalanceMode.ASYNC);
cacheConfiguration.setOnheapCacheEnabled(false);
igniteConfiguration.setCacheConfiguration(cacheConfiguration);

[error]
[2024-11-25 13:47:15.922] [WARN ] [disco-event-worker-#49%server_instance%] Local node SEGMENTED: TcpDiscoveryNode [id=62635b50-c078-42f8-b710-02c6f08462c3, consistentId=fb457a44-d27f-418e-8583-a51f9a26b518, addrs=ArrayList [127.0.0.1, 172.17.0.1, 172.18.0.1, 172.19.0.1, 172.31.207.9], sockAddrs=HashSet [/172.17.0.1:0, /127.0.0.1:0, /172.19.0.1:0, /172.18.0.1:0, /172.31.207.9:0], discPort=0, order=2, intOrder=0, lastExchangeTime=1732509929452, loc=true, ver=2.16.0#20231215-sha1:7bde6a42, isClient=true]
[2024-11-25 13:47:15.925] [ERROR] [disco-event-worker-#49%server_instance%] Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeFailureHandler [super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=SEGMENTATION, err=null]]
[2024-11-25 13:47:15.926] [ERROR] [disco-event-worker-#49%server_instance%] No deadlocked threads detected.
[2024-11-25 13:47:15.961] [ERROR] [disco-event-worker-#49%server_instance%] Thread dump at 2024/11/25 13:47:15 KST```

jinhwan116 avatar Nov 25 '24 05:11 jinhwan116

We are also observing the similar errors in Server Node too . It would be great if some apache experts solve this issue

karthikrsubs avatar Dec 29 '24 12:12 karthikrsubs