[BUG] Remote Connection Fails to Reconnect When Remote Cluster is Restarted.

[zalseryani]

Describe the bug

• Creating remote cluster connection

PUT _cluster/settings
{
  "persistent": {
    "cluster": {
      "remote": {
        "my-connection-alias": {
          "seeds": ["my-opensearch-domain/service:port"]
        }
      }
    }
  }
}

• After creation, remote status shows its connected,
• Restart remote cluster (prod) and check the remote connection status, you will see that its not connected.

Kindly advise if I am missing something here for such configuration, because in case of a cluster restart, the cross cluster replication stops, and in such case we need to recreate the remote connection since the previous one does not try to reconnect again, and re-initiate the cross-cluster replication (whether it was single index or with autofollow)

Thank you.

Related component

Indexing:Replication

To Reproduce

1- create prod and dr opensearch clusters 2- create remote cluster connection from dr site .

PUT _cluster/settings
{
  "persistent": {
    "cluster": {
      "remote": {
        "my-connection-alias": {
          "seeds": ["my-opensearch-domain/service:port"]
        }
      }
    }
  }
}

3- restart prod site and check the status of the remote connection from the dr site.

GET _remote/info

Expected behavior

• Expected behavior that remote connection from DR to PROD site should try to reconnect every specific time.

• I am not sure if this is configurable currently , as I checked the documentation settings for remote connection or cross-site replication and did not find configuration related to this case.

• Replication settings

Additional Details

• opensearch deployed in kubernetes.
• both prod and dr opensearch clusters version is 2.11.1