[Regression] security.json is not uploaded during the first initialization of SolrCloud

[erwanval]

This is a regression caused by #660.

During the first initialization of SolrCloud, the security.json doesn't exists in Zookeeper, which cause an exception during the setup-zk initContainer zkcli.sh command With the change introduced in above PR, when there is an error, it now completely skip the upload. Then the SolR just starts with no security.json at all. It seems an empty one is created during the main container initialization, so upon manual restart (or if another Solr pod takes longer to start), setup-zk run as expected and the real security.json is uploaded.

I'm not sure how to solve this while retaining the reason why this change has been introduced in the first place. Maybe the zkcli.sh could return a different error code depending on the kind of exception it encounters (missing security.json, or everything else)?

setup-zk logs during first init:

ERROR: KeeperErrorCode = NoNode for /solr/solr

Creating ZooKeeper path /solr/solr on ZooKeeper at zookeeper.solr.svc:2181 INFO - 2024-08-23 08:25:26.748; org.apache.solr.common.cloud.ConnectionManager; Waiting for client to connect to ZooKeeper INFO - 2024-08-23 08:25:26.786; org.apache.solr.common.cloud.ConnectionManager; zkClient has connected INFO - 2024-08-23 08:25:26.786; org.apache.solr.common.cloud.ConnectionManager; Client is connected to ZooKeeper Exception in thread "main" org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /security.json at org.apache.zookeeper.KeeperException.create(KeeperException.java:118) at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:2358) at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$6(SolrZkClient.java:349) at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:79) at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:349) at org.apache.solr.cloud.ZkCLI.main(ZkCLI.java:328)