solr-operator
solr-operator copied to clipboard
apache
security.json not created in zookeeper
Problem is similar to one reported in https://github.com/apache/solr-operator/issues/720
I tried install Solr Operator and Solr Cloud from scratch.
# secrets
apiVersion: v1
data:
password: <redacted>
username: <redacted>
kind: Secret
metadata:
annotations:
argocd.argoproj.io/tracking-id: solr-9:/Secret:solr-9/solr-9-solrcloud-basic-auth
creationTimestamp: "2025-04-04T20:21:16Z"
name: solr-9-solrcloud-basic-auth
namespace: solr-9
type: kubernetes.io/basic-auth
---
apiVersion: v1
data:
security.json: <BASE 64 ENCODED DATA REDACTED> kind: Secret
metadata:
annotations:
argocd.argoproj.io/tracking-id: solr-9:/Secret:solr-9/solr-9-solrcloud-security-secret-json
labels:
app.kubernetes.io/component: solr
app.kubernetes.io/instance: solr
app.kubernetes.io/name: solr
name: solr-9-solrcloud-security-secret-json
namespace: solr-9
type: Opaque
Security.json file
{
"authentication": {
"class": "solr.MultiAuthPlugin",
"schemes": [{
"scheme": "basic",
"class":"solr.BasicAuthPlugin",
"blockUnknown": true,
"realm": "solr",
"forwardCredentials": false,
"credentials":{
"admin": "<REDACTED>"
, "solr": "<REDACTED>"
, "k8s-oper": "<REDACTED>"
}
}]
},
"authorization":{
"class":"solr.RuleBasedAuthorizationPlugin",
"user-role":{
"admin": ["admin", "k8s"],
"k8s-oper": ["k8s"],
"solr": ["users", "k8s"]
},
"permissions": [
{ "name": "k8s-probe-0", "role":"null", "collection": "null", "path":"/admin/info/system" },
{ "name": "k8s-probe-1", "role":"null", "collection": "null", "path":"/admin/info/health" },
{ "name": "k8s-status", "role":"k8s", "collection": "null", "path":"/admin/collections" },
{ "name": "k8s-metrics", "role":"k8s", "collection": "null", "path":"/admin/metrics" },
{ "name": "k8s-zk", "role":"k8s", "collection": "null", "path":"/admin/zookeeper/status" },
{ "name": "k8s-ping", "role":"k8s", "collection": "*", "path":"/admin/ping" },
{ "name": "k8s-replica-balancing", "role": "k8s", "collection": null, "path": "/____v2/cluster/replicas/balance" },
{ "name": "collection-admin-edit", "role": "k8s" },
{ "name": "read", "role":["admin","users"] },
{ "name": "update", "role":["admin"] },
{ "name": "core-admin-read", "role":["admin"] },
{ "name": "security-read", "role": ["admin"] },
{ "name": "security-edit", "role": ["admin"] },
{ "name": "all", "role":["admin"] }
]
}
}
As one can see Operator PODs come up first, Zookeeper PODs come up next and then Cloud PODs come up.
helm/solr-9 % kubectl get pods -w
NAME READY STATUS RESTARTS AGE
solr-operator-69f974cfc8-tzlcw 0/1 Running 0 2s
solr-operator-zookeeper-operator-79cb947c45-jkprq 1/1 Running 0 2s
solr-operator-69f974cfc8-tzlcw 1/1 Running 0 10s
.
.
helm/solr-9 % kubectl get pods -w
NAME READY STATUS RESTARTS AGE
solr-9-solrcloud-0 0/2 Pending 0 4s
solr-9-solrcloud-1 0/2 Pending 0 4s
solr-9-solrcloud-2 0/2 Pending 0 4s
solr-9-zookeeper-0 0/1 ContainerCreating 0 4s
solr-9-zookeeper-1 0/1 Pending 0 4s
solr-9-zookeeper-2 0/1 Pending 0 4s
solr-operator-69f974cfc8-tzlcw 1/1 Running 0 118s
solr-operator-zookeeper-operator-79cb947c45-jkprq 1/1 Running 0 118s
solr-9-solrcloud-1 0/2 Pending 0 5s
solr-9-solrcloud-1 0/2 Init:0/2 0 5s
solr-9-zookeeper-2 0/1 Pending 0 8s
solr-9-solrcloud-0 0/2 Pending 0 8s
solr-9-zookeeper-2 0/1 ContainerCreating 0 8s
solr-9-solrcloud-0 0/2 Init:0/2 0 8s
solr-9-solrcloud-2 0/2 Pending 0 9s
solr-9-solrcloud-2 0/2 Init:0/2 0 9s
solr-9-zookeeper-0 0/1 Running 0 12s
solr-9-solrcloud-1 0/2 Init:1/2 0 13s
solr-9-zookeeper-2 0/1 Running 0 13s
solr-9-solrcloud-0 0/2 Init:1/2 0 13s
solr-9-solrcloud-1 0/2 Init:1/2 0 14s
solr-9-solrcloud-0 0/2 Init:1/2 0 14s
solr-9-zookeeper-1 0/1 Pending 0 16s
solr-9-zookeeper-1 0/1 ContainerCreating 0 16s
solr-9-solrcloud-2 0/2 Init:1/2 0 17s
solr-9-solrcloud-2 0/2 Init:1/2 0 18s
solr-9-zookeeper-0 1/1 Running 0 22s
solr-9-zookeeper-2 1/1 Running 0 23s
solr-9-zookeeper-1 0/1 Running 0 24s
solr-9-zookeeper-1 1/1 Running 0 34s
solr-9-solrcloud-0 0/2 PodInitializing 0 45s
solr-9-solrcloud-2 0/2 PodInitializing 0 47s
solr-9-solrcloud-0 1/2 Running 0 48s
solr-9-solrcloud-1 0/2 PodInitializing 0 50s
solr-9-solrcloud-2 1/2 Running 0 51s
solr-9-solrcloud-1 1/2 Running 0 55s
solr-9-solrcloud-0 1/2 Running 0 58s
solr-9-solrcloud-0 2/2 Running 0 60s
solr-9-solrcloud-2 1/2 Running 0 63s
solr-9-solrcloud-1 1/2 Running 0 64s
solr-9-solrcloud-2 2/2 Running 0 64s
solr-9-solrcloud-1 2/2 Running 0 66s
.
.
.
##### Zookeeper has empty security.json ###############
I have no name!@solr-9-zookeeper-0:/$ zkCli.sh
/opt/bitnami/java/bin/java
Connecting to localhost:2181
Welcome to ZooKeeper!
JLine support is enabled
WATCHER::
WatchedEvent state:SyncConnected type:None path:null zxid: -1
[zk: localhost:2181(CONNECTED) 0] get /solr/security.json
{}
Slor Cloud POD's setup-zk logs are as below
helm/solr-9 % kubectl logs solr-9-solrcloud-0 -c setup-zk
WARN - 2025-04-04 20:21:36.604; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
WARN - 2025-04-04 20:21:38.097; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
ERROR: KeeperErrorCode = NoNode for /solr
WARN - 2025-04-04 20:21:43.898; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
WARN - 2025-04-04 20:21:45.301; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
Creating ZooKeeper path /solr on ZooKeeper at solr-9-zookeeper-0.solr-9-zookeeper-headless:2181,solr-9-zookeeper-1.solr-9-zookeeper-headless:2181,solr-9-zookeeper-2.solr-9-zookeeper-headless:2181
No security.json found. Put new security.json in ZK
I restarted Slor Cloud PODs (scale down the statefulset and let operator scale the PODs back.). Now setup-zk containers say the below:
helm/solr-9 % kubectl logs solr-9-solrcloud-0 -c setup-zk
WARN - 2025-04-04 22:14:10.453; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
WARN - 2025-04-04 22:14:12.626; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
configs
overseer
aliases.json
live_nodes
collections
overseer_elect
security.json
node_roles
No security.json found. Put new security.json in ZK
So, the only solution is manually add security.json because the operator never creates default one.
Please share with us the exact SolrClouds manifest you use to bootstrap the cluster, along with the version you use of the operator and solr.
