Some Pulsar cluster pods not starting up

[jyoges]

Hi I tried to install pulsar but some pods are failing to start as below. I have attached the error logs as well. Please advise. Thanks.

CR file:

kind: PulsarCluster
metadata:
  name: example-pulsarcluster
spec:
  zookeeper:
    image:
      repository: apachepulsar/pulsar-all
      tag: 2.6.3
      pullPolicy: IfNotPresent
    size: 3
  autoRecovery:
    image:
      repository: apachepulsar/pulsar-all
      tag: 2.6.3
      pullPolicy: IfNotPresent
  bookie:
    image:
      repository: apachepulsar/pulsar-all
      tag: 2.6.3
      pullPolicy: IfNotPresent
    size: 3
  broker:
    image:
      repository: apachepulsar/pulsar-all
      tag: 2.6.3
      pullPolicy: IfNotPresent
    size: 3
  proxy:
    image:
      repository: apachepulsar/pulsar-all
      tag: 2.6.3
      pullPolicy: IfNotPresent
    size: 3
  manager:
    enable: true
    host: manager.pulsar.com
    annotations:
      kubernetes.io/ingress.class: "nginx"

Failing pods

 oc get pods
NAME                                                              READY   STATUS             RESTARTS   AGE
example-pulsarcluster-bookie-autorecovery-deployment-5f9bdkqkkl   0/1     CrashLoopBackOff   8          19m
example-pulsarcluster-bookie-autorecovery-deployment-5f9bdstzp6   0/1     CrashLoopBackOff   8          19m
example-pulsarcluster-bookie-autorecovery-deployment-5f9bdww48g   0/1     CrashLoopBackOff   8          19m
example-pulsarcluster-bookie-statefulset-0                        0/1     CrashLoopBackOff   8          19m
example-pulsarcluster-bookie-statefulset-1                        0/1     CrashLoopBackOff   8          16m
example-pulsarcluster-broker-deployment-76478578c-9tgmm           0/1     CrashLoopBackOff   8          19m
example-pulsarcluster-broker-deployment-76478578c-ssxdv           0/1     CrashLoopBackOff   8          19m
example-pulsarcluster-broker-deployment-76478578c-ttkkr           0/1     CrashLoopBackOff   8          19m
example-pulsarcluster-init-cluster-metadata-job-j65ll             0/1     Completed          0          20m
example-pulsarcluster-manager-deployment-9f8c49d99-rwbgz          1/1     Running            0          19m
example-pulsarcluster-proxy-deployment-85cb9f7666-2f9m6           1/1     Running            0          19m
example-pulsarcluster-proxy-deployment-85cb9f7666-8lnwz           1/1     Running            0          19m
example-pulsarcluster-proxy-deployment-85cb9f7666-mnfsd           1/1     Running            0          19m
example-pulsarcluster-zookeeper-statefulset-0                     1/1     Running            0          25m
example-pulsarcluster-zookeeper-statefulset-1                     1/1     Running            0          23m
example-pulsarcluster-zookeeper-statefulset-2                     1/1     Running            0          22m
pulsar-operator-64c69f896f-5mmms                                  1/1     Running            0          46m

Error log example-pulsarcluster-bookie-autorecovery-deployment-5f9bdkqkkl:

 Applying config zkServers = example-pulsarcluster-zookeeper-service
JMX enabled by default
Error: Could not find or load main class "

example-pulsarcluster-bookie-statefulset-0

 Applying config zkServers = example-pulsarcluster-zookeeper-service
Error: Could not find or load main class "

example-pulsarcluster-broker-deployment-76478578c-9tgmm

000
06:41:36.389 [pulsar-ordered-OrderedExecutor-3-0-EventThread] INFO  org.apache.pulsar.broker.namespace.OwnershipCache - Successfully acquired ownership of /namespace/public/functions/0x40000000_0x80000000
06:41:36.403 [pulsar-ordered-OrderedExecutor-7-0] INFO  org.apache.pulsar.broker.PulsarService - No ledger offloader configured, using NULL instance
06:41:36.430 [pulsar-ordered-OrderedExecutor-7-0] INFO  org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - Opening managed ledger public/functions/persistent/assignments
06:41:36.456 [bookkeeper-ml-workers-OrderedExecutor-0-0] INFO  org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [public/functions/persistent/assignments] Creating ledger, metadata: {component=[109, 97, 110, 97, 103, 101, 100, 45, 108, 101, 100, 103, 101, 114], pulsar/managed-ledger=[112, 117, 98, 108, 105, 99, 47, 102, 117, 110, 99, 116, 105, 111, 110, 115, 47, 112, 101, 114, 115, 105, 115, 116, 101, 110, 116, 47, 97, 115, 115, 105, 103, 110, 109, 101, 110, 116, 115], application=[112, 117, 108, 115, 97, 114]} - metadata ops timeout : 60 seconds
06:41:36.463 [bookkeeper-ml-workers-OrderedExecutor-0-0] WARN  org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to find 1 bookies : excludeBookies [], allBookies [].
06:41:36.470 [bookkeeper-ml-workers-OrderedExecutor-0-0] WARN  org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to find 1 bookies : excludeBookies [], allBookies [].
06:41:36.470 [bookkeeper-ml-workers-OrderedExecutor-0-0] ERROR org.apache.bookkeeper.client.LedgerCreateOp - Not enough bookies to create ledger
06:41:36.471 [bookkeeper-ml-workers-OrderedExecutor-0-0] ERROR org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl - [public/functions/persistent/assignments] Failed to initialize managed ledger: Not enough non-faulty bookies available
06:41:36.472 [bookkeeper-ml-workers-OrderedExecutor-0-0] INFO  org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [public/functions/persistent/assignments] Closing managed ledger
06:41:36.473 [bookkeeper-ml-workers-OrderedExecutor-0-0] WARN  org.apache.pulsar.broker.service.BrokerService - Failed to create topic persistent://public/functions/assignments
org.apache.bookkeeper.mledger.ManagedLedgerException: Not enough non-faulty bookies available
06:41:36.477 [pulsar-web-44-5] ERROR org.apache.pulsar.broker.admin.impl.PersistentTopicsBase - [null] Failed to create non-partitioned topic persistent://public/functions/assignments
java.util.concurrent.CompletionException: org.apache.pulsar.broker.service.BrokerServiceException$PersistenceException: org.apache.bookkeeper.mledger.ManagedLedgerException: Not enough non-faulty bookies available
        at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292) ~[?:1.8.0_282]
        at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308) ~[?:1.8.0_282]
        at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:607) ~[?:1.8.0_282]
        at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:591) ~[?:1.8.0_282]
        at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) ~[?:1.8.0_282]
        at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) ~[?:1.8.0_282]
        at org.apache.pulsar.broker.service.BrokerService$3.openLedgerFailed(BrokerService.java:1086) ~[org.apache.pulsar-pulsar-

[sky-big]

Currently, it can only support the old version of 2.5.0. The image of the new version of Pulsar has been updated. We will follow up the adaptation later。

apiVersion: pulsar.apache.org/v1alpha1
kind: PulsarCluster
metadata:
  name: example-pulsarcluster
spec:
  autoRecovery:
    size: 1
    image:
      repository: apachepulsar/pulsar-all
      tag: 2.5.0
      pullPolicy: Always
  bookie:
    size: 1
    image:
      repository: apachepulsar/pulsar-all
      tag: 2.5.0
      pullPolicy: Always
  broker:
    size: 1
    image:
      repository: apachepulsar/pulsar-all
      tag: 2.5.0
      pullPolicy: Always
  proxy:
    size: 1
    image:
      repository: apachepulsar/pulsar-all
      tag: 2.5.0
      pullPolicy: Always
  zookeeper:
    size: 1
    image:
      repository: apachepulsar/pulsar-all
      tag: 2.5.0
      pullPolicy: Always
  monitor:
    enable: true
    prometheus:
      nodePort: 30006
    grafana:
      nodePort: 30007

@johanra