noobaa install struck in noobaa-default-backing-store-noobaa

[vamseekrishna25]

Environment info

• NooBaa Version: 5.7.0
• Platform: OpenShift 4.7.4

Actual behavior

1.when trying to install noobaa 5.7.0 in ocp cluster getting this error Unable to attach or mount volumes: unmounted volumes=[tmp-logs-vol default-token-v8hdr noobaastorage], unattached volumes=[tmp-logs-vol default-token-v8hdr noobaastorage]: timed out waiting for the condition for noobaa-default-backing-store-noobaa-pod-28da7ba3 continuously terminating and restarting

Expected behavior

1. to install noobaa 5.7.0 in ocp

Steps to reproduce

1.noobaa install --db-storage-class='ibm-spectrum-scale-csi-lt' --db-volume-size-gb=2 --pv-pool-default-storage-class='ibm-spectrum-scale-csi-lt' --image-pull-secret='regcred'

More information - Screenshots / Logs / Other output

#------------------# #- Backing Stores -# #------------------#

NAME TYPE TARGET-BUCKET PHASE AGE noobaa-default-backing-store pv-pool Creating 2m59s

#------------------# #- Bucket Classes -# #------------------#

NAME PLACEMENT NAMESPACE-POLICY PHASE AGE noobaa-default-bucket-class {"tiers":[{"backingStores":["noobaa-default-backing-store"]}]} null Verifying 3m8s

#-----------------# #- Bucket Claims -# #-----------------#

No OBCs found.

oc get pods NAME READY STATUS RESTARTS AGE noobaa-core-0 1/1 Running 0 6m22s noobaa-db-0 1/1 Running 0 6m22s noobaa-default-backing-store-noobaa-pod-28da7ba3 0/1 Terminating 0 2s noobaa-endpoint-5cd69779d9-bmlnl 1/1 Running 0 4m42s noobaa-operator-89bbd5dd6-h8cfb 1/1 Running 0 6m39s

getting this error in noobaa-default-backing-store-noobaa-pod-28da7ba3 pod Unable to attach or mount volumes: unmounted volumes=[tmp-logs-vol default-token-v8hdr noobaastorage], unattached volumes=[tmp-logs-vol default-token-v8hdr noobaastorage]: timed out waiting for the condition

[nimrod-becker]

Seems like a problem in creating/attaching PVCs. Can you create a PV/PVC for another application?

[vamseekrishna25]

in the same namespace of noobaa? or different namespace

[nimrod-becker]

any NS

[vamseekrishna25]

yes I am able to create and attach it to pod no such error has come and even 5.5.0-nsfs works fine but the problem is for above versions like 5.6.0,5.7.0 this some detailed description on the error I got https://github.com/noobaa/noobaa-core/issues/6503

[nimrod-becker]

Can you attach all the logs of the operator POD and the logs of the noobaa-default-backing-store-noobaa-pod-28da7ba3 pod AND a kubectl describe on the noobaa-default-backing-store-noobaa-pod-28da7ba3 pod ?

[vamseekrishna25]

logs of noobaa operator noobaa operator log

logs of noobaa-default-backing-store-noobaa-pod-5e85efad oc logs noobaa-default-backing-store-noobaa-pod-5e85efad Error from server (BadRequest): container "noobaa-agent" in pod "noobaa-default-backing-store-noobaa-pod-5e85efad" is waiting to start: ContainerCreating

oc describe pod noobaa-default-backing-store-noobaa-pod-5e85efad

Name: noobaa-default-backing-store-noobaa-pod-5e85efad Namespace: noobaa Priority: 0 Node: worker2.deeghuge2.cp.fyre.ibm.com/10.17.91.240 Start Time: Sun, 16 May 2021 20:13:37 -0700 Labels: app=noobaa pool=noobaa-default-backing-store Annotations: k8s.v1.cni.cncf.io/network-status: [{ "name": "", "interface": "eth0", "ips": [ "10.254.19.168" ], "default": true, "dns": {} }] k8s.v1.cni.cncf.io/networks-status: [{ "name": "", "interface": "eth0", "ips": [ "10.254.19.168" ], "default": true, "dns": {} }] openshift.io/scc: restricted Status: Terminating (lasts 3d10h) Termination Grace Period: 30s IP: IPs: Controlled By: BackingStore/noobaa-default-backing-store Containers: noobaa-agent: Container ID: Image: noobaa/noobaa-core:5.7.0 Image ID: Port: 60101/TCP Host Port: 0/TCP Command: /noobaa_init_files/noobaa_init.sh agent State: Waiting Reason: ContainerCreating Ready: False Restart Count: 0 Limits: cpu: 100m memory: 400Mi Requests: cpu: 100m memory: 400Mi Environment: CONTAINER_PLATFORM: KUBERNETES AGENT_CONFIG: <set to the key 'AGENT_CONFIG' in secret 'backing-store-pv-pool-noobaa-default-backing-store'> Optional: false Mounts: /noobaa_storage from noobaastorage (rw) /usr/local/noobaa/logs from tmp-logs-vol (rw) /var/run/secrets/kubernetes.io/serviceaccount from default-token-bjr8k (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: tmp-logs-vol: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: noobaastorage: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: noobaa-default-backing-store-noobaa-pvc-5e85efad ReadOnly: false default-token-bjr8k: Type: Secret (a volume populated by a Secret) SecretName: default-token-bjr8k Optional: false QoS Class: Guaranteed Node-Selectors: Tolerations: node.kubernetes.io/memory-pressure:NoSchedule op=Exists node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events:

[steve-delillo]

Seeing the same issue after a successful noobaa manual install (using noobaa binary) on 20210526 build:

[nimrod-becker]

Intermediate builds issue, no longer relevant