CLI install from fresh takes 400 seconds on minikube

[guymguym]

It was ~100 seconds (or less) for 2.0.10:

$ noobaa install --mini
INFO[0000] CLI version: 2.1.0
INFO[0000] noobaa-image: noobaa/noobaa-core:5.3.0
INFO[0000] operator-image: noobaa/noobaa-operator:2.1.0
INFO[0000] Namespace: noobaa
INFO[0000]
INFO[0000] CRD Create:
INFO[0000] ✅ Created: CustomResourceDefinition "noobaas.noobaa.io"
INFO[0000] ✅ Created: CustomResourceDefinition "backingstores.noobaa.io"
INFO[0000] ✅ Created: CustomResourceDefinition "bucketclasses.noobaa.io"
INFO[0000] ✅ Created: CustomResourceDefinition "objectbucketclaims.objectbucket.io"
INFO[0000] ✅ Created: CustomResourceDefinition "objectbuckets.objectbucket.io"
INFO[0000]
INFO[0000] Operator Install:
INFO[0000] ✅ Already Exists: Namespace "noobaa"
INFO[0000] ✅ Created: ServiceAccount "noobaa"
INFO[0000] ✅ Created: Role "noobaa"
INFO[0000] ✅ Created: RoleBinding "noobaa"
INFO[0000] ✅ Created: ClusterRole "noobaa.noobaa.io"
INFO[0000] ✅ Created: ClusterRoleBinding "noobaa.noobaa.io"
INFO[0000] ✅ Created: Deployment "noobaa-operator"
INFO[0000]
INFO[0000] System Create:
INFO[0000] ✅ Already Exists: Namespace "noobaa"
INFO[0000] ✅ Created: NooBaa "noobaa"
INFO[0000]
INFO[0000] NOTE:
INFO[0000]   - This command has finished applying changes to the cluster.
INFO[0000]   - From now on, it only loops and reads the status, to monitor the operator work.
INFO[0000]   - You may Ctrl-C at any time to stop the loop and watch it manually.
INFO[0000]
INFO[0000] System Wait Ready:
INFO[0000] ⏳ System Phase is "". Pod "noobaa-operator-66dbc869fd-q5xmm" is not yet ready: Phase="Pending". ContainersNotReady (containers with unready status: [noobaa-operator]). ContainersNotReady (containers with unready status: [noobaa-operator]).
INFO[0003] ⏳ System Phase is "". Pod "noobaa-operator-66dbc869fd-q5xmm" is not yet ready: Phase="Pending". ContainersNotReady (containers with unready status: [noobaa-operator]). ContainersNotReady (containers with unready status: [noobaa-operator]).
INFO[0006] ⏳ System Phase is "". Pod "noobaa-operator-66dbc869fd-q5xmm" is not yet ready: Phase="Pending". ContainersNotReady (containers with unready status: [noobaa-operator]). ContainersNotReady (containers with unready status: [noobaa-operator]).
INFO[0009] ⏳ System Phase is "". Pod "noobaa-operator-66dbc869fd-q5xmm" is not yet ready: Phase="Pending". ContainersNotReady (containers with unready status: [noobaa-operator]). ContainersNotReady (containers with unready status: [noobaa-operator]).
INFO[0012] ⏳ System Phase is "". Pod "noobaa-operator-66dbc869fd-q5xmm" is not yet ready: Phase="Pending". ContainersNotReady (containers with unready status: [noobaa-operator]). ContainersNotReady (containers with unready status: [noobaa-operator]).
INFO[0015] ⏳ System Phase is "". Pod "noobaa-operator-66dbc869fd-q5xmm" is not yet ready: Phase="Pending". ContainersNotReady (containers with unready status: [noobaa-operator]). ContainersNotReady (containers with unready status: [noobaa-operator]).
INFO[0018] ⏳ System Phase is "". Pod "noobaa-operator-66dbc869fd-q5xmm" is not yet ready: Phase="Pending". ContainersNotReady (containers with unready status: [noobaa-operator]). ContainersNotReady (containers with unready status: [noobaa-operator]).
INFO[0021] ⏳ System Phase is "". Pod "noobaa-operator-66dbc869fd-q5xmm" is not yet ready: Phase="Pending". ContainersNotReady (containers with unready status: [noobaa-operator]). ContainersNotReady (containers with unready status: [noobaa-operator]).
INFO[0024] ⏳ System Phase is "Connecting". Pod "noobaa-core-0" is not yet ready: Phase="Pending". ContainersNotReady (containers with unready status: [core]). ContainersNotReady (containers with unready status: [core]).
INFO[0027] ⏳ System Phase is "Connecting". Pod "noobaa-core-0" is not yet ready: Phase="Pending". ContainersNotReady (containers with unready status: [core]). ContainersNotReady (containers with unready status: [core]).
INFO[0030] ⏳ System Phase is "Connecting". Pod "noobaa-core-0" is not yet ready: Phase="Pending". ContainersNotReady (containers with unready status: [core]). ContainersNotReady (containers with unready status: [core]).
INFO[0033] ⏳ System Phase is "Connecting". Pod "noobaa-core-0" is not yet ready: Phase="Pending". ContainersNotReady (containers with unready status: [core]). ContainersNotReady (containers with unready status: [core]).
INFO[0036] ⏳ System Phase is "Connecting". Pod "noobaa-core-0" is not yet ready: Phase="Pending". ContainersNotReady (containers with unready status: [core]). ContainersNotReady (containers with unready status: [core]).
INFO[0039] ⏳ System Phase is "Connecting". Pod "noobaa-core-0" is not yet ready: Phase="Pending". ContainersNotReady (containers with unready status: [core]). ContainersNotReady (containers with unready status: [core]).
INFO[0042] ⏳ System Phase is "Connecting". Pod "noobaa-core-0" is not yet ready: Phase="Pending". ContainersNotReady (containers with unready status: [core]). ContainersNotReady (containers with unready status: [core]).
INFO[0045] ⏳ System Phase is "Connecting". Pod "noobaa-core-0" is not yet ready: Phase="Pending". ContainersNotReady (containers with unready status: [core]). ContainersNotReady (containers with unready status: [core]).
INFO[0048] ⏳ System Phase is "Connecting". Pod "noobaa-core-0" is not yet ready: Phase="Pending". ContainersNotReady (containers with unready status: [core]). ContainersNotReady (containers with unready status: [core]).
INFO[0051] ⏳ System Phase is "Connecting". Pod "noobaa-core-0" is not yet ready: Phase="Pending". ContainersNotReady (containers with unready status: [core]). ContainersNotReady (containers with unready status: [core]).
INFO[0054] ⏳ System Phase is "Connecting". Pod "noobaa-core-0" is not yet ready: Phase="Pending". ContainersNotReady (containers with unready status: [core]). ContainersNotReady (containers with unready status: [core]).
INFO[0057] ⏳ System Phase is "Connecting". Pod "noobaa-core-0" is not yet ready: Phase="Pending". ContainersNotReady (containers with unready status: [core]). ContainersNotReady (containers with unready status: [core]).
INFO[0060] ⏳ System Phase is "Connecting". Pod "noobaa-core-0" is not yet ready: Phase="Pending". ContainersNotReady (containers with unready status: [core]). ContainersNotReady (containers with unready status: [core]).
INFO[0063] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0066] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0069] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0072] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0075] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0078] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0081] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0084] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0087] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0090] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0093] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0096] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0099] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0102] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0105] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0108] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0111] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0114] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0117] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0120] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0123] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0126] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0129] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0132] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0135] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0138] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0141] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0144] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0147] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0150] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0153] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0156] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0159] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0162] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0165] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0168] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0171] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0174] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0177] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0180] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0183] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0186] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0189] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0192] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0195] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0198] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0201] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0204] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0207] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0210] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0213] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0216] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0219] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0222] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0225] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0228] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0231] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0234] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0237] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0240] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0243] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0246] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0249] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0252] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0255] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0258] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0261] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0264] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0267] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0270] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0273] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0276] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0279] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0282] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0285] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0288] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0291] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0294] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0297] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0300] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0303] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0306] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0309] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0312] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0315] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0318] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0321] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0324] ⏳ System Phase is "Connecting". Waiting for phase ready ...
INFO[0327] ⏳ System Phase is "Configuring". Waiting for phase ready ...
INFO[0330] ⏳ System Phase is "Configuring". Waiting for phase ready ...
INFO[0333] ⏳ System Phase is "Configuring". Waiting for phase ready ...
INFO[0336] ⏳ System Phase is "Configuring". Waiting for phase ready ...
INFO[0339] ⏳ System Phase is "Configuring". Waiting for phase ready ...
INFO[0342] ⏳ System Phase is "Configuring". Waiting for phase ready ...
INFO[0345] ⏳ System Phase is "Configuring". Waiting for phase ready ...
INFO[0348] ⏳ System Phase is "Configuring". Waiting for phase ready ...
INFO[0351] ⏳ System Phase is "Configuring". Waiting for phase ready ...
INFO[0354] ⏳ System Phase is "Configuring". Waiting for phase ready ...
INFO[0357] ⏳ System Phase is "Configuring". Waiting for phase ready ...
INFO[0360] ⏳ System Phase is "Configuring". Waiting for phase ready ...
INFO[0363] ⏳ System Phase is "Configuring". Waiting for phase ready ...
INFO[0366] ⏳ System Phase is "Configuring". Waiting for phase ready ...
INFO[0369] ⏳ System Phase is "Configuring". Waiting for phase ready ...
INFO[0372] ⏳ System Phase is "Configuring". Waiting for phase ready ...
INFO[0375] ⏳ System Phase is "Configuring". Waiting for phase ready ...
INFO[0378] ⏳ System Phase is "Configuring". Waiting for phase ready ...
INFO[0381] ⏳ System Phase is "Configuring". Waiting for phase ready ...
INFO[0384] ⏳ System Phase is "Configuring". Waiting for phase ready ...
INFO[0387] ⏳ System Phase is "Configuring". Waiting for phase ready ...
INFO[0390] ⏳ System Phase is "Configuring". Waiting for phase ready ...
INFO[0393] ⏳ System Phase is "Configuring". Waiting for phase ready ...
INFO[0396] ⏳ System Phase is "Configuring". Waiting for phase ready ...
INFO[0399] ⏳ System Phase is "Configuring". Waiting for phase ready ...
INFO[0402] ⏳ System Phase is "Configuring". Waiting for phase ready ...
INFO[0405] ✅ System Phase is "Ready".
INFO[0405]
INFO[0405]
INFO[0405] CLI version: 2.1.0
INFO[0405] noobaa-image: noobaa/noobaa-core:5.3.0
INFO[0405] operator-image: noobaa/noobaa-operator:2.1.0
INFO[0405] Namespace: noobaa
INFO[0405]
INFO[0405] CRD Status:
INFO[0405] ✅ Exists: CustomResourceDefinition "noobaas.noobaa.io"
INFO[0405] ✅ Exists: CustomResourceDefinition "backingstores.noobaa.io"
INFO[0405] ✅ Exists: CustomResourceDefinition "bucketclasses.noobaa.io"
INFO[0405] ✅ Exists: CustomResourceDefinition "objectbucketclaims.objectbucket.io"
INFO[0405] ✅ Exists: CustomResourceDefinition "objectbuckets.objectbucket.io"
INFO[0405]
INFO[0405] Operator Status:
INFO[0405] ✅ Exists: Namespace "noobaa"
INFO[0405] ✅ Exists: ServiceAccount "noobaa"
INFO[0405] ✅ Exists: Role "noobaa"
INFO[0405] ✅ Exists: RoleBinding "noobaa"
INFO[0405] ✅ Exists: ClusterRole "noobaa.noobaa.io"
INFO[0405] ✅ Exists: ClusterRoleBinding "noobaa.noobaa.io"
INFO[0405] ✅ Exists: Deployment "noobaa-operator"
INFO[0405]
INFO[0405] System Status:
INFO[0405] ✅ Exists: NooBaa "noobaa"
INFO[0405] ✅ Exists: StatefulSet "noobaa-core"
INFO[0405] ✅ Exists: StatefulSet "noobaa-db"
INFO[0405] ✅ Exists: Service "noobaa-mgmt"
INFO[0405] ✅ Exists: Service "s3"
INFO[0405] ✅ Exists: Service "noobaa-db"
INFO[0405] ✅ Exists: Secret "noobaa-server"
INFO[0405] ✅ Exists: Secret "noobaa-operator"
INFO[0405] ✅ Exists: Secret "noobaa-endpoints"
INFO[0405] ✅ Exists: Secret "noobaa-admin"
INFO[0405] ✅ Exists: StorageClass "noobaa.noobaa.io"
INFO[0405] ✅ Exists: BucketClass "noobaa-default-bucket-class"
INFO[0405] ✅ Exists: Deployment "noobaa-endpoint"
INFO[0405] ✅ Exists: HorizontalPodAutoscaler "noobaa-endpoint"
INFO[0405] ⬛ (Optional) Not Found: BackingStore "noobaa-default-backing-store"
INFO[0405] ⬛ (Optional) CRD Unavailable: CredentialsRequest "noobaa-cloud-creds"
INFO[0405] ⬛ (Optional) CRD Unavailable: PrometheusRule "noobaa-prometheus-rules"
INFO[0405] ⬛ (Optional) CRD Unavailable: ServiceMonitor "noobaa-service-monitor"
INFO[0405] ⬛ (Optional) CRD Unavailable: Route "noobaa-mgmt"
INFO[0405] ⬛ (Optional) CRD Unavailable: Route "s3"
INFO[0405] ✅ Exists: PersistentVolumeClaim "db-noobaa-db-0"
INFO[0405] ✅ System Phase is "Ready"
INFO[0405] ✅ Exists:  "noobaa-admin"

#------------------#
#- Mgmt Addresses -#
#------------------#
...

[YiannisGkoufas]

Hi @guymguym what is the minikube version that you tested? Just tested with latest minikube v1.12.0 and noobaa v2.2.0 and it fails because the PVC doesn't get created. I assume there is an API incompatibility as storage-provisioner reports

E0710 08:33:31.705441       1 reflector.go:205] k8s.io/minikube/vendor/github.com/r2d4/external-storage/lib/controller/controller.go:411: Failed to list *v1.PersistentVolumeClaim: v1.PersistentVolumeClaimList: Items: []v1.PersistentVolumeClaim: v1.PersistentVolumeClaim: ObjectMeta: v1.ObjectMeta: readObjectFieldAsBytes: expect : after object field, parsing 1061 ...:{},"k:{\"... at {"kind":"PersistentVolumeClaimList","apiVersion":"v1","metadata":{"selfLink":"/api/v1/persistentvolumeclaims","resourceVersion":"504"},"items":[{"metadata":{"name":"db-noobaa-db-0","namespace":"default","selfLink":"/api/v1/namespaces/default/persistentvolumeclaims/db-noobaa-db-0","uid":"b2dee951-c9c2-435a-b587-2ed37798c6c8","resourceVersion":"504","creationTimestamp":"2020-07-10T08:31:45Z","labels":{"app":"noobaa","noobaa-db":"noobaa"},"annotations":{"volume.beta.kubernetes.io/storage-provisioner":"k8s.io/minikube-hostpath"},"ownerReferences":[{"apiVersion":"noobaa.io/v1alpha1","kind":"NooBaa","name":"noobaa","uid":"6299b045-7c13-48ef-a5a0-ee03d7f24db3","controller":true}],"finalizers":["kubernetes.io/pvc-protection"],"managedFields":[{"manager":"kube-controller-manager","operation":"Update","apiVersion":"v1","time":"2020-07-10T08:31:45Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:volume.beta.kubernetes.io/storage-provisioner":{}},"f:labels":{".":{},"f:app":{},"f:noobaa-db":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"6299b045-7c13-48ef-a5a0-ee03d7f24db3\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:spec":{"f:accessModes":{},"f:resources":{"f:requests":{".":{},"f:storage":{}}},"f:volumeMode":{}},"f:status":{"f:phase":{}}}}]},"spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"50Gi"}},"storageClassName":"standard","volumeMode":"Filesystem"},"status":{"phase":"Pending"}}]}

[guymguym]

Hey @YiannisGkoufas

Thanks, we get that too with kubernetes 1.18, so recently we were setting minikube config set kubernetes-version v1.17.3 to overcome it.

We didn't understand what is the incompatibility root cause - right @liranmauda ?

[YiannisGkoufas]

Got it, will try v1.17.3. This is a relevant discussion: https://github.com/kubernetes/minikube/issues/7218

[YiannisGkoufas]

It seems now that is stuck in:

time="2020-07-10T09:40:23Z" level=info msg="RPC: Connecting websocket (0xc0006a6cd0) &{RPC:0xc00008d090 Address:wss://noobaa-mgmt.default.svc.cluster.local:443/rpc/ State:init WS:<nil> PendingRequests:map[] NextRequestID:0 Lock:{state:9 sema:0} ReconnectDelay:3s}"
time="2020-07-10T09:40:23Z" level=error msg="RPC: closing connection (0xc0006a6cd0) &{RPC:0xc00008d090 Address:wss://noobaa-mgmt.default.svc.cluster.local:443/rpc/ State:init WS:<nil> PendingRequests:map[] NextRequestID:0 Lock:{state:9 sema:0} ReconnectDelay:3s}"
time="2020-07-10T09:40:23Z" level=warning msg="RPC: RemoveConnection wss://noobaa-mgmt.default.svc.cluster.local:443/rpc/ current=0xc0006a6cd0 conn=0xc0006a6cd0"
time="2020-07-10T09:40:23Z" level=error msg="⚠️  RPC: auth.read_auth() Call failed: RPC: connection (0xc0006a6cd0) already closed &{RPC:0xc00008d090 Address:wss://noobaa-mgmt.default.svc.cluster.local:443/rpc/ State:closed WS:<nil> PendingRequests:map[] NextRequestID:0 Lock:{state:1 sema:0} ReconnectDelay:3s}"
time="2020-07-10T09:40:23Z" level=info msg="SetPhase: temporary error during phase \"Connecting\"" sys=default/noobaa
time="2020-07-10T09:40:23Z" level=warning msg="⏳ Temporary Error: RPC: connection (0xc0006a6cd0) already closed &{RPC:0xc00008d090 Address:wss://noobaa-mgmt.default.svc.cluster.local:443/rpc/ State:closed WS:<nil> PendingRequests:map[] NextRequestID:0 Lock:{state:1 sema:0} ReconnectDelay:3s}" sys=default/noobaa
time="2020-07-10T09:40:23Z" level=warning msg="RPC: GetConnection creating connection to wss://noobaa-mgmt.default.svc.cluster.local:443/rpc/ 0xc000639680"
time="2020-07-10T09:40:23Z" level=info msg="RPC: Reconnect (0xc000639680) delay &{RPC:0xc00008d090 Address:wss://noobaa-mgmt.default.svc.cluster.local:443/rpc/ State:init WS:<nil> PendingRequests:map[] NextRequestID:0 Lock:{state:1 sema:0} ReconnectDelay:3s}"
time="2020-07-10T09:40:23Z" level=info msg="UpdateStatus: Done generation 1" sys=default/noobaa
time="2020-07-10T09:40:23Z" level=info msg="Start ..." sys=default/noobaa
time="2020-07-10T09:40:23Z" level=info msg="✅ Exists: NooBaa \"noobaa\"\n"
time="2020-07-10T09:40:24Z" level=info msg="Memory Usage: Phase \"Starting\" - Alloc = 10 MiB  Sys = 71 MiB  NumGC = 55" sys=default/noobaa
time="2020-07-10T09:40:24Z" level=info msg="SetPhase: \"Verifying\"" sys=default/noobaa
time="2020-07-10T09:40:24Z" level=info msg="Parsed image (NamedTagged) noobaa/noobaa-core:5.4.0" func=CheckSystemCR sys=default/noobaa
time="2020-07-10T09:40:24Z" level=info msg="Parsed version \"5.4.0\" from image tag \"5.4.0\"" func=CheckSystemCR sys=default/noobaa
time="2020-07-10T09:40:24Z" level=info msg="SetPhase: \"Creating\"" sys=default/noobaa
time="2020-07-10T09:40:24Z" level=info msg="ReconcileObject: Done - unchanged ServiceAccount /api/v1/namespaces/default/serviceaccounts/noobaa" sys=default/noobaa
time="2020-07-10T09:40:24Z" level=warning msg="Discovery of OAuth endpoints failed, got: Get \"https://openshift.default.svc/.well-known/oauth-authorization-server\": dial tcp: lookup openshift.default.svc on 10.96.0.10:53: no such host" sys=default/noobaa
time="2020-07-10T09:40:24Z" level=info msg="ReconcileObject: Done - unchanged Secret /api/v1/namespaces/default/secrets/noobaa-server" sys=default/noobaa
time="2020-07-10T09:40:24Z" level=info msg="ReconcileObject: Done - updated StatefulSet /apis/apps/v1/namespaces/default/statefulsets/noobaa-core" sys=default/noobaa
time="2020-07-10T09:40:24Z" level=info msg="ReconcileObject: Done - updated StatefulSet /apis/apps/v1/namespaces/default/statefulsets/noobaa-db" sys=default/noobaa
time="2020-07-10T09:40:24Z" level=info msg="ReconcileObject: Done - unchanged Service /api/v1/namespaces/default/services/noobaa-mgmt" sys=default/noobaa
time="2020-07-10T09:40:24Z" level=info msg="ReconcileObject: Done - unchanged Service /api/v1/namespaces/default/services/noobaa-db" sys=default/noobaa
time="2020-07-10T09:40:24Z" level=info msg="ReconcileObject: (Optional) CRD Unavailable: Route \n" sys=default/noobaa
time="2020-07-10T09:40:24Z" level=info msg="ReconcileObject: Done - unchanged Service /api/v1/namespaces/default/services/s3" sys=default/noobaa
time="2020-07-10T09:40:24Z" level=info msg="ReconcileObject: (Optional) CRD Unavailable: Route \n" sys=default/noobaa
time="2020-07-10T09:40:24Z" level=info msg="Not running in AWS. will attempt to create a ceph objectstore user" sys=default/noobaa
time="2020-07-10T09:40:24Z" level=info msg="❌ CRD Missing: CephObjectStoreUser \"noobaa-ceph-objectstore-user\"\n"
time="2020-07-10T09:40:24Z" level=info msg="❌ CRD Missing: : no matches for kind \"CephObjectStore\" in version \"ceph.rook.io/v1\"\n"
time="2020-07-10T09:40:24Z" level=warning msg="failed to list ceph objectstore to use as backing store" sys=default/noobaa
time="2020-07-10T09:40:24Z" level=info msg="SetPhase: \"Connecting\"" sys=default/noobaa
time="2020-07-10T09:40:24Z" level=info msg="Collected addresses: &{NodePorts:[https://172.17.0.3:30910] PodPorts:[https://172.18.0.4:8443] InternalIP:[https://10.104.243.93:443] InternalDNS:[https://noobaa-mgmt.default.svc:443] ExternalIP:[] ExternalDNS:[]}" func=CheckServiceStatus service=noobaa-mgmt sys=default/noobaa
time="2020-07-10T09:40:24Z" level=info msg="✈️  RPC: auth.read_auth() Request: <nil>"

[guymguym]

Wait a couple of minutes, it might be starting up slow like this issue is stating...

[guymguym]

BTW for the storage-provisioner issue with Kubernetes v1.18 we were tracking in this minikube issue - https://github.com/kubernetes/minikube/issues/7828

[YiannisGkoufas]

Was just about to post that it did work in the end :D The bucketclass though is Rejected reporting NooBaa BackingStore "noobaa-default-backing-store" not found or deleted This is the expected behavior in the version 2.2.x ?

[guymguym]

Yes, I think only newer versions we started to create a default pv-pool backing-store. But for now you need to choose your default manually for example:

noobaa backingstore create pv-pool noobaa-default-backing-store --num-volumes=1 --pv-size-gb=100 --storage-class=CLAZZ

[YiannisGkoufas]

Great! Thanks @guymguym !

[YiannisGkoufas]

@guymguym just to confirm that in version 2.3.0 is still significantly slower to start-up compared to 2.0.x versions

[guymguym]

@nb-ohad @YiannisGkoufas - Thanks for the discussion on today's community meeting!

Ohad explained:

The main cause for the hang is waiting for entropy on all of the noobaa process (3 process in core + 1 on the endpoint).

Couple of suggestions were discussed:

1. Guy suggested that --mini will use an empty struct for resources: {} so that both requests and limits are unspecified, and therefore it will be unlimited (though in burstable QOS class). Ohad said there were issues with empty resources struct on QE, such as OOM kills when higher QOS class pods are scheduled causing the noobaa pods to get killed. So Instead - I think this can be added as a separate flag such as --unlimited.
2. Ohad suggested to allow to disable entropy check before seeding RNG (maybe always disable it when using --mini). Guy suggested to just defer the RNG seeding to later once entropy is available without having to peg the CPU in order to generate it on start, but that's a bit of an over kill, so maybe just a disable mode will do.
3. Yiannis suggested to adjust the resource requests in --mini mode according to the available host memory (using info from kubectl get node) so that it can use more mem/cpus if available.