fusion-cloud-native
fusion-cloud-native copied to clipboard
Published
20 hours ago •
lucidworks
lucidworks
Installation failed
1. Script generated
sh customize_fusion_values.sh -c kubernetes -n lw5 --provider kubernetes --num-solr 1 --node-pool "{}"
2. Execute the generated script
./kubernetes_kubernetes_lw5_upgrade_fusion.sh
The results of
namespace/lw5 created
Created namespace lw5 with owner label
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "lucidworks" chart repository
...Successfully got an update from the "stable" chart repository
Update Complete. ⎈Happy Helming!⎈
Upgrading the 'lw5' release (Fusion chart: 5.2.0) in the 'lw5' namespace in the 'kubernetes' cluster using values:
kubernetes_kubernetes_lw5_fusion_values.yaml
NOTE: If this will be a long-running cluster for production purposes, you should save the following file(s) in version control:
kubernetes_kubernetes_lw5_fusion_values.yaml
Release "lw5" does not exist. Installing it now.
coalesce.go:199: warning: destination for client is a table. Ignoring non-table value 2181
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
manifest_sorter.go:192: info: skipping unknown hook: "crd-install"
NAME: lw5
LAST DEPLOYED: Sun Sep 13 17:53:04 2020
NAMESPACE: lw5
STATUS: deployed
REVISION: 1
Waiting up to 10 minutes to see the Fusion API Gateway deployment come online ...
Waiting for deployment spec update to be observed...
Waiting for deployment spec update to be observed...
Waiting for deployment "lw5-api-gateway" rollout to finish: 0 of 1 updated replicas are available...
error: timed out waiting for the condition
Waiting up to 5 minutes to see the Fusion Indexing deployment come online ...
Waiting for deployment "lw5-fusion-indexing" rollout to finish: 0 of 1 updated replicas are available...
error: deployment "lw5-fusion-indexing" exceeded its progress deadline
Context "kubernetes-admin@kubernetes" modified.
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
lw5 lw5 1 2020-09-13 17:53:04.223112255 +0800 CST deployed fusion-5.2.0 5.2.0
logs -> kubectl get all -n lw5
NAME READY STATUS RESTARTS AGE
pod/lw5-admin-ui-fc648b4bc-df8k6 1/1 Running 0 16m
pod/lw5-ambassador-5c56f99f85-wb98r 1/1 Running 0 16m
pod/lw5-api-gateway-56cb494746-w9xwt 0/1 Init:CrashLoopBackOff 5 16m
pod/lw5-argo-ui-97688cdd5-ww6t5 1/1 Running 0 16m
pod/lw5-auth-ui-744bf58697-rqmnl 1/1 Running 0 16m
pod/lw5-classic-rest-service-0 0/1 Pending 0 16m
pod/lw5-devops-ui-84cf4bbb9f-d68gc 1/1 Running 0 16m
pod/lw5-fusion-admin-7c66d87d99-dm75c 0/1 Init:CrashLoopBackOff 5 16m
pod/lw5-fusion-indexing-fd7f886b-hsjl9 0/1 Init:CrashLoopBackOff 5 16m
pod/lw5-fusion-log-forwarder-655f65c864-g2tmk 0/1 Init:CrashLoopBackOff 5 16m
pod/lw5-insights-6c4c6f6464-k96wl 1/1 Running 0 16m
pod/lw5-job-launcher-7bfc6d9878-49mn6 0/1 Running 7 16m
pod/lw5-job-rest-server-575685b498-cggtd 0/1 Init:CrashLoopBackOff 5 16m
pod/lw5-ml-model-service-858585f586-cr266 0/2 Init:CrashLoopBackOff 5 16m
pod/lw5-pm-ui-cf68fd6b6-w6s7g 1/1 Running 0 16m
pod/lw5-pulsar-bookkeeper-0 0/1 Pending 0 16m
pod/lw5-pulsar-broker-0 0/1 Init:0/4 0 16m
pod/lw5-pulsar-broker-1 0/1 Init:0/4 0 16m
pod/lw5-query-pipeline-5c44887974-2l5cw 0/1 Init:CrashLoopBackOff 5 16m
pod/lw5-rest-service-6f87f5f488-sf58k 0/1 Init:CrashLoopBackOff 5 16m
pod/lw5-rpc-service-59f4c7c5cb-xx7d4 0/1 Init:CrashLoopBackOff 5 16m
pod/lw5-rules-ui-7d6cc45486-jd6dw 1/1 Running 0 16m
pod/lw5-solr-0 0/1 Pending 0 16m
pod/lw5-solr-exporter-74677cf947-t46nx 0/1 Init:0/1 0 16m
pod/lw5-templating-57c96d65fc-whq8c 0/1 Init:CrashLoopBackOff 5 16m
pod/lw5-webapps-55b64587f8-gbspb 0/1 Init:CrashLoopBackOff 5 16m
pod/lw5-workflow-controller-5b877d7c67-sx4km 1/1 Running 0 16m
pod/lw5-zookeeper-0 0/1 Pending 0 16m
pod/seldon-controller-manager-7b855d7f5c-zs4cr 1/1 Running 0 16m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/admin ClusterIP 10.109.175.238 <none> 8765/TCP 16m
service/admin-ui ClusterIP 10.96.62.2 <none> 8080/TCP 16m
service/auth-ui ClusterIP 10.103.125.149 <none> 8080/TCP 16m
service/connector-plugin-service ClusterIP 10.107.48.157 <none> 9020/TCP 16m
service/connectors ClusterIP 10.97.196.223 <none> 9010/TCP 16m
service/connectors-classic ClusterIP None <none> 9000/TCP 16m
service/connectors-rpc ClusterIP 10.101.196.44 <none> 8771/TCP 16m
service/devops-ui ClusterIP 10.107.250.159 <none> 8080/TCP 16m
service/indexing ClusterIP 10.104.226.98 <none> 8765/TCP 16m
service/insights ClusterIP 10.96.173.133 <none> 8080/TCP 16m
service/job-launcher ClusterIP 10.111.187.156 <none> 8083/TCP 16m
service/job-rest-server ClusterIP 10.97.153.182 <none> 8081/TCP 16m
service/lw5-ambassador ClusterIP 10.102.138.235 <none> 80/TCP,443/TCP 16m
service/lw5-argo-ui ClusterIP 10.96.18.81 <none> 2746/TCP 16m
service/lw5-pulsar-bookkeeper ClusterIP None <none> 3181/TCP,8000/TCP 16m
service/lw5-pulsar-broker ClusterIP None <none> 8080/TCP,6650/TCP 16m
service/lw5-solr-exporter ClusterIP 10.99.38.190 <none> 9983/TCP 16m
service/lw5-solr-headless ClusterIP None <none> 8983/TCP 16m
service/lw5-solr-svc ClusterIP 10.96.213.18 <none> 8983/TCP 16m
service/lw5-zookeeper ClusterIP 10.101.97.38 <none> 2181/TCP,2281/TCP 16m
service/lw5-zookeeper-headless ClusterIP None <none> 2181/TCP,3888/TCP,2888/TCP,2281/TCP 16m
service/ml-model-grpc ClusterIP 10.101.108.200 <none> 6565/TCP 16m
service/ml-model-service ClusterIP 10.108.67.146 <none> 8086/TCP 16m
service/pm-ui ClusterIP 10.100.61.168 <none> 8080/TCP 16m
service/proxy LoadBalancer 10.108.80.194 <pending> 6764:30949/TCP 16m
service/pulsar-broker ClusterIP None <none> 8080/TCP,6650/TCP 16m
service/query ClusterIP 10.110.49.50 <none> 8787/TCP 16m
service/rules-ui ClusterIP 10.110.122.11 <none> 8080/TCP 16m
service/seldon-webhook-service ClusterIP 10.107.47.242 <none> 443/TCP 16m
service/templating ClusterIP 10.108.50.41 <none> 5250/TCP 16m
service/webapps ClusterIP 10.100.239.137 <none> 8780/TCP 16m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/lw5-admin-ui 1/1 1 1 16m
deployment.apps/lw5-ambassador 1/1 1 1 16m
deployment.apps/lw5-api-gateway 0/1 1 0 16m
deployment.apps/lw5-argo-ui 1/1 1 1 16m
deployment.apps/lw5-auth-ui 1/1 1 1 16m
deployment.apps/lw5-connector-plugin-service 0/0 0 0 16m
deployment.apps/lw5-devops-ui 1/1 1 1 16m
deployment.apps/lw5-fusion-admin 0/1 1 0 16m
deployment.apps/lw5-fusion-indexing 0/1 1 0 16m
deployment.apps/lw5-fusion-log-forwarder 0/1 1 0 16m
deployment.apps/lw5-insights 1/1 1 1 16m
deployment.apps/lw5-job-launcher 0/1 1 0 16m
deployment.apps/lw5-job-rest-server 0/1 1 0 16m
deployment.apps/lw5-ml-model-service 0/1 1 0 16m
deployment.apps/lw5-pm-ui 1/1 1 1 16m
deployment.apps/lw5-query-pipeline 0/1 1 0 16m
deployment.apps/lw5-rest-service 0/1 1 0 16m
deployment.apps/lw5-rpc-service 0/1 1 0 16m
deployment.apps/lw5-rules-ui 1/1 1 1 16m
deployment.apps/lw5-solr-exporter 0/1 1 0 16m
deployment.apps/lw5-templating 0/1 1 0 16m
deployment.apps/lw5-webapps 0/1 1 0 16m
deployment.apps/lw5-workflow-controller 1/1 1 1 16m
deployment.apps/seldon-controller-manager 1/1 1 1 16m
NAME DESIRED CURRENT READY AGE
replicaset.apps/lw5-admin-ui-fc648b4bc 1 1 1 16m
replicaset.apps/lw5-ambassador-5c56f99f85 1 1 1 16m
replicaset.apps/lw5-api-gateway-56cb494746 1 1 0 16m
replicaset.apps/lw5-argo-ui-97688cdd5 1 1 1 16m
replicaset.apps/lw5-auth-ui-744bf58697 1 1 1 16m
replicaset.apps/lw5-connector-plugin-service-6787cc8d46 0 0 0 16m
replicaset.apps/lw5-devops-ui-84cf4bbb9f 1 1 1 16m
replicaset.apps/lw5-fusion-admin-7c66d87d99 1 1 0 16m
replicaset.apps/lw5-fusion-indexing-fd7f886b 1 1 0 16m
replicaset.apps/lw5-fusion-log-forwarder-655f65c864 1 1 0 16m
replicaset.apps/lw5-insights-6c4c6f6464 1 1 1 16m
replicaset.apps/lw5-job-launcher-7bfc6d9878 1 1 0 16m
replicaset.apps/lw5-job-rest-server-575685b498 1 1 0 16m
replicaset.apps/lw5-ml-model-service-858585f586 1 1 0 16m
replicaset.apps/lw5-pm-ui-cf68fd6b6 1 1 1 16m
replicaset.apps/lw5-query-pipeline-5c44887974 1 1 0 16m
replicaset.apps/lw5-rest-service-6f87f5f488 1 1 0 16m
replicaset.apps/lw5-rpc-service-59f4c7c5cb 1 1 0 16m
replicaset.apps/lw5-rules-ui-7d6cc45486 1 1 1 16m
replicaset.apps/lw5-solr-exporter-74677cf947 1 1 0 16m
replicaset.apps/lw5-templating-57c96d65fc 1 1 0 16m
replicaset.apps/lw5-webapps-55b64587f8 1 1 0 16m
replicaset.apps/lw5-workflow-controller-5b877d7c67 1 1 1 16m
replicaset.apps/seldon-controller-manager-7b855d7f5c 1 1 1 16m
NAME READY AGE
statefulset.apps/lw5-classic-rest-service 0/1 16m
statefulset.apps/lw5-pulsar-bookkeeper 0/3 16m
statefulset.apps/lw5-pulsar-broker 0/2 16m
statefulset.apps/lw5-solr 0/1 16m
statefulset.apps/lw5-zookeeper 0/3 16m
NAME SCHEDULE SUSPEND ACTIVE LAST SCHEDULE AGE
cronjob.batch/lw5-job-launcher 0 * * * * False 0 10m 16m
cronjob.batch/lw5-job-launcher-spark-cleanup 0 * * * * False 0 10m 16m
Wrong POD log
[root@p45460v chenpeng6]# kubectl get pods -n lw5 | grep CrashLoopBackOff |awk '{print "kubectl logs " $1 " -n lw5"}'|sh -x
+ kubectl logs lw5-api-gateway-56cb494746-w9xwt -n lw5
Error from server (BadRequest): container "api-gateway" in pod "lw5-api-gateway-56cb494746-w9xwt" is waiting to start: PodInitializing
+ kubectl logs lw5-fusion-admin-7c66d87d99-dm75c -n lw5
Error from server (BadRequest): container "admin" in pod "lw5-fusion-admin-7c66d87d99-dm75c" is waiting to start: PodInitializing
+ kubectl logs lw5-fusion-indexing-fd7f886b-hsjl9 -n lw5
Error from server (BadRequest): container "fusion-indexing" in pod "lw5-fusion-indexing-fd7f886b-hsjl9" is waiting to start: PodInitializing
+ kubectl logs lw5-fusion-log-forwarder-655f65c864-g2tmk -n lw5
Error from server (BadRequest): container "fusion-log-forwarder" in pod "lw5-fusion-log-forwarder-655f65c864-g2tmk" is waiting to start: PodInitializing
+ kubectl logs lw5-job-launcher-7bfc6d9878-49mn6 -n lw5
Picked up JAVA_TOOL_OPTIONS: -XX:+ExitOnOutOfMemoryError -Dlogging.config=classpath:logback-kube.xml
Failed to connect to Pulsar topic persistent://lw5/_logs/system_logs at : pulsar://lw5-pulsar-broker:6650 due to: org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.ExecutionException: org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.CompletionException: java.net.UnknownHostException: failed to resolve 'lw5-pulsar-broker' after 2 queries ; will re-try after brief wait ...
+ kubectl logs lw5-job-rest-server-575685b498-cggtd -n lw5
Error from server (BadRequest): container "job-rest-server" in pod "lw5-job-rest-server-575685b498-cggtd" is waiting to start: PodInitializing
+ kubectl logs lw5-ml-model-service-858585f586-cr266 -n lw5
error: a container name must be specified for pod lw5-ml-model-service-858585f586-cr266, choose one of: [java-service python-service] or one of the init containers: [check-admin]
+ kubectl logs lw5-query-pipeline-5c44887974-2l5cw -n lw5
Error from server (BadRequest): container "query-pipeline" in pod "lw5-query-pipeline-5c44887974-2l5cw" is waiting to start: PodInitializing
+ kubectl logs lw5-rest-service-6f87f5f488-sf58k -n lw5
Error from server (BadRequest): container "rest-service" in pod "lw5-rest-service-6f87f5f488-sf58k" is waiting to start: PodInitializing
+ kubectl logs lw5-rpc-service-59f4c7c5cb-xx7d4 -n lw5
Error from server (BadRequest): container "rpc-service" in pod "lw5-rpc-service-59f4c7c5cb-xx7d4" is waiting to start: PodInitializing
+ kubectl logs lw5-templating-57c96d65fc-whq8c -n lw5
Error from server (BadRequest): container "templating" in pod "lw5-templating-57c96d65fc-whq8c" is waiting to start: PodInitializing
+ kubectl logs lw5-webapps-55b64587f8-gbspb -n lw5
Error from server (BadRequest): container "webapps" in pod "lw5-webapps-55b64587f8-gbspb" is waiting to start: PodInitializing
Generate the file
Wrong POD describe
lw6-api-gateway-b756bbcb5-fskl7
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 23m default-scheduler Successfully assigned lw6/lw6-api-gateway-b756bbcb5-fskl7 to p45460v.hulk.shbt.qihoo.net
Warning FailedMount 23m kubelet, p45460v.hulk.shbt.qihoo.net MountVolume.SetUp failed for volume "jks" : couldn't propagate object cache: timed out waiting for the condition
Warning FailedMount 23m kubelet, p45460v.hulk.shbt.qihoo.net MountVolume.SetUp failed for volume "lw6-api-gateway-token-8jj5t" : couldn't propagate object cache: timed out waiting for the condition
Normal Pulled 13m (x5 over 23m) kubelet, p45460v.hulk.shbt.qihoo.net Container image "lucidworks/check-fusion-dependency:v1.2.0" already present on machine
Normal Created 13m (x5 over 23m) kubelet, p45460v.hulk.shbt.qihoo.net Created container check-zk
Normal Started 13m (x5 over 23m) kubelet, p45460v.hulk.shbt.qihoo.net Started container check-zk
Warning BackOff 3m25s (x29 over 19m) kubelet, p45460v.hulk.shbt.qihoo.net Back-off restarting failed container
lw6-fusion-admin-7cc7466458-zzw7n
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 23m default-scheduler Successfully assigned lw6/lw6-fusion-admin-7cc7466458-zzw7n to p45460v.hulk.shbt.qihoo.net
Warning FailedMount 23m kubelet, p45460v.hulk.shbt.qihoo.net MountVolume.SetUp failed for volume "logback-config" : couldn't propagate object cache: timed out waiting for the condition
Warning FailedMount 23m kubelet, p45460v.hulk.shbt.qihoo.net MountVolume.SetUp failed for volume "solr-autoscaling-config" : couldn't propagate object cache: timed out waiting for the condition
Warning FailedMount 23m kubelet, p45460v.hulk.shbt.qihoo.net MountVolume.SetUp failed for volume "lw6-fusion-admin-token-jhwbj" : couldn't propagate object cache: timed out waiting for the condition
Normal Pulled 14m (x5 over 23m) kubelet, p45460v.hulk.shbt.qihoo.net Container image "lucidworks/check-fusion-dependency:v1.2.0" already present on machine
Normal Created 14m (x5 over 23m) kubelet, p45460v.hulk.shbt.qihoo.net Created container check-zk
Normal Started 14m (x5 over 23m) kubelet, p45460v.hulk.shbt.qihoo.net Started container check-zk
Warning BackOff 3m33s (x28 over 19m) kubelet, p45460v.hulk.shbt.qihoo.net Back-off restarting failed container
lw6-fusion-indexing-644568c789-n7lm4
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 23m default-scheduler Successfully assigned lw6/lw6-fusion-indexing-644568c789-n7lm4 to p45460v.hulk.shbt.qihoo.net
Warning FailedMount 23m kubelet, p45460v.hulk.shbt.qihoo.net MountVolume.SetUp failed for volume "lw6-fusion-indexing-token-m8pq9" : couldn't propagate object cache: timed out waiting for the condition
Normal Pulled 14m (x5 over 23m) kubelet, p45460v.hulk.shbt.qihoo.net Container image "lucidworks/check-fusion-dependency:v1.2.0" already present on machine
Normal Created 14m (x5 over 23m) kubelet, p45460v.hulk.shbt.qihoo.net Created container check-zk
Normal Started 14m (x5 over 23m) kubelet, p45460v.hulk.shbt.qihoo.net Started container check-zk
Warning BackOff 3m33s (x29 over 19m) kubelet, p45460v.hulk.shbt.qihoo.net Back-off restarting failed container
lw6-fusion-log-forwarder-68cc895959-dxvmx
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 23m default-scheduler Successfully assigned lw6/lw6-fusion-log-forwarder-68cc895959-dxvmx to p45461v.hulk.shbt.qihoo.net
Normal Pulled 14m (x5 over 23m) kubelet, p45461v.hulk.shbt.qihoo.net Container image "lucidworks/check-fusion-dependency:v1.2.0" already present on machine
Normal Created 14m (x5 over 23m) kubelet, p45461v.hulk.shbt.qihoo.net Created container check-pulsar
Normal Started 14m (x5 over 23m) kubelet, p45461v.hulk.shbt.qihoo.net Started container check-pulsar
Warning BackOff 3m25s (x29 over 19m) kubelet, p45461v.hulk.shbt.qihoo.net Back-off restarting failed container
lw6-job-rest-server-b64f757c7-dz8kh
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 23m default-scheduler Successfully assigned lw6/lw6-job-rest-server-b64f757c7-dz8kh to p45460v.hulk.shbt.qihoo.net
Warning FailedMount 23m kubelet, p45460v.hulk.shbt.qihoo.net MountVolume.SetUp failed for volume "lw6-job-rest-server-token-cx5wt" : couldn't propagate object cache: timed out waiting for the condition
Normal Pulled 13m (x5 over 23m) kubelet, p45460v.hulk.shbt.qihoo.net Container image "lucidworks/check-fusion-dependency:v1.2.0" already present on machine
Normal Created 13m (x5 over 23m) kubelet, p45460v.hulk.shbt.qihoo.net Created container check-zk
Normal Started 13m (x5 over 23m) kubelet, p45460v.hulk.shbt.qihoo.net Started container check-zk
Warning BackOff 3m33s (x28 over 19m) kubelet, p45460v.hulk.shbt.qihoo.net Back-off restarting failed container
lw6-ml-model-service-54bbdd5659-pvcdq
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 23m default-scheduler Successfully assigned lw6/lw6-ml-model-service-54bbdd5659-pvcdq to p45460v.hulk.shbt.qihoo.net
Warning FailedMount 23m kubelet, p45460v.hulk.shbt.qihoo.net MountVolume.SetUp failed for volume "lw6-ml-model-service-token-xf8wd" : couldn't propagate object cache: timed out waiting for the condition
Normal Pulled 13m (x5 over 23m) kubelet, p45460v.hulk.shbt.qihoo.net Container image "lucidworks/check-fusion-dependency:v1.2.0" already present on machine
Normal Created 13m (x5 over 23m) kubelet, p45460v.hulk.shbt.qihoo.net Created container check-admin
Normal Started 13m (x5 over 23m) kubelet, p45460v.hulk.shbt.qihoo.net Started container check-admin
Warning BackOff 3m34s (x27 over 19m) kubelet, p45460v.hulk.shbt.qihoo.net Back-off restarting failed container
lw6-pulsar-broker-0
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 23m default-scheduler Successfully assigned lw6/lw6-pulsar-broker-0 to p45460v.hulk.shbt.qihoo.net
Normal Pulled 23m kubelet, p45460v.hulk.shbt.qihoo.net Container image "apachepulsar/pulsar-all:2.5.2" already present on machine
Normal Created 23m kubelet, p45460v.hulk.shbt.qihoo.net Created container pulsar-bookkeeper-verify-clusterid
Normal Started 23m kubelet, p45460v.hulk.shbt.qihoo.net Started container pulsar-bookkeeper-verify-clusterid
lw6-pulsar-broker-1
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 23m default-scheduler Successfully assigned lw6/lw6-query-pipeline-6c89476654-d9jhq to p45460v.hulk.shbt.qihoo.net
Normal Pulled 14m (x5 over 23m) kubelet, p45460v.hulk.shbt.qihoo.net Container image "lucidworks/check-fusion-dependency:v1.2.0" already present on machine
Normal Created 14m (x5 over 23m) kubelet, p45460v.hulk.shbt.qihoo.net Created container check-zk
Normal Started 14m (x5 over 23m) kubelet, p45460v.hulk.shbt.qihoo.net Started container check-zk
Warning BackOff 3m28s (x29 over 19m) kubelet, p45460v.hulk.shbt.qihoo.net Back-off restarting failed container
lw6-rest-service-58ccf8cbcf-rmhvm
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 23m default-scheduler Successfully assigned lw6/lw6-rest-service-58ccf8cbcf-rmhvm to p45460v.hulk.shbt.qihoo.net
Normal Pulled 14m (x5 over 23m) kubelet, p45460v.hulk.shbt.qihoo.net Container image "lucidworks/check-fusion-dependency:v1.2.0" already present on machine
Normal Created 14m (x5 over 23m) kubelet, p45460v.hulk.shbt.qihoo.net Created container check-zk
Normal Started 14m (x5 over 23m) kubelet, p45460v.hulk.shbt.qihoo.net Started container check-zk
Warning BackOff 3m43s (x29 over 19m) kubelet, p45460v.hulk.shbt.qihoo.net Back-off restarting failed container
lw6-rpc-service-7847674ccb-5vs7v
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 23m default-scheduler Successfully assigned lw6/lw6-rpc-service-7847674ccb-5vs7v to p45461v.hulk.shbt.qihoo.net
Normal Pulled 14m (x5 over 23m) kubelet, p45461v.hulk.shbt.qihoo.net Container image "lucidworks/check-fusion-dependency:v1.2.0" already present on machine
Normal Created 14m (x5 over 23m) kubelet, p45461v.hulk.shbt.qihoo.net Created container check-zk
Normal Started 14m (x5 over 23m) kubelet, p45461v.hulk.shbt.qihoo.net Started container check-zk
Warning BackOff 3m35s (x28 over 19m) kubelet, p45461v.hulk.shbt.qihoo.net Back-off restarting failed container
lw6-solr-exporter-58d8579679-w88zh
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 23m default-scheduler Successfully assigned lw6/lw6-solr-exporter-58d8579679-w88zh to p45461v.hulk.shbt.qihoo.net
Normal Pulled 23m kubelet, p45461v.hulk.shbt.qihoo.net Container image "solr:8.4.1" already present on machine
Normal Created 23m kubelet, p45461v.hulk.shbt.qihoo.net Created container solr-init
Normal Started 23m kubelet, p45461v.hulk.shbt.qihoo.net Started container solr-init
lw6-templating-69cf4867d9-kvzcp
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 23m default-scheduler Successfully assigned lw6/lw6-templating-69cf4867d9-kvzcp to p45461v.hulk.shbt.qihoo.net
Normal Pulled 14m (x5 over 23m) kubelet, p45461v.hulk.shbt.qihoo.net Container image "lucidworks/check-fusion-dependency:v1.2.0" already present on machine
Normal Created 14m (x5 over 23m) kubelet, p45461v.hulk.shbt.qihoo.net Created container check-zk
Normal Started 14m (x5 over 23m) kubelet, p45461v.hulk.shbt.qihoo.net Started container check-zk
Warning BackOff 3m28s (x29 over 19m) kubelet, p45461v.hulk.shbt.qihoo.net Back-off restarting failed container
lw6-webapps-849cfcf887-rf6dn
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 23m default-scheduler Successfully assigned lw6/lw6-webapps-849cfcf887-rf6dn to p45460v.hulk.shbt.qihoo.net
Warning FailedMount 23m kubelet, p45460v.hulk.shbt.qihoo.net MountVolume.SetUp failed for volume "lw6-webapps-token-t5rjd" : couldn't propagate object cache: timed out waiting for the condition
Normal Pulled 14m (x5 over 23m) kubelet, p45460v.hulk.shbt.qihoo.net Container image "lucidworks/check-fusion-dependency:v1.2.0" already present on machine
Normal Created 14m (x5 over 23m) kubelet, p45460v.hulk.shbt.qihoo.net Created container check-zk
Normal Started 14m (x5 over 23m) kubelet, p45460v.hulk.shbt.qihoo.net Started container check-zk
Warning BackOff 3m42s (x28 over 19m) kubelet, p45460v.hulk.shbt.qihoo.net Back-off restarting failed container
Hi @shutter-cp, it looks like the zookeeper pod isn't scheduling and is in a Pending state, can you describe the pod to see why it's stuck in this Pending State.
