stackstorm-k8s
stackstorm-k8s copied to clipboard
Published
20 hours ago •
StackStorm
StackStorm
failed to install stackstorm-ha on kubernetes using helm
More detailed information about failure:
PS C:\Users\user\repo> helm install stackstorm-ha .\stackstorm-ha-0.110.0.tgz --namespace=myspace --debug
install.go:194: [debug] Original chart version: ""
install.go:211: [debug] CHART PATH: C:\Users\user\repo\stackstorm-ha-0.110.0.tgz
client.go:133: [debug] creating 49 resource(s)
client.go:477: [debug] Starting delete for "stackstorm-ha-job-st2-apikey-load" Job
client.go:481: [debug] Ignoring delete failure for "stackstorm-ha-job-st2-apikey-load" batch/v1, Kind=Job: jobs.batch "stackstorm-ha-job-st2-apikey-load" not found
client.go:133: [debug] creating 1 resource(s)
client.go:703: [debug] Watching for changes to Job stackstorm-ha-job-st2-apikey-load with timeout of 5m0s
client.go:731: [debug] Add/Modify event for stackstorm-ha-job-st2-apikey-load: ADDED
client.go:770: [debug] stackstorm-ha-job-st2-apikey-load: Jobs active: 0, jobs failed: 0, jobs succeeded: 0
Error: INSTALLATION FAILED: failed post-install: timed out waiting for the condition
helm.go:84: [debug] failed post-install: timed out waiting for the condition
INSTALLATION FAILED
main.newInstallCmd.func2
helm.sh/helm/v3/cmd/helm/install.go:141
github.com/spf13/cobra.(*Command).execute
github.com/spf13/[email protected]/command.go:916
github.com/spf13/cobra.(*Command).ExecuteC
github.com/spf13/[email protected]/command.go:1044
github.com/spf13/cobra.(*Command).Execute
github.com/spf13/[email protected]/command.go:968
main.main
helm.sh/helm/v3/cmd/helm/helm.go:83
runtime.main
runtime/proc.go:250
runtime.goexit
runtime/asm_amd64.s:1571
PS C:\Users\user\repo>
helm.exe version
version.BuildInfo{Version:"v3.11.2", GitCommit:"912ebc1cd10d38d340f048efaf0abda047c3468e", GitTreeState:"clean", GoVersion:"go1.18.10"}
kubectl.exe version --short
Client Version: v1.25.0
Kustomize Version: v4.5.7
Server Version: v1.20.15
PS C:\Users\user> .\kubectl.exe get all -n myspace
NAME READY STATUS RESTARTS AGE
pod/stackstorm-ha-mongodb-0 1/1 Running 0 8m53s
pod/stackstorm-ha-rabbitmq-0 1/1 Running 0 8m53s
pod/stackstorm-ha-st2actionrunner-5df8dfb44f-6p46k 0/1 CrashLoopBackOff 6 8m53s
pod/stackstorm-ha-st2actionrunner-5df8dfb44f-9bpvl 0/1 CrashLoopBackOff 6 8m53s
pod/stackstorm-ha-st2actionrunner-5df8dfb44f-dlfks 0/1 CrashLoopBackOff 6 8m53s
pod/stackstorm-ha-st2actionrunner-5df8dfb44f-hgjt2 0/1 CrashLoopBackOff 6 8m53s
pod/stackstorm-ha-st2actionrunner-5df8dfb44f-z9jw2 0/1 CrashLoopBackOff 6 8m53s
pod/stackstorm-ha-st2api-767d4f5847-55hpd 0/1 CrashLoopBackOff 6 8m53s
pod/stackstorm-ha-st2api-767d4f5847-kx7nl 0/1 CrashLoopBackOff 6 8m53s
pod/stackstorm-ha-st2auth-578c967cf8-r88lr 0/1 CrashLoopBackOff 6 8m53s
pod/stackstorm-ha-st2rulesengine-5f445b4d98-2n4hz 0/1 CrashLoopBackOff 6 8m53s
pod/stackstorm-ha-st2rulesengine-5f445b4d98-4qvmf 0/1 CrashLoopBackOff 6 8m53s
pod/stackstorm-ha-st2scheduler-ddb4b77b7-6xzsm 0/1 CrashLoopBackOff 6 8m53s
pod/stackstorm-ha-st2scheduler-ddb4b77b7-ndbr7 0/1 CrashLoopBackOff 6 8m53s
pod/stackstorm-ha-st2sensorcontainer-657cb59f7b-rt4c7 0/1 CrashLoopBackOff 6 8m54s
pod/stackstorm-ha-st2timersengine-6dcdbbc94c-nxf27 0/1 CrashLoopBackOff 6 8m53s
pod/stackstorm-ha-st2web-64644cf4c4-dtqmj 1/1 Running 0 8m54s
pod/stackstorm-ha-st2web-64644cf4c4-k669n 1/1 Running 0 8m54s
pod/stackstorm-ha-st2workflowengine-7d59b7c5c7-97lww 0/1 CrashLoopBackOff 6 8m54s
pod/stackstorm-ha-st2workflowengine-7d59b7c5c7-qw5wg 0/1 CrashLoopBackOff 6 8m54s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/stackstorm-ha-mongodb-headless ClusterIP None <none> 27017/TCP 8m55s
service/stackstorm-ha-rabbitmq ClusterIP XX.XX.14.95 <none> 5672/TCP,4369/TCP,25672/TCP,15672/TCP 8m55s
service/stackstorm-ha-rabbitmq-headless ClusterIP None <none> 4369/TCP,5672/TCP,25672/TCP,15672/TCP 8m55s
service/stackstorm-ha-redis ClusterIP XX.XX.232.114 <none> 6379/TCP,26379/TCP 8m55s
service/stackstorm-ha-redis-headless ClusterIP None <none> 6379/TCP,26379/TCP 8m55s
service/stackstorm-ha-st2api ClusterIP XX.XX.192.205 <none> 9101/TCP 8m55s
service/stackstorm-ha-st2auth ClusterIP XX.XX.163.153 <none> 9100/TCP 8m55s
service/stackstorm-ha-st2stream ClusterIP XX.XX.104.135 <none> 9102/TCP 8m55s
service/stackstorm-ha-st2web NodePort XX.XX.83.234 <none> 80:32282/TCP 8m55s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/stackstorm-ha-st2actionrunner 0/5 5 0 8m55s
deployment.apps/stackstorm-ha-st2api 0/2 2 0 8m54s
deployment.apps/stackstorm-ha-st2auth 0/2 1 0 8m54s
deployment.apps/stackstorm-ha-st2client 0/1 0 0 8m54s
deployment.apps/stackstorm-ha-st2garbagecollector 0/1 0 0 8m54s
deployment.apps/stackstorm-ha-st2notifier 0/2 0 0 8m54s
deployment.apps/stackstorm-ha-st2rulesengine 0/2 2 0 8m54s
deployment.apps/stackstorm-ha-st2scheduler 0/2 2 0 8m54s
deployment.apps/stackstorm-ha-st2sensorcontainer 0/1 1 0 8m55s
deployment.apps/stackstorm-ha-st2stream 0/2 0 0 8m54s
deployment.apps/stackstorm-ha-st2timersengine 0/1 1 0 8m54s
deployment.apps/stackstorm-ha-st2web 2/2 2 2 8m55s
deployment.apps/stackstorm-ha-st2workflowengine 0/2 2 0 8m55s
NAME DESIRED CURRENT READY AGE
replicaset.apps/stackstorm-ha-st2actionrunner-5df8dfb44f 5 5 0 8m55s
replicaset.apps/stackstorm-ha-st2api-767d4f5847 2 2 0 8m54s
replicaset.apps/stackstorm-ha-st2auth-578c967cf8 2 1 0 8m54s
replicaset.apps/stackstorm-ha-st2client-c7d56bf9c 1 0 0 8m54s
replicaset.apps/stackstorm-ha-st2garbagecollector-bf8f5bbd6 1 0 0 8m54s
replicaset.apps/stackstorm-ha-st2notifier-76d676748c 2 0 0 8m54s
replicaset.apps/stackstorm-ha-st2rulesengine-5f445b4d98 2 2 0 8m54s
replicaset.apps/stackstorm-ha-st2scheduler-ddb4b77b7 2 2 0 8m54s
replicaset.apps/stackstorm-ha-st2sensorcontainer-657cb59f7b 1 1 0 8m55s
replicaset.apps/stackstorm-ha-st2stream-6b96d4f4 2 0 0 8m54s
replicaset.apps/stackstorm-ha-st2timersengine-6dcdbbc94c 1 1 0 8m54s
replicaset.apps/stackstorm-ha-st2web-64644cf4c4 2 2 2 8m55s
replicaset.apps/stackstorm-ha-st2workflowengine-7d59b7c5c7 2 2 0 8m55s
NAME READY AGE
statefulset.apps/stackstorm-ha-mongodb 0/3 8m54s
statefulset.apps/stackstorm-ha-rabbitmq 0/3 8m54s
statefulset.apps/stackstorm-ha-redis-node 0/3 8m54s
NAME COMPLETIONS DURATION AGE
job.batch/stackstorm-ha-job-st2-apikey-load 0/1 8m53s
PS C:\Users\user>
helm list -n myspace
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
stackstorm-ha myspace 1 2023-03-16 22:23:17.5061257 +0530 IST failed stackstorm-ha-0.110.0 3.8
The problem is with the backends not starting:
NAME READY AGE
statefulset.apps/stackstorm-ha-mongodb 0/3 8m54s
statefulset.apps/stackstorm-ha-rabbitmq 0/3 8m54s
statefulset.apps/stackstorm-ha-redis-node 0/3 8m54s
StackStorm HA relies on external helm charts for MongoDB, RabbitMQ, Redis which are used by the st2 cluster.
You'll need to debug those and find the reason why they're not starting. I'd recommend looking into a storage as that was a frequent issue with the K8s cluster setup.
Once they start normally, stackstorm will start too.
