Eirini - Runner error: staging task with guid ... failed to start

[troytop]

Kubernetes v1.18.6 (SUSE CaaSP 4.5) KubeCF 2.5.0 with Eirini enabled and sle15 stack. Fresh install.

Trying to push or restage any app results in this error from the log stream:

[API.0] Failed to stage build: Runner error: staging task with guid 83ab0121-ccff-46c7-bc9c-66b7d8a67f99 failed to start

Tailing the logs from the eirini pod shows a continuous stream messages like these:

 {"timestamp":"2020-09-24T21:46:32.685348232Z","level":"debug","source":"handler","message":"handler.get-app-instances.requested","data":{"guid":"093c3436-8151-4e2c-a1fe-84028cba2c1b","session":"1003","version":"72b6e 
 7f5-0fda-4d03-bbf5-9c7880b1ec53"}}                                                                                                                                                                                       
 {"timestamp":"2020-09-24T21:46:32.723144443Z","level":"error","source":"desirer","message":"desirer.get-instance.failed-to-get-statefulset","data":{"error":"not found","guid":"093c3436-8151-4e2c-a1fe-84028cba2c1b","s 
 ession":"1002","version":"72b6e7f5-0fda-4d03-bbf5-9c7880b1ec53"}}                                                                                                                                                        
 {"timestamp":"2020-09-24T21:46:32.723273999Z","level":"error","source":"handler","message":"handler.get-app-instances.bifrost-failed","data":{"error":"failed to get instances for app: not found","guid":"093c3436-8151 
 -4e2c-a1fe-84028cba2c1b","session":"1003","version":"72b6e7f5-0fda-4d03-bbf5-9c7880b1ec53"}}                                                                                                                             

[andreas-kupries]

Failed to reproduce using:

• minikube v1.9.2 on "Opensuse-Leap" "15.1"
• kubernetes 1.17.5 (kubectl 1.15.10)
• KubeCF 2.5.0

Staging is ok, App (goenv) starts. App is accesible, i.e working.

It does take considerably longer than for diego, especially starting the app container after staging. But that seems to be in the design of eirini, at least in part. And in part maybe the limited memory for a minikube.

Regardless, no staging error, no deploy error.

[satadruroy]

KubeCF 2.5 with eirini /AKS/k8s 1.18.6 (with ing controller but don't think that's material) cf push works as expected

[fargozhu]

KubeCF: 2.5 GKE: 1.17.9-gke.1504 Eirini: Enabled

cf push works as well

[satadruroy]

@troytop would you mind running this again on a fresh CaaSP cluster?

[troytop]

I'll have to create a new CaaSP 4.5 cluster. Might take me a bit.

[troytop]

Unable to reproduce on a fresh CaaSP 4.5 cluster. Staging appears normal when testing with 12factor, though I did see one instance of this error message:

 {"timestamp":"2020-09-28T22:48:57.595851590Z","level":"debug","source":"handler","message":"handler.get-app.requested","data":{"guid":"1f98d8df-16e1-4052-a189-21be029964e7","session":"12","version":"fdd33e8d-8b49-4bcd-b 
 a0f-12b61e29335b"}}                                                                                                                                                                                                         
 {"timestamp":"2020-09-28T22:48:57.614594154Z","level":"error","source":"desirer","message":"desirer.get.failed-to-get-statefulset","data":{"error":"not found","guid":"1f98d8df-16e1-4052-a189-21be029964e7","session":"11" 
 ,"version":"fdd33e8d-8b49-4bcd-ba0f-12b61e29335b"}}                                                                                                                                                                         
 {"timestamp":"2020-09-28T22:48:57.614706048Z","level":"error","source":"handler","message":"handler.get-app.failed-to-get-lrp","data":{"error":"failed to get app: not found","guid":"1f98d8df-16e1-4052-a189-21be029964e7" 
 ,"session":"12","version":"fdd33e8d-8b49-4bcd-ba0f-12b61e29335b"}}                                                                                                                                                          
 {"timestamp":"2020-09-28T22:48:57.654176189Z","level":"debug","source":"handler","message":"handler.desire-app.requested","data":{"app_guid":"1f98d8df-16e1-4052-a189-21be029964e7","guid":"1f98d8df-16e1-4052-a189-21be029               │ 964e7-fdd33e8d-8b49-4bcd-ba0f-12b61e29335b","session":"13","version":"fdd33e8d-8b49-4bcd-ba0f-12b61e29335b"}}