kyma icon indicating copy to clipboard operation
kyma copied to clipboard

Stabilize API-gateway pipeline

Open barchw opened this issue 2 years ago • 1 comments

Description After describing API-gateway pipeline with Gherkin it is apparent that API-gateway pipeline is often failing. We need to investigate issues causing it to fail and stabilise the pipeline.

Observed problems The pipeline failed with many different errors, the most prelevant issue is Istio validation webhook context deadline exceed: https://status.build.kyma-project.io/view/gs/kyma-prow-logs/pr-logs/pull/kyma-project_kyma/14670/pre-main-kyma-gardener-gcp-goats-api-gateway/1540235233620660224

This issue also causes API Rule creation and update to fail sometimes.

"status":{
   "APIRuleStatus":{
      "code":"ERROR",
      "desc":"Internal error occurred: failed calling webhook \"[validation.istio.io](http://validation.istio.io/)\": Post \"[https://istiod.istio-system.svc:443/validate?timeout=10s](https://istiod.istio-system.svc/validate?timeout=10s)\": context deadline exceeded"
   },
   "accessRuleStatus":{
      "code":"ERROR"
   },
   "lastProcessedTime":"2022-07-11T06:45:16Z",
   "observedGeneration":2,
   "virtualServiceStatus":{
      "code":"ERROR"
   }
}

Related issues https://github.com/kyma-project/kyma/issues/14473

barchw avatar Jul 13 '22 11:07 barchw

There is no errors on gardener.

Changed concurrency to 1

Error from server (InternalError): error when creating "/home/prow/go/src/github.com/kyma-project/kyma/ory-hydra-login-consent.yaml": Internal error occurred: failed calling webhook "validation.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/validate?timeout=10s": context deadline exceeded 647 2022-07-29 07:52:03 UTC [ERROR] PID:13 --- [/home/prow/go/src/github.com/kyma-project/test-infra/prow/scripts/lib/gardener/gcp.sh] gardener::cleanup:1 AN ERROR OCCURED! Take a look at preceding log entries.

dariusztutaj avatar Aug 01 '22 07:08 dariusztutaj

since some time pipeline seems to be green. issue dummy PR and test all to confirm it's stable

strekm avatar Aug 18 '22 10:08 strekm