serving icon indicating copy to clipboard operation
serving copied to clipboard

GKE is failing to startup clusters

Open dprotaso opened this issue 1 year ago • 2 comments

GKE has been pretty flakey in the last few weeks - I'm going to open a ticket with them.

Below I'm capturing some projects and cluster name failures

Unhealth nodes during cluster creation

knative-boskos-18 Failed to start - kt2-3b3d6d5d-9158-4004-8277-11035f642-1 us-central1

knative-boskos-31 kt2-9f85b531-9fbd-4623-9053-ec89a6960-1 in us-central1...

knative-boskos-01 kt2-fb544fc1-8092-41d5-bde8-21da43d34-1 in us-central1...

knative-boskos-65 kt2-45561fe6-54b8-4e82-997c-388f420ea-1 in us-central1...

dprotaso avatar Feb 22 '24 22:02 dprotaso

Creating cluster kt2-976cbdbf-d8bf-4f25-98d2-336e3751a-1 in us-central1... knative-boskos-17

https://prow.knative.dev/view/gs/knative-prow/pr-logs/pull/knative_serving/14938/upgrade-tests_serving_main/1760984226880032768

dprotaso avatar Feb 23 '24 14:02 dprotaso

https://prow.knative.dev/view/gs/knative-prow/pr-logs/pull/knative_serving/14937/istio-latest-no-mesh-tls_serving_main/1760984197540876288

knative-boskos-81 kt2-69a9f6d8-0e87-4473-8de4-17240548b-1 in us-central1...

dprotaso avatar Feb 23 '24 14:02 dprotaso

Failure to post stuff to the API server

knative-boskos-76 https://prow.knative.dev/view/gs/knative-prow/pr-logs/pull/knative_serving/14936/istio-latest-no-mesh_serving_main/1761195224429760512

cluster kt2-e0155e07-0f9f-4cba-a70a-5fb96b430-1 in us-central1... sysctl_test.go:37: Error fetching runtime info: an error on the server ("Internal Server Error: "/apis/serving.knative.dev/v1/namespaces/serving-tests/services": the server is currently unable to handle the request") has prevented the request from succeeding (post services.serving.knative.dev)

dprotaso avatar Feb 24 '24 17:02 dprotaso

Created a case here:

https://console.cloud.google.com/support/cases/detail/v2/49969918?authuser=2&project=knative-tests

dprotaso avatar Mar 05 '24 16:03 dprotaso

knative-boskos-37 & kt2-79420f70-e8a0-47d9-beaf-98bff019a-1

https://prow.knative.dev/view/gs/knative-prow/pr-logs/pull/knative_serving/14717/istio-latest-no-mesh-tls_serving_main/1766151873733070848

dprotaso avatar Mar 08 '24 20:03 dprotaso

Latest support update

This email is to let you know that we have continuing with the investigation, and we have found that you are being affected by an issue that has already been reported previously. We have already a team working on getting this situation fixed, and they are working on a rollout for it. I'll keep you posted with the updates on this process.

dprotaso avatar Mar 12 '24 01:03 dprotaso

GKE folks recommend switching regions

https://github.com/knative/hack/pull/373

dprotaso avatar Mar 13 '24 13:03 dprotaso

Note - this will be fixed when we pull in the latest hack changes

dprotaso avatar Mar 13 '24 14:03 dprotaso