net-istio-controller crashes with error "Failed to start informers" - "failed to wait for cache at index 8 to sync"

[maylukas]

In what area(s)?

/area networking

What version of Knative?

Knative Serving: 1.15.0 Istio: 1.22.3

Expected Behavior

net-istio-controller should start

Actual Behavior

net-istio-controller fails to start and emits the following console output (I have replaced the actual service names with service(1/2/3):

2024/08/02 07:33:20 Registering 3 clients
2024/08/02 07:33:20 Registering 5 informer factories
2024/08/02 07:33:20 Registering 8 informers
2024/08/02 07:33:20 Registering 2 controllers
{"severity":"DEBUG","timestamp":"2024-08-02T07:33:20.993254057Z","caller":"logging/config.go:116","message":"Successfully created the logger."}
{"severity":"DEBUG","timestamp":"2024-08-02T07:33:20.993411819Z","caller":"logging/config.go:117","message":"Logging level set to: debug"}
{"severity":"INFO","timestamp":"2024-08-02T07:33:20.993728295Z","logger":"net-istio-controller","caller":"profiling/server.go:65","message":"Profiling enabled: false","commit":"a8bc624-dirty"}
{"severity":"INFO","timestamp":"2024-08-02T07:33:21.006166486Z","logger":"net-istio-controller","caller":"leaderelection/context.go:47","message":"Running with Standard leader election","commit":"a8bc624-dirty"}
{"severity":"DEBUG","timestamp":"2024-08-02T07:33:21.018980398Z","logger":"net-istio-controller","caller":"metrics/metrics_worker.go:76","message":"Flushing the existing exporter before setting up the new exporter.","commit":"a8bc624-dirty"}
{"severity":"DEBUG","timestamp":"2024-08-02T07:33:21.019300231Z","logger":"net-istio-controller","caller":"metrics/prometheus_exporter.go:51","message":"Created Prometheus exporter with config: &{knative.dev/net-istio net_istio_controller prometheus 5000000000 <nil>  false 9090 0.0.0.0}. Start the server for Prometheus exporter.","commit":"a8bc624-dirty"}
{"severity":"DEBUG","timestamp":"2024-08-02T07:33:21.019353212Z","logger":"net-istio-controller","caller":"metrics/metrics_worker.go:91","message":"Successfully updated the metrics exporter; old config: <nil>; new config &{knative.dev/net-istio net_istio_controller prometheus 5000000000 <nil>  false 9090 0.0.0.0}","commit":"a8bc624-dirty"}
{"severity":"DEBUG","timestamp":"2024-08-02T07:33:21.019659948Z","logger":"net-istio-controller.istio-ingress-controller","caller":"ingress/controller.go:153","message":"Creating event broadcaster","commit":"a8bc624-dirty","knative.dev/controller":"istio-ingress-controller"}
{"severity":"DEBUG","timestamp":"2024-08-02T07:33:21.020328198Z","logger":"net-istio-controller","caller":"serverlessservice/controller.go:149","message":"Creating event broadcaster","commit":"a8bc624-dirty"}
{"severity":"INFO","timestamp":"2024-08-02T07:33:21.036907198Z","logger":"net-istio-controller","caller":"sharedmain/main.go:282","message":"Starting configuration manager...","commit":"a8bc624-dirty"}
{"severity":"DEBUG","timestamp":"2024-08-02T07:33:21.127445355Z","logger":"net-istio-controller.istio-ingress-controller.config-store","caller":"configmap/store.go:155","message":"ingress config \"config-istio\" config was added or updated: &config.Istio{IngressGateways:[]config.Gateway{config.Gateway{Namespace:\"knative-serving\", Name:\"knative-ingress-gateway\", ServiceURL:\"istio-ingressgateway.istio-system.svc.cluster.local\", LabelSelector:(*v1.LabelSelector)(nil)}}, LocalGateways:[]config.Gateway{config.Gateway{Namespace:\"knative-serving\", Name:\"knative-local-gateway\", ServiceURL:\"knative-local-gateway.istio-system.svc.cluster.local\", LabelSelector:(*v1.LabelSelector)(nil)}}}","commit":"a8bc624-dirty","knative.dev/controller":"istio-ingress-controller"}
{"severity":"DEBUG","timestamp":"2024-08-02T07:33:21.127627405Z","logger":"net-istio-controller.config-store","caller":"configmap/store.go:155","message":"ingress config \"config-istio\" config was added or updated: &config.Istio{IngressGateways:[]config.Gateway{config.Gateway{Namespace:\"knative-serving\", Name:\"knative-ingress-gateway\", ServiceURL:\"istio-ingressgateway.istio-system.svc.cluster.local\", LabelSelector:(*v1.LabelSelector)(nil)}}, LocalGateways:[]config.Gateway{config.Gateway{Namespace:\"knative-serving\", Name:\"knative-local-gateway\", ServiceURL:\"knative-local-gateway.istio-system.svc.cluster.local\", LabelSelector:(*v1.LabelSelector)(nil)}}}","commit":"a8bc624-dirty"}
{"severity":"DEBUG","timestamp":"2024-08-02T07:33:21.128349559Z","logger":"net-istio-controller.istio-ingress-controller.config-store","caller":"configmap/store.go:155","message":"ingress config \"config-network\" config was added or updated: &config.Config{DefaultIngressClass:\"istio.ingress.networking.knative.dev\", DomainTemplate:\"{{.Name}}.{{.Domain}}\", TagTemplate:\"{{.Tag}}-{{.Name}}\", AutoTLS:true, ExternalDomainTLS:true, HTTPProtocol:\"redirected\", DefaultCertificateClass:\"cert-manager.certificate.networking.knative.dev\", NamespaceWildcardCertSelector:(*v1.LabelSelector)(0xc000501000), RolloutDurationSecs:0, AutocreateClusterDomainClaims:false, EnableMeshPodAddressability:false, MeshCompatibilityMode:\"auto\", DefaultExternalScheme:\"http\", InternalEncryption:false, SystemInternalTLS:\"disabled\", ClusterLocalDomainTLS:\"disabled\"}","commit":"a8bc624-dirty","knative.dev/controller":"istio-ingress-controller"}
{"severity":"DEBUG","timestamp":"2024-08-02T07:33:21.130612751Z","logger":"net-istio-controller.config-store","caller":"configmap/store.go:155","message":"ingress config \"config-network\" config was added or updated: &config.Config{DefaultIngressClass:\"istio.ingress.networking.knative.dev\", DomainTemplate:\"{{.Name}}.{{.Domain}}\", TagTemplate:\"{{.Tag}}-{{.Name}}\", AutoTLS:true, ExternalDomainTLS:true, HTTPProtocol:\"redirected\", DefaultCertificateClass:\"cert-manager.certificate.networking.knative.dev\", NamespaceWildcardCertSelector:(*v1.LabelSelector)(0xc000501100), RolloutDurationSecs:0, AutocreateClusterDomainClaims:false, EnableMeshPodAddressability:false, MeshCompatibilityMode:\"auto\", DefaultExternalScheme:\"http\", InternalEncryption:false, SystemInternalTLS:\"disabled\", ClusterLocalDomainTLS:\"disabled\"}","commit":"a8bc624-dirty"}
{"level":"info","ts":1722584001.137617,"logger":"fallback","caller":"injection/injection.go:63","msg":"Starting informers..."}
{"severity":"DEBUG","timestamp":"2024-08-02T07:33:21.219211362Z","logger":"net-istio-controller.istio-ingress-controller","caller":"controller/controller.go:418","message":"Adding to queue functions/service1 (depth: 1)","commit":"a8bc624-dirty","knative.dev/controller":"istio-ingress-controller","knative.dev/controller":"knative.dev.net-istio.pkg.reconciler.ingress.Reconciler","knative.dev/kind":"networking.internal.knative.dev.Ingress","knative.dev/key":"functions/service1"}
{"severity":"DEBUG","timestamp":"2024-08-02T07:33:21.219268903Z","logger":"net-istio-controller.istio-ingress-controller","caller":"controller/controller.go:418","message":"Adding to queue functions/service1 (depth: 1)","commit":"a8bc624-dirty","knative.dev/controller":"istio-ingress-controller","knative.dev/controller":"knative.dev.net-istio.pkg.reconciler.ingress.Reconciler","knative.dev/kind":"networking.internal.knative.dev.Ingress","knative.dev/key":"functions/service1"}
{"severity":"DEBUG","timestamp":"2024-08-02T07:33:21.21930356Z","logger":"net-istio-controller.istio-ingress-controller","caller":"controller/controller.go:418","message":"Adding to queue functions/service2 (depth: 2)","commit":"a8bc624-dirty","knative.dev/controller":"istio-ingress-controller","knative.dev/controller":"knative.dev.net-istio.pkg.reconciler.ingress.Reconciler","knative.dev/kind":"networking.internal.knative.dev.Ingress","knative.dev/key":"functions/service2"}
{"severity":"DEBUG","timestamp":"2024-08-02T07:33:21.219466948Z","logger":"net-istio-controller.istio-ingress-controller","caller":"controller/controller.go:418","message":"Adding to queue functions/service2 (depth: 1)","commit":"a8bc624-dirty","knative.dev/controller":"istio-ingress-controller","knative.dev/controller":"knative.dev.net-istio.pkg.reconciler.ingress.Reconciler","knative.dev/kind":"networking.internal.knative.dev.Ingress","knative.dev/key":"functions/service2"}
{"severity":"DEBUG","timestamp":"2024-08-02T07:33:21.219508296Z","logger":"net-istio-controller.istio-ingress-controller","caller":"controller/controller.go:418","message":"Adding to queue push-notification-gateway/service3 (depth: 2)","commit":"a8bc624-dirty","knative.dev/controller":"istio-ingress-controller","knative.dev/controller":"knative.dev.net-istio.pkg.reconciler.ingress.Reconciler","knative.dev/kind":"networking.internal.knative.dev.Ingress","knative.dev/key":"push-notification-gateway/service3"}
{"severity":"DEBUG","timestamp":"2024-08-02T07:33:21.219623036Z","logger":"net-istio-controller.istio-ingress-controller","caller":"controller/controller.go:418","message":"Adding to queue push-notification-gateway/service3 (depth: 4)","commit":"a8bc624-dirty","knative.dev/controller":"istio-ingress-controller","knative.dev/controller":"knative.dev.net-istio.pkg.reconciler.ingress.Reconciler","knative.dev/kind":"networking.internal.knative.dev.Ingress","knative.dev/key":"push-notification-gateway/service3"}
{"severity":"DEBUG","timestamp":"2024-08-02T07:33:21.414736275Z","logger":"net-istio-controller.istio-ingress-controller","caller":"controller/controller.go:418","message":"Adding to queue functions/service1 (depth: 4)","commit":"a8bc624-dirty","knative.dev/controller":"istio-ingress-controller","knative.dev/controller":"knative.dev.net-istio.pkg.reconciler.ingress.Reconciler","knative.dev/kind":"networking.internal.knative.dev.Ingress","knative.dev/key":"functions/service1"}
{"severity":"DEBUG","timestamp":"2024-08-02T07:33:21.414854426Z","logger":"net-istio-controller.istio-ingress-controller","caller":"controller/controller.go:418","message":"Adding to queue functions/service2 (depth: 5)","commit":"a8bc624-dirty","knative.dev/controller":"istio-ingress-controller","knative.dev/controller":"knative.dev.net-istio.pkg.reconciler.ingress.Reconciler","knative.dev/kind":"networking.internal.knative.dev.Ingress","knative.dev/key":"functions/service2"}
{"severity":"DEBUG","timestamp":"2024-08-02T07:33:21.41501659Z","logger":"net-istio-controller.istio-ingress-controller","caller":"controller/controller.go:418","message":"Adding to queue push-notification-gateway/service3 (depth: 4)","commit":"a8bc624-dirty","knative.dev/controller":"istio-ingress-controller","knative.dev/controller":"knative.dev.net-istio.pkg.reconciler.ingress.Reconciler","knative.dev/kind":"networking.internal.knative.dev.Ingress","knative.dev/key":"push-notification-gateway/service3"}
{"severity":"DEBUG","timestamp":"2024-08-02T07:33:34.220726032Z","logger":"net-istio-controller.istio-ingress-controller","caller":"controller/controller.go:418","message":"Adding to queue functions/service1 (depth: 4)","commit":"a8bc624-dirty","knative.dev/controller":"istio-ingress-controller","knative.dev/controller":"knative.dev.net-istio.pkg.reconciler.ingress.Reconciler","knative.dev/kind":"networking.internal.knative.dev.Ingress","knative.dev/key":"functions/service1"}
{"severity":"DEBUG","timestamp":"2024-08-02T07:33:34.221087337Z","logger":"net-istio-controller.istio-ingress-controller","caller":"controller/controller.go:418","message":"Adding to queue functions/service2 (depth: 5)","commit":"a8bc624-dirty","knative.dev/controller":"istio-ingress-controller","knative.dev/controller":"knative.dev.net-istio.pkg.reconciler.ingress.Reconciler","knative.dev/kind":"networking.internal.knative.dev.Ingress","knative.dev/key":"functions/service2"}
{"severity":"DEBUG","timestamp":"2024-08-02T07:33:34.221493671Z","logger":"net-istio-controller.istio-ingress-controller","caller":"controller/controller.go:418","message":"Adding to queue push-notification-gateway/service3 (depth: 4)","commit":"a8bc624-dirty","knative.dev/controller":"istio-ingress-controller","knative.dev/controller":"knative.dev.net-istio.pkg.reconciler.ingress.Reconciler","knative.dev/kind":"networking.internal.knative.dev.Ingress","knative.dev/key":"push-notification-gateway/service3"}
{"level":"fatal","ts":1722584029.926126,"logger":"fallback","caller":"injection/injection.go:65","msg":"Failed to start informers","error":"failed to wait for cache at index 8 to sync","stacktrace":"knative.dev/pkg/injection.EnableInjectionOrDie.func1\n\tknative.dev/[email protected]/injection/injection.go:65\nknative.dev/pkg/injection/sharedmain.MainWithConfig\n\tknative.dev/[email protected]/injection/sharedmain/main.go:309\nknative.dev/pkg/injection/sharedmain.MainWithContext\n\tknative.dev/[email protected]/injection/sharedmain/main.go:209\nmain.main\n\tknative.dev/net-istio/cmd/controller/main.go:37\nruntime.main\n\truntime/proc.go:271"}

Steps to Reproduce the Problem

Can't really tell - The same Knative configuration applied with Terraform works on our dev cluster but not in our staging cluster