controller-runtime Cache continuously out-of-date / ineffective watches

Hello!

Our org has built an operator scaffolded by Kubebuilder that uses controller-runtime (v0.11.0). For reasons I won't get into (but we're trying to move away from this!), we deploy each custom resource in its own namespace along with its own controller deployment. We have several hundred custom resource / controller deployment pairs in each of our clusters. Each controller is of course restricted to register watches only for its own namespace:

	mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
		// ...
		Namespace:              namespace,
		// ...
	})

We register watches as follows:

// SetupWithManager sets up the controller with the Manager.
func (r *AppianReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		For(&crdv1alpha1.Appian{}).
		Owns(&corev1.ConfigMap{}).
		Owns(&corev1.Secret{}).
		Owns(&corev1.Service{}).
		Owns(&corev1.ServiceAccount{}).
		Owns(&appsv1.Deployment{}).
		Owns(&appsv1.StatefulSet{}).
		Owns(&autoscalingv1.HorizontalPodAutoscaler{}).
		Owns(&batchv1.Job{}).
		Owns(&networkingv1.Ingress{}).
		Owns(&networkingv1.NetworkPolicy{}).
		Owns(&policyv1beta1.PodDisruptionBudget{}).
		Owns(&rbacv1.RoleBinding{}).
		Complete(r)
}

Recently, we've hit a bug in our infrastructure that causes many of the pods transitively owned by the custom resources to crash in each cluster near simultaneously. When this happens, many of the controllers in each cluster attempt to update the status sub-resource of their custom resources near simultaneously. We've seen, however, that many of these updates fail because of HTTP 409 Conflict errors:

{"level":"info","ts":"2022-06-22T22:08:56Z","logger":"controller.appian","msg":"Reconciling Appian","reconciler group":"crd.k8s.appian.com","reconciler kind":"Appian","name":"appian","namespace":"3026266"}
{"level":"info","ts":"2022-06-22T22:08:56Z","logger":"controller.appian","msg":"Updating status for Appian","reconciler group":"crd.k8s.appian.com","reconciler kind":"Appian","name":"appian","namespace":"3026266","from":"Ready","to":"Unready"}
{"level":"error","ts":"2022-06-22T22:08:56Z","logger":"controller.appian","msg":"Reconciler error","reconciler group":"crd.k8s.appian.com","reconciler kind":"Appian","name":"appian","namespace":"3026266","error":"Operation cannot be fulfilled on appians.crd.k8s.appian.com \"appian\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}
{"level":"debug","ts":"2022-06-22T22:08:56Z","logger":"events","msg":"Warning","object":{"kind":"Appian","namespace":"3026266","name":"appian","uid":"b05365d0-9b5e-4caf-a77b-ca568ac0635b","apiVersion":"crd.k8s.appian.com/v1alpha1","resourceVersion":"3386929400"},"reason":"ReconcileError","message":"Operation cannot be fulfilled on appians.crd.k8s.appian.com \"appian\": the object has been modified; please apply your changes to the latest version and try again"}

Normally, these errors wouldn't be a big deal as they'd phase out at the controllers update their caches, however, that doesn't appear to be happening; we've seen these errors persist for hours. This seems to indicate that there is a problem with the controllers updating their caches.

During these period of time, we've also seen the custom resources (which have finalizers) fail to delete because the controllers don't notice that the custom resources have non-zero deletion timestamps - because they're operating on outdated versions of the custom resources.

In both situations, we've seen that editing custom resources doesn't seem to result in watch events being triggered. I'm not familiar with the internals of watches. Do they use a cache similar to client.Reader instances?

In all of the above situations, we've seen that deleting the controller pod resolves the issue - as a new pod comes online with correctly initialized caches.

Please let me know what other information might be useful to get to the root of the issue. Thanks!

Some other interesting tidbits:

We've recently rolled out a new API version, but already had multiple API versions with webhook conversion (though the API versions had the same schema - don't ask 😄)

Jun 22 '22 23:06 robbie-demuth

I used the API server audit logs to get a better feel for what requests each controller actually makes to the API server both normally and when the issue occurs

Normally, each controller frequently updates its lease (I've filtered these logs out (!= "\"name\":\"3c0233a2.k8s.appian.com\"")). Each also watches the custom resources (appians.v1alpha1.crd.k8s.appian.com) and various resources that the controller "owns" every ~10 minutes:

{account_name="sites-customer", source_name="eks-control-plane"} |= "\"logStream\":\"kube-apiserver-audit-" |= "\"logGroup\":\"/aws/eks/cust-01_1-21_us-east-1/cluster\"" |= "6995" |= "\"username\":\"system:serviceaccount:6995:appian-operator-controllers\"" != "\"name\":\"3c0233a2.k8s.appian.com\"" | json | line_format "{{ .message_verb }} {{ .message_objectRef_resource }}.{{ .message_objectRef_apiVersion }}{{ if .message_objectRef_apiGroup }}.{{ .message_objectRef_apiGroup }}{{ end }} {{ .message_objectRef_name }}"

  |   | 2022-06-22 05:00:26 | watch poddisruptionbudgets.v1beta1.policy
  |   | 2022-06-22 05:00:26 | watch poddisruptionbudgets.v1beta1.policy
  |   | 2022-06-22 05:00:53 | watch configmaps.v1
  |   | 2022-06-22 05:00:53 | watch configmaps.v1
  |   | 2022-06-22 05:02:01 | watch deployments.v1.apps
  |   | 2022-06-22 05:02:01 | watch deployments.v1.apps
  |   | 2022-06-22 05:02:21 | watch ingresses.v1.networking.k8s.io
  |   | 2022-06-22 05:02:21 | watch ingresses.v1.networking.k8s.io
  |   | 2022-06-22 05:03:00 | watch services.v1
  |   | 2022-06-22 05:03:00 | watch services.v1
  |   | 2022-06-22 05:03:33 | watch statefulsets.v1.apps
  |   | 2022-06-22 05:03:33 | watch statefulsets.v1.apps
  |   | 2022-06-22 05:03:39 | watch rolebindings.v1.rbac.authorization.k8s.io
  |   | 2022-06-22 05:03:39 | watch rolebindings.v1.rbac.authorization.k8s.io
  |   | 2022-06-22 05:04:19 | watch networkpolicies.v1.networking.k8s.io
  |   | 2022-06-22 05:04:19 | watch networkpolicies.v1.networking.k8s.io
  |   | 2022-06-22 05:04:36 | watch horizontalpodautoscalers.v1.autoscaling
  |   | 2022-06-22 05:04:36 | watch horizontalpodautoscalers.v1.autoscaling
  |   | 2022-06-22 05:04:38 | watch secrets.v1
  |   | 2022-06-22 05:04:38 | watch secrets.v1
  |   | 2022-06-22 05:04:48 | watch serviceaccounts.v1
  |   | 2022-06-22 05:04:48 | watch serviceaccounts.v1
  |   | 2022-06-22 05:05:28 | watch appians.v1alpha1.crd.k8s.appian.com
  |   | 2022-06-22 05:05:28 | watch appians.v1alpha1.crd.k8s.appian.com
  |   | 2022-06-22 05:06:34 | watch jobs.v1.batch
  |   | 2022-06-22 05:06:34 | watch jobs.v1.batch
  |   | 2022-06-22 05:06:35 | watch pods.v1
  |   | 2022-06-22 05:06:35 | watch pods.v1
  |   | 2022-06-22 05:07:22 | watch ingresses.v1.networking.k8s.io
  |   | 2022-06-22 05:07:22 | watch ingresses.v1.networking.k8s.io
  |   | 2022-06-22 05:07:29 | watch deployments.v1.apps
  |   | 2022-06-22 05:07:29 | watch deployments.v1.apps
  |   | 2022-06-22 05:08:04 | watch configmaps.v1
  |   | 2022-06-22 05:08:04 | watch configmaps.v1
  |   | 2022-06-22 05:10:07 | watch statefulsets.v1.apps
  |   | 2022-06-22 05:10:07 | watch statefulsets.v1.apps
  |   | 2022-06-22 05:10:12 | watch poddisruptionbudgets.v1beta1.policy
  |   | 2022-06-22 05:10:12 | watch poddisruptionbudgets.v1beta1.policy
  |   | 2022-06-22 05:11:29 | watch services.v1
  |   | 2022-06-22 05:11:29 | watch services.v1
  |   | 2022-06-22 05:11:29 | watch rolebindings.v1.rbac.authorization.k8s.io
  |   | 2022-06-22 05:11:29 | watch rolebindings.v1.rbac.authorization.k8s.io
  |   | 2022-06-22 05:11:44 | watch networkpolicies.v1.networking.k8s.io
  |   | 2022-06-22 05:11:44 | watch networkpolicies.v1.networking.k8s.io
  |   | 2022-06-22 05:12:09 | watch serviceaccounts.v1
  |   | 2022-06-22 05:12:09 | watch serviceaccounts.v1
  |   | 2022-06-22 05:12:16 | watch horizontalpodautoscalers.v1.autoscaling
  |   | 2022-06-22 05:12:16 | watch horizontalpodautoscalers.v1.autoscaling
  |   | 2022-06-22 05:13:14 | watch appians.v1alpha1.crd.k8s.appian.com
  |   | 2022-06-22 05:13:14 | watch appians.v1alpha1.crd.k8s.appian.com
  |   | 2022-06-22 05:13:19 | watch secrets.v1
  |   | 2022-06-22 05:13:19 | watch secrets.v1
  |   | 2022-06-22 05:13:35 | watch pods.v1
  |   | 2022-06-22 05:13:35 | watch pods.v1
  |   | 2022-06-22 05:14:00 | watch deployments.v1.apps
  |   | 2022-06-22 05:14:00 | watch deployments.v1.apps
  |   | 2022-06-22 05:15:25 | watch jobs.v1.batch
  |   | 2022-06-22 05:15:25 | watch jobs.v1.batch
  |   | 2022-06-22 05:16:37 | watch services.v1
  |   | 2022-06-22 05:16:37 | watch services.v1
  |   | 2022-06-22 05:16:51 | watch configmaps.v1
  |   | 2022-06-22 05:16:51 | watch configmaps.v1
  |   | 2022-06-22 05:17:01 | watch ingresses.v1.networking.k8s.io
  |   | 2022-06-22 05:17:01 | watch ingresses.v1.networking.k8s.io
  |   | 2022-06-22 05:17:02 | watch statefulsets.v1.apps
  |   | 2022-06-22 05:17:02 | watch statefulsets.v1.apps
  |   | 2022-06-22 05:17:28 | watch serviceaccounts.v1
  |   | 2022-06-22 05:17:28 | watch serviceaccounts.v1
  |   | 2022-06-22 05:18:15 | watch poddisruptionbudgets.v1beta1.policy
  |   | 2022-06-22 05:18:15 | watch poddisruptionbudgets.v1beta1.policy
  |   | 2022-06-22 05:18:36 | watch pods.v1
  |   | 2022-06-22 05:18:36 | watch pods.v1
  |   | 2022-06-22 05:19:15 | watch horizontalpodautoscalers.v1.autoscaling
  |   | 2022-06-22 05:19:15 | watch horizontalpodautoscalers.v1.autoscaling
  |   | 2022-06-22 05:19:30 | watch appians.v1alpha1.crd.k8s.appian.com
  |   | 2022-06-22 05:19:30 | watch appians.v1alpha1.crd.k8s.appian.com
  |   | 2022-06-22 05:20:12 | watch rolebindings.v1.rbac.authorization.k8s.io
  |   | 2022-06-22 05:20:12 | watch rolebindings.v1.rbac.authorization.k8s.io
  |   | 2022-06-22 05:20:17 | watch deployments.v1.apps
  |   | 2022-06-22 05:20:17 | watch deployments.v1.apps
  |   | 2022-06-22 05:20:53 | watch networkpolicies.v1.networking.k8s.io
  |   | 2022-06-22 05:20:53 | watch networkpolicies.v1.networking.k8s.io
  |   | 2022-06-22 05:21:43 | watch services.v1
  |   | 2022-06-22 05:21:43 | watch services.v1
  |   | 2022-06-22 05:21:56 | watch secrets.v1
  |   | 2022-06-22 05:21:56 | watch secrets.v1
  |   | 2022-06-22 05:22:15 | watch configmaps.v1
  |   | 2022-06-22 05:22:15 | watch configmaps.v1
  |   | 2022-06-22 05:22:54 | watch jobs.v1.batch
  |   | 2022-06-22 05:22:54 | watch jobs.v1.batch
  |   | 2022-06-22 05:23:34 | watch statefulsets.v1.apps
  |   | 2022-06-22 05:23:34 | watch statefulsets.v1.apps
  |   | 2022-06-22 05:24:18 | watch horizontalpodautoscalers.v1.autoscaling
  |   | 2022-06-22 05:24:18 | watch horizontalpodautoscalers.v1.autoscaling
  |   | 2022-06-22 05:25:03 | watch ingresses.v1.networking.k8s.io
  |   | 2022-06-22 05:25:03 | watch ingresses.v1.networking.k8s.io
  |   | 2022-06-22 05:25:07 | watch appians.v1alpha1.crd.k8s.appian.com
  |   | 2022-06-22 05:25:07 | watch appians.v1alpha1.crd.k8s.appian.com
  |   | 2022-06-22 05:26:14 | watch deployments.v1.apps
  |   | 2022-06-22 05:26:14 | watch deployments.v1.apps
  |   | 2022-06-22 05:26:38 | watch rolebindings.v1.rbac.authorization.k8s.io
  |   | 2022-06-22 05:26:38 | watch rolebindings.v1.rbac.authorization.k8s.io
  |   | 2022-06-22 05:27:00 | watch serviceaccounts.v1
  |   | 2022-06-22 05:27:00 | watch serviceaccounts.v1
  |   | 2022-06-22 05:27:33 | watch pods.v1
  |   | 2022-06-22 05:27:33 | watch pods.v1
  |   | 2022-06-22 05:27:35 | watch networkpolicies.v1.networking.k8s.io
  |   | 2022-06-22 05:27:35 | watch networkpolicies.v1.networking.k8s.io
  |   | 2022-06-22 05:28:10 | watch poddisruptionbudgets.v1beta1.policy
  |   | 2022-06-22 05:28:10 | watch poddisruptionbudgets.v1beta1.policy
  |   | 2022-06-22 05:28:36 | watch services.v1
  |   | 2022-06-22 05:28:36 | watch services.v1
  |   | 2022-06-22 05:29:37 | watch statefulsets.v1.apps
  |   | 2022-06-22 05:29:37 | watch statefulsets.v1.apps
  |   | 2022-06-22 05:30:06 | watch ingresses.v1.networking.k8s.io
  |   | 2022-06-22 05:30:06 | watch ingresses.v1.networking.k8s.io
  |   | 2022-06-22 05:30:10 | watch secrets.v1
  |   | 2022-06-22 05:30:10 | watch secrets.v1
  |   | 2022-06-22 05:30:13 | watch configmaps.v1
  |   | 2022-06-22 05:30:13 | watch configmaps.v1
  |   | 2022-06-22 05:31:57 | watch horizontalpodautoscalers.v1.autoscaling
  |   | 2022-06-22 05:31:57 | watch horizontalpodautoscalers.v1.autoscaling
  |   | 2022-06-22 05:32:14 | watch jobs.v1.batch
  |   | 2022-06-22 05:32:14 | watch jobs.v1.batch
  |   | 2022-06-22 05:32:48 | watch pods.v1
  |   | 2022-06-22 05:32:48 | watch pods.v1
  |   | 2022-06-22 05:33:54 | watch networkpolicies.v1.networking.k8s.io
  |   | 2022-06-22 05:33:54 | watch networkpolicies.v1.networking.k8s.io
  |   | 2022-06-22 05:34:24 | watch appians.v1alpha1.crd.k8s.appian.com
  |   | 2022-06-22 05:34:24 | watch appians.v1alpha1.crd.k8s.appian.com
  |   | 2022-06-22 05:35:20 | watch deployments.v1.apps
  |   | 2022-06-22 05:35:20 | watch deployments.v1.apps
  |   | 2022-06-22 05:35:29 | watch rolebindings.v1.rbac.authorization.k8s.io
  |   | 2022-06-22 05:35:29 | watch rolebindings.v1.rbac.authorization.k8s.io
  |   | 2022-06-22 05:36:01 | watch serviceaccounts.v1
  |   | 2022-06-22 05:36:01 | watch serviceaccounts.v1
  |   | 2022-06-22 05:36:25 | watch ingresses.v1.networking.k8s.io
  |   | 2022-06-22 05:36:25 | watch ingresses.v1.networking.k8s.io
  |   | 2022-06-22 05:36:27 | watch configmaps.v1
  |   | 2022-06-22 05:36:27 | watch configmaps.v1
  |   | 2022-06-22 05:36:52 | watch secrets.v1
  |   | 2022-06-22 05:36:52 | watch secrets.v1
  |   | 2022-06-22 05:36:57 | watch poddisruptionbudgets.v1beta1.policy
  |   | 2022-06-22 05:36:57 | watch poddisruptionbudgets.v1beta1.policy
  |   | 2022-06-22 05:37:16 | watch statefulsets.v1.apps
  |   | 2022-06-22 05:37:16 | watch statefulsets.v1.apps
  |   | 2022-06-22 05:37:19 | watch services.v1
  |   | 2022-06-22 05:37:19 | watch services.v1
  |   | 2022-06-22 05:39:34 | watch horizontalpodautoscalers.v1.autoscaling
  |   | 2022-06-22 05:39:34 | watch horizontalpodautoscalers.v1.autoscaling
  |   | 2022-06-22 05:40:31 | watch rolebindings.v1.rbac.authorization.k8s.io
  |   | 2022-06-22 05:40:31 | watch rolebindings.v1.rbac.authorization.k8s.io
  |   | 2022-06-22 05:40:43 | watch pods.v1
  |   | 2022-06-22 05:40:43 | watch pods.v1
  |   | 2022-06-22 05:41:11 | watch deployments.v1.apps
  |   | 2022-06-22 05:41:11 | watch deployments.v1.apps
  |   | 2022-06-22 05:41:55 | watch jobs.v1.batch
  |   | 2022-06-22 05:41:55 | watch jobs.v1.batch
  |   | 2022-06-22 05:42:23 | watch appians.v1alpha1.crd.k8s.appian.com
  |   | 2022-06-22 05:42:23 | watch appians.v1alpha1.crd.k8s.appian.com
  |   | 2022-06-22 05:42:46 | watch networkpolicies.v1.networking.k8s.io
  |   | 2022-06-22 05:42:46 | watch networkpolicies.v1.networking.k8s.io
  |   | 2022-06-22 05:43:14 | watch ingresses.v1.networking.k8s.io
  |   | 2022-06-22 05:43:14 | watch ingresses.v1.networking.k8s.io
  |   | 2022-06-22 05:43:55 | watch secrets.v1
  |   | 2022-06-22 05:43:55 | watch secrets.v1
  |   | 2022-06-22 05:44:58 | watch statefulsets.v1.apps
  |   | 2022-06-22 05:44:58 | watch statefulsets.v1.apps
  |   | 2022-06-22 05:45:09 | watch poddisruptionbudgets.v1beta1.policy
  |   | 2022-06-22 05:45:09 | watch poddisruptionbudgets.v1beta1.policy
  |   | 2022-06-22 05:45:28 | watch serviceaccounts.v1
  |   | 2022-06-22 05:45:28 | watch serviceaccounts.v1
  |   | 2022-06-22 05:45:52 | watch configmaps.v1
  |   | 2022-06-22 05:45:52 | watch configmaps.v1
  |   | 2022-06-22 05:46:01 | watch rolebindings.v1.rbac.authorization.k8s.io
  |   | 2022-06-22 05:46:01 | watch rolebindings.v1.rbac.authorization.k8s.io
  |   | 2022-06-22 05:46:20 | watch services.v1
  |   | 2022-06-22 05:46:20 | watch services.v1
  |   | 2022-06-22 05:48:14 | watch deployments.v1.apps
  |   | 2022-06-22 05:48:14 | watch deployments.v1.apps
  |   | 2022-06-22 05:48:23 | watch pods.v1
  |   | 2022-06-22 05:48:23 | watch pods.v1
  |   | 2022-06-22 05:49:05 | watch horizontalpodautoscalers.v1.autoscaling
  |   | 2022-06-22 05:49:05 | watch horizontalpodautoscalers.v1.autoscaling
  |   | 2022-06-22 05:49:26 | watch secrets.v1
  |   | 2022-06-22 05:49:26 | watch secrets.v1
  |   | 2022-06-22 05:49:56 | watch jobs.v1.batch
  |   | 2022-06-22 05:49:56 | watch jobs.v1.batch
  |   | 2022-06-22 05:50:53 | watch ingresses.v1.networking.k8s.io
  |   | 2022-06-22 05:50:53 | watch ingresses.v1.networking.k8s.io
  |   | 2022-06-22 05:51:21 | watch poddisruptionbudgets.v1beta1.policy
  |   | 2022-06-22 05:51:21 | watch poddisruptionbudgets.v1beta1.policy
  |   | 2022-06-22 05:51:31 | watch configmaps.v1
  |   | 2022-06-22 05:51:31 | watch configmaps.v1
  |   | 2022-06-22 05:52:10 | watch appians.v1alpha1.crd.k8s.appian.com
  |   | 2022-06-22 05:52:10 | watch appians.v1alpha1.crd.k8s.appian.com
  |   | 2022-06-22 05:52:32 | watch networkpolicies.v1.networking.k8s.io
  |   | 2022-06-22 05:52:32 | watch networkpolicies.v1.networking.k8s.io
  |   | 2022-06-22 05:53:28 | watch deployments.v1.apps
  |   | 2022-06-22 05:53:28 | watch deployments.v1.apps
  |   | 2022-06-22 05:53:50 | watch statefulsets.v1.apps
  |   | 2022-06-22 05:53:50 | watch statefulsets.v1.apps
  |   | 2022-06-22 05:54:04 | watch services.v1
  |   | 2022-06-22 05:54:04 | watch services.v1
  |   | 2022-06-22 05:55:01 | watch rolebindings.v1.rbac.authorization.k8s.io
  |   | 2022-06-22 05:55:01 | watch rolebindings.v1.rbac.authorization.k8s.io
  |   | 2022-06-22 05:55:13 | watch serviceaccounts.v1
  |   | 2022-06-22 05:55:13 | watch serviceaccounts.v1
  |   | 2022-06-22 05:56:28 | watch poddisruptionbudgets.v1beta1.policy
  |   | 2022-06-22 05:56:28 | watch poddisruptionbudgets.v1beta1.policy
  |   | 2022-06-22 05:56:28 | watch horizontalpodautoscalers.v1.autoscaling
  |   | 2022-06-22 05:56:28 | watch horizontalpodautoscalers.v1.autoscaling
  |   | 2022-06-22 05:57:06 | watch secrets.v1
  |   | 2022-06-22 05:57:06 | watch secrets.v1
  |   | 2022-06-22 05:58:03 | watch jobs.v1.batch
  |   | 2022-06-22 05:58:03 | watch jobs.v1.batch
  |   | 2022-06-22 05:58:04 | watch pods.v1
  |   | 2022-06-22 05:58:04 | watch pods.v1
  |   | 2022-06-22 05:59:05 | watch ingresses.v1.networking.k8s.io
  |   | 2022-06-22 05:59:05 | watch ingresses.v1.networking.k8s.io
  |   | 2022-06-22 05:59:10 | watch networkpolicies.v1.networking.k8s.io
  |   | 2022-06-22 05:59:10 | watch networkpolicies.v1.networking.k8s.io
  |   | 2022-06-22 05:59:25 | watch statefulsets.v1.apps
  |   | 2022-06-22 05:59:25 | watch statefulsets.v1.apps

When the issue occurs, each controller frequently updates its lease (I've filtered these logs out (!= "\"name\":\"3c0233a2.k8s.appian.com\"")). Each also watches various resources that the controller "owns" every ~10 minutes. Notably, however, each does not watch the custom resources (appians.v1alpha1.crd.k8s.appian.com):

{account_name="sites-customer", source_name="eks-control-plane"} |= "\"logStream\":\"kube-apiserver-audit-" |= "\"logGroup\":\"/aws/eks/cust-01_1-21_us-east-1/cluster\"" |= "6995" |= "\"username\":\"system:serviceaccount:6995:appian-operator-controllers\"" != "\"name\":\"3c0233a2.k8s.appian.com\"" | json | line_format "{{ .message_verb }} {{ .message_objectRef_resource }}.{{ .message_objectRef_apiVersion }}{{ if .message_objectRef_apiGroup }}.{{ .message_objectRef_apiGroup }}{{ end }} {{ .message_objectRef_name }}"

  |   | 2022-06-22 01:30:46 | watch ingresses.v1.networking.k8s.io Show context
  |   | 2022-06-22 01:30:46 | watch ingresses.v1.networking.k8s.io
  |   | 2022-06-22 01:31:21 | watch statefulsets.v1.apps
  |   | 2022-06-22 01:31:21 | watch statefulsets.v1.apps
  |   | 2022-06-22 01:31:54 | watch horizontalpodautoscalers.v1.autoscaling
  |   | 2022-06-22 01:31:54 | watch horizontalpodautoscalers.v1.autoscaling
  |   | 2022-06-22 01:31:58 | watch pods.v1
  |   | 2022-06-22 01:31:58 | watch pods.v1
  |   | 2022-06-22 01:32:06 | watch poddisruptionbudgets.v1beta1.policy
  |   | 2022-06-22 01:32:06 | watch poddisruptionbudgets.v1beta1.policy
  |   | 2022-06-22 01:33:25 | watch configmaps.v1
  |   | 2022-06-22 01:33:25 | watch configmaps.v1
  |   | 2022-06-22 01:33:29 | watch jobs.v1.batch
  |   | 2022-06-22 01:33:29 | watch jobs.v1.batch
  |   | 2022-06-22 01:35:36 | watch networkpolicies.v1.networking.k8s.io
  |   | 2022-06-22 01:35:36 | watch networkpolicies.v1.networking.k8s.io
  |   | 2022-06-22 01:35:39 | watch rolebindings.v1.rbac.authorization.k8s.io
  |   | 2022-06-22 01:35:39 | watch rolebindings.v1.rbac.authorization.k8s.io
  |   | 2022-06-22 01:35:39 | watch services.v1
  |   | 2022-06-22 01:35:39 | watch services.v1
  |   | 2022-06-22 01:35:52 | watch secrets.v1
  |   | 2022-06-22 01:35:52 | watch secrets.v1
  |   | 2022-06-22 01:36:50 | watch ingresses.v1.networking.k8s.io
  |   | 2022-06-22 01:36:50 | watch ingresses.v1.networking.k8s.io
  |   | 2022-06-22 01:37:41 | watch pods.v1
  |   | 2022-06-22 01:37:41 | watch pods.v1
  |   | 2022-06-22 01:37:54 | watch deployments.v1.apps
  |   | 2022-06-22 01:37:54 | watch deployments.v1.apps
  |   | 2022-06-22 01:38:58 | watch serviceaccounts.v1
  |   | 2022-06-22 01:38:58 | watch serviceaccounts.v1
  |   | 2022-06-22 01:39:03 | watch poddisruptionbudgets.v1beta1.policy
  |   | 2022-06-22 01:39:03 | watch poddisruptionbudgets.v1beta1.policy
  |   | 2022-06-22 01:39:12 | watch statefulsets.v1.apps
  |   | 2022-06-22 01:39:12 | watch statefulsets.v1.apps
  |   | 2022-06-22 01:40:39 | watch services.v1
  |   | 2022-06-22 01:40:39 | watch services.v1
  |   | 2022-06-22 01:40:56 | watch jobs.v1.batch
  |   | 2022-06-22 01:40:56 | watch jobs.v1.batch
  |   | 2022-06-22 01:41:19 | watch networkpolicies.v1.networking.k8s.io
  |   | 2022-06-22 01:41:19 | watch networkpolicies.v1.networking.k8s.io
  |   | 2022-06-22 01:41:30 | watch horizontalpodautoscalers.v1.autoscaling
  |   | 2022-06-22 01:41:30 | watch horizontalpodautoscalers.v1.autoscaling
  |   | 2022-06-22 01:41:38 | watch configmaps.v1
  |   | 2022-06-22 01:41:38 | watch configmaps.v1
  |   | 2022-06-22 01:42:43 | watch rolebindings.v1.rbac.authorization.k8s.io
  |   | 2022-06-22 01:42:43 | watch rolebindings.v1.rbac.authorization.k8s.io
  |   | 2022-06-22 01:43:21 | watch pods.v1
  |   | 2022-06-22 01:43:21 | watch pods.v1
  |   | 2022-06-22 01:43:36 | watch deployments.v1.apps
  |   | 2022-06-22 01:43:36 | watch deployments.v1.apps
  |   | 2022-06-22 01:43:58 | watch ingresses.v1.networking.k8s.io
  |   | 2022-06-22 01:43:58 | watch ingresses.v1.networking.k8s.io
  |   | 2022-06-22 01:44:24 | watch serviceaccounts.v1
  |   | 2022-06-22 01:44:24 | watch serviceaccounts.v1
  |   | 2022-06-22 01:45:45 | watch secrets.v1
  |   | 2022-06-22 01:45:45 | watch secrets.v1
  |   | 2022-06-22 01:46:39 | watch horizontalpodautoscalers.v1.autoscaling
  |   | 2022-06-22 01:46:39 | watch horizontalpodautoscalers.v1.autoscaling
  |   | 2022-06-22 01:46:44 | watch poddisruptionbudgets.v1beta1.policy
  |   | 2022-06-22 01:46:44 | watch poddisruptionbudgets.v1beta1.policy
  |   | 2022-06-22 01:48:13 | watch statefulsets.v1.apps
  |   | 2022-06-22 01:48:13 | watch statefulsets.v1.apps
  |   | 2022-06-22 01:48:40 | watch services.v1
  |   | 2022-06-22 01:48:40 | watch services.v1
  |   | 2022-06-22 01:49:25 | watch deployments.v1.apps
  |   | 2022-06-22 01:49:25 | watch deployments.v1.apps
  |   | 2022-06-22 01:50:14 | watch networkpolicies.v1.networking.k8s.io
  |   | 2022-06-22 01:50:14 | watch networkpolicies.v1.networking.k8s.io
  |   | 2022-06-22 01:50:33 | watch jobs.v1.batch
  |   | 2022-06-22 01:50:33 | watch jobs.v1.batch
  |   | 2022-06-22 01:50:51 | watch configmaps.v1
  |   | 2022-06-22 01:50:51 | watch configmaps.v1
  |   | 2022-06-22 01:50:54 | watch rolebindings.v1.rbac.authorization.k8s.io
  |   | 2022-06-22 01:50:54 | watch rolebindings.v1.rbac.authorization.k8s.io
  |   | 2022-06-22 01:51:59 | watch pods.v1
  |   | 2022-06-22 01:51:59 | watch pods.v1
  |   | 2022-06-22 01:52:22 | watch secrets.v1
  |   | 2022-06-22 01:52:22 | watch secrets.v1
  |   | 2022-06-22 01:53:32 | watch ingresses.v1.networking.k8s.io
  |   | 2022-06-22 01:53:32 | watch ingresses.v1.networking.k8s.io
  |   | 2022-06-22 01:53:33 | watch serviceaccounts.v1
  |   | 2022-06-22 01:53:33 | watch serviceaccounts.v1
  |   | 2022-06-22 01:54:23 | watch services.v1
  |   | 2022-06-22 01:54:23 | watch services.v1
  |   | 2022-06-22 01:55:10 | watch horizontalpodautoscalers.v1.autoscaling
  |   | 2022-06-22 01:55:10 | watch horizontalpodautoscalers.v1.autoscaling
  |   | 2022-06-22 01:55:40 | watch networkpolicies.v1.networking.k8s.io
  |   | 2022-06-22 01:55:40 | watch networkpolicies.v1.networking.k8s.io
  |   | 2022-06-22 01:56:19 | watch poddisruptionbudgets.v1beta1.policy
  |   | 2022-06-22 01:56:19 | watch poddisruptionbudgets.v1beta1.policy
  |   | 2022-06-22 01:57:34 | watch statefulsets.v1.apps
  |   | 2022-06-22 01:57:34 | watch statefulsets.v1.apps
  |   | 2022-06-22 01:58:14 | watch rolebindings.v1.rbac.authorization.k8s.io
  |   | 2022-06-22 01:58:14 | watch rolebindings.v1.rbac.authorization.k8s.io
  |   | 2022-06-22 01:58:15 | watch pods.v1
  |   | 2022-06-22 01:58:15 | watch pods.v1
  |   | 2022-06-22 01:58:48 | watch secrets.v1
  |   | 2022-06-22 01:58:48 | watch secrets.v1
  |   | 2022-06-22 01:58:49 | watch configmaps.v1
  |   | 2022-06-22 01:58:49 | watch configmaps.v1
  |   | 2022-06-22 01:59:00 | watch jobs.v1.batch
  |   | 2022-06-22 01:59:00 | watch jobs.v1.batch
  |   | 2022-06-22 01:59:12 | watch deployments.v1.apps
  |   | 2022-06-22 01:59:12 | watch deployments.v1.apps
  |   | 2022-06-22 02:00:32 | watch horizontalpodautoscalers.v1.autoscaling
  |   | 2022-06-22 02:00:32 | watch horizontalpodautoscalers.v1.autoscaling
  |   | 2022-06-22 02:00:58 | watch serviceaccounts.v1
  |   | 2022-06-22 02:00:58 | watch serviceaccounts.v1
  |   | 2022-06-22 02:01:27 | watch services.v1
  |   | 2022-06-22 02:01:27 | watch services.v1
  |   | 2022-06-22 02:03:21 | watch ingresses.v1.networking.k8s.io
  |   | 2022-06-22 02:03:21 | watch ingresses.v1.networking.k8s.io
  |   | 2022-06-22 02:03:39 | watch rolebindings.v1.rbac.authorization.k8s.io
  |   | 2022-06-22 02:03:39 | watch rolebindings.v1.rbac.authorization.k8s.io
  |   | 2022-06-22 02:03:54 | watch secrets.v1
  |   | 2022-06-22 02:03:54 | watch secrets.v1
  |   | 2022-06-22 02:03:59 | watch networkpolicies.v1.networking.k8s.io
  |   | 2022-06-22 02:03:59 | watch networkpolicies.v1.networking.k8s.io
  |   | 2022-06-22 02:05:27 | watch pods.v1
  |   | 2022-06-22 02:05:27 | watch pods.v1
  |   | 2022-06-22 02:06:15 | watch poddisruptionbudgets.v1beta1.policy
  |   | 2022-06-22 02:06:15 | watch poddisruptionbudgets.v1beta1.policy
  |   | 2022-06-22 02:06:44 | watch jobs.v1.batch
  |   | 2022-06-22 02:06:44 | watch jobs.v1.batch
  |   | 2022-06-22 02:07:16 | watch statefulsets.v1.apps
  |   | 2022-06-22 02:07:16 | watch statefulsets.v1.apps
  |   | 2022-06-22 02:07:23 | watch configmaps.v1
  |   | 2022-06-22 02:07:23 | watch configmaps.v1
  |   | 2022-06-22 02:07:28 | watch serviceaccounts.v1
  |   | 2022-06-22 02:07:28 | watch serviceaccounts.v1
  |   | 2022-06-22 02:08:03 | watch deployments.v1.apps
  |   | 2022-06-22 02:08:03 | watch deployments.v1.apps
  |   | 2022-06-22 02:08:44 | watch services.v1
  |   | 2022-06-22 02:08:44 | watch services.v1
  |   | 2022-06-22 02:09:53 | watch horizontalpodautoscalers.v1.autoscaling
  |   | 2022-06-22 02:09:53 | watch horizontalpodautoscalers.v1.autoscaling
  |   | 2022-06-22 02:10:54 | watch rolebindings.v1.rbac.authorization.k8s.io
  |   | 2022-06-22 02:10:54 | watch rolebindings.v1.rbac.authorization.k8s.io
  |   | 2022-06-22 02:11:21 | watch secrets.v1
  |   | 2022-06-22 02:11:21 | watch secrets.v1
  |   | 2022-06-22 02:11:51 | watch networkpolicies.v1.networking.k8s.io
  |   | 2022-06-22 02:11:51 | watch networkpolicies.v1.networking.k8s.io
  |   | 2022-06-22 02:12:32 | watch statefulsets.v1.apps
  |   | 2022-06-22 02:12:32 | watch statefulsets.v1.apps
  |   | 2022-06-22 02:12:33 | watch poddisruptionbudgets.v1beta1.policy
  |   | 2022-06-22 02:12:33 | watch poddisruptionbudgets.v1beta1.policy
  |   | 2022-06-22 02:12:37 | watch ingresses.v1.networking.k8s.io
  |   | 2022-06-22 02:12:37 | watch ingresses.v1.networking.k8s.io
  |   | 2022-06-22 02:13:01 | watch jobs.v1.batch
  |   | 2022-06-22 02:13:01 | watch jobs.v1.batch
  |   | 2022-06-22 02:14:19 | watch pods.v1
  |   | 2022-06-22 02:14:19 | watch pods.v1
  |   | 2022-06-22 02:14:24 | watch services.v1
  |   | 2022-06-22 02:14:24 | watch services.v1
  |   | 2022-06-22 02:14:50 | watch deployments.v1.apps
  |   | 2022-06-22 02:14:50 | watch deployments.v1.apps
  |   | 2022-06-22 02:15:43 | watch horizontalpodautoscalers.v1.autoscaling
  |   | 2022-06-22 02:15:43 | watch horizontalpodautoscalers.v1.autoscaling
  |   | 2022-06-22 02:16:09 | watch rolebindings.v1.rbac.authorization.k8s.io
  |   | 2022-06-22 02:16:09 | watch rolebindings.v1.rbac.authorization.k8s.io
  |   | 2022-06-22 02:16:57 | watch serviceaccounts.v1
  |   | 2022-06-22 02:16:57 | watch serviceaccounts.v1
  |   | 2022-06-22 02:17:11 | watch configmaps.v1
  |   | 2022-06-22 02:17:11 | watch configmaps.v1
  |   | 2022-06-22 02:19:06 | watch statefulsets.v1.apps
  |   | 2022-06-22 02:19:06 | watch statefulsets.v1.apps
  |   | 2022-06-22 02:19:18 | watch jobs.v1.batch
  |   | 2022-06-22 02:19:18 | watch jobs.v1.batch
  |   | 2022-06-22 02:19:29 | watch secrets.v1
  |   | 2022-06-22 02:19:29 | watch secrets.v1
  |   | 2022-06-22 02:20:19 | watch poddisruptionbudgets.v1beta1.policy
  |   | 2022-06-22 02:20:19 | watch poddisruptionbudgets.v1beta1.policy
  |   | 2022-06-22 02:21:33 | watch networkpolicies.v1.networking.k8s.io
  |   | 2022-06-22 02:21:33 | watch networkpolicies.v1.networking.k8s.io
  |   | 2022-06-22 02:21:36 | watch deployments.v1.apps
  |   | 2022-06-22 02:21:36 | watch deployments.v1.apps
  |   | 2022-06-22 02:21:56 | watch ingresses.v1.networking.k8s.io
  |   | 2022-06-22 02:21:56 | watch ingresses.v1.networking.k8s.io
  |   | 2022-06-22 02:21:58 | watch services.v1
  |   | 2022-06-22 02:21:58 | watch services.v1
  |   | 2022-06-22 02:22:01 | watch serviceaccounts.v1
  |   | 2022-06-22 02:22:01 | watch serviceaccounts.v1
  |   | 2022-06-22 02:22:53 | watch pods.v1
  |   | 2022-06-22 02:22:53 | watch pods.v1
  |   | 2022-06-22 02:23:18 | watch rolebindings.v1.rbac.authorization.k8s.io
  |   | 2022-06-22 02:23:18 | watch rolebindings.v1.rbac.authorization.k8s.io
  |   | 2022-06-22 02:24:33 | watch horizontalpodautoscalers.v1.autoscaling
  |   | 2022-06-22 02:24:33 | watch horizontalpodautoscalers.v1.autoscaling
  |   | 2022-06-22 02:25:11 | watch configmaps.v1
  |   | 2022-06-22 02:25:11 | watch configmaps.v1
  |   | 2022-06-22 02:26:01 | watch poddisruptionbudgets.v1beta1.policy
  |   | 2022-06-22 02:26:01 | watch poddisruptionbudgets.v1beta1.policy
  |   | 2022-06-22 02:26:16 | watch statefulsets.v1.apps
  |   | 2022-06-22 02:26:16 | watch statefulsets.v1.apps
  |   | 2022-06-22 02:27:28 | watch ingresses.v1.networking.k8s.io
  |   | 2022-06-22 02:27:28 | watch ingresses.v1.networking.k8s.io
  |   | 2022-06-22 02:28:07 | watch pods.v1
  |   | 2022-06-22 02:28:07 | watch pods.v1
  |   | 2022-06-22 02:28:52 | watch secrets.v1
  |   | 2022-06-22 02:28:52 | watch secrets.v1
  |   | 2022-06-22 02:28:53 | watch serviceaccounts.v1
  |   | 2022-06-22 02:28:53 | watch serviceaccounts.v1
  |   | 2022-06-22 02:29:04 | watch jobs.v1.batch
  |   | 2022-06-22 02:29:04 | watch jobs.v1.batch
  |   | 2022-06-22 02:30:05 | watch rolebindings.v1.rbac.authorization.k8s.io
  |   | 2022-06-22 02:30:05 | watch rolebindings.v1.rbac.authorization.k8s.io
  |   | 2022-06-22 02:30:43 | watch networkpolicies.v1.networking.k8s.io
  |   | 2022-06-22 02:30:43 | watch networkpolicies.v1.networking.k8s.io
  |   | 2022-06-22 02:30:52 | watch configmaps.v1
  |   | 2022-06-22 02:30:52 | watch configmaps.v1
  |   | 2022-06-22 02:30:54 | watch deployments.v1.apps
  |   | 2022-06-22 02:30:54 | watch deployments.v1.apps
  |   | 2022-06-22 02:31:29 | watch services.v1
  |   | 2022-06-22 02:31:29 | watch services.v1
  |   | 2022-06-22 02:32:15 | watch horizontalpodautoscalers.v1.autoscaling
  |   | 2022-06-22 02:32:15 | watch horizontalpodautoscalers.v1.autoscaling
  |   | 2022-06-22 02:32:46 | watch poddisruptionbudgets.v1beta1.policy
  |   | 2022-06-22 02:32:46 | watch poddisruptionbudgets.v1beta1.policy
  |   | 2022-06-22 02:34:38 | watch serviceaccounts.v1
  |   | 2022-06-22 02:34:38 | watch serviceaccounts.v1
  |   | 2022-06-22 02:35:15 | watch statefulsets.v1.apps
  |   | 2022-06-22 02:35:15 | watch statefulsets.v1.apps
  |   | 2022-06-22 02:35:18 | watch pods.v1
  |   | 2022-06-22 02:35:18 | watch pods.v1
  |   | 2022-06-22 02:36:39 | watch rolebindings.v1.rbac.authorization.k8s.io
  |   | 2022-06-22 02:36:39 | watch rolebindings.v1.rbac.authorization.k8s.io
  |   | 2022-06-22 02:36:41 | watch ingresses.v1.networking.k8s.io
  |   | 2022-06-22 02:36:41 | watch ingresses.v1.networking.k8s.io
  |   | 2022-06-22 02:36:49 | watch networkpolicies.v1.networking.k8s.io
  |   | 2022-06-22 02:36:49 | watch networkpolicies.v1.networking.k8s.io
  |   | 2022-06-22 02:37:19 | watch services.v1
  |   | 2022-06-22 02:37:19 | watch services.v1
  |   | 2022-06-22 02:37:59 | watch secrets.v1
  |   | 2022-06-22 02:37:59 | watch secrets.v1
  |   | 2022-06-22 02:38:45 | watch jobs.v1.batch
  |   | 2022-06-22 02:38:45 | watch jobs.v1.batch
  |   | 2022-06-22 02:38:48 | watch poddisruptionbudgets.v1beta1.policy
  |   | 2022-06-22 02:38:48 | watch poddisruptionbudgets.v1beta1.policy
  |   | 2022-06-22 02:39:02 | watch deployments.v1.apps
  |   | 2022-06-22 02:39:02 | watch deployments.v1.apps
  |   | 2022-06-22 02:39:39 | watch configmaps.v1
  |   | 2022-06-22 02:39:39 | watch configmaps.v1
  |   | 2022-06-22 02:40:32 | watch serviceaccounts.v1
  |   | 2022-06-22 02:40:32 | watch serviceaccounts.v1
  |   | 2022-06-22 02:41:22 | watch horizontalpodautoscalers.v1.autoscaling
  |   | 2022-06-22 02:41:22 | watch horizontalpodautoscalers.v1.autoscaling
  |   | 2022-06-22 02:41:37 | watch pods.v1
  |   | 2022-06-22 02:41:37 | watch pods.v1
  |   | 2022-06-22 02:41:38 | watch statefulsets.v1.apps
  |   | 2022-06-22 02:41:38 | watch statefulsets.v1.apps
  |   | 2022-06-22 02:42:03 | watch ingresses.v1.networking.k8s.io
  |   | 2022-06-22 02:42:03 | watch ingresses.v1.networking.k8s.io
  |   | 2022-06-22 02:43:21 | watch secrets.v1
  |   | 2022-06-22 02:43:21 | watch secrets.v1
  |   | 2022-06-22 02:45:10 | watch deployments.v1.apps
  |   | 2022-06-22 02:45:10 | watch deployments.v1.apps
  |   | 2022-06-22 02:45:31 | watch jobs.v1.batch
  |   | 2022-06-22 02:45:31 | watch jobs.v1.batch
  |   | 2022-06-22 02:45:40 | watch networkpolicies.v1.networking.k8s.io
  |   | 2022-06-22 02:45:40 | watch networkpolicies.v1.networking.k8s.io
  |   | 2022-06-22 02:46:28 | watch rolebindings.v1.rbac.authorization.k8s.io
  |   | 2022-06-22 02:46:28 | watch rolebindings.v1.rbac.authorization.k8s.io
  |   | 2022-06-22 02:46:29 | watch services.v1
  |   | 2022-06-22 02:46:29 | watch services.v1
  |   | 2022-06-22 02:47:46 | watch configmaps.v1
  |   | 2022-06-22 02:47:46 | watch configmaps.v1
  |   | 2022-06-22 02:48:15 | watch poddisruptionbudgets.v1beta1.policy
  |   | 2022-06-22 02:48:15 | watch poddisruptionbudgets.v1beta1.policy
  |   | 2022-06-22 02:48:35 | watch serviceaccounts.v1
  |   | 2022-06-22 02:48:35 | watch serviceaccounts.v1
  |   | 2022-06-22 02:50:18 | watch pods.v1
  |   | 2022-06-22 02:50:18 | watch pods.v1
  |   | 2022-06-22 02:50:50 | watch networkpolicies.v1.networking.k8s.io
  |   | 2022-06-22 02:50:50 | watch networkpolicies.v1.networking.k8s.io
  |   | 2022-06-22 02:50:53 | watch jobs.v1.batch
  |   | 2022-06-22 02:50:53 | watch jobs.v1.batch
  |   | 2022-06-22 02:50:59 | watch statefulsets.v1.apps
  |   | 2022-06-22 02:50:59 | watch statefulsets.v1.apps
  |   | 2022-06-22 02:51:11 | watch horizontalpodautoscalers.v1.autoscaling
  |   | 2022-06-22 02:51:11 | watch horizontalpodautoscalers.v1.autoscaling
  |   | 2022-06-22 02:51:13 | watch ingresses.v1.networking.k8s.io
  |   | 2022-06-22 02:51:13 | watch ingresses.v1.networking.k8s.io
  |   | 2022-06-22 02:51:32 | watch secrets.v1
  |   | 2022-06-22 02:51:32 | watch secrets.v1
  |   | 2022-06-22 02:51:32 | watch services.v1
  |   | 2022-06-22 02:51:32 | watch services.v1
  |   | 2022-06-22 02:51:59 | watch deployments.v1.apps
  |   | 2022-06-22 02:51:59 | watch deployments.v1.apps
  |   | 2022-06-22 02:53:42 | watch rolebindings.v1.rbac.authorization.k8s.io
  |   | 2022-06-22 02:53:42 | watch rolebindings.v1.rbac.authorization.k8s.io
  |   | 2022-06-22 02:54:23 | watch configmaps.v1
  |   | 2022-06-22 02:54:23 | watch configmaps.v1
  |   | 2022-06-22 02:55:23 | watch serviceaccounts.v1
  |   | 2022-06-22 02:55:23 | watch serviceaccounts.v1
  |   | 2022-06-22 02:56:31 | watch horizontalpodautoscalers.v1.autoscaling
  |   | 2022-06-22 02:56:31 | watch horizontalpodautoscalers.v1.autoscaling
  |   | 2022-06-22 02:57:06 | watch secrets.v1
  |   | 2022-06-22 02:57:06 | watch secrets.v1
  |   | 2022-06-22 02:57:23 | watch ingresses.v1.networking.k8s.io
  |   | 2022-06-22 02:57:23 | watch ingresses.v1.networking.k8s.io
  |   | 2022-06-22 02:57:30 | watch networkpolicies.v1.networking.k8s.io
  |   | 2022-06-22 02:57:30 | watch networkpolicies.v1.networking.k8s.io
  |   | 2022-06-22 02:57:55 | watch jobs.v1.batch
  |   | 2022-06-22 02:57:55 | watch jobs.v1.batch
  |   | 2022-06-22 02:58:07 | watch poddisruptionbudgets.v1beta1.policy
  |   | 2022-06-22 02:58:07 | watch poddisruptionbudgets.v1beta1.policy
  |   | 2022-06-22 02:58:36 | watch deployments.v1.apps
  |   | 2022-06-22 02:58:36 | watch deployments.v1.apps
  |   | 2022-06-22 02:58:58 | watch services.v1
  |   | 2022-06-22 02:58:58 | watch services.v1
  |   | 2022-06-22 02:59:10 | watch pods.v1
  |   | 2022-06-22 02:59:10 | watch pods.v1
  |   | 2022-06-22 02:59:43 | watch statefulsets.v1.apps
  |   | 2022-06-22 02:59:43 | watch statefulsets.v1.apps
  |   | 2022-06-22 03:00:10 | watch configmaps.v1
  |   | 2022-06-22 03:00:10 | watch configmaps.v1
  |   | 2022-06-22 03:00:33 | watch rolebindings.v1.rbac.authorization.k8s.io
  |   | 2022-06-22 03:00:33 | watch rolebindings.v1.rbac.authorization.k8s.io
  |   | 2022-06-22 03:02:29 | watch secrets.v1
  |   | 2022-06-22 03:02:29 | watch secrets.v1
  |   | 2022-06-22 03:02:35 | watch serviceaccounts.v1
  |   | 2022-06-22 03:02:35 | watch serviceaccounts.v1
  |   | 2022-06-22 03:02:39 | watch horizontalpodautoscalers.v1.autoscaling
  |   | 2022-06-22 03:02:39 | watch horizontalpodautoscalers.v1.autoscaling
  |   | 2022-06-22 03:03:13 | watch ingresses.v1.networking.k8s.io
  |   | 2022-06-22 03:03:13 | watch ingresses.v1.networking.k8s.io
  |   | 2022-06-22 03:04:27 | watch deployments.v1.apps
  |   | 2022-06-22 03:04:27 | watch deployments.v1.apps
  |   | 2022-06-22 03:05:03 | watch jobs.v1.batch
  |   | 2022-06-22 03:05:03 | watch jobs.v1.batch
  |   | 2022-06-22 03:05:17 | watch networkpolicies.v1.networking.k8s.io
  |   | 2022-06-22 03:05:17 | watch networkpolicies.v1.networking.k8s.io
  |   | 2022-06-22 03:05:50 | watch statefulsets.v1.apps
  |   | 2022-06-22 03:05:50 | watch statefulsets.v1.apps
  |   | 2022-06-22 03:06:08 | watch poddisruptionbudgets.v1beta1.policy
  |   | 2022-06-22 03:06:08 | watch poddisruptionbudgets.v1beta1.policy
  |   | 2022-06-22 03:06:58 | watch pods.v1
  |   | 2022-06-22 03:06:58 | watch pods.v1
  |   | 2022-06-22 03:07:14 | watch rolebindings.v1.rbac.authorization.k8s.io
  |   | 2022-06-22 03:07:14 | watch rolebindings.v1.rbac.authorization.k8s.io
  |   | 2022-06-22 03:07:29 | watch configmaps.v1
  |   | 2022-06-22 03:07:29 | watch configmaps.v1
  |   | 2022-06-22 03:07:35 | watch services.v1
  |   | 2022-06-22 03:07:35 | watch services.v1
  |   | 2022-06-22 03:07:52 | watch horizontalpodautoscalers.v1.autoscaling
  |   | 2022-06-22 03:07:52 | watch horizontalpodautoscalers.v1.autoscaling
  |   | 2022-06-22 03:08:22 | watch ingresses.v1.networking.k8s.io
  |   | 2022-06-22 03:08:22 | watch ingresses.v1.networking.k8s.io
  |   | 2022-06-22 03:09:39 | watch deployments.v1.apps
  |   | 2022-06-22 03:09:39 | watch deployments.v1.apps
  |   | 2022-06-22 03:10:00 | watch serviceaccounts.v1
  |   | 2022-06-22 03:10:00 | watch serviceaccounts.v1
  |   | 2022-06-22 03:10:47 | watch networkpolicies.v1.networking.k8s.io
  |   | 2022-06-22 03:10:47 | watch networkpolicies.v1.networking.k8s.io
  |   | 2022-06-22 03:11:41 | watch secrets.v1
  |   | 2022-06-22 03:11:41 | watch secrets.v1
  |   | 2022-06-22 03:12:35 | watch jobs.v1.batch
  |   | 2022-06-22 03:12:35 | watch jobs.v1.batch
  |   | 2022-06-22 03:12:52 | watch configmaps.v1
  |   | 2022-06-22 03:12:52 | watch configmaps.v1
  |   | 2022-06-22 03:13:19 | watch pods.v1
  |   | 2022-06-22 03:13:19 | watch pods.v1
  |   | 2022-06-22 03:13:26 | watch poddisruptionbudgets.v1beta1.policy
  |   | 2022-06-22 03:13:26 | watch poddisruptionbudgets.v1beta1.policy
  |   | 2022-06-22 03:14:07 | watch horizontalpodautoscalers.v1.autoscaling
  |   | 2022-06-22 03:14:07 | watch horizontalpodautoscalers.v1.autoscaling
  |   | 2022-06-22 03:14:21 | watch rolebindings.v1.rbac.authorization.k8s.io
  |   | 2022-06-22 03:14:21 | watch rolebindings.v1.rbac.authorization.k8s.io
  |   | 2022-06-22 03:15:18 | watch services.v1
  |   | 2022-06-22 03:15:18 | watch services.v1
  |   | 2022-06-22 03:15:45 | watch statefulsets.v1.apps
  |   | 2022-06-22 03:15:45 | watch statefulsets.v1.apps
  |   | 2022-06-22 03:15:54 | watch ingresses.v1.networking.k8s.io
  |   | 2022-06-22 03:15:54 | watch ingresses.v1.networking.k8s.io
  |   | 2022-06-22 03:16:32 | watch networkpolicies.v1.networking.k8s.io
  |   | 2022-06-22 03:16:32 | watch networkpolicies.v1.networking.k8s.io
  |   | 2022-06-22 03:17:27 | watch deployments.v1.apps
  |   | 2022-06-22 03:17:27 | watch deployments.v1.apps
  |   | 2022-06-22 03:17:49 | watch serviceaccounts.v1
  |   | 2022-06-22 03:17:49 | watch serviceaccounts.v1
  |   | 2022-06-22 03:18:49 | watch secrets.v1
  |   | 2022-06-22 03:18:49 | watch secrets.v1
  |   | 2022-06-22 03:20:08 | watch rolebindings.v1.rbac.authorization.k8s.io
  |   | 2022-06-22 03:20:08 | watch rolebindings.v1.rbac.authorization.k8s.io
  |   | 2022-06-22 03:21:08 | watch jobs.v1.batch
  |   | 2022-06-22 03:21:08 | watch jobs.v1.batch
  |   | 2022-06-22 03:21:13 | watch poddisruptionbudgets.v1beta1.policy
  |   | 2022-06-22 03:21:13 | watch poddisruptionbudgets.v1beta1.policy
  |   | 2022-06-22 03:21:20 | watch configmaps.v1
  |   | 2022-06-22 03:21:20 | watch configmaps.v1
  |   | 2022-06-22 03:21:30 | watch horizontalpodautoscalers.v1.autoscaling
  |   | 2022-06-22 03:21:30 | watch horizontalpodautoscalers.v1.autoscaling
  |   | 2022-06-22 03:22:20 | watch pods.v1
  |   | 2022-06-22 03:22:20 | watch pods.v1
  |   | 2022-06-22 03:22:49 | watch statefulsets.v1.apps
  |   | 2022-06-22 03:22:49 | watch statefulsets.v1.apps
  |   | 2022-06-22 03:23:19 | watch ingresses.v1.networking.k8s.io
  |   | 2022-06-22 03:23:19 | watch ingresses.v1.networking.k8s.io
  |   | 2022-06-22 03:24:46 | watch services.v1
  |   | 2022-06-22 03:24:46 | watch services.v1
  |   | 2022-06-22 03:25:28 | watch networkpolicies.v1.networking.k8s.io
  |   | 2022-06-22 03:25:28 | watch networkpolicies.v1.networking.k8s.io
  |   | 2022-06-22 03:26:00 | watch deployments.v1.apps
  |   | 2022-06-22 03:26:00 | watch deployments.v1.apps
  |   | 2022-06-22 03:26:11 | watch serviceaccounts.v1
  |   | 2022-06-22 03:26:11 | watch serviceaccounts.v1
  |   | 2022-06-22 03:26:35 | watch jobs.v1.batch
  |   | 2022-06-22 03:26:35 | watch jobs.v1.batch
  |   | 2022-06-22 03:27:50 | watch poddisruptionbudgets.v1beta1.policy
  |   | 2022-06-22 03:27:50 | watch poddisruptionbudgets.v1beta1.policy
  |   | 2022-06-22 03:27:57 | watch secrets.v1
  |   | 2022-06-22 03:27:57 | watch secrets.v1
  |   | 2022-06-22 03:28:01 | watch configmaps.v1
  |   | 2022-06-22 03:28:01 | watch configmaps.v1
  |   | 2022-06-22 03:28:11 | watch horizontalpodautoscalers.v1.autoscaling
  |   | 2022-06-22 03:28:11 | watch horizontalpodautoscalers.v1.autoscaling
  |   | 2022-06-22 03:28:32 | watch ingresses.v1.networking.k8s.io
  |   | 2022-06-22 03:28:32 | watch ingresses.v1.networking.k8s.io
  |   | 2022-06-22 03:29:15 | watch rolebindings.v1.rbac.authorization.k8s.io
  |   | 2022-06-22 03:29:15 | watch rolebindings.v1.rbac.authorization.k8s.io
  |   | 2022-06-22 03:29:23 | watch pods.v1
  |   | 2022-06-22 03:29:23 | watch pods.v1
  |   | 2022-06-22 03:29:48 | watch services.v1
  |   | 2022-06-22 03:29:48 | watch services.v1

Jun 23 '22 16:06 robbie-demuth

Based on the above, I have several questions:

Most importantly, what would cause a custom controller to continue watching resources it "owns" but stop watching its custom resources?
How do changes to watched resources typically result in immediate reconciliation when watches are periodic on the order of ~10 minutes?

Jun 23 '22 16:06 robbie-demuth

A teammate of mine stumbled across https://github.com/kubernetes/kubernetes/issues/108344. The issue seems identical to this one minus the fact that is related to Aggregated APIs instead of CRDs (though my understanding is that they're extremely similar)

Jun 23 '22 20:06 robbie-demuth

+1 facing the same issue

@robbie-demuth what are the k8s version and client version you facing this issue with ?

Jul 02 '22 16:07 AdheipSingh

Hi @robbie-demuth @AdheipSingh,

How can we build an POC to check this one? If we get a sample from Kubebuilder like: https://github.com/kubernetes-sigs/kubebuilder/tree/master/testdata/project-v3-with-deploy-image can we reproduce this issue? Also, what are the versions used?

Jul 05 '22 18:07 camilamacedo86

Here's my go mod for my operator

go 1.17

require (
	github.com/DataDog/datadog-go v4.8.3+incompatible
	github.com/go-logr/logr v1.2.2
	github.com/onsi/ginkgo v1.16.5
	github.com/onsi/gomega v1.17.0
	github.com/sirupsen/logrus v1.8.1
	github.com/stretchr/testify v1.7.0
	go.etcd.io/etcd/client/v3 v3.5.0
	k8s.io/api v0.23.1
	k8s.io/apimachinery v0.23.1
	k8s.io/client-go v0.23.1
	sigs.k8s.io/controller-runtime v0.11.0
)

Kubernetes version is v1.22.8-gke.202

I have 17 CR's reconciled by a single operator.

@camilamacedo86

Jul 13 '22 06:07 AdheipSingh

All of our k8s.io deps (including client-go) currently use v0.23.5. We currently use v0.11.2 of controller-runtime. Server version is v1.21.12-eks-a64ea69

As for reproducing, we think we have a better idea of the (near) root cause. We correlated the time at which the watches die with when we rollout updates to our CRD. We rollout updates to our CRD using kubectl replace (it's too big for kubectl apply). We also use cert-manager to manage the certificate and CA bundle for our conversion webhook, so we make use of the cert-manager.io/inject-ca-from annotation and let cert-manager inject the CA bundle into our CRD. In turns out our rollout implementation actually results in the CA bundle being temporarily removed from the CRD. We believe that this causes a brief outage of our CRD API - analogous to the networking blip described in https://github.com/kubernetes/kubernetes/issues/108344. We're currently working to address this to see if it resolves the issue

That being said, the brief outage / networking blip doesn't induce the issue in all controllers in our cluster. Since the initial occurrence of the issue when we saw this en masse, the issue has been infrequent. Just yesterday, we rolled out a CRD update, but only saw the issue with 3 out of several hundred controllers

Also, it still seems like controller-runtime / client-go / Kubernetes should be resilient to this - if it is indeed contributing to the issue (hence why I said "near root cause" above). Even if our CRD API takes a brief outage, I'd expect all controllers to resume watching once it's available again

Thanks, @camilamacedo86 !

Jul 13 '22 17:07 robbie-demuth

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle stale
Mark this issue or PR as rotten with /lifecycle rotten
Close this issue or PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

Oct 11 '22 18:10 k8s-triage-robot

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle rotten
Close this issue or PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

Nov 10 '22 18:11 k8s-triage-robot

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Reopen this issue with /reopen
Mark this issue as fresh with /remove-lifecycle rotten
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Dec 10 '22 19:12 k8s-triage-robot

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied

After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied

After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Reopen this issue with /reopen

Mark this issue as fresh with /remove-lifecycle rotten

Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Dec 10 '22 19:12 k8s-ci-robot