`kustomize-controller` fails with `knative` 1.2.0

[zzvara]

Describe the bug

The kustomize-controller fails with knative 1.2.0 as follows:

{"level":"info","ts":"2022-03-15T08:31:00.204Z","logger":"controller-runtime.metrics","msg":"Metrics server is starting to listen","addr":":8080"}
{"level":"info","ts":"2022-03-15T08:31:00.205Z","logger":"setup","msg":"starting manager"}
{"level":"info","ts":"2022-03-15T08:31:00.206Z","msg":"Starting server","path":"/metrics","kind":"metrics","addr":"[::]:8080"}
{"level":"info","ts":"2022-03-15T08:31:00.206Z","msg":"Starting server","kind":"health probe","addr":"[::]:9440"}
I0315 08:31:00.307441       7 leaderelection.go:248] attempting to acquire leader lease flux-system/kustomize-controller-leader-election...
I0315 08:31:00.341196       7 leaderelection.go:258] successfully acquired lease flux-system/kustomize-controller-leader-election
{"level":"info","ts":"2022-03-15T08:31:00.341Z","logger":"controller.kustomization","msg":"Starting EventSource","reconciler group":"kustomize.toolkit.fluxcd.io","reconciler kind":"Kustomization","source":"kind source: *v1beta2.Kustomization"}
{"level":"info","ts":"2022-03-15T08:31:00.341Z","logger":"controller.kustomization","msg":"Starting EventSource","reconciler group":"kustomize.toolkit.fluxcd.io","reconciler kind":"Kustomization","source":"kind source: *v1beta1.GitRepository"}
{"level":"info","ts":"2022-03-15T08:31:00.341Z","logger":"controller.kustomization","msg":"Starting EventSource","reconciler group":"kustomize.toolkit.fluxcd.io","reconciler kind":"Kustomization","source":"kind source: *v1beta1.Bucket"}
{"level":"info","ts":"2022-03-15T08:31:00.341Z","logger":"controller.kustomization","msg":"Starting Controller","reconciler group":"kustomize.toolkit.fluxcd.io","reconciler kind":"Kustomization"}
{"level":"debug","ts":"2022-03-15T08:31:00.341Z","logger":"events","msg":"Normal","object":{"kind":"ConfigMap","namespace":"flux-system","name":"kustomize-controller-leader-election","uid":"d51f56db-5866-4a8e-a4db-f310ae869047","apiVersion":"v1","resourceVersion":"11676305"},"reason":"LeaderElection","message":"kustomize-controller-649cc86fdd-95k79_a7ca69e5-09d7-4421-9f28-9b9b4c59fff6 became leader"}
{"level":"debug","ts":"2022-03-15T08:31:00.342Z","logger":"events","msg":"Normal","object":{"kind":"Lease","namespace":"flux-system","name":"kustomize-controller-leader-election","uid":"7f8bc379-2121-4041-9408-b2a9a45c5171","apiVersion":"coordination.k8s.io/v1","resourceVersion":"11676308"},"reason":"LeaderElection","message":"kustomize-controller-649cc86fdd-95k79_a7ca69e5-09d7-4421-9f28-9b9b4c59fff6 became leader"}
{"level":"info","ts":"2022-03-15T08:31:00.443Z","logger":"controller.kustomization","msg":"Starting workers","reconciler group":"kustomize.toolkit.fluxcd.io","reconciler kind":"Kustomization","worker count":4}
{"level":"info","ts":"2022-03-15T08:31:10.443Z","logger":"controller.kustomization","msg":"server-side apply completed","reconciler group":"kustomize.toolkit.fluxcd.io","reconciler kind":"Kustomization","name":"flux-system","namespace":"flux-system","output":{"CustomResourceDefinition/alerts.notification.toolkit.fluxcd.io":"unchanged","CustomResourceDefinition/buckets.source.toolkit.fluxcd.io":"unchanged","CustomResourceDefinition/gitrepositories.source.toolkit.fluxcd.io":"unchanged","CustomResourceDefinition/helmcharts.source.toolkit.fluxcd.io":"unchanged","CustomResourceDefinition/helmreleases.helm.toolkit.fluxcd.io":"unchanged","CustomResourceDefinition/helmrepositories.source.toolkit.fluxcd.io":"unchanged","CustomResourceDefinition/imagepolicies.image.toolkit.fluxcd.io":"unchanged","CustomResourceDefinition/imagerepositories.image.toolkit.fluxcd.io":"unchanged","CustomResourceDefinition/imageupdateautomations.image.toolkit.fluxcd.io":"unchanged","CustomResourceDefinition/knativeeventings.operator.knative.dev":"unchanged","CustomResourceDefinition/knativeservings.operator.knative.dev":"unchanged","CustomResourceDefinition/kustomizations.kustomize.toolkit.fluxcd.io":"unchanged","CustomResourceDefinition/providers.notification.toolkit.fluxcd.io":"unchanged","CustomResourceDefinition/receivers.notification.toolkit.fluxcd.io":"unchanged","Namespace/flux-system":"unchanged"}}
{"level":"error","ts":"2022-03-15T08:31:11.872Z","logger":"controller.kustomization","msg":"Reconciliation failed after 11.42867209s, next try in 10m0s","reconciler group":"kustomize.toolkit.fluxcd.io","reconciler kind":"Kustomization","name":"flux-system","namespace":"flux-system","revision":"master/2a7ae64191fa4e30882097f46d3c04f5b294b156","error":"ConfigMap/system/config-logging dry-run failed, reason: BadRequest, error: admission webhook \"config.webhook.serving.knative.dev\" denied the request: validation failed: the update modifies a key in \"_example\" which is probably not what you want. Instead, copy the respective setting to the top-level of the ConfigMap, directly below \"data\"\n","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}
{"level":"debug","ts":"2022-03-15T08:31:11.872Z","logger":"events","msg":"Warning","object":{"kind":"Kustomization","namespace":"flux-system","name":"flux-system","uid":"b36d31d3-66b3-4545-a9bc-b67a574df99d","apiVersion":"kustomize.toolkit.fluxcd.io/v1beta2","resourceVersion":"11670096"},"reason":"error","message":"ConfigMap/system/config-logging dry-run failed, reason: BadRequest, error: admission webhook \"config.webhook.serving.knative.dev\" denied the request: validation failed: the update modifies a key in \"_example\" which is probably not what you want. Instead, copy the respective setting to the top-level of the ConfigMap, directly below \"data\"\n"}
{"level":"info","ts":"2022-03-15T08:41:13.523Z","logger":"controller.kustomization","msg":"server-side apply completed","reconciler group":"kustomize.toolkit.fluxcd.io","reconciler kind":"Kustomization","name":"flux-system","namespace":"flux-system","output":{"CustomResourceDefinition/alerts.notification.toolkit.fluxcd.io":"unchanged","CustomResourceDefinition/buckets.source.toolkit.fluxcd.io":"unchanged","CustomResourceDefinition/gitrepositories.source.toolkit.fluxcd.io":"unchanged","CustomResourceDefinition/helmcharts.source.toolkit.fluxcd.io":"unchanged","CustomResourceDefinition/helmreleases.helm.toolkit.fluxcd.io":"unchanged","CustomResourceDefinition/helmrepositories.source.toolkit.fluxcd.io":"unchanged","CustomResourceDefinition/imagepolicies.image.toolkit.fluxcd.io":"unchanged","CustomResourceDefinition/imagerepositories.image.toolkit.fluxcd.io":"unchanged","CustomResourceDefinition/imageupdateautomations.image.toolkit.fluxcd.io":"unchanged","CustomResourceDefinition/knativeeventings.operator.knative.dev":"unchanged","CustomResourceDefinition/knativeservings.operator.knative.dev":"unchanged","CustomResourceDefinition/kustomizations.kustomize.toolkit.fluxcd.io":"unchanged","CustomResourceDefinition/providers.notification.toolkit.fluxcd.io":"unchanged","CustomResourceDefinition/receivers.notification.toolkit.fluxcd.io":"unchanged","Namespace/flux-system":"unchanged"}}
{"level":"error","ts":"2022-03-15T08:41:14.988Z","logger":"controller.kustomization","msg":"Reconciliation failed after 3.106334761s, next try in 10m0s","reconciler group":"kustomize.toolkit.fluxcd.io","reconciler kind":"Kustomization","name":"flux-system","namespace":"flux-system","revision":"master/2a7ae64191fa4e30882097f46d3c04f5b294b156","error":"ConfigMap/system/config-observability dry-run failed, reason: BadRequest, error: admission webhook \"config.webhook.serving.knative.dev\" denied the request: validation failed: the update modifies a key in \"_example\" which is probably not what you want. Instead, copy the respective setting to the top-level of the ConfigMap, directly below \"data\"\n","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}
{"level":"debug","ts":"2022-03-15T08:41:14.989Z","logger":"events","msg":"Warning","object":{"kind":"Kustomization","namespace":"flux-system","name":"flux-system","uid":"b36d31d3-66b3-4545-a9bc-b67a574df99d","apiVersion":"kustomize.toolkit.fluxcd.io/v1beta2","resourceVersion":"11676488"},"reason":"error","message":"ConfigMap/system/config-observability dry-run failed, reason: BadRequest, error: admission webhook \"config.webhook.serving.knative.dev\" denied the request: validation failed: the update modifies a key in \"_example\" which is probably not what you want. Instead, copy the respective setting to the top-level of the ConfigMap, directly below \"data\"\n"}

Steps to reproduce

1. Install Flux as follows:

flux bootstrap git --namespace flux-system --author-name Flux CD --branch master --cluster-domain core --commit-message-appendix [Flux CD] --url ssh://redacted --username redacted --log-level debug --network-policy true --components-extra=image-reflector-controller,image-automation-controller --insecure-skip-tls-verify true --private-key-file redacted

2. Install knative from [1] as suggested by [2]. We replaced all namespace: default to namespace: system so that knative would be installed onto the system namespace instead.

[1] https://github.com/knative/operator/releases/download/knative-v1.2.0/operator.yaml [2] https://knative.dev/docs/install/operator/knative-with-operators/#install-the-latest-knative-operator-release

3. Add knative-serving.yaml to Flux as follows:

# @see [https://knative.dev/docs/install/operator/knative-with-operators/#create-the-knative-serving-custom-resource]
apiVersion: operator.knative.dev/v1alpha1
kind: KnativeServing
metadata:
  name: knative-serving
  namespace: system
spec:
  config:
    autoscaler:
      enable-scale-to-zero: "false"

4. See that within system namespace, the following ConfigMaps have been created:

• config-logging
• config-observability

5. See Flux CD logs.

Expected behavior

The kustomize-controller not to fail on config-logging and config-observability resources.

Screenshots and recordings

No response

OS / Distro

20.04.3 LTS (Focal Fossa) Linux 5.4.0-100-generic #113-Ubuntu SMP Thu Feb 3 18:43:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Flux version

ghcr.io/fluxcd/kustomize-controller:v0.21.1

Flux check

► checking prerequisites ✗ flux 0.27.2 <0.27.3 (new version is available, please upgrade) ✔ Kubernetes 1.22.6 >=1.20.6-0 ► checking controllers ✔ helm-controller: deployment ready ► ghcr.io/fluxcd/helm-controller:v0.17.1 ✔ image-automation-controller: deployment ready ► ghcr.io/fluxcd/image-automation-controller:v0.20.0 ✔ image-reflector-controller: deployment ready ► ghcr.io/fluxcd/image-reflector-controller:v0.16.0 ✔ kustomize-controller: deployment ready ► ghcr.io/fluxcd/kustomize-controller:v0.21.1 ✔ notification-controller: deployment ready ► ghcr.io/fluxcd/notification-controller:v0.22.2 ✔ source-controller: deployment ready ► ghcr.io/fluxcd/source-controller:v0.21.2 ✔ all checks passed

Git provider

No response

Container Registry provider

No response

Additional context

No response

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[stefanprodan]

ConfigMap/system/config-observability dry-run failed, reason: BadRequest, error: admission webhook \"config.webhook.serving.knative.dev\" denied the request: validation failed: the update modifies a key in "_example" which is probably not what you want. Instead, copy the respective setting to the top-level of the ConfigMap, directly below \"data\""

I don't see how this error has anything to do with Flux, the ConfigMap is invalid according to knative.

[zzvara]

@stefanprodan The ConfigMap exists as a resource in Kubernetes. I suspect that it is valid then:

What is interesting to me is that what does the kustomize-controller has to do with it in the first place? Why is this (if an issue), appears in kustomize-controller and hinders the synchronization of all resources? What I can confirm is that Flux CD is not syncing resources from any Git repositories.

[stefanprodan]

@zzvara you should create a dedicated Flux Kustomization for knative, so it doesn't affect all the other syncs. See here an example: https://github.com/fluxcd/flux2-kustomize-helm-example

[zzvara]

@stefanprodan thanks, that fixes it!

[zzvara]

Unfortunately, this is still an issue in case a HelmRelease contains a knative Service. The issue, in this case, is similar, that, is:

Helm upgrade failed: failed to replace object: admission webhook "validation.webhook.serving.knative.dev" denied the request: validation failed: annotation value is immutable: metadata.annotations.serving.knative.dev/creator && failed to replace object: admission webhook "validation.webhook.serving.knative.dev" denied the request: validation failed: annotation value is immutable: metadata.annotations.serving.knative.dev/creator && failed to replace object: admission webhook "validation.webhook.serving.knative.dev" denied the request: validation failed: annotation value is immutable: metadata.annotations.serving.knative.dev/creator && failed to replace object: admission webhook "validation.webhook.serving.knative.dev" denied the request: validation failed: annotation value is immutable: metadata.annotations.serving.knative.dev/creator && failed to replace object: admission webhook "validation.webhook.serving.knative.dev" denied the request: validation failed: annotation value is immutable: metadata.annotations.serving.knative.dev/creator && failed to replace

As far as I can see, knative Services can not be used with fluxcd.

[stefanprodan]

@zzvara I’m. not sure that error is caused by Flux, we don’t modify the metadata.annotations.serving.knative.dev/creator annotation. Does this work with the Helm CLI if you run an upgrade?