Using Fleet to deploy Rancher Monitoring results in two webhooks shown as "modified"

[belgaied2]

Description

I would like to deploy Rancher Monitoring, Logging, etc. using Fleet and the following GitHub Repos:

• https://github.com/belgaied2/fleet-rancher-monitoring/
• https://github.com/belgaied2/fleet-rancher-logging/

GitRepo object (for Monitoring):

apiVersion: fleet.cattle.io/v1alpha1
kind: GitRepo
metadata:
  name: rancher-monitoring
  namespace: fleet-default
spec:
  branch: main
  forceSyncGeneration: 3
  insecureSkipTLSVerify: false
  paths:
  - /rancher-monitoring/
  - /rancher-monitoring-crd/
  paused: false
  repo: https://github.com/belgaied2/fleet-rancher-monitoring.git
  targets:
  - clusterName: c-vrfqq

Logging works just fine but GitRepo for Monitoring stays in the Modifed state as shown here:

A look at the resources of the GitRepo shows:

Here is the end of status section for Monitoring bundle:

  summary:
    desiredReady: 1
    modified: 1
    nonReadyResources:
    - bundleState: Modified
      modifiedStatus:
      - apiVersion: admissionregistration.k8s.io/v1
        kind: MutatingWebhookConfiguration
        name: rancher-monitoring-admission
        patch: '{"webhooks":[{"admissionReviewVersions":["v1","v1beta1"],"clientConfig":{"caBundle":"LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJkakNDQVJ5Z0F3SUJBZ0lSQVBXRzcyb0xJZUhObzQ0SVBpYmZzM0F3Q2dZSUtvWkl6ajBFQXdJd0R6RU4KTUFzR0ExVUVDaE1FYm1sc01UQWdGdzB5TWpBeU1UUXhORFE1TWpkYUdBOHlNVEl5TURFeU1URTBORGt5TjFvdwpEekVOTUFzR0ExVUVDaE1FYm1sc01UQlpNQk1HQnlxR1NNNDlBZ0VHQ0NxR1NNNDlBd0VIQTBJQUJMMWNVQU92CkZ0SDBidmVXdlBsM1NzNEptdVRBbkk0blVMbmZKVWErRFRoWVdKb055UExkTTllU004eEM2M3kxUkVVR0JrM2YKV3Q4SE1ud0xTOWJsbDdDalZ6QlZNQTRHQTFVZER3RUIvd1FFQXdJQ0JEQVRCZ05WSFNVRUREQUtCZ2dyQmdFRgpCUWNEQVRBUEJnTlZIUk1CQWY4RUJUQURBUUgvTUIwR0ExVWREZ1FXQkJTaVRWK0NxUEE5Q09ad1c1VmU2UTFtCitBemE2REFLQmdncWhrak9QUVFEQWdOSUFEQkZBaUVBNUxpdnRqUUROQjBRa01jNm4rS0ZhdFdEMmw5VzlNeUYKNTJYVWtOd3lTNkFDSUFEaS9CZ3l0TksxSkpBYzBheGl1WDBjY2dxYnVnN1Z4OXVxTkc1TTBsbGsKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=","service":{"name":"rancher-monitoring-operator","namespace":"cattle-monitoring-system","path":"/admission-prometheusrules/mutate","port":443}},"failurePolicy":"Ignore","matchPolicy":"Equivalent","name":"prometheusrulemutate.monitoring.coreos.com","namespaceSelector":{},"objectSelector":{},"reinvocationPolicy":"Never","rules":[{"apiGroups":["monitoring.coreos.com"],"apiVersions":["*"],"operations":["CREATE","UPDATE"],"resources":["prometheusrules"],"scope":"*"}],"sideEffects":"None","timeoutSeconds":10}]}'
      - apiVersion: admissionregistration.k8s.io/v1
        kind: ValidatingWebhookConfiguration
        name: rancher-monitoring-admission
        patch: '{"webhooks":[{"admissionReviewVersions":["v1","v1beta1"],"clientConfig":{"caBundle":"LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJkakNDQVJ5Z0F3SUJBZ0lSQVBXRzcyb0xJZUhObzQ0SVBpYmZzM0F3Q2dZSUtvWkl6ajBFQXdJd0R6RU4KTUFzR0ExVUVDaE1FYm1sc01UQWdGdzB5TWpBeU1UUXhORFE1TWpkYUdBOHlNVEl5TURFeU1URTBORGt5TjFvdwpEekVOTUFzR0ExVUVDaE1FYm1sc01UQlpNQk1HQnlxR1NNNDlBZ0VHQ0NxR1NNNDlBd0VIQTBJQUJMMWNVQU92CkZ0SDBidmVXdlBsM1NzNEptdVRBbkk0blVMbmZKVWErRFRoWVdKb055UExkTTllU004eEM2M3kxUkVVR0JrM2YKV3Q4SE1ud0xTOWJsbDdDalZ6QlZNQTRHQTFVZER3RUIvd1FFQXdJQ0JEQVRCZ05WSFNVRUREQUtCZ2dyQmdFRgpCUWNEQVRBUEJnTlZIUk1CQWY4RUJUQURBUUgvTUIwR0ExVWREZ1FXQkJTaVRWK0NxUEE5Q09ad1c1VmU2UTFtCitBemE2REFLQmdncWhrak9QUVFEQWdOSUFEQkZBaUVBNUxpdnRqUUROQjBRa01jNm4rS0ZhdFdEMmw5VzlNeUYKNTJYVWtOd3lTNkFDSUFEaS9CZ3l0TksxSkpBYzBheGl1WDBjY2dxYnVnN1Z4OXVxTkc1TTBsbGsKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=","service":{"name":"rancher-monitoring-operator","namespace":"cattle-monitoring-system","path":"/admission-prometheusrules/validate","port":443}},"failurePolicy":"Ignore","matchPolicy":"Equivalent","name":"prometheusrulemutate.monitoring.coreos.com","namespaceSelector":{},"objectSelector":{},"rules":[{"apiGroups":["monitoring.coreos.com"],"apiVersions":["*"],"operations":["CREATE","UPDATE"],"resources":["prometheusrules"],"scope":"*"}],"sideEffects":"None","timeoutSeconds":10}]}'
      name: fleet-default/c-vrfqq
    ready: 0
  unavailable: 0
  unavailablePartitions: 0

If I patch manually using kubectl one of the above patches, it works and the bundle and GitRepo are then both in the Active state.

Environment

• Rancher version : v2.6.2
• Fleet : 100.0.1+up0.3.7
• Monitoring: 100.0.0+up16.6.0 and 100.1.0+up19.0.3 (Both shown the problem)
• Downstream Cluster type: RKE1
• Cluster architecture: 1 Control plane/etcd , 4 workers (each 4 vCPU and 32GB of RAM) , Reservation (~10%) and usage on the cluster are low.

[slavonicsniper]

i had the same problem. rancher-monitoring chart has hooks with jobs that patch these resource (which are modified for the fleet in the end) after the helm installation, i guess fleet cannot handle this at the moment.

[ibrokethecloud]

@belgaied2 a diff is needed to inform fleet to ignore the modified objects. I have an example here: https://github.com/ibrokethecloud/core-bundles/blob/master/monitoring/fleet.yaml

[timofey-drozhzhin]

Disclosure: I'm somewhat new to Kubernetes.

If anyone else is struggling with this, it's probably because the official fleet-examples are outdated.

Solution:

To fix the issue, add the following code to the Rancher Monitoring fleet.yaml file and redeploy the changes.

diff:
  comparePatches:
    - apiVersion: admissionregistration.k8s.io/v1
      kind: MutatingWebhookConfiguration
      name: rancher-monitoring-admission
      operations:
        - {"op":"remove", "path":"/webhooks"}
    - apiVersion: admissionregistration.k8s.io/v1
      kind: ValidatingWebhookConfiguration
      name: rancher-monitoring-admission
      operations:
        - {"op":"remove", "path":"/webhooks"}

Explanation

This happens when Fleet catches the final object being different from what you originally requested.

In the above screenshot we can see that two objects got modified at runtime - MutatingWebhookConfiguration and ValidatingWebhookConfiguration. If we download the YAML file of the affected bundle and navigate to the summary: section, we'll see that the /webhooks object has been patched during runtime. Therefore, we append the above diff to let Fleet know to ignore these changes.

Note: A more proper way might be to fine-tune the excluded objects, i.e. /webhooks/0/<object name>.

For more information, refer to https://fleet.rancher.io/bundle-diffs/

[raif-ahmed]

The patch diff logic is really buggy, so

diff:
  comparePatches:
  - apiVersion: admissionregistration.k8s.io/v1
    kind: MutatingWebhookConfiguration
    name: rancher-monitoring-admission
    operations:
    - {"op":"remove", "path":"/webhooks/0/admissionReviewVersions"}
    - {"op":"remove", "path":"/webhooks/0/clientConfig"}
    - {"op":"remove", "path":"/webhooks/0/failurePolicy"}
    - {"op":"remove", "path":"/webhooks/0/matchPolicy"}
    - {"op":"remove", "path":"/webhooks/0/name"}
    - {"op":"remove", "path":"/webhooks/0/namespaceSelector"}
    - {"op":"remove", "path":"/webhooks/0/objectSelector"}
    - {"op":"remove", "path":"/webhooks/0/reinvocationPolicy"}
    - {"op":"remove", "path":"/webhooks/0/rules/0"}
    - {"op":"remove", "path":"/webhooks/0/sideEffects"}
    - {"op":"remove", "path":"/webhooks/0/timeoutSeconds"}

doens't work !!

While

diff:
  comparePatches:
  - apiVersion: admissionregistration.k8s.io/v1
    kind: MutatingWebhookConfiguration
    name: rancher-monitoring-admission
    operations:
    - {"op":"remove", "path":"/webhooks/0"}

work.

although both should be the same !! as i exclude every single child json path !!!!