Feature Request: Skip Bundle based on Cluster Label

[matteogazzadi]

Is your feature request related to a problem?

We work on a solution where we have hundred of helm charts developed by our company to be deployed, as is, in hundreds of clusters. To keep track of what actually is installed and due to impact on the production environment, deployment is done by selecting a tag of components and not using always the latest version with autoamtic upgrade.

To avoid having one GitRepo per Helm chart (and due to tag approach actually multiple git repo for the same chart just for different tag), we are following the approach that helm charts are grouped together by "functionality" and basically have one git repo per each functionality with multiple fleet.yaml.

However, several times we would like to being able to specify that for a given GitRepo, a Bunde should be installed or not based on a cluster labels, because this bundle could contain functionality not required in that environment.

This is not possible with today fleet solution, we can just specify which GitRepo are linked to a Cluster but not having a sort of "feature flag" approach to the bundle.

This is limiting a lot our capabilities.

Solution you'd like

Having a flag in the fleet.yaml, where we can "skip" a bundle via cluster label selector.

Alternatives you've considered

Adding a boolean to the Values.yaml of the helm populated via global.cluster.fleet.label, and if that boolean is false add to every helm chart template an {{ if .Values.Enabled}} {{ end }} block to basically not create the kubernetes resource.

This however does not work if the chart has any dependency, because the value from a label is a string and the only way to skip a dependency chart is to set condition: false, but Chart.Yaml does not allow any tpl operation and we cannot convert the given "string" into a boolean.

Anything else?

No response

[vincentbernaud]

We would also be very interested to have such feature. It would allow to use a single GitRepo to deploy different resources for on-premise and cloud provider managed clusters. For exemple we would like to deploy metalLB only on on-premise clusters and skip the bundle on EKS

[atsai1220]

Does using this selection in the gitrepo object provide the functionality you need?

    targets:
      - clusterSelector:
          matchExpressions:
          - key: environment
            operator: NotIn
            values:
            - dev
            - test

[atsai1220]

We would also be very interested to have such feature. It would allow to use a single GitRepo to deploy different resources for on-premise and cloud provider managed clusters. For exemple we would like to deploy metalLB only on on-premise clusters and skip the bundle on EKS

In this example you could do (verbose example. one selector on a common key should suffice):

  targets:
  - clusterSelector:
      matchExpressions:
      - key: provider
        operator: NotIn
        values:
        - on-prem
        - aks
        - gks
      - key: location
        operator: In
        values:
        - eks

or

  targets:
  - clusterSelector:
      matchExpressions:
      - key: provider
        operator: In
        values:
        - eks

There are plenty of ways to select:

matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". matchExpressions is a list of pod selector requirements. Valid operators include In, NotIn, Exists, and DoesNotExist. The values set must be non-empty in the case of In and NotIn. All of the requirements, from both matchLabels and matchExpressions are ANDed together -- they must all be satisfied in order to match.

https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#resources-that-support-set-based-requirements

[vincentbernaud]

Hi @atsai1220,

Thanks for answering. In our case we don't want to skip an entire GitRepo but only a particular bundle based on a selector.

Lets take 2 clusters, one EKS and one RKE, both having the provider.cattle.io: rke or eks automatic label.

Currently we maintain multiple git repos with the same configuration except for metalLB. Our fleet.yaml for metalLB look like this :

defaultNamespace: ingress-system

helm:
  repo: https://metallb.github.io/metallb
  chart: metallb
  releaseName: metallb
  valuesFiles:
    - values.yaml

diff:
  comparePatches:
    - apiVersion: apiextensions.k8s.io/v1
      kind: CustomResourceDefinition
      name: addresspools.metallb.io
      operations:
        - {"op":"remove", "path":"/spec/conversion/webhook/clientConfig"}
        - {"op":"remove", "path":"/status"}

    - apiVersion: apiextensions.k8s.io/v1
      kind: CustomResourceDefinition
      name: bgppeers.metallb.io
      operations:
        - {"op":"remove", "path":"/spec/conversion/webhook"}
        - {"op":"remove", "path":"/status"}

What would have been nice is a skip option in targetCustomizations that would skip this bundle when the selector match.

Example :

targetCustomizations:
  - name: cluster1
    skip: true
    clusterSelector:
      matchLabels:
        provider.cattle.io: eks

It would allow us to use the same GitRepo on both EKS and RKE while ensuring that RKE clusters still get load balancers. Otherwise we have to maintain multiple git repos for our different providers and keep track of changes between repos.

Thanks !

[janosmiko]

+1 for this. It would be great if you could make it possible to skip only specific bundles using targetCustomization.

[tkazmierczak-tt]

+1 I'm also interested in this functionality