[argo-cd] Sync podAntiAffinity for application-controller with upstream manifests

[mkilchhofer]

Describe the bug Saw this in Slack:

Jesper Berg Axelsen: Hi guys. We are trying to make the argocd-application-controller highly available and in the helm chart I have changed the value enableStatefulSet to true and created 3 replicas and added ARGOCD_CONTROLLER_REPLICAS to env variables. However, when Kubernetes schedules this statefulset, all my replicas are scheduled on the same node. This means that when this node fails, I completely lose my application controller. Is this intended? Is there any way to secure that these are scheduled on different nodes? Or some other way to work around it? Ref: https://cloud-native.slack.com/archives/C01TSERG0KZ/p1623140214208100

@jannfis from the upstream project answered:

jannfis Hey, in the original manifests (i.e. native installation, not via Helm) there are anti affinity rules for the controller pods. You may want to address this to the Helm chart maintainers (possibly in #argo-helm-charts)

To Reproduce See above

Expected behavior The component application-controller has set an podAntiAffinity.

Screenshots See above

Additional context

• https://github.com/argoproj/argo-cd/issues/4549
• https://github.com/argoproj/argo-cd/pull/4599

[github-actions[bot]]

Stale issue message

[silverraven691]

Not stale

[pdrastil]

@silverraven691 @mkilchhofer I've just checked HA deployment that we are running with 3.11.1 version of the chart and I see both application-controllers on different nodes. Is this still an issue if you have similar configuration?

controller:
  enableStatefulSet: true
  replicas: 2
  env:
    # Must match number of replicas
    - name: ARGOCD_CONTROLLER_REPLICAS
      value: "2"
  # Add pod anti-affinity for HA deployment
  affinity:
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
        - podAffinityTerm:
            labelSelector:
              matchLabels:
                app.kubernetes.io/name: argocd-application-controller
            topologyKey: kubernetes.io/hostname
          weight: 100
        - podAffinityTerm:
            labelSelector:
              matchLabels:
                app.kubernetes.io/part-of: argocd
            topologyKey: kubernetes.io/hostname
          weight: 5

[mkilchhofer]

Yup, you can inject an affinity config through values, but my intent was to implement something which is present automatically (maybe if replica is >=1 or so). When someone uses the kustomize manifest provided by the upstream repo, this is automatically set:

      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchLabels:
                  app.kubernetes.io/name: argocd-application-controller
              topologyKey: kubernetes.io/hostname
          - weight: 5
            podAffinityTerm:
              labelSelector:
                matchLabels:
                  app.kubernetes.io/part-of: argocd
              topologyKey: kubernetes.io/hostname

Ref: https://github.com/argoproj/argo-cd/blob/a894d4b128c724129752bac9971c903ab6c650ba/manifests/base/application-controller/argocd-application-controller-statefulset.yaml#L145-L159

[pdrastil]

For automatic affinity I would be careful. Example from upstream repo will work for the most users but sometimes you want to tweak this to avoid certain nodes or place the pods into specific availability zones in cloud offerings. I would probably place an example in values.yaml so anyone can quickly start from default configuration but I wouldn't enforce something smart because you don't know anything about environment that will be used for the deployment.

[mkilchhofer]

Hmm and what about implement something like Bitnami in the global section and set the default to soft?

Name	Description	Value
podAffinityPreset	Pod affinity preset. Ignored if affinity is set. Allowed values: soft or hard	""
podAntiAffinityPreset	Pod anti-affinity preset. Ignored if affinity is set. Allowed values: soft or hard	soft

Source: https://github.com/bitnami/charts/tree/master/bitnami/external-dns

If a user override the affinity per component, then the preset would be inactive for this component.

[pdrastil]

That is probably good idea - have some soft default to support minikube / kind / k3s setups that run with single node and if someone needs enterprise deployment with custom affinity rules they can be still configured individually via affinity field for each component.

[pdrastil]

Closed via https://github.com/argoproj/argo-helm/pull/1794