Proposal: support multiple clusters (destinations) for an Application

[guilhermeoki]

Hi all,

Currently, an Application supports a single ApplicationDestination. So, if you have an app spread in multiple clusters, you should create a resource type Application for each cluster.

In this approach, you need to set a different name for each app. Besides that, it's not so clear to track drifts in configuration parameters among applications.

I don't know if this makes sense but my idea is to pass in Application something like a list of ClusterConfig, and in this resource we have the ApplicationDestination and ApplicationSource(Helm|Kustomize|....). What do you think about it?

[alexec]

Multi-cluster support is something we've been discussing. It'd be great if you can add details of the use cases you're trying to solve.

[asvasyanin]

We're running multiple k8s clusters in different datacenters. To deploy a microservice to multiple clusters we have to create almost same Application resource with different name and deploy it multiple times. I guess it could be useful if we can specify more then one ApplicationDestination

[k]

I ran into a problem here as well.

Lets take a monitoring chart, like prometheus-operator. I would like an instance of prometheus-operator on all of my n clusters.

I would also like the application to be deployed with the name prometheus-operator and same namespace to all clusters so when I change contexts I can use the same command to fetch pods and read secrets/configs etc.

Currently there is no way to do this besides launching a separate argocd in each cluster, which means if I want to adhere to the above schema basically renders the cluster feature of argocd useless.

I tried to launch the application Resource in a different namespace with the same name but argocd did not pick it up. (I used the cluster install)

What I would like is one application deployed to multiple clusters, with the ability to select unique helm values files for each Application Destination. Is there a current way to support this that still takes advantage of Argocd's ability to add multiple clusters in one Argocd instance?

BTW: I'm loving Argocd so far really appreciate this project :)

[k]

I just thought of a potential workaround for now, perhaps setting nameOverride and fullnameOverride in the helm chart will work. I realize this is only really a problem for helm charts because the name is taken from the application name, if we could decouple that it would suffice

[mvazquezc]

I see this feature as a must when working on a multi-cluster scenario.

One potential use case is running applications across multiple clusters, let's say I want to run my application in all my clusters that match X criteria, I want ArgoCD to take care of the deployment without needing to define multiple applications with different names within ArgoCD.

X criteria could be:

• Cluster has a specific label
• Cluster has been selected by the user manually using ArgoCD application wizard

With the introduction of a single application that is deployed to multiple clusters, users might need to define "overrides", so they can change the behaviour of the application on specific clusters. e.g: I want to run v1 in prod clusters and v2 on test clusters.

[emichaf]

Running in to the same problem deploying the same microservice in several clusters. Would be great if DRY could be applied in the GITOPS repo, and handling multiple Application destinations in the Application manifest file.

[alexmt]

I agree that Argo CD needs better support for the multi-cluster scenario. It would be very difficult to add this feature into Application CRD. I think instead it makes sense to have higher level CRD (e.g. MultiClusterApplication) which manages applications

[alexec]

I'm trying to get my head around this use case here. It seems there maybe two:

1. A single app that has owns resources in different clusters.
2. A single app that is deployed identically (or with small variations) in different clusters.

Can you please clarify what your after and maybe provide a few real-world examples please?

[dgoodwin]

In our case we've got a system that dynamically provisions clusters based on user interaction with a web UI. Your use case (2) is the one that led me to this issue @alexec.

It looks like Argo's "cluster registry" is effectively the credentials secrets with a given label.

Would it be reasonable to have the ability to dynamically assign applications to clusters with particular additional labels? I'm not sure if this fits with your Application construct, or perhaps some new higher level construct which has a controller to create and prune the specific per-cluster Applications that match a label selector.

i.e. "ApplicationSet", mirrors data in Application but has a label selector for Clusters. Controller monitors the cluster secrets and maintains creation of Applications for each cluster that matches.

[asvasyanin]

I'm trying to get my head around this use case here. It seems there maybe two:

1. A single app that has owns resources in different clusters.
2. A single app that is deployed identically (or with small variations) in different clusters.

Can you please clarify what your after and maybe provide a few real-world examples please?

Our case is second one We have multiple identical clusters in different datacenters. All our clusters are on-prem, cloud is not our case. We are using same identical set of microservices deployed in multiple clusters both for loadbalancing and high availability. Argocd deployed in separate management cluster. At this moment we had to create identical apps in argocd with different names to target different clusters

[salanki]

I have the same need as @alexmojo. Thanks for your consideration.

[asvasyanin]

I can help to implement this feature, but design must be discussed before that

[OmerKahani]

We also have the same use case, I will describe one possible solution so we can start to discuss on the design.

I think that there should be one application with multiple destinations, each destination will have its own settings:

• cluster
• namespace
• value files - helm
• auto-sync

In the UI I think we need tabs inside the application UI, every destination (cluster + namespace) is a tab.

manual sync should be done to every destination (inside the tab), also sync status should be inside the tab. refresh is for all of them at once - outside the tab. health should be inside and also outside at the application level.

[dgoodwin]

Could this be done with a higher level construct, keeping Application as the mapping to a specific cluster and controllers to generate the Applications automatically? It would be nice to be able to map based on label selectors in configurations where the clusters are dynamic (coming and going).

[OmerKahani]

@dgoodwin if you want the application to be the mapping then you can use app-of apps, where you have a child app per destination. I don't like this as I think we don't need to solve everything with apps-on-apps.

[dgoodwin]

@OmerKahani you are thinking more in terms of making the current concept of Application multi-cluster? It seemed like keeping that constant as a building block and adding a higher level construct managing the Applications via a controller would be a less invasive change both in terms of UI, backend and API compatability. (thinking in terms of Deployment->ReplicaSet, cluster API MachineSet -> Machine, etc) I am new to this though so that might be misguided and I'm sure doing multi-cluster support within Application could still meet the goals.

[OmerKahani]

Yes, I prefer a solution where the application can be multi-destination. It is a big change in the UI, but it will make ArgoCD more guided, and therefore more easy to use (comparing to app-of apps)

[alexec]

Hi,

We do not currently plan to implement this - it is an eschewed feature.

Why?

1. Argo CD deploys an app into a single cluster. This allows you to perform a single query to fully understand the state of the app. For an app in multiple clusters we'd need to query each cluster.
2. The single cluster design is fundamental to the code, so the changes we'd need to make would impact performance and reliability too negatively.
3. We know from deploying apps to multiple clusters ourselves, each app requires some customization that cannot be automatically determined. E.g. ingress, load balancers, scaled down or up.

How Can I Do Instead?

For cases where variants of a set of manifests is deployed in two clusters (e.g. for HA/DR), we create two apps, e.g. guestbook-usw2 and guestbook-use2. These can managed by the same Argo CD as well as having any customizations you need.

Currently, you can use the app-of-apps pattern to manage them together. In v1.4 you will be able create apps with labels, and then sync, or wait on them using the same labels. See cluster bootstrapping docs and #1932 for examples.

Thank you,

Alex

[travisghansen]

I know this request is pie in the sky type stuff, but as the number of clusters grow using a tool like argo to manage the 'fleet' of clusters would be a fantastic fit. Creating multiple apps doesn't scale well beyond a handful of clusters unfortunately (consider 'edge' clusters at each POS location for example).

For what it's worth, my vision currently includes building out first-class integration with helmfile which allows for dynamic generation of values and conditionals via ENV variables (to my knowledge not possible with helm directly...at least inside of values.yaml files, obviously could be used with --set). ENV used in conjunction with helmfile's labels and selectors it would be possible to manage a fleet of cluster and tailor quite nicely on a per-cluster basis. In order to facilitate the per-cluster basis argo would need to support ENV vars defined at the cluster level, and at the cluster-to-app level.

Thanks for the consideration!

[adrienjt]

FYI, @gokulchandrap wrote a blog post about using multicluster-scheduler with Argo CD (scroll down to read the relevant part of this very long blog post that also touches Workflows): https://itnext.io/multicluster-scheduler-argo-workflows-across-kubernetes-clusters-ea98016499ca This may be an alternate solution for proposal #3403.

[travisghansen]

This idea is being materialized by the rancher team as well: https://rancher.com/blog/2020/fleet-management-kubernetes/

May be of interest/use to people following this issue.

[jessesuen]

The solution to this issue that Argo team is envisioning, is the ApplicationSet CRD proposal, which can be tracked here: https://github.com/argoproj-labs/applicationset.

Rather than overloading the existing Application object to support multiple destinations, we go higher, with a new abstraction that produces multiple Applications, each to a single destination.

This was inspired in part by the proposal in https://github.com/argoproj/argo-cd/issues/3403. I also believe this will address issue https://github.com/argoproj/argo-cd/issues/2986.

[infa-kparida]

This will be a useful feature indeed.+1

[matoszz]

I have the same use case as @alexmojo - excited to see the ApplicationSet evolve

[virtuman]

How to add support for ApplicationSet into existing argo-cd server? Can't find any docs. thank you

[caiobegotti]

I'm interested on that as well as we're evaluating both Argo and Rancher's Fleet at the company and for others here's the current status about merging ApplicationSet to Argo: Graduating ApplicationSets, and shipping it with ArgoCD https://github.com/argoproj-labs/applicationset/issues/86

[travisghansen]

Does anyone know if applicationsets let you set env vars on a per cluster basis? I haven’t tried them out yet so was just curious.

[ltmleo]

It would be great if there was a way to configure multiple destinations: i.e.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: ''
spec:
  destinations:
    - name: ''
      namespace: ''
      server: ''
      values: ''
    - name: ''
      namespace: ''
      server: ''
      values: ''
  source:
    path: ''
    repoURL: ''
    targetRevision: HEAD
  project: ''

this example is thinking in helm cases.

[gdubya]

@ltmleo have you looked at ApplicationSets?

[lgogolin]

@ltmleo have you looked at ApplicationSets?

ApplicationSet is nice, but then You hitting issues like targetRevision You are unable to deploy single generated Application with different targetRevision.

Let say I want to deploy dev cluster into branch