When GitOps operator is run locally (not installed via OLM), it does not correctly setup the 'argoproj.io' Role rules for the 'argocd-application-controller'

[jgwest]

When GitOps operator is run locally, it does not correctly setup the 'argoproj.io' Role rules for the 'argocd-application-controller' pod/service account.

When all of these are true:

• You run the GitOps operator locally, for example via scripts/run_e2e_tests.sh or with make run-local
• You run it on a cluster that has not previously had the GitOps operator installed via OLM (and likewise not Argo CD operator)

You will find:

• That the argocd-application-controller created within the openshift-gitops namespace is missing permissions to read/write argoproj.io CRs.
• That the following error occurs in the argocd-application-controller pod log: time="2021-06-28T19:46:10Z" level=error msg="Error persisting normalized application spec: applications.argoproj.io \"(application name)\" is forbidden: User \"system:serviceaccount:openshift-gitops:openshift-gitops-argocd-application- controller\" cannot patch resource \"applications\" in API group \"argoproj.io\" in the namespace \"openshift-gitops\"" application=(application name)

This is due to how the role is configured, as of https://github.com/argoproj-labs/argocd-operator/pull/318.

When OLM installs an operator, it adds read/write access to the admin ClusterRole. However, if you don't use OLM to install the GitOps operator, then the admin cluster role will NOT contain the Argo CD CR permissions. Thus the admin cluster role cannot be guaranteed to contain the necessary cluster CRDs.

Unfortunately, as of https://github.com/argoproj-labs/argocd-operator/pull/318, the admin ClusterRole is appended to the openshift-gitops-argocd-application-controller namespace Role. So if the admin ClusterRole is missing the Argo CD CRs (because the operator was not installed via OLM), then the openshift-gitops namespace will be missing permissions to write to Argo CD CRs.

TLDR: we can't rely on the admin ClusterRole to necessarily contain the rules we need, because it is written to by OLM, and not every CRD is installed via OLM.

To reproduce:

1. Start with a clean cluster (one on which no one has previously installed GitOps operator):

   • To verify that you have a clean cluster, run the following command:
   • k get clusterrole/admin -o yaml | grep argoproj.io | wc -l
   • It should return 0

2. cd (path to gitops-operator)

3. Run kubectl apply -f deploy/crds to install the CRDs (ignore the error 'The GitopsService "example-gitopsservice" is invalid')

4. Run make run-local

5. Wait for the openshift-gitops namespace to be created, and wait for Argo CD to be installed in it.

6. Run kubectl get role/openshift-gitops-argocd-application-controller -n openshift-gitops -o yaml

   • Notice that it does NOT contain any rules for 'argoproj.io'

7. Delete the openshift-gitops namespace

8. Now install the latest OpenShift GitOp operator from OperatorHub

9. Wait for the openshift-gitops namespace to be created, and wait for Argo CD to be installed in it.

10. Run kubectl get role/openshift-gitops-argocd-application-controller -n openshift-gitops -o yaml

    • Notice that it DOES contain rules that reference to 'argoproj.io'

The difference between steps 6 and 10 is the issue.

[jgwest]

FYI @shubhamagarwal19

[shubhamagarwal19]

Thanks @jgwest !! Will look into it!

[chetan-rns]

@jgwest @shubhamagarwal19 When running locally, the controller creates the default instance with namespaced mode, preventing it from creating cluster scoped resources. Is this part of the same issue?

[saumeya]

This issue is fixed with #479 but will only work for Openshift Clusters.