Service account permission issue

[timbuchinger]

Checklist:

• [ ] I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• [ ] I've included steps to reproduce the bug.
• [ ] I've pasted the output of argocd version.

Describe the bug

Occasionally, we are seeing error messages stating that the ArgoCD Server service account does not have permission to create Events.

This could be resolved by adding the create verb here: https://github.com/argoproj/argo-helm/blob/45861fadb78bc79456b97f636a5ab05bf1b8890a/charts/argo-cd/templates/argocd-server/clusterrole.yaml#L22

I can make a PR for this if this is a big and not a misconfiguration on my part.

To Reproduce

We are not clear on what is triggering this action. It is not happening frequently for us.

Expected behavior

The ClusterRole provided with the Helm chart contains appropriate permissions.

Version

argocd: v2.5.3+0c7de21
  BuildDate: 2022-11-28T16:51:33Z
  GitCommit: 0c7de210ae66bf631cc4f27ee1b5cdc0d04c1c96
  GitTreeState: clean
  GoVersion: go1.18.8
  Compiler: gc
  Platform: linux/amd64

Logs

time="2022-12-29T13:47:58Z" level=error msg="Unable to create audit event: events is forbidden: User \"system:serviceaccount:argocd:argocd-server\" cannot create resource \"events\" in API group \"\" in the namespace \"grafana\": Azure does not have opinion for this user." name=grafana-prod reason=ResourceActionRan type=Normal

[scrocquesel]

We do have a similar issue but it seems argo-cd is trying to create event in the target cluster/namespace with its service account token and not the token configured in the target cluster secret

[scrocquesel]

After digging, the issue is because the LogResourceEvent create an event in the cluster the server is running in. Thus ResourceRef of ResourceNode is meaningless to this cluster. logResourceEvent should use the cluster config from the app. It means that the cluster config account must be able to create event to target cluster namespace.

[scrocquesel]

I pushed #12796 to fix argocd was not using the correct cluster config. But I don't know if the clusterrole should still be modified.

[crenshaw-dev]

Hm. I think it makes sense to create the resource events on the Argo CD host cluster. Maybe the resource ref should just be changed to point to the Application?

I can imagine there could be value in creating the events on the destination cluster as well. But I think that should be an opt-in feature rather than on by default.

[scrocquesel]

Hm. I think it makes sense to create the resource events on the Argo CD host cluster. Maybe the resource ref should just be changed to point to the Application?

I can imagine there could be value in creating the events on the destination cluster as well. But I think that should be an opt-in feature rather than on by default.

Make sense, there is already the same event attach to the application. I don't know what was the purpose of this additional log then.

[otherguy]

I just noticed this issue today on 2.10.2.