Resources are created during dry-run when using `Force=true,Replace=true` sync options

[rafal-jan]

If resources have the argocd.argoproj.io/sync-options: Force=true,Replace=true annotation, they are recreated twice:

• when Argo CD performs dry-run
• when Argo CD performs the actual sync

This behavior can be observed when using multiple Jobs with the argocd.argoproj.io/sync-options: Force=true,Replace=true annotation and different sync waves. All Jobs are created immediately on the cluster during the dry-run phase and then recreated one-by-one as their respective sync wave is processed by Argo CD.

More information about using replace with dry-run and force options can be found in https://github.com/kubernetes/kubectl/issues/1222. kubectl is used as library in gitops-engine but only Run() method is called. However, the fix in https://github.com/kubernetes/kubernetes/pull/110326 updated the Validate() method.

I believe that the force option should not be set to true when performing a dry-run replace operation. This would avoid unnecessary resource recreation and make it usable with sync waves.

[mmclane]

I believe I am seeing this too. I have this annotation set on a K8s job. When that job gets replaced, it happens twice therefore the job runs twice even though it succeeds the first time.

[mmclane]

Having done a little more testing, I have found that the job gets recreated twice when something else has the annotation: argocd.argoproj.io/sync-wave: "-1"

To test this out I created an app that installs a custom helm chart. That helm chart creates two things, a configmap and a job. The job runs a hello world container.

If I have no annotations, ArgoCD can not update the job. To accommodate this I add the annotation argocd.argoproj.io/sync-options: Replace=true,Force=true to the job. Everything works as expected if I change the image tag for that job via a parameter override and the job gets replaced once.

If however I add the annotation argocd.argoproj.io/sync-wave: "-1" to the configmap, then when I update the jobs image tag, ArgoCD will replace the job twice causing it to run twice.

Here are some additional observations:

• If I set the configmap's annotation to argocd.argoproj.io/sync-wave: "0", things work as expected.
• If I set the job to have annotation to argocd.argoproj.io/sync-wave: "1", the job gets replaced twice.

I am running ArgoCD version v2.12.3+6b9cd82

[majusmisiak]

I am observing exactly the same issue.

Contrary to @mmclane I did not find that setting same argocd.argoproj.io/sync-wave on all resources in the application resolves this. What I observe instead is following:

1. If a Job has argocd.argoproj.io/sync-options: Force=true,Replace=true and argocd.argoproj.io/sync-wave: 1 , and there are other resources with argocd.argoproj.io/sync-wave: 0 in the Application, then 2 Job objects are created with the same name, in about 3 seconds interval of each other. Each of these Jobs spawns a duplicate Pod and the first one is immediately set to the Terminating state.
2. If a Job has argocd.argoproj.io/sync-options: Force=true,Replace=true and argocd.argoproj.io/sync-wave: 0 basically same thing is happening. Only difference is that interval between the Jobs is shorter (1 second or maybe less), so it's harder to observe.

This is how it looks in kubectl get pods:

I would also expect that setting ServerSideApply=false flag on an Application or Resource level should fix this (since Argo would internally use kubectl instead of kubernetes library). But ServerSideApply=false seems to have no effect on this bug.

@rafal-jan any chance for having your PR merged?

@rafal-jan or @mmclane did you find any workaround to this issue?

I am running ArgoCD version v2.14.11+8283115

[Jerrimikkihvatai]

Getting the same issue with jobs and different sync waves Any chance for fixing that?