Add `progressDeadlineSeconds`

[maruina]

To make sure the operator doesn't get stuck in an infinite loop, we need a safeguard mechanism. We want this to be eventual consistent, because there might be a delay in ArgoCD (it might be down, or unable to sync for a time window).

We can use the progressDeadlineSeconds concept as seen in https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.19/#deploymentspec-v1-apps and https://docs.flagger.app/usage/how-it-works

[maruina]

This is a good example about how this is used: https://github.com/fluxcd/flagger/blob/main/pkg/canary/daemonset_ready.go#L74

[maruina]

Considering how ArgoCD support retries with backoff, as per https://argo-cd.readthedocs.io/en/stable/operator-manual/application.yaml

    # The retry feature is available since v1.7
    retry:
      limit: 5 # number of failed sync attempt retries; unlimited number of attempts if less than 0
      backoff:
        duration: 5s # the amount to back off. Default unit is seconds, but could also be a duration (e.g. "2m", "1h")
        factor: 2 # a factor to multiply the base duration after each failed retry
        maxDuration: 3m # the maximum amount of time allowed for the backoff strategy

we can tune ArgoCD to retry how many times we need with the appropriate backoff.

This behavior makes the ProgressDeadline feature less urgent.