Repetitive update of the reconciled VaultSecret status

[sadovnikov]

While preparing to test #37, another problem was discovered. It can be reproduced with both 1.1.2 and 1.2.1.

The scenario is:

• the Vault operator runs in argocd namespace and watches on its own namespace. It runs using default SA
• the git-github VaultSecret is being created in the namespace. It should be synced with the git-github secret

Everything works smoothly, but the following gets logged by the Vault operator

2022-03-11T15:00:56.732Z	ERROR	controller	Reconciler error	{"reconcilerGroup": "maupu.org", "reconcilerKind": "VaultSecret", "controller": "vaultsecret", "name": "git-github", "namespace": "argocd", "error": "Operation cannot be fulfilled on vaultsecrets.maupu.org \"git-github\": the object has been modified; please apply your changes to the latest version and try again"}
github.com/go-logr/zapr.(*zapLogger).Error
	/go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:128
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:237
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:209
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:188
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.Until
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90

Seems the operator attempts to reconcile the secret three times. The first two attempts start at nearly the same time and one of the threads gets into the conflict trying to update the status of the VaultSecret.

This is confirmed by the logs of the API server. it gets two requests to update /apis/maupu.org/v1beta1/namespaces/argocd/vaultsecrets/git-github/status from system:serviceaccount:argocd:default. The second update is rejected with "Conflict" 409.

❯ kubectl -n argocd logs vault-secret-7978575d49-88b77 | grep "Reconciling VaultSecret"
2022-03-11T15:00:41.050Z	INFO	vaultsecret-operator	Reconciling VaultSecret	{"Request.Namespace": "argocd", "Request.Name": "git"}
2022-03-11T15:00:41.244Z	INFO	vaultsecret-operator	Reconciling VaultSecret	{"Request.Namespace": "argocd", "Request.Name": "git-github-internal"}
2022-03-11T15:00:41.322Z	INFO	vaultsecret-operator	Reconciling VaultSecret	{"Request.Namespace": "argocd", "Request.Name": "argocd-secret"}
2022-03-11T15:00:41.387Z	INFO	vaultsecret-operator	Reconciling VaultSecret	{"Request.Namespace": "argocd", "Request.Name": "git"}
2022-03-11T15:00:56.574Z	INFO	vaultsecret-operator	Reconciling VaultSecret	{"Request.Namespace": "argocd", "Request.Name": "git-github"}
2022-03-11T15:00:56.668Z	INFO	vaultsecret-operator	Reconciling VaultSecret	{"Request.Namespace": "argocd", "Request.Name": "git-github"}
2022-03-11T15:00:57.732Z	INFO	vaultsecret-operator	Reconciling VaultSecret	{"Request.Namespace": "argocd", "Request.Name": "git-github"}