Backups created by policy can be deleted by ArgoCD

[wyattwalter]

We are using ArgoCD to manage both the kube-arangodb operator install as well as a number of ArangoDB deployment objects. I noticed today that some backups that are being scheduled by our BackupPolicy are simply missing. Looking through the event logs, it looks like those backups were indeed created, and then subsequently deleted by ArgoCD.

I believe that what has been happening is that the backup is created on the schedule without an ownerReference object by the policy handler without an ownerReference but it does do a copy of the annotations from the deployment such as argocd.argoproj.io/instance. Then some job is coming along and adding that ownerReference to the backup object. However, if ArgoCD tries to sync with prune set to true on the application in the meantime, it deletes the backup object. This happens because at that moment the object says it's managed by an ArgoCD project (argocd.argoproj.io/instance is set), but the application doesn't have that resource and there's no ownerReference to follow so ArgoCD can know that it's owned by the cluster.

I think that ArgoCD is behaving correctly in this case, and that the ownerReference should be added to the object by the policy handler at creation time.

[jwierzbo]

We have ownerReference set in place in every Backup object

➜ kubectl get arangobackup backup-1 -o yaml
apiVersion: backup.arangodb.com/v1
kind: ArangoBackup
metadata:
  annotations:
  finalizers:
  - arangobackups.backup.arangodb.com/cleanup
  generation: 2
  name: backup-1
  namespace: test
  ownerReferences:
  - apiVersion: database.arangodb.com/v1
    controller: true
    kind: ArangoDeployment
    name: single
    uid: d264d3a9-afb3-4acf-96e7-e177570b3f10

[wyattwalter]

Yes, but not immediately upon creation when it's created by a policy. There's a timing-related issue here where it doesn't at first and the operator comes along at some point and adds that field. Not all backups are deleted by ArgoCD, only ones where a sync happens in that time between when the backup object is created and the ownerReferences field is added.

[ajanikow]

Hello!

We will modify creation of backups for policy.

Best, Adam.

[wyattwalter]

Hey @ajanikow , it looks like this is still an issue in the latest version of the operator. We've worked around it last year by disabling prune on the ArgoCD app, but that has its own problems in certain scenarios. Can we get this in an upcoming release?