ReplicationDestination does not populate data from the pvc when cleanupCachePVC is true "sometimes"

[bornav]

Describe the bug given the manifest specified bellow 2 things can happen, 1: all provisions correctly and works ~20% of the time 2: on replication after successfully getting data from the bucket, it deletes the cache and does not populate the data to the target pvc

Steps to reproduce running the manifest specified bellow should suffice Expected behavior target pvc is supplied with data, and after that the pvc's are cleaned up

Actual results after pulling data it creates the vs-prime volume, and deletes the cache and temp, after which it locks the pvc in maintenance mode, since it can no longer find the data for transfer(assumption)

Additional context

manifest used to replicate the bug
kind: Pod
apiVersion: v1
metadata:
  name: nfs-test
  namespace: apps
spec:
  containers:
    - name: my-frontend
      image: busybox:1.28
      volumeMounts:
      - mountPath: "/ts"
        name: ts
      command: [ "sleep", "1000000" ]
  volumes:
    - name: ts
      persistentVolumeClaim:
        claimName: test
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  labels:
    app: test
  name: test
  namespace: apps
  annotations:
    numberOfReplicas: '1'
spec:
  accessModes: ["ReadWriteMany"]
  storageClassName: longhorn-storage-rwm
  dataSourceRef:
    kind: ReplicationDestination
    apiGroup: volsync.backube
    name: test-restore
  resources:
    requests:
      storage: 1Gi
status: {}
---
apiVersion: volsync.backube/v1alpha1
kind: ReplicationDestination
metadata:
  name: test-restore
  namespace: apps
spec:
  trigger:
    manual: restore-once
  restic:
    repository: restic-test
    storageClassName: longhorn-storage-im
    accessModes: ["ReadWriteOnce"]
    copyMethod: Snapshot
    capacity: 1Gi
    cleanupCachePVC: true
    cleanupTempPVC: true
---
apiVersion: v1
kind: Secret
metadata:
    name: restic-test
    namespace: apps
type: Opaque
stringData:
    RESTIC_REPOSITORY: s3:<>
    RESTIC_PASSWORD: <>
    AWS_ACCESS_KEY_ID: <>
    AWS_SECRET_ACCESS_KEY: <>