velero still backup a pv with exclude label

[felicianmv]

What steps did you take and what happened:

oc label pvc my-nfs velero.io/exclude-from-backup=true
oc label pv my-nfs velero.io/exclude-from-backup=true
velero backup create pwb-testing --include-namespaces pwb --wait --default-volumes-to-restic

What did you expect to happen: We expected all objects from pwb namespace to be backup, except the PV and pvc above. That's not the case, PV is still backup ( and failing after a while, but we are not interested in that issue ).

The following information will help us better understand what's going on: Our aim here is to backup all objects from a pwb namespace, which has 2 runnings pods. One of the pods has a PV created manually on a nfs share and we want the data from inside PV to be excluded ( we have other means to backup nfs data ), hence we added "exclude-from-backup" label to both PV and pvc.

If you are using velero v1.7.0+:
Please use velero debug --backup <backupname> --restore <restorename> to generate the support bundle, and attach to this issue, more options please refer to velero debug --help

time="2022-10-27T11:21:26Z" level=info msg="Adding pvc my-nfs to additionalItems" backup=openshift-adp/pwb-testing cmd=/velero logSource="pkg/backup/pod_action.go:75" pluginName=velero
time="2022-10-27T11:21:26Z" level=info msg="Excluding item because it has label velero.io/exclude-from-backup=true" backup=openshift-adp/pwb-testing logSource="pkg/backup/item_backupper.go:82" name=my-nfs namespace=pwb resource=persistentvolumeclaims
time="2022-10-27T11:51:00Z" level=info msg="Processing item" backup=openshift-adp/pwb-testing logSource="pkg/backup/backup.go:340" name=my-nfs namespace=pwb progress= resource=persistentvolumeclaims
time="2022-10-27T11:51:00Z" level=info msg="Excluding item because it has label velero.io/exclude-from-backup=true" backup=openshift-adp/pwb-testing logSource="pkg/backup/item_backupper.go:82" name=my-nfs namespace=pwb resource=persistentvolumeclaims
time="2022-10-27T11:51:00Z" level=info msg="Backed up 2 items out of an estimated total of 78 (estimate will change throughout the backup)" backup=openshift-adp/pwb-testing logSource="pkg/backup/backup.go:380" name=my-nfs namespace=pwb progress= resource=persistentvolumeclaims

If you are using earlier versions:
Please provide the output of the following commands (Pasting long output into a GitHub gist or other pastebin is fine.)

• kubectl logs deployment/velero -n velero
• velero backup describe <backupname> or kubectl get backup/<backupname> -n velero -o yaml
• velero backup logs <backupname>
• velero restore describe <restorename> or kubectl get restore/<restorename> -n velero -o yaml
• velero restore logs <restorename>

Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.]

Environment:

• Velero version (use velero version): 1.9
• Velero features (use velero client config get features): none
• Kubernetes version (use kubectl version):1.23.5
• Kubernetes installer & version:
• Cloud provider or hardware configuration: vsphere
• OS (e.g. from /etc/os-release): Openshift 4.11 / OADP 1.1.0

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

• :+1: for "I would like to see this bug fixed as soon as possible"
• :-1: for "There are more important bugs to focus on right now"

[felicianmv]

Hi all,

I was able to achieve what I wanted using https://velero.io/docs/v1.9/restic/#using-the-opt-out-approach:

oc annotate pod/pwbjavaapp-c748df668-k2gpb backup.velero.io/backup-volumes-excludes=my-nfs

velero backup create pwb-testing --include-namespaces pwb --wait --default-volumes-to-restic

Initially I only checked https://velero.io/docs/v1.9/resource-filtering/#veleroioexclude-from-backuptrue , and my impression was that velero.io/exclude-from-backup=true label has the highest priority, but it seems this method doesn't apply to PVs. Maybe adding a note on this document will help other people avoiding this confusion. Thank you.

[ShubhamTatvamasi]

Hi all,
we are also facing this issue, we have created a schedule for taking a backup every 6 hours, but we always get PVC backup errors when the backup is done. We have added velero.io/exclude-from-backup=true label on PVC to skip those volumes but still no use, the velero schedule cron job is picking it up anyway.

➜  ~ velero get schedules
NAME                    STATUS    CREATED                         SCHEDULE          BACKUP TTL   LAST BACKUP   SELECTOR   PAUSED
staging-6hrly-backup    Enabled   2022-07-04 14:12:38 +0530 IST   0 6,12,18 * * *   336h0m0s     1h ago        <none>     false

If we trigger the backup manually like this, then we don't get any errors.

velero create backup --from-schedule staging-6hrly-backup

Velero Backup	Triggered	Items backed up	Errors
staging-6hrly-backup-20240307070336	Manually	10895	0
staging-6hrly-backup-20240307060058	By schedule cron job	10916	34

[FredNass]

@Lyndon-Li Is this a documentation issue?

PVs data still being backed up despite velero.io/exclude-from-backup=true being set on PVCs is unexpected.

Either the backuper should ignore PV's data when this label is set on PVCs, or the backup reference documentation should clearly state:

• that this label will only exclude the K8s 'Kind' on which it's set from the backup (and still backup the data in case of a PVC)
• that to exclude PV's data from a backup, one should use the opt-out approach and set backup.velero.io/backup-volumes-excludes: <spec.volumes.name> annotation on pods (via inheritance).

What do you think?

[kaovilai]

+1 doc issue.

However, to clarify, backup.velero.io/backup-volumes-excludes: <spec.volumes.name> is for restic/kopia file system backups which is set to the pod mounting those volumes (doesn't have to be PV/PVC).

So if you have restic/kopia enabled, you would have to in addition to excluding velero.io/exclude-from-backup=true which would exclude yaml from backup, also annotate the pod to prevent file system backup of volumes mounted by those pods.