Fatal: unable to open config file results in PartiallyFailed Backup

[amrap030]

What steps did you take and what happened:

Unfortunately my backups end up being PartiallyFailed due to the following error:

Errors:
  Velero:   message: /pod volume backup failed: data path backup failed: error running restic backup command restic backup --repo=s3:https://***.net/<bucketname>/velero/restic/kube-system --password-file=/tmp/credentials/velero/velero-repo-credentials-repository-password --cache-dir=/scratch/.cache/restic . --tag=pod-uid=1927b692-dda3-4994-b047-335921d6dc2c --tag=volume=socket-dir --tag=backup=velero-daily-20241004171637 --tag=backup-uid=4afba32a-2995-48e1-bd80-cc811de09aeb --tag=ns=kube-system --tag=pod=openstack-cinder-csi-controllerplugin-7f8cf7f5cb-r8ppl --host=velero --json with error: exit status 1 stderr: Fatal: unable to open config file: Stat: Get "https://***.net/<bucketname>/?location=": dial tcp: lookup ***.net: i/o timeout
Is there a repository at the following location?
s3:https://***.net/<bucketname>/velero/restic/kube-system

However, when looking into my bucket with an S3 viewer, there is the repository /velero/restic/kube-system and it also contains the config file along with the snapshots etc.

I already tried setting various proxy settings, because I run this on-premise and the S3 bucket is an on-premise enterprise object storage, but without success. Since the backup files are uploaded to the S3 buckets just fine, I assume the proxy settings are not relevant. I also tried to install restic on my local machine and tried to verify the repository via restic -r s3:https://***.net/<bucketname>/velero/restic/kube-system snapshots which works just fine.

Additionally, I am using the velero/velero-plugin-for-aws:v1.9.0 plugin, as it is an S3 compatible storage.

Since I am running everything in our on-premise environment, I don't really want to add the debug information bundle as it might contain sensitive data.

What did you expect to happen:

I expect that the backup executes just fine without being PartiallyFailed.

The following information will help us better understand what's going on:

If you are using velero v1.7.0+:
Please use velero debug --backup <backupname> --restore <restorename> to generate the support bundle, and attach to this issue, more options please refer to velero debug --help

If you are using earlier versions:
Please provide the output of the following commands (Pasting long output into a GitHub gist or other pastebin is fine.)

• kubectl logs deployment/velero -n velero
• velero backup describe <backupname> or kubectl get backup/<backupname> -n velero -o yaml
• velero backup logs <backupname>
• velero restore describe <restorename> or kubectl get restore/<restorename> -n velero -o yaml
• velero restore logs <restorename>

Anything else you would like to add:

Environment:

• Velero version (use velero version): 1.14.1
• Velero features (use velero client config get features): n/a
• Kubernetes version (use kubectl version): 1.31
• Kubernetes installer & version: Juju Charms
• Cloud provider or hardware configuration: Openstack
• OS (e.g. from /etc/os-release): n/a

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

• :+1: for "I would like to see this bug fixed as soon as possible"
• :-1: for "There are more important bugs to focus on right now"