percona-server-mongodb-operator
percona-server-mongodb-operator copied to clipboard
Issue with AWS IRSA Configuration for Restore: S3 Object Header Forbidden (403)
Report
When configuring AWS IAM Roles for Service Accounts (IRSA) for backups in the Percona MongoDB Operator, the backup process works correctly. However, restoring from the backup fails with a "Forbidden" error when trying to access S3 objects.
More about the problem
"error": "set resync backup list from the store: init storage: get S3 object header: Forbidden: Forbidden\n\tstatus code: 403"
Steps to reproduce
- Configure AWS IRSA: Set up IAM roles and policies for the Percona MongoDB Operator. Annotate the Kubernetes service account with the IAM role ARN. refer to this doumentation: https://docs.percona.com/percona-backup-mongodb/manage/automate-s3-access.html#iam-roles-for-service-accounts-irsa
- Configure Backup: Set up the Percona MongoDB helm chart to use IRSA for S3 backups. backups are created and listed successfully. backups are configured with psmdb-db helm chart
backup:
enabled: true
pitr:
enabled: false
storages:
s3-us-west:
type: s3
s3:
bucket: S3-BACKUP-BUCKET-NAME-HERE
credentialsSecret: "" # empty and it works, the backup use the aws IRSA
prefix: "someprefix"
region: us-west-3
- Initiate a restore process using the PerconaServerMongoDBRestore CRD.
apiVersion: psmdb.percona.com/v1
kind: PerconaServerMongoDBRestore
metadata:
name: restore1
spec:
clusterName: my-cluster-name
backupName: backup1
storageName: s3-us-west
backupSource:
type: logical
storageName: s3-us-west
destination: s3://S3-BACKUP-BUCKET-NAME-HERE/BACKUP-DESTINATION
s3:
credentialsSecret: "" # empty, i want to use the aws IRSA
region: us-west-2
bucket: S3-BACKUP-BUCKET-NAME-HERE
prefix: "some prefix"
Versions
- Kubernetes: 1.29
- Operator: 1.16.1
- Helm chart psmdb-db: 1.16.2
- Database: mongodb 7.0.8-5
Anything else?
No response