percona-server-mongodb-operator icon indicating copy to clipboard operation
percona-server-mongodb-operator copied to clipboard

Issue with AWS IRSA Configuration for Restore: S3 Object Header Forbidden (403)

Open HamoucheTF1 opened this issue 6 months ago • 0 comments

Report

When configuring AWS IAM Roles for Service Accounts (IRSA) for backups in the Percona MongoDB Operator, the backup process works correctly. However, restoring from the backup fails with a "Forbidden" error when trying to access S3 objects.

More about the problem

"error": "set resync backup list from the store: init storage: get S3 object header: Forbidden: Forbidden\n\tstatus code: 403"

Steps to reproduce

  1. Configure AWS IRSA: Set up IAM roles and policies for the Percona MongoDB Operator. Annotate the Kubernetes service account with the IAM role ARN. refer to this doumentation: https://docs.percona.com/percona-backup-mongodb/manage/automate-s3-access.html#iam-roles-for-service-accounts-irsa
  2. Configure Backup: Set up the Percona MongoDB helm chart to use IRSA for S3 backups. backups are created and listed successfully. backups are configured with psmdb-db helm chart
backup:
  enabled: true
  pitr:
    enabled: false
  storages:
    s3-us-west:
      type: s3
      s3:
        bucket: S3-BACKUP-BUCKET-NAME-HERE
        credentialsSecret: "" # empty and it works, the backup use the aws IRSA
        prefix: "someprefix"
        region: us-west-3
  1. Initiate a restore process using the PerconaServerMongoDBRestore CRD.
apiVersion: psmdb.percona.com/v1
kind: PerconaServerMongoDBRestore
metadata:
  name: restore1
spec:
  clusterName: my-cluster-name
  backupName: backup1
  storageName: s3-us-west
 backupSource:
   type: logical
   storageName: s3-us-west
   destination: s3://S3-BACKUP-BUCKET-NAME-HERE/BACKUP-DESTINATION
   s3:
     credentialsSecret: "" # empty, i want to use the aws IRSA
     region: us-west-2
     bucket: S3-BACKUP-BUCKET-NAME-HERE
     prefix: "some prefix"

Versions

  1. Kubernetes: 1.29
  2. Operator: 1.16.1
  3. Helm chart psmdb-db: 1.16.2
  4. Database: mongodb 7.0.8-5

Anything else?

No response

HamoucheTF1 avatar Aug 06 '24 09:08 HamoucheTF1