Associate IAM role with a Service Account (AWS)

[feikesteenbergen]

Documentation

• [ ] Enable https://docs.aws.amazon.com/eks/latest/userguide/enable-iam-roles-for-service-accounts.html
• [ ] eks, associate iam
• [ ] Attach policy to service account with eksctl

Code

• [ ] Generate credentials from AWS_WEB_IDENTITY_TOKEN_FILE: https://aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts/
• [ ] Rotate refresh of credentials (only valid for max 24 hour)
• [ ] Use iam-roles credentials iff available for pgBackRest

Currently, the pgBackRest backup requires secrets to do its backups to S3. These secrets are stored as k8s secrets, however they are specified as plain text in the values.yaml file. By associating an IAM role with a Service Account we remove the need for specifying these secrets in the values altogether.

https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-technical-overview.html

[feikesteenbergen]

We'll need to inject the variables as environment variables for now, as it seems pgBackRest does (not yet) allow transparent use of the IAM role:

https://github.com/pgbackrest/pgbackrest/issues/721

[drpebcak]

Just wanted to update this issue with an open one for pgbackrest where they are talking about this functionality: https://github.com/pgbackrest/pgbackrest/issues/1126

[dwsteele]

Note that this feature has been implemented (but not yet merged) in pgbackrest/pgbackrest#1137. We fully expect it to be included in the next release, currently scheduled for July 25.

[sebastianvoss]

@dwsteele This is great and highly appreciated. Do you have a revised timeline for the release?

[dwsteele]

Do you have a revised timeline for the release?

Well, I meant to say August 25 above. But in any case we have moved it to August 31 so we can get more review of the patch. We're looking pretty good now so I don't anticipate any more delays.

[drpebcak]

Just as a note on this, we are using kube2iam for this sort of thing. For kube2iam its just a matter of adding an annotation to the pod, instead if the serviceaccount (iam.amazonaws.com/role: "role-arn"). It would be nice if that was kept in mind here.

Currently it is already supported by adding to the podAnnotations but just something to keep in mind :)

[rbdevops]

Just add the ability to set annotations to the service account

[pfrydids]

So I was able to get this to work with the following (using Terraform), without any changes to the helm chart

module "pgbackrest_role" {
  source  = "terraform-aws-modules/iam/aws//modules/iam-assumable-role-with-oidc"
  version = "4.1.0"

  create_role      = true
  role_description = "pgbackrest role"
  role_name        = "pgbackrest_role"

  provider_url     = var.cluster_oidc_issuer_url
  role_policy_arns = [
    aws_iam_policy.pgbackrest.arn,
  ]

  oidc_fully_qualified_subjects = [
    "system:serviceaccount:${kubernetes_namespace.main.metadata[0].name}:timescaledb-single",
  ]
}

resource "aws_iam_policy" "pgbackrest" {
  name = "pgbackrest_s3_policy"
  description = "pgBackRest policy for access to a specific S3 bucket"

  policy = jsonencode({
    Version = "2012-10-17",
    Statement = [
      {
        Action = [
          "s3:ListBucket",
          "s3:GetObject",
          "s3:PutObject",
          "s3:DeleteObject",
          "s3:ListMultipartUploadParts",
          "s3:AbortMultipartUpload",
        ],
        Effect = "Allow",
        Resource = [
          S3_BUCKET_BACKUP_ARN,  # Replace with your S3 bucket name
          "S3_BUCKET_BACKUP_ARN/*", # Replace with your S3 bucket name
        ],
      },
    ],
  })
}

Then in the helm chart values I added

secrets:
  pgbackrest:
    PGBACKREST_REPO1_S3_REGION: ...
    PGBACKREST_REPO1_S3_BUCKET: ...
    PGBACKREST_REPO1_S3_ENDPOINT: ... (needs to include region)
    PGBACKREST_REPO1_S3_KEY_TYPE: web-id

serviceAccount:
  annotations:
    "eks.amazonaws.com/role-arn": the arn of the role created above
backup:
  enabled: true
  pgBackRest:
    repo1-type: s3
    repo1-s3-region: ...
    repo1-s3-endpoint: ... (needs to include region)