Ability to add KMS access to IAM policy for ebs-csi-driver when using EBS default encryption

[jrbeilke]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

What is the outcome that you are trying to reach?

Maybe I'm missing something, but we have an AWS account with default EBS encryption enabled via one of our KMS keys, and this is causing issues when deploying PVs with the ebs-csi-driver since it doesn't have access to the KMS key.

Seems the AWS docs recommend adding an IAM Policy granting access to the KMS key for the ebs-csi-driver, but I'm not seeing an easy way to do this via the aws-ebs-csi-driver module: https://docs.aws.amazon.com/eks/latest/userguide/csi-iam-role.html

Describe the solution you would like

Perhaps the aws-ebs-csi-driver module could be updated to include a variable for a KMS key id, and if that variable is present then the necessary IAM policy would be added.

Describe alternatives you have considered

Another option would be to provide the IAM role for the aws-ebs-csi-driver module as an output, and then we could add the KMS policy to the role ourselves after the module is applied.

If all else fails I suppose we'll have to go the self-managed route and manage our own role with var.helm_config.additional_iam_policies

Additional context

Out of the box on a new AWS account with EBS default encryption via KMS this causes the Prometheus add-on deployment to fail because the ebs-csi-driver cannot provision the necessary volumes: Failed to provision volume with StorageClass "gp2": failed to create encrypted volume: the volume disappeared after creation, most likely due to inaccessible KMS encryption key

Here's an example of the Terraform config:

module "aws_controllers" {
  source = "github.com/aws-ia/terraform-aws-eks-blueprints//modules/kubernetes-addons?ref=v4.6.0"

  eks_cluster_id = data.aws_eks_cluster.cluster.id

  #---------------------------------------------------------------
  # Use AWS controllers separately
  # So that it can delete resources it created from other addons or workloads
  #---------------------------------------------------------------

  # EKS Addons
  enable_amazon_eks_vpc_cni            = true
  enable_amazon_eks_coredns            = true
  enable_amazon_eks_kube_proxy         = true
  enable_amazon_eks_aws_ebs_csi_driver = true

  # K8s Add-ons for AWS
  enable_aws_for_fluentbit            = true
  enable_aws_load_balancer_controller = true
  enable_external_dns                 = true
  eks_cluster_domain                  = var.eks_cluster_domain
}

module "eks_common_addons" {
  source = "github.com/aws-ia/terraform-aws-eks-blueprints//modules/kubernetes-addons?ref=v4.6.0"

  eks_cluster_id = data.aws_eks_cluster.cluster.id

  # K8s Add-ons
  enable_cluster_autoscaler = true
  enable_metrics_server     = true
  enable_prometheus         = true

  depends_on = [module.aws_controllers]
}

[florentio]

Hello @jrbeilke you can provide additional policies to the ebs-csi-driver policy by using the using amazon_eks_aws_ebs_csi_driver_config https://github.com/aws-ia/terraform-aws-eks-blueprints/blob/f9a7ebdae3454f7997590268166ad519df9f6617/modules/kubernetes-addons/variables.tf#L103

In your use case his can solve the problem, here is the policy as described here https://docs.aws.amazon.com/eks/latest/userguide/csi-iam-role.html

data "aws_iam_policy_document" "aws_ebs_csi_driver_kms" {
  statement {
    sid       = ""
    effect    = "Allow"
    resources = ["custom-key-id"]
    actions   = [
        "kms:CreateGrant",
        "kms:ListGrants",
        "kms:RevokeGrant"
      ]

    condition {
      test     = "Bool"
      variable = "kms:GrantIsForAWSResource"
      values   = ["true"]
    }
  }

  statement {
    sid       = ""
    effect    = "Allow"
    resources = ["custom-key-id"]
    actions   = [
        "kms:Encrypt",
        "kms:Decrypt",
        "kms:ReEncrypt*",
        "kms:GenerateDataKey*",
        "kms:DescribeKey"
      ]

  }
}

The policy document will be

resource "aws_iam_policy" "aws_ebs_csi_driver_kms" {
  name        = "KMS_Key_For_Encryption_On_EBS_Policy "
  description = "IAM Policy for KMS permission for AWS EBS CSI Driver"
  policy      = data.aws_iam_policy_document.aws_ebs_csi_driver_kms.json
}

then your aws_controllers will use the policy to set additional_iam_policies

module "aws_controllers" {
  source = "github.com/aws-ia/terraform-aws-eks-blueprints//modules/kubernetes-addons?ref=v4.6.0"

  eks_cluster_id = data.aws_eks_cluster.cluster.id

  #---------------------------------------------------------------
  # Use AWS controllers separately
  # So that it can delete resources it created from other addons or workloads
  #---------------------------------------------------------------

  # EKS Addons
  enable_amazon_eks_vpc_cni            = true
  enable_amazon_eks_coredns            = true
  enable_amazon_eks_kube_proxy         = true
  enable_amazon_eks_aws_ebs_csi_driver = true
  amazon_eks_aws_ebs_csi_driver_config = { 
     additional_iam_policies = [aws_iam_policy.aws_ebs_csi_driver_kms.arn] 
}

  # K8s Add-ons for AWS
  enable_aws_for_fluentbit            = true
  enable_aws_load_balancer_controller = true
  enable_external_dns                 = true
  eks_cluster_domain                  = var.eks_cluster_domain
}

[askulkarni2]

@jrbeilke did @florentio's solution work for you?

[bryantbiggs]

thank you @florentio - closing with the correct guidance provided above