aws-load-balancer-controller
aws-load-balancer-controller copied to clipboard
kubernetes-sigs
Failing to create target group with default policy
Describe the bug The controller is failing to create a target group based on the provided policy.
User: arn:aws:sts::123:assumed-role/AWSALBIngressController_3f2bb898eae5ea79ebdb9cb3514f5ec6/1655374570081000102 is not authorized to perform: elasticloadbalancing:AddTags on resource: arn:aws:elasticloadbalancing:us-west-2:123:targetgroup/foo/*
This error occurs after trying to create the target group and before reconciliation.
Based on my understanding of the policy, AddTags requires a null cluster tag in the request but also requires a non-null resource tag. I'm not sure how that can be possible during target group creation. Especially since the request has the cluster tag and presumably the target group has no tags on creation.
Steps to reproduce
Expected outcome The target group should be created with expected tags
Environment
- AWS Load Balancer controller version 2.2.3
- Kubernetes version v1.17.12-eks-7684af
- Using EKS (yes/no), if so version? yes, v1.17.12-eks-7684af
Additional Context:
@croachrose do you mind sharing a little more details about your EKS environment? Also what does the IAM policy used by the role AWSALBIngressController_3f2bb898eae5ea79ebdb9cb3514f5ec6 looks like? Look for block like this
{
"Effect": "Allow",
"Action": [
"elasticloadbalancing:AddTags",
"elasticloadbalancing:RemoveTags"
],
"Resource": [
"arn:aws:elasticloadbalancing:*:*:targetgroup/*/*",
"arn:aws:elasticloadbalancing:*:*:loadbalancer/net/*/*",
"arn:aws:elasticloadbalancing:*:*:loadbalancer/app/*/*"
],
"Condition": {
"Null": {
"aws:RequestTag/elbv2.k8s.aws/cluster": "true",
"aws:ResourceTag/elbv2.k8s.aws/cluster": "false"
}
}
},
@berry2012 Not sure what other details would be helpful.
Our policy looks exactly like that. We applied the policy from docs/install.
In order to fix this, I temporarily removed
"Condition": {
"Null": {
"aws:RequestTag/elbv2.k8s.aws/cluster": "true",
"aws:ResourceTag/elbv2.k8s.aws/cluster": "false"
}
}
@berry2012 It's almost like AWS changed the implementation of CreateTargetGroup to invoke AddTags if tags are part of the request. In that case, aws:ResourceTag/elbv2.k8s.aws/cluster is null and aws:RequestTag/elbv2.k8s.aws/cluster is not null.
@croachrose from a more recent alb controller, e.g. v2.4.1 with EKS 1.22, I see tags as part of the request so that explains it.
I reproduced this in the environments below without altering the default IAM policy of the alb controller in the documentation and did not encounter the error.
Env 1 AWS Load Balancer controller version 2.3.1 EKS 1.21 Endpoint: Public
Env 2 AWS Load Balancer controller version 2.4.2 EKS 1.22 Endpoint: Public
Your workaround make sense as well for your specific environment. Below works as well
"Condition": {
"Null": {
"aws:RequestTag/elbv2.k8s.aws/cluster": "false",
"aws:ResourceTag/elbv2.k8s.aws/cluster": "false"
}
}
@berry2012 Any idea what is different about our environment? Is it the controller version? For what it's worth, this was working just fine and we changed nothing. It just stopped working suddenly.
@berry2012 update - AWS got back to us and our clusters were upgraded automatically to 1.18 when this started happening. Could k8s 1.18 have issues with controller 2.2.3?
My EKS version is 1.20 (not fresh, but upgraded from 1.15 -> 1.16 -> 1.17 -> 1.18 -> 1.19 -> 1.20) with ALB Ingress Controller v 1.1.5.
When upgrading it to:
-
AWS Load Balancer Controller v2.3.1 No error with the default IAM policy of the alb controller. So, I will keep this version for my environment.
-
AWS Load Balancer Controller v2.4.2 I had to apply @croachrose's workaround to get rid of the error.
In order to fix this, I temporarily removed
"Condition": {
"Null": {
"aws:RequestTag/elbv2.k8s.aws/cluster": "true",
"aws:ResourceTag/elbv2.k8s.aws/cluster": "false"
}
}
Below didn't work either.
"Condition": {
"Null": {
"aws:RequestTag/elbv2.k8s.aws/cluster": "false",
"aws:ResourceTag/elbv2.k8s.aws/cluster": "false"
}
}
@croachrose, if you ran the aws-alb-ingress-controller previously, then you'd need to be on v1.1.3 or later before upgrading to the aws-load-balancer-controller v2.x. You'd also need additional IAM permissions. With appropriate IAM permissions, controller v2.2.3 works with k8s 1.18.
In your case, is the target group an existing one, or controller is trying to create a new target group?
@kishorj We were happily running v2.2.3 for months with no issue and no changes. We were using the exact policy from the install docs (with the tag conditions). It just stopped working one day and removing those conditions fixed it. The target group was not an existing group.
@kishorj We were happily running v2.2.3 for months with no issue and no changes. We were using the exact policy from the install docs (with the tag conditions). It just stopped working one day and removing those conditions fixed it. The target group was not an existing group.
We also ran into the same problem with existing target groups across multiple EKS clusters.
@Glaaj, @croachrose, do you have any automation removing tags on resources? Please share the cloudtrail events from the failure. You can email to k8s-alb-controller-triage AT amazon.com
@Glaaj, were you able to get the cloudtrail events? Also share the policy permissions from the role. You can email them to k8s-alb-controller-triage AT amazon.com
We've run into this issue (or something with very similar symptoms) on a brand new eks cluster using k8s 1.24 and the latest release of the controller. we are still investigating
We've also run into that issue Today. Surprisingly that happened to only one of our clusters that was running fine for over 4 months. We are currently using EKS 1.22 and aws-lb-controller v2.4.2 (we tried also to update to v2.4.6, it didn't help though). Only a cluster in ap-northeast-1 is affected.
Event record
{
"eventVersion": "1.08",
"userIdentity": {
"type": "AssumedRole",
"principalId": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
"arn": "arn:aws:sts::XXXXXXXXXXXX:assumed-role/role-aws-lb-controllerXXXXXXXXXXXXXXXXXXXXXXXXXX/XXXXXXXXXXXXXXXXXXX",
"accountId": "XXXXXXXXXXXX",
"accessKeyId": "XXXXXXXXXXXXXXXXXXXX",
"sessionContext": {
"sessionIssuer": {
"type": "Role",
"principalId": "XXXXXXXXXXXXXXXXXXXXX",
"arn": "arn:aws:iam::XXXXXXXXXXXX:role/role-aws-lb-controllerXXXXXXXXXXXXXXXXXXXXXXXXXX",
"accountId": "XXXXXXXXXXXX",
"userName": "role-aws-lb-controllerXXXXXXXXXXXXXXXXXXXXXXXXXX"
},
"webIdFederationData": {
"federatedProvider": "arn:aws:iam::XXXXXXXXXXXX:oidc-provider/oidc.eks.ap-northeast-1.amazonaws.com/id/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
"attributes": {}
},
"attributes": {
"creationDate": "2023-02-08T02:58:05Z",
"mfaAuthenticated": "false"
}
}
},
"eventTime": "2023-02-08T03:40:24Z",
"eventSource": "elasticloadbalancing.amazonaws.com",
"eventName": "CreateTargetGroup",
"awsRegion": "ap-northeast-1",
"sourceIPAddress": "XXX.XXX.XXX.XXX",
"userAgent": "elbv2.k8s.aws/v2.4.2 aws-sdk-go/1.42.27 (go1.17.10; linux; amd64)",
"errorCode": "AccessDenied",
"errorMessage": "User: arn:aws:sts::XXXXXXXXXXXX:assumed-role/role-aws-lb-controllerXXXXXXXXXXXXXXXXXXXXXXXXXX/XXXXXXXXXXXXXXXXXXX is not authorized to perform: elasticloadbalancing:AddTags on resource: arn:aws:elasticloadbalancing:ap-northeast-1:XXXXXXXXXXXX:targetgroup/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/* because no identity-based policy allows the elasticloadbalancing:AddTags action",
"requestParameters": null,
"responseElements": null,
"requestID": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
"eventID": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
"readOnly": false,
"eventType": "AwsApiCall",
"managementEvent": true,
"recipientAccountId": "XXXXXXXXXXXX",
"eventCategory": "Management",
"tlsDetails": {
"tlsVersion": "TLSv1.2",
"cipherSuite": "ECDHE-RSA-AES128-GCM-SHA256",
"clientProvidedHostHeader": "elasticloadbalancing.ap-northeast-1.amazonaws.com"
}
}
For now we have set up elasticloadbalancing:AddTags without restrictions.
Same issue as described above, only one of ap-southeast-1 clusters affected, happed for the first time today.
aws-load-balancer-controller v2.4.4 EKS 1.24, Platform version eks.3
Policy attached to the role
{
"Statement": [
{
"Action": "iam:CreateServiceLinkedRole",
"Condition": {
"StringEquals": {
"iam:AWSServiceName": "elasticloadbalancing.amazonaws.com"
}
},
"Effect": "Allow",
"Resource": "*",
"Sid": ""
},
{
"Action": [
"elasticloadbalancing:DescribeTargetHealth",
"elasticloadbalancing:DescribeTargetGroups",
"elasticloadbalancing:DescribeTargetGroupAttributes",
"elasticloadbalancing:DescribeTags",
"elasticloadbalancing:DescribeSSLPolicies",
"elasticloadbalancing:DescribeRules",
"elasticloadbalancing:DescribeLoadBalancers",
"elasticloadbalancing:DescribeLoadBalancerAttributes",
"elasticloadbalancing:DescribeListeners",
"elasticloadbalancing:DescribeListenerCertificates",
"ec2:GetCoipPoolUsage",
"ec2:DescribeVpcs",
"ec2:DescribeVpcPeeringConnections",
"ec2:DescribeTags",
"ec2:DescribeSubnets",
"ec2:DescribeSecurityGroups",
"ec2:DescribeNetworkInterfaces",
"ec2:DescribeInternetGateways",
"ec2:DescribeInstances",
"ec2:DescribeCoipPools",
"ec2:DescribeAvailabilityZones",
"ec2:DescribeAddresses",
"ec2:DescribeAccountAttributes"
],
"Effect": "Allow",
"Resource": "*",
"Sid": ""
},
{
"Action": [
"wafv2:GetWebACLForResource",
"wafv2:GetWebACL",
"wafv2:DisassociateWebACL",
"wafv2:AssociateWebACL",
"waf-regional:GetWebACLForResource",
"waf-regional:GetWebACL",
"waf-regional:DisassociateWebACL",
"waf-regional:AssociateWebACL",
"shield:GetSubscriptionState",
"shield:DescribeProtection",
"shield:DeleteProtection",
"shield:CreateProtection",
"iam:ListServerCertificates",
"iam:GetServerCertificate",
"cognito-idp:DescribeUserPoolClient",
"acm:ListCertificates",
"acm:DescribeCertificate"
],
"Effect": "Allow",
"Resource": "*",
"Sid": ""
},
{
"Action": [
"ec2:RevokeSecurityGroupIngress",
"ec2:CreateSecurityGroup",
"ec2:AuthorizeSecurityGroupIngress"
],
"Effect": "Allow",
"Resource": "*",
"Sid": ""
},
{
"Action": "ec2:CreateTags",
"Condition": {
"Null": {
"aws:RequestTag/elbv2.k8s.aws/cluster": "false"
},
"StringEquals": {
"ec2:CreateAction": "CreateSecurityGroup"
}
},
"Effect": "Allow",
"Resource": "arn:aws:ec2:*:*:security-group/*",
"Sid": ""
},
{
"Action": [
"ec2:DeleteTags",
"ec2:CreateTags"
],
"Condition": {
"Null": {
"aws:RequestTag/elbv2.k8s.aws/cluster": "true",
"aws:ResourceTag/elbv2.k8s.aws/cluster": "false"
}
},
"Effect": "Allow",
"Resource": "arn:aws:ec2:*:*:security-group/*",
"Sid": ""
},
{
"Action": [
"ec2:RevokeSecurityGroupIngress",
"ec2:DeleteSecurityGroup",
"ec2:AuthorizeSecurityGroupIngress"
],
"Condition": {
"Null": {
"aws:ResourceTag/elbv2.k8s.aws/cluster": "false"
}
},
"Effect": "Allow",
"Resource": "*",
"Sid": ""
},
{
"Action": [
"elasticloadbalancing:CreateTargetGroup",
"elasticloadbalancing:CreateLoadBalancer"
],
"Condition": {
"Null": {
"aws:RequestTag/elbv2.k8s.aws/cluster": "false"
}
},
"Effect": "Allow",
"Resource": "*",
"Sid": ""
},
{
"Action": [
"elasticloadbalancing:DeleteRule",
"elasticloadbalancing:DeleteListener",
"elasticloadbalancing:CreateRule",
"elasticloadbalancing:CreateListener"
],
"Effect": "Allow",
"Resource": "*",
"Sid": ""
},
{
"Action": [
"elasticloadbalancing:RemoveTags",
"elasticloadbalancing:AddTags"
],
"Condition": {
"Null": {
"aws:RequestTag/elbv2.k8s.aws/cluster": "true",
"aws:ResourceTag/elbv2.k8s.aws/cluster": "false"
}
},
"Effect": "Allow",
"Resource": [
"arn:aws:elasticloadbalancing:*:*:targetgroup/*/*",
"arn:aws:elasticloadbalancing:*:*:loadbalancer/net/*/*",
"arn:aws:elasticloadbalancing:*:*:loadbalancer/app/*/*"
],
"Sid": ""
},
{
"Action": [
"elasticloadbalancing:RemoveTags",
"elasticloadbalancing:AddTags"
],
"Effect": "Allow",
"Resource": [
"arn:aws:elasticloadbalancing:*:*:listener/net/*/*/*",
"arn:aws:elasticloadbalancing:*:*:listener/app/*/*/*",
"arn:aws:elasticloadbalancing:*:*:listener-rule/net/*/*/*",
"arn:aws:elasticloadbalancing:*:*:listener-rule/app/*/*/*"
],
"Sid": ""
},
{
"Action": [
"elasticloadbalancing:SetSubnets",
"elasticloadbalancing:SetSecurityGroups",
"elasticloadbalancing:SetIpAddressType",
"elasticloadbalancing:ModifyTargetGroupAttributes",
"elasticloadbalancing:ModifyTargetGroup",
"elasticloadbalancing:ModifyLoadBalancerAttributes",
"elasticloadbalancing:DeleteTargetGroup",
"elasticloadbalancing:DeleteLoadBalancer"
],
"Condition": {
"Null": {
"aws:ResourceTag/elbv2.k8s.aws/cluster": "false"
}
},
"Effect": "Allow",
"Resource": "*",
"Sid": ""
},
{
"Action": [
"elasticloadbalancing:RegisterTargets",
"elasticloadbalancing:DeregisterTargets"
],
"Effect": "Allow",
"Resource": "arn:aws:elasticloadbalancing:*:*:targetgroup/*/*",
"Sid": ""
},
{
"Action": [
"elasticloadbalancing:SetWebAcl",
"elasticloadbalancing:RemoveListenerCertificates",
"elasticloadbalancing:ModifyRule",
"elasticloadbalancing:ModifyListener",
"elasticloadbalancing:AddListenerCertificates"
],
"Effect": "Allow",
"Resource": "*",
"Sid": ""
}
],
"Version": "2012-10-17"
}
event with error
{
"eventVersion": "1.08",
"userIdentity": {
"type": "AssumedRole",
"principalId": "XXXXXXXXXXXX",
"arn": "arn:aws:sts::XXXXXXXXXXXX:assumed-role/XXX-aws-load-balancer-controller/fXXXXXXXXXXXXXXXXXXX",
"accountId": "XXXXXXXXXXXX",
"accessKeyId": "XXXXXXXXXXXXXXXXXXXXX",
"sessionContext": {
"sessionIssuer": {
"type": "Role",
"principalId": "XXXXXXXXXXXXXXXXXXXXXX",
"arn": "arn:aws:iam::XXXXXXXXXXXX:role/XXX-aws-load-balancer-controller",
"accountId": "XXXXXXXXXXXX",
"userName": "XXX-aws-load-balancer-controller"
},
"webIdFederationData": {
"federatedProvider": "arn:aws:iam::XXXXXXXXXXXX:oidc-provider/oidc.eks.ap-southeast-1.amazonaws.com/id/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
"attributes": {}
},
"attributes": {
"creationDate": "2023-02-09T14:00:28Z",
"mfaAuthenticated": "false"
}
}
},
"eventTime": "2023-02-09T14:39:00Z",
"eventSource": "elasticloadbalancing.amazonaws.com",
"eventName": "CreateTargetGroup",
"awsRegion": "ap-southeast-1",
"sourceIPAddress": "XXX.XXX.XXX.XXX",
"userAgent": "elbv2.k8s.aws/v2.4.4 aws-sdk-go/1.42.27 (go1.18.6; linux; amd64)",
"errorCode": "AccessDenied",
"errorMessage": "User: arn:aws:sts::XXXXXXXXXXXX:assumed-role/XXX-aws-load-balancer-controller/XXXXXXXXXXXXXXXXXXX is not authorized to perform: elasticloadbalancing:AddTags on resource: arn:aws:elasticloadbalancing:ap-southeast-1:XXXXXXXXXXXX:targetgroup/XXXXXXXXXXXXXXXXXXXX/* because no identity-based policy allows the elasticloadbalancing:AddTags action",
"requestParameters": null,
"responseElements": null,
"requestID": "xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx",
"eventID": "xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx",
"readOnly": false,
"eventType": "AwsApiCall",
"managementEvent": true,
"recipientAccountId": "XXXXXXXXXXXX",
"eventCategory": "Management",
"tlsDetails": {
"tlsVersion": "TLSv1.2",
"cipherSuite": "ECDHE-RSA-AES128-GCM-SHA256",
"clientProvidedHostHeader": "elasticloadbalancing.ap-southeast-1.amazonaws.com"
}
}
i also just started seeing this today when spinning up a new eks cluster on kubernetes 1.24 in us-east-1, running aws-load-balancer-controller 2.4.2, using the policy from the docs (https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.4.2/docs/install/iam_policy.json)
We also just started seeing this issue today. Using v2.4.4 (via helm chart v1.4.5) on EKS 1.23 We installed 1 cluster and it worked as expected. Then we installed a second and third cluster and it was failing with the AccessDenied error.
{
"Effect": "Allow",
"Action": [
"elasticloadbalancing:RemoveTags",
"elasticloadbalancing:AddTags"
],
"Resource": [
"arn:aws:elasticloadbalancing:*:*:targetgroup/*/*",
"arn:aws:elasticloadbalancing:*:*:loadbalancer/net/*/*",
"arn:aws:elasticloadbalancing:*:*:loadbalancer/app/*/*"
],
"Condition": {
"Null": {
"aws:RequestTag/elbv2.k8s.aws/cluster": [
"true"
],
"aws:ResourceTag/elbv2.k8s.aws/cluster": [
"false"
]
}
}
},
Removing the Condition block allowed the ALB Controller to function again.
This is so strange because this has been working for us for literally months and this is the first time we've run into it.
We're having the same issue with a new cluster running kubernetes 1.24 and aws-load-balancer-controller 2.4.6 with the policy from https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/main/docs/install/iam_policy.json.
Following for advice.
The same problems occurs on Cluster 1.24 eks.4 alb 2.4.6.
It works if only aws:RequestTag/elbv2.k8s.aws/cluster line is removed, but this seems like a temporary solution.
Hopefully there will be a quick update. 🥲
Same for me here: EKS 1.24 and aws-load-balancer-controller 2.4.2
I'll remove the Condition block temporary.
This is also happening in GovCloud FWIW.
Removing the aforementioned condition fixes this problem.
This is due to recent behavior change on the requirement of explicit elasticloadbalancing:AddTags permission for create resources, and potentially affect new AWS accounts. Please add the following additional IAM policies if you encounter this error:
{
"Effect": "Allow",
"Action": [
"elasticloadbalancing:AddTags"
],
"Resource": [
"arn:aws:elasticloadbalancing:*:*:targetgroup/*/*",
"arn:aws:elasticloadbalancing:*:*:loadbalancer/net/*/*",
"arn:aws:elasticloadbalancing:*:*:loadbalancer/app/*/*"
],
"Condition": {
"StringEquals": {
"elasticloadbalancing:CreateAction": [
"CreateTargetGroup",
"CreateLoadBalancer"
]
},
"Null": {
"aws:RequestTag/elbv2.k8s.aws/cluster": "false"
}
}
}
We will update our recommended policies and documentation.
The above policy change does not work for me unless I also add CreateListener to the elasticloadbalancing:CreateAction list.
Versions: aws-load-balancer-controller v2.4.1, EKS 1.22
ELB team have temporarily rolled back their change regards IAM permissions
I ran into this issue as well. Deleting and creating the service resolved it even without changing the IAM Policy.
I had done this many times before without issue.
The issue occurred on EKS 1.24 ap-southeast-1, controller v2.4.6
