aws-controllers-k8s
Lambda Controller - appears to require extra IAM permissions when first run for a given account
Describe the bug
Controller needs extra IAM permissions, but only when first installed.
I am using the documented IAM policy from https://raw.githubusercontent.com/aws-controllers-k8s/lambda-controller/main/config/iam/recommended-inline-policy.
For each AWS account and region where I have the lambda controller installed, the first Function I create reports AccessDeniedException: \n\tstatus code: 403 in the controller logs.
If I edit the policy to add Action *, the creation succeeds. I can subsequently remove this Action, and the controller continues to work as advertised.
Does the controller perhaps need to create some resource one time, if it has never run before?
Relevant debug-level logs:
2023-08-31T14:51:44.198Z DEBUG ackrt > r.Sync {"account": "xxx", "role": "", "region": "us-east-1", "kind": "Function", "namespace": "default", "name": "test", "generation": 1}
2023-08-31T14:51:44.198Z DEBUG ackrt >> r.resetConditions {"account": "xxx", "role": "", "region": "us-east-1", "kind": "Function", "namespace": "default", "name": "test", "generation": 1}
2023-08-31T14:51:44.198Z DEBUG ackrt << r.resetConditions {"account": "xxx", "role": "", "region": "us-east-1", "kind": "Function", "namespace": "default", "name": "test", "generation": 1}
2023-08-31T14:51:44.198Z DEBUG ackrt >> rm.ResolveReferences {"account": "xxx", "role": "", "region": "us-east-1", "kind": "Function", "namespace": "default", "name": "test", "is_adopted": false, "generation": 1}
2023-08-31T14:51:44.219Z DEBUG ackrt << rm.ResolveReferences {"account": "xxx", "role": "", "region": "us-east-1", "kind": "Function", "namespace": "default", "name": "test", "is_adopted": false, "generation": 1}
2023-08-31T14:51:44.219Z DEBUG ackrt >> rm.EnsureTags {"account": "xxx", "role": "", "region": "us-east-1", "kind": "Function", "namespace": "default", "name": "test", "is_adopted": false, "generation": 1}
2023-08-31T14:51:44.219Z DEBUG ackrt << rm.EnsureTags {"account": "xxx", "role": "", "region": "us-east-1", "kind": "Function", "namespace": "default", "name": "test", "is_adopted": false, "generation": 1}
2023-08-31T14:51:44.219Z DEBUG ackrt >> rm.ReadOne {"account": "xxx", "role": "", "region": "us-east-1", "kind": "Function", "namespace": "default", "name": "test", "is_adopted": false, "generation": 1}
2023-08-31T14:51:44.219Z DEBUG ackrt >>> rm.sdkFind {"account": "xxx", "role": "", "region": "us-east-1", "kind": "Function", "namespace": "default", "name": "test", "is_adopted": false, "generation": 1}
2023-08-31T14:51:44.238Z DEBUG ackrt <<< rm.sdkFind {"account": "xxx", "role": "", "region": "us-east-1", "kind": "Function", "namespace": "default", "name": "test", "is_adopted": false, "generation": 1, "error": "resource not found"}
2023-08-31T14:51:44.238Z DEBUG ackrt << rm.ReadOne {"account": "xxx", "role": "", "region": "us-east-1", "kind": "Function", "namespace": "default", "name": "test", "is_adopted": false, "generation": 1, "error": "resource not found"}
2023-08-31T14:51:44.238Z DEBUG ackrt >> r.createResource {"account": "xxx", "role": "", "region": "us-east-1", "kind": "Function", "namespace": "default", "name": "test", "is_adopted": false, "generation": 1}
2023-08-31T14:51:44.238Z DEBUG ackrt >>> rm.Create {"account": "xxx", "role": "", "region": "us-east-1", "kind": "Function", "namespace": "default", "name": "test", "is_adopted": false, "generation": 1}
2023-08-31T14:51:44.238Z DEBUG ackrt >>>> rm.sdkCreate {"account": "xxx", "role": "", "region": "us-east-1", "kind": "Function", "namespace": "default", "name": "test", "is_adopted": false, "generation": 1}
2023-08-31T14:51:44.484Z DEBUG ackrt <<<< rm.sdkCreate {"account": "xxx", "role": "", "region": "us-east-1", "kind": "Function", "namespace": "default", "name": "test", "is_adopted": false, "generation": 1, "error": "AccessDeniedException: \n\tstatus code: 403, request id: 29d650e4-5dc3-4971-8627-e6700b3f9b80"}
2023-08-31T14:51:44.484Z DEBUG ackrt <<< rm.Create {"account": "xxx", "role": "", "region": "us-east-1", "kind": "Function", "namespace": "default", "name": "test", "is_adopted": false, "generation": 1, "error": "AccessDeniedException: \n\tstatus code: 403, request id: 29d650e4-5dc3-4971-8627-e6700b3f9b80"}
2023-08-31T14:51:44.484Z DEBUG ackrt << r.createResource {"account": "xxx", "role": "", "region": "us-east-1", "kind": "Function", "namespace": "default", "name": "test", "is_adopted": false, "generation": 1, "error": "AccessDeniedException: \n\tstatus code: 403, request id: 29d650e4-5dc3-4971-8627-e6700b3f9b80"}
2023-08-31T14:51:44.484Z DEBUG ackrt >> r.ensureConditions {"account": "xxx", "role": "", "region": "us-east-1", "kind": "Function", "namespace": "default", "name": "test", "is_adopted": false, "generation": 1}
2023-08-31T14:51:44.484Z DEBUG ackrt >>> rm.IsSynced {"account": "xxx", "role": "", "region": "us-east-1", "kind": "Function", "namespace": "default", "name": "test", "is_adopted": false, "generation": 1}
2023-08-31T14:51:44.484Z DEBUG ackrt <<< rm.IsSynced {"account": "xxx", "role": "", "region": "us-east-1", "kind": "Function", "namespace": "default", "name": "test", "is_adopted": false, "generation": 1}
2023-08-31T14:51:44.484Z DEBUG ackrt << r.ensureConditions {"account": "xxx", "role": "", "region": "us-east-1", "kind": "Function", "namespace": "default", "name": "test", "is_adopted": false, "generation": 1}
2023-08-31T14:51:44.484Z DEBUG ackrt < r.Sync {"account": "xxx", "role": "", "region": "us-east-1", "kind": "Function", "namespace": "default", "name": "test", "is_adopted": false, "generation": 1, "error": "AccessDeniedException: \n\tstatus code: 403, request id: 29d650e4-5dc3-4971-8627-e6700b3f9b80"}
2023-08-31T14:51:44.484Z DEBUG ackrt > r.patchResourceStatus {"account": "xxx", "role": "", "region": "us-east-1", "kind": "Function", "namespace": "default", "name": "test", "is_adopted": false, "generation": 1}
2023-08-31T14:51:44.484Z DEBUG ackrt >> kc.Patch (status) {"account": "xxx", "role": "", "region": "us-east-1", "kind": "Function", "namespace": "default", "name": "test", "is_adopted": false, "generation": 1}
2023-08-31T14:51:44.499Z DEBUG ackrt patched resource status {"account": "xxx", "role": "", "region": "us-east-1", "kind": "Function", "namespace": "default", "name": "test", "is_adopted": false, "generation": 1, "json": "{\"metadata\":{\"resourceVersion\":\"848660735\"},\"spec\":{\"role\":null},\"status\":{\"conditions\":[{\"lastTransitionTime\":\"2023-08-31T14:51:44Z\",\"status\":\"True\",\"type\":\"ACK.ReferencesResolved\"},{\"message\":\"AccessDeniedException: \\n\\tstatus code: 403, request id: 29d65
0e4-5dc3-4971-8627-e6700b3f9b80\",\"status\":\"True\",\"type\":\"ACK.Recoverable\"},{\"lastTransitionTime\":\"2023-08-31T14:51:44Z\",\"message\":\"Unable to determine if desired resource state matches latest observed state\",\"reason\":\"AccessDeniedException: \\n\\tstatus code: 403, request id: 29d650e4-5dc3-4971-8627-e6700b3f9b80\",\"status\":\"Unknown\",\"type\":\"ACK.ResourceSynced\"}]}}"}
2023-08-31T14:51:44.499Z DEBUG ackrt << kc.Patch (status) {"account": "xxx", "role": "", "region": "us-east-1", "kind": "Function", "namespace": "default", "name": "test", "is_adopted": false, "generation": 1}
2023-08-31T14:51:44.499Z DEBUG ackrt < r.patchResourceStatus {"account": "xxx", "role": "", "region": "us-east-1", "kind": "Function", "namespace": "default", "name": "test", "is_adopted": false, "generation": 1}
2023-08-31T14:51:44.499Z ERROR Reconciler error {"controller": "function", "controllerGroup": "lambda.services.k8s.aws", "controllerKind": "Function", "Function": {"name":"test","namespace":"default"}, "namespace": "default", "name": "test", "reconcileID": "32a2612b-2db2-4d3f-89c0-cfa38c956a72", "error": "AccessDeniedException: \n\tstatus code: 403, request id: 29d650e4-5dc3-4971-8627-e6700b3f9b80"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:235
2023-08-31T14:51:44.510Z DEBUG exporter.field-export-reconciler error did not need requeue {"error": "the source resource is not synced yet"}
Steps to reproduce
- have an account/region which has never run lambda controller
- install lambda controller
- create a Function, which will fail
- edit controller IAM policy to add Action
* - Function creation will succeed
- edit controller IAM policy to remove Action
*
Expected outcome
Initial install of controller should have sufficient permissions.
Environment
- Kubernetes version: v1.27.4-eks-2d98532
- Using EKS (yes/no), if so version? yes 1.27
- AWS service targeted (S3, RDS, etc.) Lambda
Hey @rlister,
Thanks for bringing this into our attention. Can you give more details on the template you used to create the function?
We tried to reproduce it in a new region and it works fine with the IAM role you described. Maybe there's something special with the template you are trying. Thanks!
Example function template (applied using helm):
apiVersion: lambda.services.k8s.aws/v1alpha1
kind: Function
metadata:
name: {{ .Release.Name }}
namespace: {{ .Release.Namespace }}
spec:
name: {{ .Release.Name }}
description: {{ now | unixEpoch | quote }}
packageType: Image
code:
imageURI: {{ .Values.image }}
timeout: 180
environment:
variables: {{ toJson .Values.env }}
roleRef:
from:
name: {{ .Release.Name }}
spec.description is set from date to force an update on every helm update, due to issues getting function to update from new images.
Exec role is created with ACK IAM controller:
apiVersion: iam.services.k8s.aws/v1alpha1
kind: Role
metadata:
name: {{ .Release.Name }}
namespace: {{ .Release.Namespace }}
spec:
name: {{ .Release.Name }}
description: {{ .Release.Name }}
assumeRolePolicyDocument: |
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "lambda.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
policies:
- arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
and spec.environment.variables is set from a values object like:
env:
SENTRY_ENVIRONMENT: production
Hey @rlister, thanks for sharing more details on the issue.
I looked into your issue and figured the reason behind access denied is because your ECR repository where your image is stored might not have all the permissions. If you see this documentation, it lists out the two ECR repository policies ecr:BatchGetImage and ecr:GetDownloadUrlForLayer needed in your ECR repo for Lambda to retrieve the image.
In case when Amazon ECR repository does not include these permissions, Lambda adds ecr:BatchGetImage and ecr:GetDownloadUrlForLayer to the container image repository permissions. However Lambda can add these permissions only if the principal calling Lambda has ecr:getRepositoryPolicy and ecr:setRepositoryPolicy permissions.
In your case the recommended-inline-policy do not have the permissions required for Lambda to add the permissions in ECR. When you change policy action to * it adds permissions to do everything including the permissions required by Lambda to set the ECR permissions. This way ECR policy is updated and the next time you create a function using same image, the ECR now has required permission and functions get created properly.
So please check if your ECR repository has the required permissions.
Closing this issue for now. Please let us know if you have any more questions regarding this. Thanks!