eksctl
eksctl copied to clipboard
eksctl-io
[Bug] Not authorized to access pricing API errors in Karpenter log
What were you trying to accomplish?
Create EKS cluster with Karpenter support.
What happened?
EKS cluster and Karpenter components were created, but have error messages in Karpenter controller log:
2022-09-05T12:16:18.813Z ERROR controller.aws.pricing updating spot pricing, UnauthorizedOperation: You are not authorized to perform this operation.
status code: 403, request id: 8f99fa14-175e-48a5-bfc9-747616e81ded, using existing pricing data from 2022-08-17T00:19:52Z {"commit": "3d87474"}
...
2022-09-05T12:16:19.227Z ERROR controller.aws.pricing updating on-demand pricing, AccessDeniedException: User: arn:aws:sts::269621987045:assumed-role/eksctl-cluster-with-karpenter-iamservice-role/1662380178445292353 is not authorized to perform: pricing:GetProducts because no identity-based policy allows the pricing:GetProducts action
status code: 400, request id: ca453ae3-e67e-4d3f-9c9a-a1b54958a6e6; AccessDeniedException: User: arn:aws:sts::269621987045:assumed-role/eksctl-cluster-with-karpenter-iamservice-role/1662380178445292353 is not authorized to perform: pricing:GetProducts because no identity-based policy allows the pricing:GetProducts action
status code: 400, request id: 4976ddb4-3943-4f34-ba76-d647e638d2f6, using existing pricing data from 2022-08-17T00:19:52Z {"commit": "3d87474"}
How to reproduce it?
Create EKS cluster with following config file:
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
name: cluster-with-karpenter
region: ap-northeast-1
version: '1.22'
tags:
karpenter.sh/discovery: cluster-with-karpenter
iam:
withOIDC: true
karpenter:
version: '0.15.0'
createServiceAccount: true # default is false
managedNodeGroups:
- name: managed-ng-1
minSize: 1
maxSize: 2
desiredCapacity: 1
Logs
Log of Karpenter controller:
2022-09-05T12:16:18.399Z INFO Successfully created the logger.
2022-09-05T12:16:18.399Z INFO Logging level set to: debug
{"level":"info","ts":1662380178.4083061,"logger":"fallback","caller":"injection/injection.go:61","msg":"Starting informers..."}
2022-09-05T12:16:18.408Z INFO controller Initializing with version v0.15.0 {"commit": "3d87474"}
2022-09-05T12:16:18.445Z DEBUG controller.aws Using AWS region ap-northeast-1 {"commit": "3d87474"}
2022-09-05T12:16:18.661Z DEBUG controller.aws Discovered caBundle, length 1099 {"commit": "3d87474"}
2022-09-05T12:16:18.661Z INFO controller loading config from karpenter/karpenter-global-settings {"commit": "3d87474"}
2022-09-05T12:16:18.661Z INFO controller.aws.pricing Updating EC2 pricing information {"commit": "3d87474"}
I0905 12:16:18.771350 1 leaderelection.go:243] attempting to acquire leader lease karpenter/karpenter-leader-election...
2022-09-05T12:16:18.771Z INFO controller starting metrics server {"commit": "3d87474", "path": "/metrics"}
I0905 12:16:18.800252 1 leaderelection.go:253] successfully acquired lease karpenter/karpenter-leader-election
2022-09-05T12:16:18.813Z ERROR controller.aws.pricing updating spot pricing, UnauthorizedOperation: You are not authorized to perform this operation.
status code: 403, request id: 8f99fa14-175e-48a5-bfc9-747616e81ded, using existing pricing data from 2022-08-17T00:19:52Z {"commit": "3d87474"}
2022-09-05T12:16:18.872Z DEBUG controller.aws.launchtemplate Hydrating the launch template cache with tags matching "karpenter.k8s.aws/cluster: cluster-with-karpenter" {"commit": "3d87474"}
2022-09-05T12:16:18.872Z INFO controller.controller.provisioning Starting EventSource {"commit": "3d87474", "reconciler group": "", "reconciler kind": "Pod", "source": "kind source: /, Kind="}
2022-09-05T12:16:18.872Z INFO controller.controller.provisioning Starting Controller {"commit": "3d87474", "reconciler group": "", "reconciler kind": "Pod"}
2022-09-05T12:16:18.872Z INFO controller.controller.provisioning Starting workers {"commit": "3d87474", "reconciler group": "", "reconciler kind": "Pod", "worker count": 10}
2022-09-05T12:16:18.872Z INFO controller.controller.node-state Starting EventSource {"commit": "3d87474", "reconciler group": "", "reconciler kind": "Node", "source": "kind source: /, Kind="}
2022-09-05T12:16:18.872Z INFO controller.controller.node-state Starting Controller {"commit": "3d87474", "reconciler group": "", "reconciler kind": "Node"}
2022-09-05T12:16:18.872Z INFO controller.controller.pod-state Starting EventSource {"commit": "3d87474", "reconciler group": "", "reconciler kind": "Pod", "source": "kind source: /, Kind="}
2022-09-05T12:16:18.872Z INFO controller.controller.pod-state Starting Controller {"commit": "3d87474", "reconciler group": "", "reconciler kind": "Pod"}
2022-09-05T12:16:18.872Z INFO controller.controller.node Starting EventSource {"commit": "3d87474", "reconciler group": "", "reconciler kind": "Node", "source": "kind source: /, Kind="}
2022-09-05T12:16:18.872Z INFO controller.controller.node Starting EventSource {"commit": "3d87474", "reconciler group": "", "reconciler kind": "Node", "source": "kind source: /, Kind="}
2022-09-05T12:16:18.872Z INFO controller.controller.node Starting EventSource {"commit": "3d87474", "reconciler group": "", "reconciler kind": "Node", "source": "kind source: /, Kind="}
2022-09-05T12:16:18.872Z INFO controller.controller.node Starting Controller {"commit": "3d87474", "reconciler group": "", "reconciler kind": "Node"}
2022-09-05T12:16:18.873Z INFO controller.controller.termination Starting EventSource {"commit": "3d87474", "reconciler group": "", "reconciler kind": "Node", "source": "kind source: /, Kind="}
2022-09-05T12:16:18.873Z INFO controller.controller.termination Starting Controller {"commit": "3d87474", "reconciler group": "", "reconciler kind": "Node"}
2022-09-05T12:16:18.873Z INFO controller.controller.podmetrics Starting EventSource {"commit": "3d87474", "reconciler group": "", "reconciler kind": "Pod", "source": "kind source: /, Kind="}
2022-09-05T12:16:18.873Z INFO controller.controller.podmetrics Starting Controller {"commit": "3d87474", "reconciler group": "", "reconciler kind": "Pod"}
2022-09-05T12:16:18.873Z INFO controller.controller.provisionermetrics Starting EventSource {"commit": "3d87474", "reconciler group": "karpenter.sh", "reconciler kind": "Provisioner", "source": "kind source: /, Kind="}
2022-09-05T12:16:18.873Z INFO controller.controller.provisionermetrics Starting Controller {"commit": "3d87474", "reconciler group": "karpenter.sh", "reconciler kind": "Provisioner"}
2022-09-05T12:16:18.873Z INFO controller.controller.counter Starting EventSource {"commit": "3d87474", "reconciler group": "karpenter.sh", "reconciler kind": "Provisioner", "source": "kind source: /, Kind="}
2022-09-05T12:16:18.873Z INFO controller.controller.counter Starting EventSource {"commit": "3d87474", "reconciler group": "karpenter.sh", "reconciler kind": "Provisioner", "source": "kind source: /, Kind="}
2022-09-05T12:16:18.873Z INFO controller.controller.counter Starting Controller {"commit": "3d87474", "reconciler group": "karpenter.sh", "reconciler kind": "Provisioner"}
2022-09-05T12:16:18.965Z DEBUG controller.aws.launchtemplate Finished hydrating the launch template cache with 0 items {"commit": "3d87474"}
2022-09-05T12:16:18.973Z INFO controller.controller.pod-state Starting workers {"commit": "3d87474", "reconciler group": "", "reconciler kind": "Pod", "worker count": 10}
2022-09-05T12:16:18.973Z INFO controller.controller.node-state Starting workers {"commit": "3d87474", "reconciler group": "", "reconciler kind": "Node", "worker count": 10}
2022-09-05T12:16:18.973Z INFO controller.controller.termination Starting workers {"commit": "3d87474", "reconciler group": "", "reconciler kind": "Node", "worker count": 10}
2022-09-05T12:16:18.973Z INFO controller.controller.podmetrics Starting workers {"commit": "3d87474", "reconciler group": "", "reconciler kind": "Pod", "worker count": 1}
2022-09-05T12:16:19.073Z INFO controller.controller.provisionermetrics Starting workers {"commit": "3d87474", "reconciler group": "karpenter.sh", "reconciler kind": "Provisioner", "worker count": 1}
2022-09-05T12:16:19.074Z INFO controller.controller.node Starting workers {"commit": "3d87474", "reconciler group": "", "reconciler kind": "Node", "worker count": 10}
2022-09-05T12:16:19.075Z INFO controller.controller.counter Starting workers {"commit": "3d87474", "reconciler group": "karpenter.sh", "reconciler kind": "Provisioner", "worker count": 10}
2022-09-05T12:16:19.227Z ERROR controller.aws.pricing updating on-demand pricing, AccessDeniedException: User: arn:aws:sts::269621987045:assumed-role/eksctl-cluster-with-karpenter-iamservice-role/1662380178445292353 is not authorized to perform: pricing:GetProducts because no identity-based policy allows the pricing:GetProducts action
status code: 400, request id: ca453ae3-e67e-4d3f-9c9a-a1b54958a6e6; AccessDeniedException: User: arn:aws:sts::269621987045:assumed-role/eksctl-cluster-with-karpenter-iamservice-role/1662380178445292353 is not authorized to perform: pricing:GetProducts because no identity-based policy allows the pricing:GetProducts action
status code: 400, request id: 4976ddb4-3943-4f34-ba76-d647e638d2f6, using existing pricing data from 2022-08-17T00:19:52Z {"commit": "3d87474"}
Anything else we need to know?
The IAM policy for Karpenter lacks of some actions when compared to Karpenter CloudFormation: Karpenter IAM policy in eksctl: https://github.com/weaveworks/eksctl/blob/main/pkg/cfn/builder/karpenter.go#L112 Karpenter IAM policy in Karpenter Cloudformation: https://github.com/aws/karpenter/blob/main/website/content/en/v0.15.0/getting-started/getting-started-with-eksctl/cloudformation.yaml#L43
Versions
$ eksctl info
eksctl version: 0.110.0
kubectl version: v1.22.6-eks-7d68063
OS: linux
