The security token included in the request is invalid, status code: 403

[hcho1989]

I followed this blog (https://aws.amazon.com/blogs/containers/api-gateway-as-an-ingress-controller-for-eks/) to setup an eks cluster with managed nodes and api-gateway ingress controller.

Right after Step 6 in the blog, I was supposed to find api-reverse-proxy pods, but I cannot find them there.

In the blog it suggested to view the logs. Here is the error I found:

{"level":"error","ts":1605260574.0323148,"caller":"ingress/ingress_controller.go:281","msg":"error describing stack","error":"InvalidClientTokenId: The security token included in the request is invalid\n\tstatus code: 403, request id: 30ed34f6-d088-4ec6-b751-94098cf358b7","stacktrace":"github.com/awslabs/amazon-apigateway-ingress-controller/pkg/controller/ingress.(*ReconcileIngress).Reconcile\n\t/go/src/github.com/awslabs/amazon-apigateway-ingress-controller/pkg/controller/ingress/ingress_controller.go:281\ngithub.com/awslabs/amazon-apigateway-ingress-controller/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/awslabs/amazon-apigateway-ingress-controller/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:215\ngithub.com/awslabs/amazon-apigateway-ingress-controller/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/awslabs/amazon-apigateway-ingress-controller/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/awslabs/amazon-apigateway-ingress-controller/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/awslabs/amazon-apigateway-ingress-controller/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/awslabs/amazon-apigateway-ingress-controller/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/awslabs/amazon-apigateway-ingress-controller/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/awslabs/amazon-apigateway-ingress-controller/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/awslabs/amazon-apigateway-ingress-controller/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"error","ts":1605260574.032469,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"ingress-controller","request":"default/api-95d8427d","error":"InvalidClientTokenId: The security token included in the request is invalid\n\tstatus code: 403, request id: 30ed34f6-d088-4ec6-b751-94098cf358b7","stacktrace":"github.com/awslabs/amazon-apigateway-ingress-controller/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/awslabs/amazon-apigateway-ingress-controller/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/awslabs/amazon-apigateway-ingress-controller/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/awslabs/amazon-apigateway-ingress-controller/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\ngithub.com/awslabs/amazon-apigateway-ingress-controller/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/awslabs/amazon-apigateway-ingress-controller/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/awslabs/amazon-apigateway-ingress-controller/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/awslabs/amazon-apigateway-ingress-controller/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/awslabs/amazon-apigateway-ingress-controller/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/awslabs/amazon-apigateway-ingress-controller/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/awslabs/amazon-apigateway-ingress-controller/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/awslabs/amazon-apigateway-ingress-controller/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}

It seems that there is something wrong with the security token.

Also I am supposed to find the corresponding api-gateway created, but I cannot find it.

Anyone can help?

[aditya-inapp]

Are you creating it manually or using Terraform?

[hcho1989]

Are you creating it manually or using Terraform?

I followed the blog, created the cluster with eksctl cli tool.

[hcho1989]

Here is the prettified log message

github.com/awslabs/amazon-apigateway-ingress-controller/pkg/controller/ingress.(*ReconcileIngress).Reconcile
    /go/src/github.com/awslabs/amazon-apigateway-ingress-controller/pkg/controller/ingress/ingress_controller.go:281
    github.com/awslabs/amazon-apigateway-ingress-controller/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /go/src/github.com/awslabs/amazon-apigateway-ingress-controller/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:215
    github.com/awslabs/amazon-apigateway-ingress-controller/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1
    /go/src/github.com/awslabs/amazon-apigateway-ingress-controller/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158
    github.com/awslabs/amazon-apigateway-ingress-controller/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1
    /go/src/github.com/awslabs/amazon-apigateway-ingress-controller/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
    github.com/awslabs/amazon-apigateway-ingress-controller/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil
    /go/src/github.com/awslabs/amazon-apigateway-ingress-controller/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134
    github.com/awslabs/amazon-apigateway-ingress-controller/vendor/k8s.io/apimachinery/pkg/util/wait.Until
    /go/src/github.com/awslabs/amazon-apigateway-ingress-controller/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88

something to do with cloudFormationAPI.DescribeStacks ?

What am I missing? credentials?