eksutil
eksutil copied to clipboard
chankh
runtime error, while cluster is found.
{
"errorMessage": "runtime error: invalid memory address or nil pointer dereference",
"errorType": "errorString",
"stackTrace": [
{
"path": "lambda-eks-sample/vendor/github.com/aws/aws-lambda-go/lambda/function.go",
"line": 27,
"label": "(*Function).Invoke.func1"
},
{
"path": "runtime/asm_amd64.s",
"line": 573,
"label": "call32"
},
{
"path": "runtime/panic.go",
"line": 502,
"label": "gopanic"
},
{
"path": "runtime/panic.go",
"line": 63,
"label": "panicmem"
},
{
"path": "runtime/signal_unix.go",
"line": 388,
"label": "sigpanic"
},
{
"path": "lambda-eks-sample/main.go",
"line": 109,
"label": "(*ClusterConfig).loadConfig"
},
{
"path": "lambda-eks-sample/main.go",
"line": 51,
"label": "handler"
},
{
"path": "runtime/asm_amd64.s",
"line": 577,
"label": "call512"
},
{
"path": "reflect/value.go",
"line": 447,
"label": "Value.call"
},
{
"path": "reflect/value.go",
"line": 308,
"label": "Value.Call"
},
{
"path": "lambda-eks-sample/vendor/github.com/aws/aws-lambda-go/lambda/handler.go",
"line": 111,
"label": "newHandler.func1"
},
{
"path": "lambda-eks-sample/vendor/github.com/aws/aws-lambda-go/lambda/handler.go",
"line": 22,
"label": "lambdaHandler.Invoke"
},
{
"path": "lambda-eks-sample/vendor/github.com/aws/aws-lambda-go/lambda/function.go",
"line": 59,
"label": "(*Function).Invoke"
},
{
"path": "runtime/asm_amd64.s",
"line": 574,
"label": "call64"
},
{
"path": "reflect/value.go",
"line": 447,
"label": "Value.call"
},
{
"path": "reflect/value.go",
"line": 308,
"label": "Value.Call"
},
{
"path": "net/rpc/server.go",
"line": 384,
"label": "(*service).call"
},
{
"path": "runtime/asm_amd64.s",
"line": 2361,
"label": "goexit"
}
]
}
START RequestId: 4e8ecdc7-a548-11e8-9f64-250e268ae38f Version: $LATEST
time="2018-08-21T13:44:20Z" level=info msg="Looking up EKS cluster" cluster=xxx
time="2018-08-21T13:44:22Z" level=info msg="Found cluster" cluster=xxx
time="2018-08-21T13:44:22Z" level=debug msg="Cluster details" cluster="<nil>"
runtime error: invalid memory address or nil pointer dereference: errorString
[{"path":"lambda-eks-sample/vendor/github.com/aws/aws-lambda-go/lambda/function.go","line":27,"label":"(*Function).Invoke.func1"},{"path":"runtime/asm_amd64.s","line":573,"label":"call32"},{"path":"runtime/panic.go","line":502,"label":"gopanic"},{"path":"runtime/panic.go","line":63,"label":"panicmem"},{"path":"runtime/signal_unix.go","line":388,"label":"sigpanic"},{"path":"lambda-eks-sample/main.go","line":109,"label":"(*ClusterConfig).loadConfig"},{"path":"lambda-eks-sample/main.go","line":51,"label":"handler"},{"path":"runtime/asm_amd64.s","line":577,"label":"call512"},{"path":"reflect/value.go","line":447,"label":"Value.call"},{"path":"reflect/value.go","line":308,"label":"Value.Call"},{"path":"lambda-eks-sample/vendor/github.com/aws/aws-lambda-go/lambda/handler.go","line":111,"label":"newHandler.func1"},{"path":"lambda-eks-sample/vendor/github.com/aws/aws-lambda-go/lambda/handler.go","line":22,"label":"lambdaHandler.Invoke"},{"path":"lambda-eks-sample/vendor/github.com/aws/aws-lambda-go/lambda/function.go","line":59,"label":"(*Function).Invoke"},{"path":"runtime/asm_amd64.s","line":574,"label":"call64"},{"path":"reflect/value.go","line":447,"label":"Value.call"},{"path":"reflect/value.go","line":308,"label":"Value.Call"},{"path":"net/rpc/server.go","line":384,"label":"(*service).call"},{"path":"runtime/asm_amd64.s","line":2361,"label":"goexit"}]
END RequestId: 4e8ecdc7-a548-11e8-9f64-250e268ae38f
REPORT RequestId: 4e8ecdc7-a548-11e8-9f64-250e268ae38f Duration: 1241.58 ms Billed Duration: 1300 ms Memory Size: 128 MB Max Memory Used: 44 MB
runtime error: invalid memory address or nil pointer dereference
errorString
Getting the above errors while trying to run a test event. The privileges of my role are proper I think, because I tested them outside of AWS lambda environment and work as expected. Any idea if I misconfigured something? Thanks, in advance.
@chankh time="2018-08-21T13:44:22Z" level=debug msg="Cluster details" cluster="<nil>" It looks like my role has not the ability to describe EKS cluster, while I attached the relevant IAM Policy. Initially also my role had more restricted K8s rbac than system:masters but seems that the problem is in a previous state on describing cluster. Any Ideas why attaching AmazonEKSClusterPolicy does not solve this failure?
@stafot you will need the eks:DescribeCluster permission in your Lambda's execution role. I have just updated the README with this info. Thank you for pointing out.
@chankh AmazonEKSClusterPolicy does not have eks:DescribeCluster so this was the reason of not working. Thanks for your answer and your documentation update.
@chankh In a new lambda function I recently created, get the same error, but not always, randomly. Is self-cured so it is not a big deal, but it is very strange. Based on this I checked my code and your eksauth util and all error controls look as expected. I am not reopening because as I said it happens randomly and not affects the reliability of my function, just sharing it if you have any ideas about the cause.
@stafot the cluster name is retrieved from environment variables, so what I can suggest as a way to debug is to print that CLUSTER_NAME environment variable out in the log, and whenever it fails, check to see if the value is set properly.
From cloudwatch logs:
18:47:45
time="2019-03-16T18:47:45Z" level=info msg="Found cluster" cluster=my_cluster
18:47:45
time="2019-03-16T18:47:45Z" level=debug msg="Cluster details" cluster="<nil>"
@chankh Seems that c.ClusterName is set but result.cluster doesn't being set. ref. code
I assume needs to be added a line that will log the aws api error to have more visibility on what gone wrong. Do you agree?Are you ok if I raise a PR with this addition?
@stafot you are right, the errors in loadConfig() needs to be propagated back to the calling function. Looking at your logs, does your Lambda function has the eks:DescribeCluster IAM permission?
@chankh Yes it has the proper permissions. Error is shown in random pattern. It is a cron and 9/10 works fine but ~ 1/10 fails.
@chankh I will raise the relevant PR to increase my visibility to the problem.
time="2019-03-26T18:47:45Z" level=error msg="RequestError: send request failed\ncaused by: Get https://eks.us-east-1.amazonaws.com/clusters/my_cluster: write tcp x.x.x.x:47804->y.y.y.y:443: write: broken pipe" cluster=my_cluster
18:47:45
time="2019-03-26T18:47:45Z" level=fatal msg="Unable to load Kubernetes Client Config: RequestError: send request failed\ncaused by: Get https://eks.us-east-1.amazonaws.com/clusters/my_cluster: write tcp x.x.x.x:47804->y.y.y.y:443: write: broken pipe: RequestError: send request failed\ncaused by: Get https://eks.us-east-1.amazonaws.com/clusters/my_cluster: write tcp x.x.x.x:47804->y.y.y.y:443: write: broken pipe" error="Unable to load Kubernetes Client Config: RequestError: send request failed\ncaused by: Get https://eks.us-east-1.amazonaws.com/clusters/my_cluster: write tcp x.x.x.x:47804->y.y.y.y:443: write: broken pipe: RequestError: send request failed\ncaused by: Get https://eks.us-east-1.amazonaws.com/clusters/my_cluster: write tcp x.x.x.x:47804->y.y.y.y:443: write: broken pipe"
18:47:45
2019/03/26 18:47:45 unexpected EOF
18:47:45
2019/03/26 18:47:45 unexpected EOF
The relevant errors after applied logging enhancement to the lambdas of mine.
/cc @chankh