kubectl-eks icon indicating copy to clipboard operation
kubectl-eks copied to clipboard

eks nodes fails with karpenter nodes

Open rothgar opened this issue 2 years ago • 8 comments

I haven't had time to troubleshoot this yet but I have a cluster with karpenter nodes

k get no                                                                                                                     
NAME                                             STATUS     ROLES    AGE     VERSION                                           
i-001b401a8f4cb3b48.us-east-2.compute.internal   Ready      <none>   61m     v1.27.3                                           
i-002c826ec1bc8c7c9.us-east-2.compute.internal   Ready      <none>   53m     v1.27.3                                           
i-003ca3cd1e27c400c.us-east-2.compute.internal   Ready      <none>   63m     v1.27.3                                           
i-003f808cae84dd6cb.us-east-2.compute.internal   Ready      <none>   69m     v1.27.3                                           
i-004125fe7b0156226.us-east-2.compute.internal   Ready      <none>   45m     v1.27.3

Trying to use the eks plugin gives me this error

k eks nodes                                                                                                                  
NAME     INSTANCE-TYPE     OS     CAPACITY-TYPE     ZONE     AMI-ID     AMI-NAME     AGE                                       
panic: runtime error: invalid memory address or nil pointer dereference                                                        
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x3c739a6]                                                        
                                                                                                                               
goroutine 1 [running]:                                                                                                         
github.com/surajincloud/kubectl-eks/cmd.nodes(0x6193700?, {0x441a0f2?, 0x0?, 0x0?})                                            
        /home/runner/work/kubectl-eks/kubectl-eks/cmd/nodes.go:56 +0x886                                                       
github.com/spf13/cobra.(*Command).execute(0x6193700, {0x62d3ba8, 0x0, 0x0})                                                    
        /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:916 +0x862                                            
github.com/spf13/cobra.(*Command).ExecuteC(0x61939e0)                                                                          
        /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:1044 +0x3bd                                           
github.com/spf13/cobra.(*Command).Execute(...)                                                                                 
        /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:968                                                   
github.com/surajincloud/kubectl-eks/cmd.Execute()                                                                              
        /home/runner/work/kubectl-eks/kubectl-eks/cmd/root.go:25 +0x25                                                         
main.main()                                                                                                                    
        /home/runner/work/kubectl-eks/kubectl-eks/main.go:6 +0x17

I have 2 nodes in a node group

k eks nodegroups                                                                                                             
NAME     RELEASE             AMI_TYPE       INSTANCE_TYPES     STATUS                                                          
main     1.27.1-20230703     AL2_x86_64     t3.large           ACTIVE

version

k eks version
v0.4.1

rothgar avatar Aug 29 '23 19:08 rothgar

Having the same issue when using karpenter nodes.

sgreer81 avatar Sep 01 '23 09:09 sgreer81

/me taking a look

surajnarwade avatar Sep 03 '23 08:09 surajnarwade

Hi @rothgar @sgreer81 I tried reproducing the error but I couldn't get the same error as you

kubectl eks version
v0.4.1

kubectl eks nodes
NAME                                              INSTANCE-TYPE     OS        CAPACITY-TYPE     ZONE           AMI-ID                    AMI-NAME                           AGE
ip-192-168-152-204.eu-west-2.compute.internal     t3.large          linux     ON_DEMAND         eu-west-2c     ami-09d73830fd8e6d486     amazon-eks-node-1.27-v20230825     16m
ip-192-168-168-19.eu-west-2.compute.internal      t3.large          linux     ON_DEMAND         eu-west-2b     ami-09d73830fd8e6d486     amazon-eks-node-1.27-v20230825     16m

kubectl eks nodegroups
NAME     RELEASE             AMI_TYPE       INSTANCE_TYPES     STATUS
main     1.27.4-20230825     AL2_x86_64     t3.large           ACTIVE

surajnarwade avatar Sep 06 '23 10:09 surajnarwade

@rothgar @sgreer81 error is pointing to https://github.com/surajincloud/kubectl-eks/blob/main/cmd/nodes.go#L56. as per code, if there's no nodegroup image label it will look for karpenter image label on the node. I think your nodes don't have that label as well. and code is not really handling that error, i will get it fixed

surajnarwade avatar Sep 06 '23 10:09 surajnarwade

fixed in this PR: https://github.com/surajincloud/kubectl-eks/pull/100

surajnarwade avatar Sep 06 '23 10:09 surajnarwade

@rothgar @sgreer81 can you try v0.4.2

surajnarwade avatar Sep 06 '23 12:09 surajnarwade

That doesn't crash for me although the AMI-NAME looks like it's using the ssm path

k eks nodes                                                                                                                           
NAME                                               INSTANCE-TYPE     OS        CAPACITY-TYPE     ZONE           AMI-ID                  
  AMI-NAME                                                                        AGE                                                   
i-000a8e02bf3ff4081.us-east-2.compute.internal     m5zn.6xlarge      linux     ON-DEMAND         us-east-2c     ami-000c38e2fb6e05120   
  ubuntu-eks/k8s_1.27/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-20230816     6d21h                                                 
i-0052f4cf0b3d058c6.us-east-2.compute.internal     c6a.12xlarge      linux     ON-DEMAND         us-east-2a     ami-000c38e2fb6e05120   
  ubuntu-eks/k8s_1.27/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-20230816     6d21h                                                 
i-00ce125d01e31ffea.us-east-2.compute.internal     c6a.4xlarge       linux     ON-DEMAND         us-east-2c     ami-000c38e2fb6e05120   
  ubuntu-eks/k8s_1.27/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-20230816     6d21h

From your PR fix it looks like you could do another split on / and take [-1] to get the name. I have Amazon Linux nodes in the cluster too and they render properly in the output

rothgar avatar Sep 06 '23 21:09 rothgar

@rothgar it's fixed now: https://github.com/surajincloud/kubectl-eks/pull/104

surajnarwade avatar Sep 08 '23 07:09 surajnarwade