Cannot generate report with Amazon EKS

[arun-gupta]

Trying to run the Docker command on an EKS cluster and getting the following error. kubectl proxy is already running:

eks $ docker run -it --user=$(id -u) --net=host -v $(pwd)/output:/output hjacobs/kube-resource-report:0.2.1 /output
INFO:kube_resource_report.report:Querying cluster localhost:8001 (http://localhost:8001/)..
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): localhost:8001
ERROR:kube_resource_report.report:HTTPConnectionPool(host='localhost', port=8001): Max retries exceeded with url: /api/v1/nodes (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f8257213c18>: Failed to establish a new connection: [Errno 111] Connection refused'))
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 171, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw)
  File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 79, in create_connection
    raise err
  File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 69, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 354, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.7/http/client.py", line 1229, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.7/http/client.py", line 1275, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.7/http/client.py", line 1224, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.7/http/client.py", line 1016, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.7/http/client.py", line 956, in send
    self.connect()
  File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 196, in connect
    conn = self._new_conn()
  File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 180, in _new_conn
    self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f8257213c18>: Failed to establish a new connection: [Errno 111] Connection refused

Any ideas?

[arun-gupta]

Here is how the cluster was created:

eksctl create cluster --name myeks --nodes 4 --region us-west-2
2018-09-17T19:58:35-07:00 [ℹ]  setting availability zones to [us-west-2b us-west-2c us-west-2a]
2018-09-17T19:58:39-07:00 [ℹ]  using "ami-08cab282f9979fc7a" for nodes
2018-09-17T19:58:39-07:00 [ℹ]  creating EKS cluster "myeks" in "us-west-2" region
2018-09-17T19:58:39-07:00 [ℹ]  will create 2 separate CloudFormation stacks for cluster itself and the initial nodegroup
2018-09-17T19:58:39-07:00 [ℹ]  if you encounter any issues, check CloudFormation console or try 'eksctl utils describe-stacks --region=us-west-2 --name=myeks'
2018-09-17T19:58:39-07:00 [ℹ]  creating cluster stack "eksctl-myeks-cluster"
2018-09-17T20:10:35-07:00 [ℹ]  creating nodegroup stack "eksctl-myeks-nodegroup-0"
2018-09-17T20:14:06-07:00 [✔]  all EKS cluster resource for "myeks" had been created
2018-09-17T20:14:06-07:00 [✔]  saved kubeconfig as "kubectl-eks-gpu"
2018-09-17T20:14:25-07:00 [ℹ]  the cluster has 1 nodes
2018-09-17T20:14:25-07:00 [ℹ]  node "ip-192-168-120-73.us-west-2.compute.internal" is not ready
2018-09-17T20:14:25-07:00 [ℹ]  waiting for at least 4 nodes to become ready
2018-09-17T20:14:56-07:00 [ℹ]  the cluster has 4 nodes
2018-09-17T20:14:56-07:00 [ℹ]  node "ip-192-168-115-46.us-west-2.compute.internal" is ready
2018-09-17T20:14:56-07:00 [ℹ]  node "ip-192-168-120-73.us-west-2.compute.internal" is ready
2018-09-17T20:14:56-07:00 [ℹ]  node "ip-192-168-159-205.us-west-2.compute.internal" is not ready
2018-09-17T20:14:56-07:00 [ℹ]  node "ip-192-168-204-115.us-west-2.compute.internal" is ready
2018-09-17T20:15:10-07:00 [ℹ]  kubectl command should work with "kubectl-eks-gpu", try 'kubectl --kubeconfig=kubectl-eks-gpu get nodes'
2018-09-17T20:15:10-07:00 [✔]  EKS cluster "myeks" in "us-west-2" region is ready

Generated output directory is at output.tar.gz

[hjacobs]

@arun-gupta are you on Mac? Can you try deploying kube-resource-report directly (instead of kubectl proxy)? --net=host is not the actual machine network if you use Docker for Mac (because of the VM). See also https://github.com/hjacobs/kube-resource-report/issues/38

[arun-gupta]

@hjacobs Yes on Mac but this is an EKS cluster running in the cloud.

[tabossert]

Any Update on this?

[hjacobs]

@tabossert I'm pretty sure kube-resource-report just works when deploying directly (kubectl apply). Please try it out.

[tabossert]

Hi @hjacobs, thanks for responding. I did just that and deployed it out, here is what I am seeing

WARNING:kube_resource_report.report:Failed to query metrics: 404 Client Error: Not Found for url: https://10.100.0.1:443/apis/metrics.k8s.io/v1beta1/nodes WARNING:kube_resource_report.report:Failed to query metrics: 404 Client Error: Not Found for url: https://10.100.0.1:443/apis/metrics.k8s.io/v1beta1/pods ERROR:kube_resource_report.report:'rules' Traceback (most recent call last): File "/kube_resource_report/report.py", line 330, in get_cluster_summaries summary = future.result() File "/usr/local/lib/python3.7/concurrent/futures/_base.py", line 425, in result return self.__get_result() File "/usr/local/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result raise self._exception File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs) File "/kube_resource_report/report.py", line 261, in query_cluster for rule in item["spec"]["rules"]: KeyError: 'rules'