kube-hunter
kube-hunter copied to clipboard
Kube Hunter couldn't find any clusters - Pod Mode - AWS EKS
What happened
Running the standard kube-hunter pod/job on an AWS EKS cluster returns the message Kube Hunter couldn't find any clusters
I've verified that i can manually connect to the API endpoint from within the pod using curl and the secrets available in the pod.
/kube-hunter # KUBE_TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
/kube-hunter # curl -sSk -H "Authorization: Bearer $KUBE_TOKEN" https://$KUBERNETES_SERVICE_HOST:$KUBERNETES_PORT_443_TCP_PORT/version
{
"major": "1",
"minor": "16+",
"gitVersion": "v1.16.8-eks-e16311",
"gitCommit": "e163110a04dcb2f39c3325af96d019b4925419eb",
"gitTreeState": "clean",
"buildDate": "2020-03-27T22:37:12Z",
"goVersion": "go1.13.8",
"compiler": "gc",
"platform": "linux/amd64"
With DEBUG on kube-hunter enabled i see the following exceptions...
2020-06-16 10:07:13,890 INFO kube_hunter.modules.report.collector Found vulnerability "Access to pod's secrets" in Local to Pod (kube-hunter-cert-manager-m7h4z)
2020-06-16 10:07:13,892 DEBUG urllib3.connectionpool Starting new HTTP connection (1): 169.254.169.254:80
2020-06-16 10:07:13,894 DEBUG urllib3.connectionpool http://169.254.169.254:80 "GET /metadata/instance?api-version=2017-08-01 HTTP/1.1" 404 337
2020-06-16 10:07:13,896 DEBUG urllib3.connectionpool Starting new HTTPS connection (1): canhazip.com:443
2020-06-16 10:07:13,928 DEBUG kube_hunter.core.events.handler ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 665, in urlopen
httplib_response = self._make_request(
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 376, in _make_request
self._validate_conn(conn)
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 994, in _validate_conn
conn.connect()
File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 352, in connect
self.sock = ssl_wrap_socket(
File "/usr/local/lib/python3.8/site-packages/urllib3/util/ssl_.py", line 370, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "/usr/local/lib/python3.8/ssl.py", line 500, in wrap_socket
return self.sslsocket_class._create(
File "/usr/local/lib/python3.8/ssl.py", line 1040, in _create
self.do_handshake()
File "/usr/local/lib/python3.8/ssl.py", line 1309, in do_handshake
self._sslobj.do_handshake()
ConnectionResetError: [Errno 104] Connection reset by peer
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
resp = conn.urlopen(
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 719, in urlopen
retries = retries.increment(
File "/usr/local/lib/python3.8/site-packages/urllib3/util/retry.py", line 400, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/usr/local/lib/python3.8/site-packages/urllib3/packages/six.py", line 734, in reraise
raise value.with_traceback(tb)
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 665, in urlopen
httplib_response = self._make_request(
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 376, in _make_request
self._validate_conn(conn)
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 994, in _validate_conn
conn.connect()
File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 352, in connect
self.sock = ssl_wrap_socket(
File "/usr/local/lib/python3.8/site-packages/urllib3/util/ssl_.py", line 370, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "/usr/local/lib/python3.8/ssl.py", line 500, in wrap_socket
return self.sslsocket_class._create(
File "/usr/local/lib/python3.8/ssl.py", line 1040, in _create
self.do_handshake()
File "/usr/local/lib/python3.8/ssl.py", line 1309, in do_handshake
self._sslobj.do_handshake()
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/kube_hunter/core/events/handler.py", line 134, in worker
hook.execute()
File "/usr/local/lib/python3.8/site-packages/kube_hunter/modules/discovery/hosts.py", line 107, in execute
subnets, ext_ip = self.traceroute_discovery()
File "/usr/local/lib/python3.8/site-packages/kube_hunter/modules/discovery/hosts.py", line 140, in traceroute_discovery
external_ip = requests.get("https://canhazip.com", timeout=config.network_timeout).text
File "/usr/local/lib/python3.8/site-packages/requests/api.py", line 76, in get
return request('get', url, params=params, **kwargs)
File "/usr/local/lib/python3.8/site-packages/requests/api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 530, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 643, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 498, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
2020-06-16 10:07:13,931 DEBUG kube_hunter.core.events.handler Event <class 'kube_hunter.core.events.types.HuntFinished'> got published with <kube_hunter.core.events.types.HuntFinished object at 0x7f7574c10760>
2020-06-16 10:07:13,931 DEBUG kube_hunter.core.events.handler Executing <class 'kube_hunter.modules.report.collector.SendFullReport'> with {'previous': None, 'hunter': None}
2020-06-16 10:07:13,947 DEBUG kube_hunter.modules.report.dispatchers Dispatching report via stdout
2020-06-16 10:07:13,947 DEBUG __main__ Cleaned Queue
Expected behavior
I should see the cluster/services vulnerabilities printed.
I think this has been fixed as part of #342, but you've seen it because the kube-hunter image was broken and didn't get updated for a while. This is now sorted out and the latest image on Docker Hub got built two days ago, so this should be corrected now.
The logs show the following line
File "/usr/local/lib/python3.8/site-packages/kube_hunter/modules/discovery/hosts.py", line 140, in traceroute_discovery
external_ip = requests.get("https://canhazip.com", timeout=config.network_timeout).text
but the call to canhaszip.com
was removed as part of #342.
@luciano-nbs please could you try it again, making sure the image is refreshed, and let us know how you get on?
hi Liz,
Im using the master branch with latest commit (commit 2428e2e869b7e07d5cc154d25ebeaceb4cc2085f) and running the pod.yaml in the eks cluster still gives the message of cluster not being found. Despite that message, few information disclosure related issues are being flagged for example; explosure of token inside the pod, /var/run/secrets... Is this the normal behaviour expected? Thanks.
I don't think this is specific to EKS - I've seen the same on a kind cluster running with --pod
:
k logs kube-hunter-hcg98
2020-09-03 14:15:19,483 INFO kube_hunter.modules.report.collector Started hunting
2020-09-03 14:15:19,483 INFO kube_hunter.modules.report.collector Discovering Open Kubernetes Services
2020-09-03 14:15:19,494 INFO kube_hunter.modules.report.collector Found vulnerability "CAP_NET_RAW Enabled" in Local to Pod (kube-hunter-hcg98)
2020-09-03 14:15:19,495 INFO kube_hunter.modules.report.collector Found vulnerability "Read access to pod's service account token" in Local to Pod (kube-hunter-hcg98)
2020-09-03 14:15:19,496 INFO kube_hunter.modules.report.collector Found vulnerability "Access to pod's secrets" in Local to Pod (kube-hunter-hcg98)
Vulnerabilities
For further information about a vulnerability, search its ID in:
https://github.com/aquasecurity/kube-hunter/tree/master/docs/_kb
+--------+----------------------+-------------+----------------------+----------------------+----------------------+
| ID | LOCATION | CATEGORY | VULNERABILITY | DESCRIPTION | EVIDENCE |
+--------+----------------------+-------------+----------------------+----------------------+----------------------+
| None | Local to Pod (kube- | Access Risk | CAP_NET_RAW Enabled | CAP_NET_RAW is | |
| | hunter-hcg98) | | | enabled by default | |
| | | | | for pods. | |
| | | | | If an attacker | |
| | | | | manages to | |
| | | | | compromise a pod, | |
| | | | | they could | |
| | | | | potentially take | |
| | | | | advantage of this | |
| | | | | capability to | |
| | | | | perform network | |
| | | | | attacks on other | |
| | | | | pods running on the | |
| | | | | same node | |
+--------+----------------------+-------------+----------------------+----------------------+----------------------+
| None | Local to Pod (kube- | Access Risk | Access to pod's | Accessing the pod's | ['/var/run/secrets/k |
| | hunter-hcg98) | | secrets | secrets within a | ubernetes.io/service |
| | | | | compromised pod | ... |
| | | | | might disclose | |
| | | | | valuable data to a | |
| | | | | potential attacker | |
+--------+----------------------+-------------+----------------------+----------------------+----------------------+
| KHV050 | Local to Pod (kube- | Access Risk | Read access to pod's | Accessing the pod | eyJhbGciOiJSUzI1NiIs |
| | hunter-hcg98) | | service account | service account | ImtpZCI6IiJ9.eyJpc3M |
| | | | token | token gives an | ... |
| | | | | attacker the option | |
| | | | | to use the server | |
| | | | | API | |
+--------+----------------------+-------------+----------------------+----------------------+----------------------+
Kube Hunter couldn't find any clusters
Yeah, I had similar issue on EKS today. The message appears on the bottom after the vulns are reported just like @lizrice posted above. Nothing major. Seems like normal behavior but the message at the bottom is a big confusing indeed.
@magnologan
Could you please run the master branch again in your setup with --log debug
?
I would like to see why this is happening. Are you able to get the version from within the new master branch pod using curl?
i got same issue for eks : ths debug info: @danielsagi The Job "kube-hunter" is invalid: spec.template: Invalid value: core.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:"", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"controller-uid":"7065ea70-693d-414b-a150-d2f0d030922f", "job-name":"kube-hunter"}, Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(*v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:core.PodSpec{Volumes:[]core.Volume(nil), InitContainers:[]core.Container(nil), Containers:[]core.Container{core.Container{Name:"kube-hunter", Image:"aquasec/kube-hunter", Command:[]string{"kube-hunter"}, Args:[]string{"--interface --log debug"}, WorkingDir:"", Ports:[]core.ContainerPort(nil), EnvFrom:[]core.EnvFromSource(nil), Env:[]core.EnvVar(nil), Resources:core.ResourceRequirements{Limits:core.ResourceList(nil), Requests:core.ResourceList(nil)}, VolumeMounts:[]core.VolumeMount(nil), VolumeDevices:[]core.VolumeDevice(nil), LivenessProbe:(*core.Probe)(nil), ReadinessProbe:(*core.Probe)(nil), Lifecycle:(*core.Lifecycle)(nil), TerminationMessagePath:"/dev/termination-log", TerminationMessagePolicy:"File", ImagePullPolicy:"Always", SecurityContext:(*core.SecurityContext)(nil), Stdin:false, StdinOnce:false, TTY:false}}, RestartPolicy:"Never", TerminationGracePeriodSeconds:(*int64)(0xc0468d9f28), ActiveDeadlineSeconds:(*int64)(nil), DNSPolicy:"ClusterFirst", NodeSelector:map[string]string(nil), ServiceAccountName:"", AutomountServiceAccountToken:(*bool)(nil), NodeName:"", SecurityContext:(*core.PodSecurityContext)(0xc03ebf1260), ImagePullSecrets:[]core.LocalObjectReference(nil), Hostname:"", Subdomain:"", Affinity:(*core.Affinity)(nil), SchedulerName:"default-scheduler", Tolerations:[]core.Toleration(nil), HostAliases:[]core.HostAlias(nil), PriorityClassName:"", Priority:(*int32)(nil), PreemptionPolicy:(*core.PreemptionPolicy)(nil), DNSConfig:(*core.PodDNSConfig)(nil), ReadinessGates:[]core.PodReadinessGate(nil), RuntimeClassName:(*string)(nil), EnableServiceLinks:(*bool)(nil)}}: field is immutable
btw, this k8s version is 1.15
Hi @leozheng2000 Your issue is unrelated, you get a problem in the yaml file. you need to delete the job before you create a new one.
So is there an update for this? Is the message just a red herring if it spits out vulnerabilities beforehand like https://github.com/aquasecurity/kube-hunter/issues/358#issuecomment-686530347