kubernetes
kubernetes copied to clipboard
kubernetes
TOB-K8S-024: kubelet liveness probes can be used to enumerate host network
This issue was reported in the Kubernetes Security Audit Report
Description Kubernetes supports both readiness and liveness probes to detect when a Pod is operating correctly, and when to begin or stop directing traffic to a Pod. Three methods are available to facilitate these probes: command execution, HTTP, and TCP.
Using the HTTP and TCP probes, it is possible for an operator with limited access to the cluster (purely kubernetes-service related access) to enumerate the underlying host network. This is possible due to the scope in which these probes execute. Unlike the command execution probe, which will execute a command within the container, the TCP and HTTP probes execute from the context of the kubelet process. Thus, host networking interfaces are used, and the operator is now able to specify hosts which may not be available to Pods kubelet is managing.
The enumeration of the host network uses the container’s health and readiness to determine the status of the remote host. If the pod is killed and restarted due to a failed liveness probe, this indicates that the host is inaccessible. If the Pod successfully passes the liveness check and is presented as ready, the host is accessible. These two states create boolean states of accessible and inaccessible hosts to the underlying host running kubelet.
Additionally, an attacker can append headers through the Pod specification, which are interpreted by the Go HTTP library as authentication or additional request headers. This can allow an attacker to abuse liveness probes to access a wider-range of cluster resources.
An example Pod file that can enumerate the host network is available in Appendix E.
Exploit Scenario Alice configures a cluster which restricts communications between services on the cluster. Eve gains access to Alice’s cluster, and subsequently submits many Pods enumerating the host network in an attempt to gain information about Alice’s underlying host network.
Recommendations Short term, restrict the kubelet in a way that prevents the kubelet from probing hosts it does not manage directly.
Long term, consider restricting probes to the container runtime, allowing liveness to be determined within the scope of the container-networking interface.
Anything else we need to know?:
See #81146 for current status of all issues created from these findings.
The vendor gave this issue an ID of TOB-K8S-024 and it was finding 21 of the report.
The vendor considers this issue Medium Severity.
To view the original finding, begin on page 60 of the Kubernetes Security Review Report
Environment:
- Kubernetes version: 1.13.4
Preventing pods without hostNetwork from probing other hosts should be sufficient. Pods with hostNetwork can already probe the host. PSP controls hostNetwork.
In a previous discussion, we had discussed marking the Host fields of HTTPGetAction and TCPSocketAction as deprecated. It looks like that never actually happened.
/cc @thockin
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale
In a previous discussion, we had discussed marking the
Hostfields ofHTTPGetActionandTCPSocketActionas deprecated. It looks like that never actually happened./cc @thockin
@thockin what do you think about this proposal from @tallclair
This fell into a lot of cracks.
Deprecating the host field seems reasonable, but I don't think it addresses the problem, unless I am missing something?
We could, as Clayton suggests, limit use of this field to hostNetwork pods - either at API time (preferred) or at runtime. But it would have to be opt-in (e.g. another admission controller like externalIPs).
Technically this is not difficult - is it sufficient?
@thockin glad you're back into this, let me prep an answer on Monday. Enjoy your weekend!
Deprecating the host field seems reasonable, but I don't think it addresses the problem, unless I am missing something?
Indeed, it would be like killing a fly with a sledgehammer...
We could, as Clayton suggests, limit use of this field to hostNetwork pods - either at API time (preferred) or at runtime. But it would have to be opt-in (e.g. another admission controller like externalIPs).
Ok, works for me, I think if should be sufficient to mitigate the vulnerability. I guess we'll need a KEP for that?
I'd be open to discussing whether this would be a fit for the Pod Security Standards and inclusion in Pod Security admission by extension. IMO this is hovering around the border of the scope of the pod security standards, but they do already have opinions on HostNetwork, HostPorts, and CAP_NET_RAW.
Any core solution to this is going to need a KEP.
This issue has not been updated in over 1 year, and should be re-triaged.
You can:
- Confirm that this issue is still relevant with
/triage accepted(org members only) - Close this issue with
/close
For more details on the triage process, see https://www.kubernetes.dev/docs/guide/issue-triage/
/remove-triage accepted
This issue has not been updated in over 1 year, and should be re-triaged.
You can:
- Confirm that this issue is still relevant with
/triage accepted(org members only) - Close this issue with
/close
For more details on the triage process, see https://www.kubernetes.dev/docs/guide/issue-triage/
/remove-triage accepted