CNP vs kubelet probes

[danwinship]

NP has the annoying rule that all pods always allow all connections from their own node, to ensure that health/liveness/etc probes work. ANP currently does not need to worry about this, because there is currently no way to write an ANP that blocks traffic from kubelet to a pod However, once we add "deny all" (#248) to CNP (or extend Nodes/Networks to allow them in ingress rules), then we will need to deal with this.

KEP-4559 is supposed to fix this for us, but it will not be done before CNP beta (we haven't even picked a solution yet), so we can't rely on that.

So...

1. We could do nothing and say that if the admin writes a "deny all ingress" policy, then the admin is saying that they don't want kubelet probes to work. (At least not http/tcp probes; exec probes would still work.) Note also that if the admin writes a "deny all ingress" Baseline policy, users would not easily be able to write an "allow kubelet probes" NetworkPolicy to override it. (They would have to write an ipBlock rule with the correct node IPs/CIDRs.)
2. We could allow using Nodes from ingress rules, and probably also add a way to say "SameNode"/"NotSameNode", and then say that, as above, if the admin writes "deny all ingress", then it denies kubelet probes too, but now the admin can instead write "allow from SameNode, and deny all other ingress" (and they would have to write that rather than just "deny all" if they wanted probes to work). (This still has the same problem the NP rule does, where the enforcement hole is too large, since it allows hostNetwork processes other than kubelet too, and has the same problem as the first option, that if the admin does Baseline "deny all ingress", it's hard to override it with NetworkPolicy.
3. We could add a KubeletProbe ingress type, and then otherwise as in # 2.
4. We could add the same automatic/unconfigurable "all pods always allow all connections from their own node" rule that NP has to CNP. (Ugh. Terrible in multiple ways.)
5. We could add an automatic/unconfigurable rule that "all pods always allow connections from their kubelet, but not from other hostNetwork processes on the same node" and leave it to implementations to make that work.

I feel like the last one is probably the best solution? For iptables, nftables, and eBPF-based implementations, "allow only from kubelet" could be done by doing a cgroups check on the packet. OVS doesn't have access to cgroup information on sockets, but since this would only apply to packets starting in the host-network namespace anyway, OVS-based implementations could do something like using a single nftables rule to set a packet mark on connections from kubelet's cgroup, and then check the mark from an OVS rule.

There could potentially be crazy configurations where this would be more problematic to implement, but it seems like those configurations would probably make ordinary NP problematic to implement too? Also, no clue about Windows, but then, according to the e2e tests, Windows doesn't currently implement NetworkPolicy anyway.

[fasaxc]

Calico has always allowed host to pod traffic so if we tighten that, we'd need to put it behind an option.

[npinaeva]

For iptables, nftables, and eBPF-based implementations, "allow only from kubelet" could be done by doing a cgroups check on the packet

As someone who implemented this, I want to add that cgroup-based rules (in ip/nftables) need to be re-created on every kubelet restart (https://patchwork.ozlabs.org/project/netfilter-devel/patch/[email protected]/#1511797) which adds some extra complexity :)

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale