helm-charts icon indicating copy to clipboard operation
helm-charts copied to clipboard

Published 20 hours ago • verified publisher icon Mailu

[BUG] Open relay again with hostPort

Open yelhouti opened this issue 3 years ago • 5 comments

Describe the bug When using flannel as CNI (after my last upgrade), it seemed to handle hostPort with NAT, meaning that source IP is changed. This is not the case when using hostNetwok as it doesn't go through CNI. I would be willing to add a hostNetwork mode to the chart, but I don't have enough knowledge on how the components work. The main constraint for using hostNetwork IMO is that the ports are currently not marked as hostPorts. maybe we could split the front into two parts:

  1. with the mail components (I don't know if we need nginx or we can expose them directly) and we can use hostMetwork on these.
  2. the web components (admin, webclient...) which are exposed using ingress...

Would anyone in the mailu team be willing to discuss this with me and I make the implementation.

Thanks in advance.

Environment

  • Bare Metal (Kubespray)

yelhouti avatar Jan 16 '22 18:01 yelhouti

I don't think this is a good idea. With hostNetwork you expose your whole pods, potentially introducing more issues (e.g. front pod has several ports for internal use only).

If flanel does not preserve the source IP, it's seriously broken (seems so, check https://github.com/kubernetes/kubernetes/issues/29742).

I'd recommend to use either a different CNI or to install some LoadBalancer (like MetalLB) that allows to expose the services in a correct way.

micw avatar Jan 20 '22 20:01 micw

@micw this is why I suggested splitting the front into two parts, one that we can expose and the other that is hidden.

yelhouti avatar Jan 20 '22 20:01 yelhouti

Sorry, I was reading this only half-ways. Splitting would exactly solve what I wrote in the 1st paragraph.

Nevertheless, I'm still not convinced to implement this as a workaround.

I digged a bit deeper. Seems that "hostPort" is independent of the CNI you use and it's a general issue. But I wonder when this occurs. I have set up countless k8s systems using hostPort in the last years and had such an issue only once (messed up firewall rules after cluster upgrade that were fixed after a reboot).

If hostPort is really broken on current K8S, we need to re-think the whole concept of exposing ports without an LB.

Edit: just an idea. If you run on a recent system, there's probably NFTables installed by default which is not compatible with K8S and can also cause such issues. Can you verify that you switched to iptables?

micw avatar Jan 20 '22 21:01 micw

If hostPort is really broken on current K8S, we need to re-think the whole concept of exposing ports without an LB.

It really seems to be the case, these issues appear and deseappear on each upgrade. This is why I think hostNetwork would be a better fit when a loadBalancer is not available.

Edit: just an idea. If you run on a recent system, there's probably NFTables installed by default which is not compatible with K8S and can also cause such issues. Can you verify that you switched to iptables?

~~indeed recent OS here with NFTables thanks for the tip, could you please send me a link to the doc about why and how it's broken ?~~ False alarm nftsables is disabled in my system by default

yelhouti avatar Jan 20 '22 21:01 yelhouti

@micw the preserving of source IP seems to be unspecified behavior, I really think we should switch to hostNetwrok for what is exposed externally, and split with other half. what do you think ?

yelhouti avatar Jan 24 '22 03:01 yelhouti