talos
talos copied to clipboard
Published
20 hours ago •
siderolabs
siderolabs
hostPort conflicts with IPVS on 1.7.x
Bug Report
After upgrading to 1.7.6. If a pod listen to hostPort, any request to services with the same port number on the same node will be redirected to this hostPort.
Description
I have a traefik pod listen to hostPort 80, 443 that masks requests to kubernetes API 10.96.0.1:443.
Logs
After 1.7.x, some iptables rules use iptables-nft while hostPort-related items continue to use iptables-legacy.
On 1.6.8:
$ iptables-legacy-save
# Generated by iptables-save v1.8.9 on Sun Aug 25 18:08:14 2024
*nat
:PREROUTING ACCEPT [99:12216]
:INPUT ACCEPT [50:7761]
:OUTPUT ACCEPT [44:7576]
:POSTROUTING ACCEPT [86:11523]
:CNI-DN-b618f359d5a1c38a30eef - [0:0]
:CNI-DN-f7d536801c41904670fe8 - [0:0]
:CNI-DN-f94cfd0b249b096d6a37a - [0:0]
:CNI-HOSTPORT-DNAT - [0:0]
:CNI-HOSTPORT-MASQ - [0:0]
:CNI-HOSTPORT-SETMARK - [0:0]
:KUBE-KUBELET-CANARY - [0:0]
:KUBE-LOAD-BALANCER - [0:0]
:KUBE-MARK-DROP - [0:0]
:KUBE-MARK-MASQ - [0:0]
:KUBE-NODE-PORT - [0:0]
:KUBE-POSTROUTING - [0:0]
:KUBE-SERVICES - [0:0]
-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A PREROUTING -m addrtype --dst-type LOCAL -j CNI-HOSTPORT-DNAT
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT -m addrtype --dst-type LOCAL -j CNI-HOSTPORT-DNAT
-A POSTROUTING -m comment --comment "CNI portfwd requiring masquerade" -j CNI-HOSTPORT-MASQ
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
-A POSTROUTING -s 10.244.0.0/16 -d 10.244.0.0/16 -m comment --comment "flanneld masq" -j RETURN
-A POSTROUTING -s 10.244.0.0/16 ! -d 224.0.0.0/4 -m comment --comment "flanneld masq" -j MASQUERADE --random-fully
-A POSTROUTING ! -s 10.244.0.0/16 -d 10.244.9.0/24 -m comment --comment "flanneld masq" -j RETURN
-A POSTROUTING ! -s 10.244.0.0/16 -d 10.244.0.0/16 -m comment --comment "flanneld masq" -j MASQUERADE --random-fully
-A CNI-DN-b618f359d5a1c38a30eef -s 10.244.9.0/24 -p udp -m udp --dport 3478 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-b618f359d5a1c38a30eef -s 127.0.0.1/32 -p udp -m udp --dport 3478 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-b618f359d5a1c38a30eef -p udp -m udp --dport 3478 -j DNAT --to-destination 10.244.9.67:3478
-A CNI-DN-f7d536801c41904670fe8 -s 10.244.9.0/24 -p tcp -m tcp --dport 80 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-f7d536801c41904670fe8 -s 127.0.0.1/32 -p tcp -m tcp --dport 80 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-f7d536801c41904670fe8 -p tcp -m tcp --dport 80 -j DNAT --to-destination 10.244.9.63:8000
-A CNI-DN-f7d536801c41904670fe8 -s 10.244.9.0/24 -p tcp -m tcp --dport 443 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-f7d536801c41904670fe8 -s 127.0.0.1/32 -p tcp -m tcp --dport 443 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-f7d536801c41904670fe8 -p tcp -m tcp --dport 443 -j DNAT --to-destination 10.244.9.63:8443
-A CNI-DN-f7d536801c41904670fe8 -s 10.244.9.0/24 -p tcp -m tcp --dport 4430 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-f7d536801c41904670fe8 -s 127.0.0.1/32 -p tcp -m tcp --dport 4430 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-f7d536801c41904670fe8 -p tcp -m tcp --dport 4430 -j DNAT --to-destination 10.244.9.63:4430
-A CNI-DN-f94cfd0b249b096d6a37a -s 10.244.9.0/24 -p udp -m udp --dport 8443 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-f94cfd0b249b096d6a37a -s 127.0.0.1/32 -p udp -m udp --dport 8443 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-f94cfd0b249b096d6a37a -p udp -m udp --dport 8443 -j DNAT --to-destination 10.244.9.66:8443
-A CNI-HOSTPORT-DNAT -p tcp -m comment --comment "dnat name: \"cbr0\" id: \"a2b80f7a9e40d142b740cd6a6c2a812c98c963ac1ba72c49e3c58bc872fcdd3c\"" -m multiport --dports 80,443,4430 -j CNI-DN-f7d536801c41904670fe8
-A CNI-HOSTPORT-DNAT -p udp -m comment --comment "dnat name: \"cbr0\" id: \"85ccb4c2d18139b2505184edca9f9fdd3ba8a07b48e7a6c4ae5d5ee6689d7f07\"" -m multiport --dports 8443 -j CNI-DN-f94cfd0b249b096d6a37a
-A CNI-HOSTPORT-DNAT -p udp -m comment --comment "dnat name: \"cbr0\" id: \"1f633b9873e3ec61f505732bf872b6dfa45b448997c70d838553918b2dda67b9\"" -m multiport --dports 3478 -j CNI-DN-b618f359d5a1c38a30eef
-A CNI-HOSTPORT-MASQ -m mark --mark 0x2000/0x2000 -j MASQUERADE
-A CNI-HOSTPORT-SETMARK -m comment --comment "CNI portfwd masquerade mark" -j MARK --set-xmark 0x2000/0x2000
-A KUBE-LOAD-BALANCER -j KUBE-MARK-MASQ
-A KUBE-MARK-DROP -j MARK --set-xmark 0x8000/0x8000
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
-A KUBE-NODE-PORT -p tcp -m comment --comment "Kubernetes nodeport TCP port for masquerade purpose" -m set --match-set KUBE-NODE-PORT-TCP dst -j KUBE-MARK-MASQ
-A KUBE-POSTROUTING -m comment --comment "Kubernetes endpoints dst ip:port, source ip for solving hairpin purpose" -m set --match-set KUBE-LOOP-BACK dst,dst,src -j MASQUERADE
-A KUBE-POSTROUTING -m mark ! --mark 0x4000/0x4000 -j RETURN
-A KUBE-POSTROUTING -j MARK --set-xmark 0x4000/0x0
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -j MASQUERADE --random-fully
-A KUBE-SERVICES ! -s 10.244.0.0/16 -m comment --comment "Kubernetes service cluster ip + port for masquerade purpose" -m set --match-set KUBE-CLUSTER-IP dst,dst -j KUBE-MARK-MASQ
-A KUBE-SERVICES -m addrtype --dst-type LOCAL -j KUBE-NODE-PORT
-A KUBE-SERVICES -m set --match-set KUBE-CLUSTER-IP dst,dst -j ACCEPT
COMMIT
# Completed on Sun Aug 25 18:08:14 2024
# Generated by iptables-save v1.8.9 on Sun Aug 25 18:08:14 2024
*filter
:INPUT ACCEPT [796:154156]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [752:67420]
:KUBE-FIREWALL - [0:0]
:KUBE-FORWARD - [0:0]
:KUBE-KUBELET-CANARY - [0:0]
:KUBE-NODE-PORT - [0:0]
:KUBE-PROXY-FIREWALL - [0:0]
:KUBE-SOURCE-RANGES-FIREWALL - [0:0]
-A INPUT -m comment --comment "kube-proxy firewall rules" -j KUBE-PROXY-FIREWALL
-A INPUT -m comment --comment "kubernetes health check rules" -j KUBE-NODE-PORT
-A INPUT -j KUBE-FIREWALL
-A FORWARD -m comment --comment "kube-proxy firewall rules" -j KUBE-PROXY-FIREWALL
-A FORWARD -m comment --comment "kubernetes forwarding rules" -j KUBE-FORWARD
-A FORWARD -s 10.244.0.0/16 -m comment --comment "flanneld forward" -j ACCEPT
-A FORWARD -d 10.244.0.0/16 -m comment --comment "flanneld forward" -j ACCEPT
-A OUTPUT -j KUBE-FIREWALL
-A KUBE-FIREWALL ! -s 127.0.0.0/8 -d 127.0.0.0/8 -m comment --comment "block incoming localnet connections" -m conntrack ! --ctstate RELATED,ESTABLISHED,DNAT -j DROP
-A KUBE-FIREWALL -m comment --comment "kubernetes firewall for dropping marked packets" -m mark --mark 0x8000/0x8000 -j DROP
-A KUBE-FORWARD -m comment --comment "kubernetes forwarding rules" -m mark --mark 0x4000/0x4000 -j ACCEPT
-A KUBE-FORWARD -m comment --comment "kubernetes forwarding conntrack rule" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A KUBE-NODE-PORT -m comment --comment "Kubernetes health check node port" -m set --match-set KUBE-HEALTH-CHECK-NODE-PORT dst -j ACCEPT
-A KUBE-SOURCE-RANGES-FIREWALL -j DROP
COMMIT
# Completed on Sun Aug 25 18:08:14 2024
# Generated by iptables-save v1.8.9 on Sun Aug 25 18:08:14 2024
On 1.7.6:
$ iptables-legacy-save
# Generated by iptables-save v1.8.9 on Sun Aug 25 17:49:54 2024
*nat
:PREROUTING ACCEPT [1740:192951]
:INPUT ACCEPT [1040:122485]
:OUTPUT ACCEPT [1674:133000]
:POSTROUTING ACCEPT [2176:183720]
:CNI-DN-3fbb375130723d1a84cbe - [0:0]
:CNI-DN-81077802a72ed5ac760e9 - [0:0]
:CNI-DN-83011c2a8020d81033ef2 - [0:0]
:CNI-HOSTPORT-DNAT - [0:0]
:CNI-HOSTPORT-MASQ - [0:0]
:CNI-HOSTPORT-SETMARK - [0:0]
:KUBE-KUBELET-CANARY - [0:0]
:KUBE-MARK-DROP - [0:0]
:KUBE-MARK-MASQ - [0:0]
:KUBE-POSTROUTING - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j CNI-HOSTPORT-DNAT
-A OUTPUT -m addrtype --dst-type LOCAL -j CNI-HOSTPORT-DNAT
-A POSTROUTING -m comment --comment "CNI portfwd requiring masquerade" -j CNI-HOSTPORT-MASQ
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
-A CNI-DN-3fbb375130723d1a84cbe -s 10.244.9.0/24 -p udp -m udp --dport 8443 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-3fbb375130723d1a84cbe -s 127.0.0.1/32 -p udp -m udp --dport 8443 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-3fbb375130723d1a84cbe -p udp -m udp --dport 8443 -j DNAT --to-destination 10.244.9.58:8443
-A CNI-DN-81077802a72ed5ac760e9 -s 10.244.9.0/24 -p tcp -m tcp --dport 80 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-81077802a72ed5ac760e9 -s 127.0.0.1/32 -p tcp -m tcp --dport 80 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-81077802a72ed5ac760e9 -p tcp -m tcp --dport 80 -j DNAT --to-destination 10.244.9.60:8000
-A CNI-DN-81077802a72ed5ac760e9 -s 10.244.9.0/24 -p tcp -m tcp --dport 443 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-81077802a72ed5ac760e9 -s 127.0.0.1/32 -p tcp -m tcp --dport 443 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-81077802a72ed5ac760e9 -p tcp -m tcp --dport 443 -j DNAT --to-destination 10.244.9.60:8443
-A CNI-DN-81077802a72ed5ac760e9 -s 10.244.9.0/24 -p tcp -m tcp --dport 4430 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-81077802a72ed5ac760e9 -s 127.0.0.1/32 -p tcp -m tcp --dport 4430 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-81077802a72ed5ac760e9 -p tcp -m tcp --dport 4430 -j DNAT --to-destination 10.244.9.60:4430
-A CNI-DN-83011c2a8020d81033ef2 -s 10.244.9.0/24 -p udp -m udp --dport 3478 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-83011c2a8020d81033ef2 -s 127.0.0.1/32 -p udp -m udp --dport 3478 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-83011c2a8020d81033ef2 -p udp -m udp --dport 3478 -j DNAT --to-destination 10.244.9.59:3478
-A CNI-HOSTPORT-DNAT -p udp -m comment --comment "dnat name: \"cbr0\" id: \"f4cfae4c920976730e9649b9085239b58ceb374d0de72f6aa90e8678323fb437\"" -m multiport --dports 8443 -j CNI-DN-3fbb375130723d1a84cbe
-A CNI-HOSTPORT-DNAT -p udp -m comment --comment "dnat name: \"cbr0\" id: \"1bb40bd499077e221a51b01c61910c2dfd8a2f4c7e40f28c574faeb2e90f6579\"" -m multiport --dports 3478 -j CNI-DN-83011c2a8020d81033ef2
-A CNI-HOSTPORT-DNAT -p tcp -m comment --comment "dnat name: \"cbr0\" id: \"10d281defe95b891415a9f5da51109459626ab584b58827822834b6959f425d2\"" -m multiport --dports 80,443,4430 -j CNI-DN-81077802a72ed5ac760e9
-A CNI-HOSTPORT-MASQ -m mark --mark 0x2000/0x2000 -j MASQUERADE
-A CNI-HOSTPORT-SETMARK -m comment --comment "CNI portfwd masquerade mark" -j MARK --set-xmark 0x2000/0x2000
-A KUBE-MARK-DROP -j MARK --set-xmark 0x8000/0x8000
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
-A KUBE-POSTROUTING -m mark ! --mark 0x4000/0x4000 -j RETURN
-A KUBE-POSTROUTING -j MARK --set-xmark 0x4000/0x0
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -j MASQUERADE --random-fully
COMMIT
# Completed on Sun Aug 25 17:49:54 2024
# Generated by iptables-save v1.8.9 on Sun Aug 25 17:49:54 2024
*filter
:INPUT ACCEPT [18325:6493414]
:FORWARD ACCEPT [68118:14066587]
:OUTPUT ACCEPT [18124:2101431]
:KUBE-FIREWALL - [0:0]
:KUBE-KUBELET-CANARY - [0:0]
-A INPUT -j KUBE-FIREWALL
-A OUTPUT -j KUBE-FIREWALL
-A KUBE-FIREWALL ! -s 127.0.0.0/8 -d 127.0.0.0/8 -m comment --comment "block incoming localnet connections" -m conntrack ! --ctstate RELATED,ESTABLISHED,DNAT -j DROP
-A KUBE-FIREWALL -m comment --comment "kubernetes firewall for dropping marked packets" -m mark --mark 0x8000/0x8000 -j DROP
COMMIT
# Completed on Sun Aug 25 17:49:54 2024
# Generated by iptables-save v1.8.9 on Sun Aug 25 17:49:54 2024
*mangle
:PREROUTING ACCEPT [84803:20325123]
:INPUT ACCEPT [18333:6495024]
:FORWARD ACCEPT [68118:14066587]
:OUTPUT ACCEPT [18662:2137631]
:POSTROUTING ACCEPT [86250:16169628]
:KUBE-IPTABLES-HINT - [0:0]
:KUBE-KUBELET-CANARY - [0:0]
COMMIT
# Completed on Sun Aug 25 17:49:54 2024
$ iptables-save
# Generated by iptables-save v1.8.9 (nf_tables) on Sun Aug 25 17:49:11 2024
*mangle
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:KUBE-IPTABLES-HINT - [0:0]
COMMIT
# Completed on Sun Aug 25 17:49:11 2024
# Generated by iptables-save v1.8.9 (nf_tables) on Sun Aug 25 17:49:11 2024
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:KUBE-FORWARD - [0:0]
:KUBE-NODE-PORT - [0:0]
:KUBE-PROXY-FIREWALL - [0:0]
:KUBE-SOURCE-RANGES-FIREWALL - [0:0]
-A INPUT -m comment --comment "kube-proxy firewall rules" -j KUBE-PROXY-FIREWALL
-A INPUT -m comment --comment "kubernetes health check rules" -j KUBE-NODE-PORT
-A FORWARD -m comment --comment "kube-proxy firewall rules" -j KUBE-PROXY-FIREWALL
-A FORWARD -m comment --comment "kubernetes forwarding rules" -j KUBE-FORWARD
-A FORWARD -s 10.244.0.0/16 -m comment --comment "flanneld forward" -j ACCEPT
-A FORWARD -d 10.244.0.0/16 -m comment --comment "flanneld forward" -j ACCEPT
-A KUBE-FORWARD -m comment --comment "kubernetes forwarding rules" -m mark --mark 0x4000/0x4000 -j ACCEPT
-A KUBE-FORWARD -m comment --comment "kubernetes forwarding conntrack rule" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A KUBE-NODE-PORT -m comment --comment "Kubernetes health check node port" -m set --match-set KUBE-HEALTH-CHECK-NODE-PORT dst -j ACCEPT
-A KUBE-SOURCE-RANGES-FIREWALL -j DROP
COMMIT
# Completed on Sun Aug 25 17:49:11 2024
# Generated by iptables-save v1.8.9 (nf_tables) on Sun Aug 25 17:49:11 2024
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:KUBE-LOAD-BALANCER - [0:0]
:KUBE-MARK-MASQ - [0:0]
:KUBE-NODE-PORT - [0:0]
:KUBE-POSTROUTING - [0:0]
:KUBE-SERVICES - [0:0]
-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
-A POSTROUTING -s 10.244.0.0/16 -d 10.244.0.0/16 -m comment --comment "flanneld masq" -j RETURN
-A POSTROUTING -s 10.244.0.0/16 ! -d 224.0.0.0/4 -m comment --comment "flanneld masq" -j MASQUERADE --random-fully
-A POSTROUTING ! -s 10.244.0.0/16 -d 10.244.9.0/24 -m comment --comment "flanneld masq" -j RETURN
-A POSTROUTING ! -s 10.244.0.0/16 -d 10.244.0.0/16 -m comment --comment "flanneld masq" -j MASQUERADE --random-fully
-A KUBE-LOAD-BALANCER -j KUBE-MARK-MASQ
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
-A KUBE-NODE-PORT -p tcp -m comment --comment "Kubernetes nodeport TCP port for masquerade purpose" -m set --match-set KUBE-NODE-PORT-TCP dst -j KUBE-MARK-MASQ
-A KUBE-POSTROUTING -m comment --comment "Kubernetes endpoints dst ip:port, source ip for solving hairpin purpose" -m set --match-set KUBE-LOOP-BACK dst,dst,src -j MASQUERADE
-A KUBE-POSTROUTING -m mark ! --mark 0x4000/0x4000 -j RETURN
-A KUBE-POSTROUTING -j MARK --set-xmark 0x4000/0x0
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -j MASQUERADE --random-fully
-A KUBE-SERVICES ! -s 10.244.0.0/16 -m comment --comment "Kubernetes service cluster ip + port for masquerade purpose" -m set --match-set KUBE-CLUSTER-IP dst,dst -j KUBE-MARK-MASQ
-A KUBE-SERVICES -m addrtype --dst-type LOCAL -j KUBE-NODE-PORT
-A KUBE-SERVICES -m set --match-set KUBE-CLUSTER-IP dst,dst -j ACCEPT
COMMIT
# Completed on Sun Aug 25 17:49:11 2024
# Warning: iptables-legacy tables present, use iptables-legacy-save to see them
Environment
- Talos version: v1.7.6
- Kubernetes version: 1.25.2
- Platform: Oracle
Have you upgraded your Kubernetes manifests after an upgrade?
Look what's in -legacy and try to see which component is still using it.
This issue is stale because it has been open 180 days with no activity. Remove stale label or comment or this will be closed in 7 days.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
This issue was closed because it has been stalled for 7 days with no activity.