nomad
nomad copied to clipboard
Published
20 hours ago •
hashicorp
hashicorp
iptables entries still not reconciled
Nomad version
Nomad v1.6.1 BuildDate 2023-07-21T13:49:42Z Revision 515895c7690cdc72278018dc5dc58aca41204ccc
Operating system and Environment details
Ubuntu 20.04 ARM64 (nVidia Jetson)
Issue
Hi there.
I'm still suffering with leftover iptables rules after node reboot. I still don't understand how to reproduce it, it seems that sometimes simply rebooting the node is enough, but it doesn't always work. Here is an example of iptables output where you can see old rules pointing to addresses that no longer exist. I'll be happy to provide more information if you can guide me on how to debug this further.
Most probably, related to: https://github.com/hashicorp/nomad/issues/6385
iptables -t nat -L
# iptables -t nat -L
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
CNI-HOSTPORT-DNAT all -- anywhere anywhere ADDRTYPE match dst-type LOCAL
DOCKER all -- anywhere anywhere ADDRTYPE match dst-type LOCAL
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
CNI-HOSTPORT-DNAT all -- anywhere anywhere ADDRTYPE match dst-type LOCAL
DOCKER all -- anywhere !localhost/8 ADDRTYPE match dst-type LOCAL
Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
MASQUERADE all -- 172.17.0.0/16 anywhere
CNI-HOSTPORT-MASQ all -- anywhere anywhere /* CNI portfwd requiring masquerade */
CNI-6dac8c7cb0b2faaf1b2abd20 all -- 172.26.64.47 anywhere /* name: "nomad" id: "f9b0e061-d25b-e390-5e65-3b58a94db8bd" */
MASQUERADE all -- anywhere anywhere
CNI-d1adb7cc80046f2c66f1a4d7 all -- 172.26.64.68 anywhere /* name: "nomad" id: "284720c6-4bb9-82ab-5752-b3fd7051c53b" */
Chain CNI-6dac8c7cb0b2faaf1b2abd20 (1 references)
target prot opt source destination
ACCEPT all -- anywhere 172.26.64.0/20 /* name: "nomad" id: "f9b0e061-d25b-e390-5e65-3b58a94db8bd" */
MASQUERADE all -- anywhere !base-address.mcast.net/4 /* name: "nomad" id: "f9b0e061-d25b-e390-5e65-3b58a94db8bd" */
Chain CNI-DN-6dac8c7cb0b2faaf1b2ab (2 references)
target prot opt source destination
CNI-HOSTPORT-SETMARK tcp -- 172.26.64.0/20 anywhere tcp dpt:3001
CNI-HOSTPORT-SETMARK tcp -- localhost anywhere tcp dpt:3001
DNAT tcp -- anywhere anywhere tcp dpt:3001 to:172.26.64.47:3001
CNI-HOSTPORT-SETMARK udp -- 172.26.64.0/20 anywhere udp dpt:3001
CNI-HOSTPORT-SETMARK udp -- localhost anywhere udp dpt:3001
DNAT udp -- anywhere anywhere udp dpt:3001 to:172.26.64.47:3001
CNI-HOSTPORT-SETMARK tcp -- 172.26.64.0/20 anywhere tcp dpt:postgresql
CNI-HOSTPORT-SETMARK tcp -- localhost anywhere tcp dpt:postgresql
DNAT tcp -- anywhere anywhere tcp dpt:postgresql to:172.26.64.47:5432
CNI-HOSTPORT-SETMARK udp -- 172.26.64.0/20 anywhere udp dpt:5432
CNI-HOSTPORT-SETMARK udp -- localhost anywhere udp dpt:5432
DNAT udp -- anywhere anywhere udp dpt:5432 to:172.26.64.47:5432
CNI-HOSTPORT-SETMARK tcp -- 172.26.64.0/20 anywhere tcp dpt:3000
CNI-HOSTPORT-SETMARK tcp -- localhost anywhere tcp dpt:3000
DNAT tcp -- anywhere anywhere tcp dpt:3000 to:172.26.64.47:3000
CNI-HOSTPORT-SETMARK udp -- 172.26.64.0/20 anywhere udp dpt:3000
CNI-HOSTPORT-SETMARK udp -- localhost anywhere udp dpt:3000
DNAT udp -- anywhere anywhere udp dpt:3000 to:172.26.64.47:3000
CNI-HOSTPORT-SETMARK tcp -- 172.26.64.0/20 anywhere tcp dpt:http
CNI-HOSTPORT-SETMARK tcp -- localhost anywhere tcp dpt:http
DNAT tcp -- anywhere anywhere tcp dpt:http to:172.26.64.47:80
CNI-HOSTPORT-SETMARK udp -- 172.26.64.0/20 anywhere udp dpt:80
CNI-HOSTPORT-SETMARK udp -- localhost anywhere udp dpt:80
DNAT udp -- anywhere anywhere udp dpt:80 to:172.26.64.47:80
Chain CNI-DN-d1adb7cc80046f2c66f1a (2 references)
target prot opt source destination
CNI-HOSTPORT-SETMARK tcp -- 172.26.64.0/20 anywhere tcp dpt:3001
CNI-HOSTPORT-SETMARK tcp -- localhost anywhere tcp dpt:3001
DNAT tcp -- anywhere anywhere tcp dpt:3001 to:172.26.64.68:3001
CNI-HOSTPORT-SETMARK udp -- 172.26.64.0/20 anywhere udp dpt:3001
CNI-HOSTPORT-SETMARK udp -- localhost anywhere udp dpt:3001
DNAT udp -- anywhere anywhere udp dpt:3001 to:172.26.64.68:3001
CNI-HOSTPORT-SETMARK tcp -- 172.26.64.0/20 anywhere tcp dpt:postgresql
CNI-HOSTPORT-SETMARK tcp -- localhost anywhere tcp dpt:postgresql
DNAT tcp -- anywhere anywhere tcp dpt:postgresql to:172.26.64.68:5432
CNI-HOSTPORT-SETMARK udp -- 172.26.64.0/20 anywhere udp dpt:5432
CNI-HOSTPORT-SETMARK udp -- localhost anywhere udp dpt:5432
DNAT udp -- anywhere anywhere udp dpt:5432 to:172.26.64.68:5432
CNI-HOSTPORT-SETMARK tcp -- 172.26.64.0/20 anywhere tcp dpt:3000
CNI-HOSTPORT-SETMARK tcp -- localhost anywhere tcp dpt:3000
DNAT tcp -- anywhere anywhere tcp dpt:3000 to:172.26.64.68:3000
CNI-HOSTPORT-SETMARK udp -- 172.26.64.0/20 anywhere udp dpt:3000
CNI-HOSTPORT-SETMARK udp -- localhost anywhere udp dpt:3000
DNAT udp -- anywhere anywhere udp dpt:3000 to:172.26.64.68:3000
CNI-HOSTPORT-SETMARK tcp -- 172.26.64.0/20 anywhere tcp dpt:http
CNI-HOSTPORT-SETMARK tcp -- localhost anywhere tcp dpt:http
DNAT tcp -- anywhere anywhere tcp dpt:http to:172.26.64.68:80
CNI-HOSTPORT-SETMARK udp -- 172.26.64.0/20 anywhere udp dpt:80
CNI-HOSTPORT-SETMARK udp -- localhost anywhere udp dpt:80
DNAT udp -- anywhere anywhere udp dpt:80 to:172.26.64.68:80
Chain CNI-HOSTPORT-DNAT (2 references)
target prot opt source destination
CNI-DN-6dac8c7cb0b2faaf1b2ab tcp -- anywhere anywhere /* dnat name: "nomad" id: "f9b0e061-d25b-e390-5e65-3b58a94db8bd" */ multiport dports 3001,postgresql,3000,http
CNI-DN-6dac8c7cb0b2faaf1b2ab udp -- anywhere anywhere /* dnat name: "nomad" id: "f9b0e061-d25b-e390-5e65-3b58a94db8bd" */ multiport dports 3001,5432,3000,80
CNI-DN-d1adb7cc80046f2c66f1a tcp -- anywhere anywhere /* dnat name: "nomad" id: "284720c6-4bb9-82ab-5752-b3fd7051c53b" */ multiport dports 3001,postgresql,3000,http
CNI-DN-d1adb7cc80046f2c66f1a udp -- anywhere anywhere /* dnat name: "nomad" id: "284720c6-4bb9-82ab-5752-b3fd7051c53b" */ multiport dports 3001,5432,3000,80
Chain CNI-HOSTPORT-MASQ (1 references)
target prot opt source destination
MASQUERADE all -- anywhere anywhere mark match 0x2000/0x2000
Chain CNI-HOSTPORT-SETMARK (32 references)
target prot opt source destination
MARK all -- anywhere anywhere /* CNI portfwd masquerade mark */ MARK or 0x2000
Chain CNI-d1adb7cc80046f2c66f1a4d7 (1 references)
target prot opt source destination
ACCEPT all -- anywhere 172.26.64.0/20 /* name: "nomad" id: "284720c6-4bb9-82ab-5752-b3fd7051c53b" */
MASQUERADE all -- anywhere !base-address.mcast.net/4 /* name: "nomad" id: "284720c6-4bb9-82ab-5752-b3fd7051c53b" */
Chain DOCKER (2 references)
target prot opt source destination
RETURN all -- anywhere anywhere
