No access from container to container via host ip

[AlekseyMelikov]

Hi. Thanks for your work supporting nftables in CNI plugins.

We use nomad, docker and CNI. We are having problems accessing from a container to another container via host ip.

For example, if execute in a container

$ curl HOST_IP:9090
curl: (7) Failed to connect to HOST_IP port 9090: Connection refused

If execute on the host, then ok

$ curl 10.88.0.54:9090
<a href="/graph">Found</a>.

CNI settings

{
  "cniVersion": "0.4.0",
  "name": "mynet",
  "plugins": [
    {
      "type": "bridge",
      "bridge": "mynet0",
      "isGateway": true,
      "ipMasq": false,
      "ipam": {
        "type": "host-local",
        "routes": [
          {
            "dst": "0.0.0.0/0"
          }
        ],
        "ranges": [
          [
            {
              "subnet": "10.88.0.0/16",
              "gateway": "10.88.0.1"
            }
          ]
        ]
      }
    },
    {
      "type": "cni-nftables-portmap",
      "capabilities": {
        "portMappings": true
      }
    },
    {
      "type": "cni-nftables-firewall",
      "forward_chain_name": "forward"
    }
  ]
}

sysctl

$ sysctl -a | grep net.ipv4.conf.all.route_localnet
net.ipv4.conf.all.route_localnet = 1

Nftables rules

table ip filter {
	chain input {
		type filter hook input priority filter; policy accept;
	}

	chain forward {
		type filter hook forward priority filter; policy accept;
		jump cni-ffw-f207c1a2a8709f670b8b03f
		oifname "mynet0" ip daddr 10.88.0.54 udp dport 9090 counter packets 0 bytes 0 accept
		oifname "mynet0" ip daddr 10.88.0.54 tcp dport 9090 counter packets 7 bytes 388 accept
		jump cni-ffw-05073c0b0d1d279a0b103db
		oifname "mynet0" ip daddr 10.88.0.53 udp dport 5432 counter packets 0 bytes 0 accept
		oifname "mynet0" ip daddr 10.88.0.53 tcp dport 5432 counter packets 4 bytes 220 accept
	}

	chain output {
		type filter hook output priority filter; policy accept;
	}

	chain cni-ffw-05073c0b0d1d279a0b103db {
		oifname "mynet0" ip daddr 10.88.0.53 ct state established,related counter packets 486 bytes 11358135 accept
		iifname "mynet0" ip saddr 10.88.0.53 counter packets 332 bytes 21999 accept
		iifname "mynet0" oifname "mynet0" counter packets 0 bytes 0 accept
	}

	chain cni-ffw-f207c1a2a8709f670b8b03f {
		oifname "mynet0" ip daddr 10.88.0.54 ct state established,related counter packets 643 bytes 39798 accept
		iifname "mynet0" ip saddr 10.88.0.54 counter packets 936 bytes 2442404 accept
		iifname "mynet0" oifname "mynet0" counter packets 3 bytes 180 accept
	}
}
table ip nat {
	chain postrouting {
		type nat hook postrouting priority srcnat; policy accept;
		jump cni-npo-f207c1a2a8709f670b8b03f
		jump cni-npo-05073c0b0d1d279a0b103db
	}

	chain prerouting {
		type nat hook prerouting priority dstnat; policy accept;
		ip daddr 10.17.0.1 jump cni-npr-f207c1a2a8709f670b8b03f
		ip daddr 172.16.0.2 jump cni-npr-f207c1a2a8709f670b8b03f
		ip daddr HOST_IP1 jump cni-npr-f207c1a2a8709f670b8b03f
		ip daddr HOST_IP1 jump cni-npr-f207c1a2a8709f670b8b03f
		ip daddr 127.0.0.1 jump cni-npr-f207c1a2a8709f670b8b03f
		ip daddr 10.17.0.1 jump cni-npr-05073c0b0d1d279a0b103db
		ip daddr 172.16.0.2 jump cni-npr-05073c0b0d1d279a0b103db
		ip daddr HOST_IP1 jump cni-npr-05073c0b0d1d279a0b103db
		ip daddr HOST_IP2 jump cni-npr-05073c0b0d1d279a0b103db
		ip daddr 127.0.0.1 jump cni-npr-05073c0b0d1d279a0b103db
	}

	chain output {
		type nat hook output priority -100; policy accept;
		ip daddr 10.17.0.1 jump cni-npr-f207c1a2a8709f670b8b03f
		ip daddr 172.16.0.2 jump cni-npr-f207c1a2a8709f670b8b03f
		ip daddr HOST_IP1 jump cni-npr-f207c1a2a8709f670b8b03f
		ip daddr HOST_IP2 jump cni-npr-f207c1a2a8709f670b8b03f
		ip daddr 127.0.0.1 jump cni-npr-f207c1a2a8709f670b8b03f
		ip daddr 10.17.0.1 jump cni-npr-05073c0b0d1d279a0b103db
		ip daddr 172.16.0.2 jump cni-npr-05073c0b0d1d279a0b103db
		ip daddr HOST_IP1 jump cni-npr-05073c0b0d1d279a0b103db
		ip daddr HOST_IP2 jump cni-npr-05073c0b0d1d279a0b103db
		ip daddr 127.0.0.1 jump cni-npr-05073c0b0d1d279a0b103db
	}

	chain input {
		type nat hook input priority 100; policy accept;
	}

	chain cni-npr-05073c0b0d1d279a0b103db {
		iifname != "mynet0" tcp dport 5432 dnat to 10.88.0.53:5432
		iifname != "mynet0" udp dport 5432 dnat to 10.88.0.53:5432
	}

	chain cni-npo-05073c0b0d1d279a0b103db {
		oifname "mynet0" ip daddr 10.88.0.53 counter packets 4 bytes 220 masquerade
		iifname "mynet0" ip saddr 10.88.0.53 ip daddr 224.0.0.0/24 counter packets 0 bytes 0 return
		iifname "mynet0" ip saddr 10.88.0.53 ip daddr 255.255.255.255 counter packets 0 bytes 0 return
		iifname "mynet0" ip saddr 10.88.0.53 counter packets 19 bytes 1281 masquerade
	}

	chain cni-npr-f207c1a2a8709f670b8b03f {
		iifname != "mynet0" tcp dport 9090 dnat to 10.88.0.54:9090
		iifname != "mynet0" udp dport 9090 dnat to 10.88.0.54:9090
	}

	chain cni-npo-f207c1a2a8709f670b8b03f {
		oifname "mynet0" ip daddr 10.88.0.54 counter packets 11 bytes 628 masquerade
		iifname "mynet0" ip saddr 10.88.0.54 ip daddr 224.0.0.0/24 counter packets 0 bytes 0 return
		iifname "mynet0" ip saddr 10.88.0.54 ip daddr 255.255.255.255 counter packets 0 bytes 0 return
		iifname "mynet0" ip saddr 10.88.0.54 counter packets 0 bytes 0 masquerade
	}
}
table ip raw {
	chain prerouting {
		type filter hook prerouting priority raw; policy accept;
	}
}

We found that if change the chain cni-npr-f207c1a2a8709f670b8b03f to

	chain cni-npr-f207c1a2a8709f670b8b03f {
		tcp dport 9090 dnat to 10.88.0.54:9090
		iifname != "mynet0" tcp dport 9090 dnat to 10.88.0.54:9090
		iifname != "mynet0" udp dport 9090 dnat to 10.88.0.54:9090
	}

that is, adding dnat without iifname != "mynet0" then the connection from container to container via the host IP address is successful.

$ curl HOST_IP:9090
<a href="/graph">Found</a>.

[greenpau]

@AlekseyMelikov , i think it is doable, but I don’t have time to dive into it.

[AlekseyMelikov]

I committed fix. And it looks like it works. But I'm not sure if this fix won't break something.

[greenpau]

I committed fix. And it looks like it works. But I'm not sure if this fix won't break something.

@AlekseyMelikov , I will try testing it this week.

[greenpau]

@AlekseyMelikov , are you still using this plugin?