Netbird ACL doesn't work when accessing peer with Docker container

[alexcupertme]

Describe the problem When one peer inside Netbird network tries to reach another peer's container, it fails. Destination peer has the container with published port, for example 80:8080 (external port should differ)

And in Netbird ACL i only allowed 80 and 443 ports, which should be enough

To Reproduce

Steps to reproduce the behavior:

1. Connect to the netbird as user peer
2. Connect host peer
3. Set up docker container with external port. External port should be 80 (could be any port). Internal should be any except 80. Container should return any response locally on port 80
4. Add single rule in the list: ALLOW TCP, All Source, All destination, ports 80, 443, bidirectional
5. curl . Connection timed out
6. Add second rule in ACL. ALLOW TCP, All source, All destination, port - internal port of the container
7. curl . Getting response as expected

Expected behavior

It should be enough to add ACL rule, which allows external container port, because why i need to know all port forwardings? I dont want to specify all internal ports of all my containers, because i will expose only 80 port, not their system ports.

Are you using NetBird Cloud?

no

NetBird version

0.28.6

[lixmal]

To be clear, netbird runs on the host and not inside the container, right?

Could you provide the nftables output please? nft list ruleset Might need to install the nftables package first.

[alexcupertme]

@lixmal I have this setup, 3 different hosts:

1. Netbird controller running inside docker, with zitadel, caddy, postgres etc.
2. Peer 1 - my PC
3. Peer 2 - app with docker container

$ nft list ruleset
table ip nat {
        chain DOCKER {
                iifname "br-b64dc14d949f" counter packets 0 bytes 0 return
                iifname "docker0" counter packets 0 bytes 0 return
                iifname != "br-b64dc14d949f" meta l4proto tcp tcp dport 80 counter packets 89859 bytes 5378483 dnat to 172.18.0.2:80
                iifname != "br-b64dc14d949f" meta l4proto tcp tcp dport 443 counter packets 4771 bytes 247506 dnat to 172.18.0.2:443
                iifname != "br-b64dc14d949f" meta l4proto tcp tcp dport 8080 counter packets 567 bytes 24868 dnat to 172.18.0.2:8080
        }

        chain POSTROUTING {
                type nat hook postrouting priority srcnat; policy accept;
                oifname != "br-b64dc14d949f" ip saddr 172.18.0.0/16 counter packets 83 bytes 4980 masquerade
                oifname != "docker0" ip saddr 172.17.0.0/16 counter packets 0 bytes 0 masquerade
                meta l4proto tcp ip saddr 172.18.0.2 ip daddr 172.18.0.2 tcp dport 80 counter packets 0 bytes 0 masquerade
                meta l4proto tcp ip saddr 172.18.0.2 ip daddr 172.18.0.2 tcp dport 443 counter packets 0 bytes 0 masquerade
                meta l4proto tcp ip saddr 172.18.0.2 ip daddr 172.18.0.2 tcp dport 8080 counter packets 0 bytes 0 masquerade
        }

        chain PREROUTING {
                type nat hook prerouting priority dstnat; policy accept;
                fib daddr type local counter packets 1015834 bytes 52066735 jump DOCKER
        }

        chain OUTPUT {
                type nat hook output priority -100; policy accept;
                ip daddr != 127.0.0.0/8 fib daddr type local counter packets 286 bytes 40040 jump DOCKER
        }
}
table ip filter {
        chain DOCKER {
                iifname != "br-b64dc14d949f" oifname "br-b64dc14d949f" meta l4proto tcp ip daddr 172.18.0.2 tcp dport 80 counter packets 89859 bytes 5378483 accept
                iifname != "br-b64dc14d949f" oifname "br-b64dc14d949f" meta l4proto tcp ip daddr 172.18.0.2 tcp dport 443 counter packets 4773 bytes 247626 accept
                iifname != "br-b64dc14d949f" oifname "br-b64dc14d949f" meta l4proto tcp ip daddr 172.18.0.2 tcp dport 8080 counter packets 567 bytes 24868 accept
        }

        chain DOCKER-ISOLATION-STAGE-1 {
                iifname "br-b64dc14d949f" oifname != "br-b64dc14d949f" counter packets 899655 bytes 312809594 jump DOCKER-ISOLATION-STAGE-2
                iifname "docker0" oifname != "docker0" counter packets 0 bytes 0 jump DOCKER-ISOLATION-STAGE-2
                counter packets 41820128 bytes 7041359642 return
        }

        chain DOCKER-ISOLATION-STAGE-2 {
                oifname "br-b64dc14d949f" counter packets 0 bytes 0 drop
                oifname "docker0" counter packets 0 bytes 0 drop
                counter packets 908707 bytes 323436671 return
        }

        chain FORWARD {
                type filter hook forward priority filter; policy drop;
                counter packets 41706520 bytes 6998463817 jump DOCKER-USER
                counter packets 41706520 bytes 6998463817 jump DOCKER-ISOLATION-STAGE-1
                oifname "br-b64dc14d949f" ct state related,established counter packets 39171857 bytes 6587614706 accept
                oifname "br-b64dc14d949f" counter packets 1635008 bytes 98039517 jump DOCKER
                iifname "br-b64dc14d949f" oifname != "br-b64dc14d949f" counter packets 899655 bytes 312809594 accept
                iifname "br-b64dc14d949f" oifname "br-b64dc14d949f" counter packets 1539809 bytes 92388540 accept
                oifname "docker0" ct state related,established counter packets 0 bytes 0 accept
                oifname "docker0" counter packets 0 bytes 0 jump DOCKER
                iifname "docker0" oifname != "docker0" counter packets 0 bytes 0 accept
                iifname "docker0" oifname "docker0" counter packets 0 bytes 0 accept
        }

        chain DOCKER-USER {
                counter packets 41820128 bytes 7041359642 return
        }
}
table ip6 nat {
        chain DOCKER {
        }
}
table ip6 filter {
        chain DOCKER {
        }

        chain DOCKER-ISOLATION-STAGE-1 {
                iifname "br-b64dc14d949f" oifname != "br-b64dc14d949f" counter packets 0 bytes 0 jump DOCKER-ISOLATION-STAGE-2
                iifname "docker0" oifname != "docker0" counter packets 0 bytes 0 jump DOCKER-ISOLATION-STAGE-2
                counter packets 0 bytes 0 return
        }

        chain DOCKER-ISOLATION-STAGE-2 {
                oifname "br-b64dc14d949f" counter packets 0 bytes 0 drop
                oifname "docker0" counter packets 0 bytes 0 drop
                counter packets 0 bytes 0 return
        }

        chain FORWARD {
                type filter hook forward priority filter; policy drop;
                counter packets 0 bytes 0 jump DOCKER-USER
        }

        chain DOCKER-USER {
                counter packets 0 bytes 0 return
        }
}

[lixmal]

Peer 2 - app with docker container

I don't see netbird rules in the nft output. So netbird runs in the container?

If that's the case then The VPN and access rules are set up within the container. Any forwardings set up by the container runtime on the host are not known to netbird nor would we like to mess with them.

[alexcupertme]

I don't see netbird rules in the nft output. So netbird runs in the container?

Yes netbird controller runs in the container

Sorry didn't understood first which host nftables you want to see. Here's info about host with container. I think thats what you want to see

~# nft list ruleset
table ip nat {
        chain DOCKER {
                iifname "br-a8513a7899b9" counter packets 0 bytes 0 return
                iifname "docker0" counter packets 0 bytes 0 return
                iifname != "br-a8513a7899b9" meta l4proto tcp tcp dport 8250 counter packets 0 bytes 0 dnat to 172.29.0.5:8250
                iifname != "br-a8513a7899b9" meta l4proto tcp tcp dport 80 counter packets 1954 bytes 111305 dnat to 172.29.0.5:8200
                iifname != "docker0" meta l4proto udp udp dport 8600 counter packets 0 bytes 0 dnat to 172.17.0.2:8600
                iifname != "docker0" meta l4proto tcp tcp dport 8500 counter packets 32 bytes 1664 dnat to 172.17.0.2:8500
                iifname != "docker0" meta l4proto tcp tcp dport 9001 counter packets 0 bytes 0 dnat to 172.17.0.4:9001
        }

        chain POSTROUTING {
                type nat hook postrouting priority srcnat; policy accept;
                oifname != "br-a8513a7899b9" ip saddr 172.29.0.0/16 counter packets 520 bytes 31514 masquerade
                oifname != "docker0" ip saddr 172.17.0.0/16 counter packets 84 bytes 6440 masquerade
                meta l4proto tcp ip saddr 172.29.0.5 ip daddr 172.29.0.5 tcp dport 8250 counter packets 0 bytes 0 masquerade
                meta l4proto tcp ip saddr 172.29.0.5 ip daddr 172.29.0.5 tcp dport 8200 counter packets 0 bytes 0 masquerade
                meta l4proto udp ip saddr 172.17.0.2 ip daddr 172.17.0.2 udp dport 8600 counter packets 0 bytes 0 masquerade
                meta l4proto tcp ip saddr 172.17.0.2 ip daddr 172.17.0.2 tcp dport 8500 counter packets 0 bytes 0 masquerade
                meta l4proto tcp ip saddr 172.17.0.4 ip daddr 172.17.0.4 tcp dport 9001 counter packets 0 bytes 0 masquerade
        }

        chain PREROUTING {
                type nat hook prerouting priority dstnat; policy accept;
                fib daddr type local counter packets 5049 bytes 315770 jump DOCKER
        }

        chain OUTPUT {
                type nat hook output priority -100; policy accept;
                ip daddr != 127.0.0.0/8 fib daddr type local counter packets 0 bytes 0 jump DOCKER
        }
}
table ip filter {
        chain DOCKER {
                iifname != "br-a8513a7899b9" oifname "br-a8513a7899b9" meta l4proto tcp ip daddr 172.29.0.5 tcp dport 8250 counter packets 0 bytes 0 accept
                iifname != "br-a8513a7899b9" oifname "br-a8513a7899b9" meta l4proto tcp ip daddr 172.29.0.5 tcp dport 8200 counter packets 1665 bytes 96077 accept
                iifname != "docker0" oifname "docker0" meta l4proto udp ip daddr 172.17.0.2 udp dport 8600 counter packets 0 bytes 0 accept
                iifname != "docker0" oifname "docker0" meta l4proto tcp ip daddr 172.17.0.2 tcp dport 8500 counter packets 32 bytes 1664 accept
                iifname != "docker0" oifname "docker0" meta l4proto tcp ip daddr 172.17.0.4 tcp dport 9001 counter packets 0 bytes 0 accept
        }

        chain DOCKER-ISOLATION-STAGE-1 {
                iifname "br-a8513a7899b9" oifname != "br-a8513a7899b9" counter packets 95688 bytes 151849709 jump DOCKER-ISOLATION-STAGE-2
                iifname "docker0" oifname != "docker0" counter packets 6786 bytes 22253491 jump DOCKER-ISOLATION-STAGE-2
                counter packets 17944502 bytes 2752645095 return
        }

        chain DOCKER-ISOLATION-STAGE-2 {
                oifname "br-a8513a7899b9" counter packets 0 bytes 0 drop
                oifname "docker0" counter packets 0 bytes 0 drop
                counter packets 264567 bytes 387219540 return
        }

        chain FORWARD {
                type filter hook forward priority filter; policy drop;
                counter packets 4618318 bytes 801152504 jump DOCKER-USER
                counter packets 4618318 bytes 801152504 jump DOCKER-ISOLATION-STAGE-1
                oifname "br-a8513a7899b9" ct state related,established counter packets 95633 bytes 19926068 accept
                oifname "br-a8513a7899b9" counter packets 1692 bytes 97769 jump DOCKER
                iifname "br-a8513a7899b9" oifname != "br-a8513a7899b9" counter packets 95688 bytes 151849709 accept
                iifname "br-a8513a7899b9" oifname "br-a8513a7899b9" counter packets 21 bytes 1380 accept
                oifname "docker0" ct state related,established counter packets 4316755 bytes 584321436 accept
                oifname "docker0" counter packets 71701 bytes 4302425 jump DOCKER
                iifname "docker0" oifname != "docker0" counter packets 6786 bytes 22253491 accept
                iifname "docker0" oifname "docker0" counter packets 71669 bytes 4300761 accept
        }

        chain DOCKER-USER {
                counter packets 17944502 bytes 2752645095 return
        }
}
table ip6 filter {
}
table ip6 nat {
}
table ip netbird {
        set nb0000094 {
                type ipv4_addr
                flags dynamic
                elements = { 0.0.0.0 }
        }

        set nb0000095 {
                type ipv4_addr
                flags dynamic
                elements = { 0.0.0.0 }
        }

        set nb0000105 {
                type ipv4_addr
                flags dynamic
                elements = { 0.0.0.0, 100.115.5.195,
                             100.115.238.191 }
        }

        set nb0000273 {
                type ipv4_addr
                flags dynamic
                elements = { 0.0.0.0 }
        }

        set nb0000278 {
                type ipv4_addr
                flags dynamic
                elements = { 0.0.0.0 }
        }

        set nb0000279 {
                type ipv4_addr
                flags dynamic
                elements = { 0.0.0.0 }
        }

        set nb0000280 {
                type ipv4_addr
                flags dynamic
                elements = { 0.0.0.0 }
        }

        set nb0000281 {
                type ipv4_addr
                flags dynamic
                elements = { 0.0.0.0 }
        }

        chain netbird-rt-fwd {
        }

        chain netbird-rt-nat {
                type nat hook postrouting priority srcnat - 1; policy accept;
                oifname "lo" return
        }

        chain netbird-acl-input-rules {
                iifname "wt0" tcp dport 443 accept
                iifname "wt0" tcp dport 80 accept
                iifname "wt0" tcp sport 443 accept
                iifname "wt0" tcp sport 80 accept
                iifname "wt0" tcp dport 8200 accept
                iifname "wt0" ip saddr @nb0000105 tcp dport 44338 accept
                iifname "wt0" ip protocol icmp accept
        }

        chain netbird-acl-output-rules {
                oifname "wt0" tcp sport 443 accept
                oifname "wt0" tcp sport 80 accept
                oifname "wt0" tcp dport 443 accept
                oifname "wt0" tcp dport 80 accept
                oifname "wt0" tcp sport 8200 accept
                oifname "wt0" ip daddr @nb0000105 tcp sport 44338 accept
                oifname "wt0" ip protocol icmp accept
        }

        chain netbird-acl-input-filter {
                type filter hook input priority filter; policy accept;
                iifname "wt0" ip saddr 100.115.0.0/16 ip daddr != 100.115.0.0/16 accept
                iifname "wt0" ip saddr != 100.115.0.0/16 ip daddr 100.115.0.0/16 accept
                iifname "wt0" ip saddr 100.115.0.0/16 ip daddr 100.115.0.0/16 jump netbird-acl-input-rules
                iifname "wt0" drop
        }

        chain netbird-acl-output-filter {
                type filter hook output priority filter; policy accept;
                oifname "wt0" ip saddr != 100.115.0.0/16 ip daddr 100.115.0.0/16 accept
                oifname "wt0" ip saddr 100.115.0.0/16 ip daddr != 100.115.0.0/16 accept
                oifname "wt0" ip saddr 100.115.0.0/16 ip daddr 100.115.0.0/16 jump netbird-acl-output-rules
                oifname "wt0" drop
        }

        chain netbird-acl-forward-filter {
                type filter hook forward priority filter; policy accept;
                iifname "wt0" jump netbird-rt-fwd
                oifname "wt0" jump netbird-rt-fwd
                iifname "wt0" meta mark 0x000007e4 accept
                oifname "wt0" meta mark 0x000007e4 accept
                iifname "wt0" jump netbird-acl-input-rules
                iifname "wt0" drop
        }

        chain netbird-acl-prerouting-filter {
                type filter hook prerouting priority mangle; policy accept;
                ip saddr @nb0000281 ip daddr 100.115.103.245 tcp dport 443 meta mark set 0x000007e4
                ip saddr @nb0000280 ip daddr 100.115.103.245 tcp dport 80 meta mark set 0x000007e4
                ip saddr @nb0000273 ip daddr 100.115.103.245 tcp dport 8200 meta mark set 0x000007e4
                ip saddr @nb0000105 ip daddr 100.115.103.245 tcp dport 44338 meta mark set 0x000007e4
                ip saddr @nb0000095 ip daddr 100.115.103.245 ip protocol icmp meta mark set 0x000007e4
                iifname "wt0" ip saddr != 100.115.0.0/16 ip daddr 100.115.103.245 meta mark set 0x000007e4
        }
}

[alexcupertme]

Big thanks for investigating, issue disappeared by itself. Closing