Deployment blocks port 443 in nftables

[Black616Angel]

Describe the bug tldr: Deployment blocks port 443 in nftables.

For a few days now our server doesn't work. I looked into it and no connection to the port 443 even for non-docker containers works. And curl gives no route to host instead of the usual.

Some part of the deployment breaks the firewall. It may be docker or this ansible-playbook itself, I don't know, but deleting the rules opens the port up, but destroys the routing to the containers. A new deployment fixes the routing, but also blocks the port (somehow).

To Reproduce My vars.yml file looks like this:

# general
matrix_domain: matrix.example.com
matrix_homeserver_implementation: synapse

matrix_synapse_macaroon_secret_key: "secure"
matrix_ma1sd_enabled: false
matrix_homeserver_generic_secret_key: "{{ matrix_synapse_macaroon_secret_key }}"
matrix_sliding_sync_enabled: true
matrix_coturn_turn_static_auth_secret: "secure"

matrix_synapse_admin_enabled: true
# shared secret
#matrix_synapse_ext_password_provider_shared_secret_auth_shared_secret: secure
matrix_synapse_ext_password_provider_shared_secret_auth_enabled: true
matrix_synapse_ext_password_provider_shared_secret_auth_shared_secret: "secure"

# federation
matrix_homeserver_federation_enabled: true
matrix_synapse_allow_public_rooms_over_federation: true

# traefik
#traefik_version: v3.3.1
matrix_playbook_reverse_proxy_type: playbook-managed-traefik
#traefik_config_certificatesResolvers_acme_email: [email protected]
matrix_static_files_container_labels_base_domain_enabled: true
traefik_dashboard_enabled: true
traefik_dashboard_hostname: not.example.com
traefik_dashboard_basicauth_enabled: true
traefik_dashboard_basicauth_user: username
traefik_dashboard_basicauth_password: secure

#Postgres
postgres_connection_password: 'secure'

# telegram
matrix_mautrix_telegram_enabled: true
matrix_mautrix_telegram_api_id: 2255250
matrix_mautrix_telegram_api_hash: HASH

#signal
matrix_mautrix_signal_enabled: true

# whatsapp
matrix_mautrix_whatsapp_enabled: true

# facebook
matrix_mautrix_facebook_enabled: true

Expected behavior Port 443 is open and all services are available.

Matrix Server:

• OS: Ubuntu 24.04
• Architecture amd64
• Docker version 28.2.2, build e6534b4

Additional context I tested with python3 -m http.server 443 as server to not having to set up traefik over and over again, when getting to the cause and even that didn't go through from outside. I also tested an old traefik version, since I found out about other problems, but that didn't help either. (see vars.yml) If I do nft flush ruleset however, everything works fine, but docker (or ansible) then reapplies it.

[spantaleev]

Docker probably doesn't play well with nftables. Consider installing iptables-legacy instead and see if that makes things better.

[Black616Angel]

iptables-legacy is installed and docker handles all the config through iptables, but nftables then does the networking. nftables even says: # Warning: table ip nat is managed by iptables-nft, do not touch!

But I tried both and only nftables did the trick.

[Black616Angel]

I found the problem and a workaround.

The problem seems to be docker adding multiple NAT rules for different stuff on the same port and then nftables just uses one of them. In my case the port 443 was routed to the sliding sync container.

I fixed it the following way:

1. Find out, which IPs the traffic is routed to

sudo nft -a list chain ip nat DOCKER | grep 443

Look for the first line where it says "dnat to IP:PORT" and remember the IP.

2. Search the IP in the traefik network

sudo docker inspect traefik

3. If it's not traefik, delete it from nftables

search all rules with the IP:

sudo nft -a list chain ip nat DOCKER | grep <REPLACE THIS WITH THE IP>

and remember the handle numbers

then delete line by line:

sudo nft delete rule ip nat DOCKER handle <HANDLE-NUMBER>