Redirection not working: 504 Gateway Time-out

[aredey]

I have set up a test container in a public VPC listening on http://18.33.44.55:32769 . I can access the container directly using this url in the port indicated. I have then set up the Proxy Host in NginxProxyManager as follows:

Domain Names: ifconfig.mydomain.com
Scheme: http
Forward IP: 18.33.44.55
Forward Port: 32769

I don't need https access for this test for the time being so I left he SSL tab blank: None. Navigating to "http://ifconfig.mydomain.com" does show up in the container's nginx proxy log, but the there is no sign of actually forwarding the request in the log tail -f /config/log/proxy_host-4.log: [27/Mar/2021:13:19:40 +0000] - 504 504 - GET http ifconfig.mydomain.com "/" [Client 123.4.5.67] [Length 164] [Gzip -] [Sent-to 18.33.44.55] "curl/7.58.0" "-" I was testing this operation with curl (to disregard fat browsers' attempts to silently redirect http to https).

... and neither is the forwarding of packets to port80 or 443 showing up in tcpdump -n -i eth0 'port 32769 || port 80 || port 443' in the container's hosting VPC (no trace of the port 32769 packets). IP addresses and domain names are edited to protect the innocent...

Does anyone have any pointers for how to troubleshoot? I suspect the issue might be with the nginx config

[jlesage]

Looks like the Nginx Proxy Manager container has trouble to access 18.33.44.55 ?

According the your log message, NPM forwarded to 18.33.44.55, but timed out (error 504).

[aredey]

Thanks for answering! Yes, this is what's puzzling for me. From the same docker host I can access the target container (using curl http://18.33.44.55:32769) but not sure why nginx from the container is unable. This is the same for multiple containerised web apps. When "tcpdump" ing on the host I can see the request coming in the to the NPM app but tit is not forwarded to the target (no further tcp packe seen). How can I figure out why is this? I tried setting the docker network to be the standard bridge network or various container specific "bridges" (with inter-porocess-communicaton enabled). None helped and am a bit stuck. Do you have any suggestion for what to look for?

[jlesage]

So is the same host running both NPM and the target container ?

[aredey]

Yes, in this case, they are both on the same host. But this host is also Internet accessible as well. I tried to use the default bridge network (Internet accessible) and the private bridge (also internet accessible). Tried to refer to the target container by the external (public) IP address and the socket "file". Neither worked: I seem to be able to hit nginx but it doesn't forward (reverse proxy) traffic to the intended destination.

[jlesage]

To keep things simple, let's configure both containers to use the default bridge network, with proper port mappings.

In this scenario, you should set in the proxy host the IP address of the host itself, with the port of the target container.

Also, is the host also has a private/LAN IP address, or only a public one ?

[aredey]

I have changed the docker network used for both (NPM/nginx-proxy-manager and a test container called ifconfig) to the standard "bridge" network. Both containers get a private IP address on this network, and I can ping from the NPM container all private IP addresses.

KEY/Explanation: redey@npm: is my username on the docker host computer (SAAS in the Cloud) redey@mylaptop: is my username on my laptop

Here are the diagnostics I attempted:

#1 on docker host VPS:

There are 3 containers connected to the standard "bridge" network and all pingable from the NPM container:

docker ps | grep -E 'NAMES|nginx|ifconfig'

CONTAINER ID        IMAGE                                  COMMAND                CREATED             STATUS              PORTS                                                               NAMES
3c8fa1d95118        jlesage/nginx-proxy-manager:latest     "/init"                3 weeks ago         Up 45 minutes       0.0.0.0:443->4443/tcp, 0.0.0.0:80->8080/tcp, 0.0.0.0:81->8181/tcp   nginx-proxy-manager
0e8f5b78e79e        mpolden/echoip:latest                  "/opt/echoip/echoip"   4 weeks ago         Up 52 minutes       0.0.0.0:32769->8080/tcp                                             ifconfig

redey@npm:~$ docker network inspect bridge | grep -E 'Containers|Name|Subnet|IPv4Address'

        "Name": "bridge",
                    "Subnet": "172.17.0.0/16",
        "Containers": {
                "Name": "ifconfig",
                "IPv4Address": "172.17.0.6/16",
                "Name": "nginx-proxy-manager",
                "IPv4Address": "172.17.0.5/16",

redey@npm:~$ for i in 5 6 ; do docker exec -it 3c8fa1d95118 ping -c 1 172.17.0.$i ; done | grep -E '172.17|received'

PING 172.17.0.5 (172.17.0.5): 56 data bytes
64 bytes from 172.17.0.5: seq=0 ttl=64 time=0.060 ms
--- 172.17.0.5 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
PING 172.17.0.6 (172.17.0.6): 56 data bytes
64 bytes from 172.17.0.6: seq=0 ttl=64 time=0.100 ms
--- 172.17.0.6 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss

#2 NPM proxy configuration http://18.33.44.55:81/nginx/proxy (this is the web interface):

SOURCE                   DESTINATION                 SSL             ACCESS    STATUS
ifconfig.mydomain.com    http://18.33.44.55:32769    Let's Encrypt   Public    Online
npm.mydomain.com         http://18.33.44.55:81       Let's Encrypt   Public    Online 

#3 Web access (with curl) of both containers from the NPM host computer and from my laptop via the Internet:

redey@npm:~$ curl http://18.33.44.55:32769
18.33.44.55

redey@npm:~$ curl 18.33.44.55:81
<!doctype html>
<html lang="en">
<head>
    <meta charset="utf-8">
<snip>...</snip>
</body>
</html>

#4 but trying to access the reverse proxy from the Internet (my laptop) it fails: ##ifconfig.mydomain.com:

redey@mylaptop:~$ curl ifconfig.mydomain.com
^C
redey@mylaptop:~$ curl https://ifconfig.mydomain.com
curl: (60) SSL certificate problem: self signed certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

##npm.mydomain.com

redey@mylaptop:~$ curl npm.mydomain.com
<html>
<head><title>301 Moved Permanently</title></head>
<body>
<center><h1>301 Moved Permanently</h1></center>
<hr><center>openresty</center>
</body>
</html>

redey@mylaptop:~$ curl https://npm.mydomain.com
^C

In summary: same result. The Internet connected host/browser can send a packet to the NPM host but it is not "proxied" to the target web server...

[aredey]

Massive facepalm situation: the docker host was running a host based firewall (ufw), for some reason (well for some earlier experimenting). I was going to run curl (not installed), telnet (not installed) then nc from within the NPM container, unsuccessfully. This gave me to prompt to check if iptables was enabled. Rest is history, many thanks for looking into this!!!