TLS does not work on NATed IPv4 literal

[ledlamp]

caddy version v2.7.6

Host has private IP 10.0.0.79 and public IP 129.159.139.0 NAT forwarded (Oracle Cloud).

This works as expected:

10.0.0.79 {
        tls internal
        respond "ok"
}

curl --insecure https://10.0.0.79 -> "ok". curl -D - --insecure https://129.159.139.0 -> blank 200

But this does not:

129.159.139.0 {
        tls internal
        respond "ok"
}

# curl --insecure https://129.159.139.0
curl: (35) error:0A000438:SSL routines::tlsv1 alert internal error

Windows:

> curl.exe --insecure https://129.159.139.0
curl: (35) schannel: next InitializeSecurityContext failed: SEC_E_ILLEGAL_MESSAGE (0x80090326) - This error usually occurs when a fatal SSL/TLS alert is received (e.g. handshake failed). More detail may be available in the Windows System event log.

Google Chrome: ERR_SSL_PROTOCOL_ERROR

Firefox: SSL_ERROR_INTERNAL_ERROR_ALERT

However, having both sites make them both work:

129.159.139.0 {
        tls internal
        respond "foo"
}

10.0.0.79 {
        tls internal
        respond "bar"
}

curl --insecure https://129.159.139.0 -> "foo" curl --insecure https://10.0.0.79 -> "bar"

HOWEVER, if I explicitly specify my own cert files, it STILL uses the caddy local cert. With any of the following configs, the Caddy Local Authority cert is always used regardless.

129.159.139.0 {
        tls ssl/129.159.139.0/certificate.crt ssl/129.159.139.0/private.key
        # ...
}

10.0.0.79 {
        tls internal
        respond "test"
}

129.159.139.0 {
        tls ssl/129.159.139.0/certificate.crt ssl/129.159.139.0/private.key
        # ...
}

10.0.0.79 {
        tls ssl/129.159.139.0/certificate.crt ssl/129.159.139.0/private.key
        respond "test"
}

129.159.139.0 10.0.0.79 {
        tls ssl/129.159.139.0/certificate.crt ssl/129.159.139.0/private.key
        # ...
}

10.0.0.79 {
        tls ssl/129.159.139.0/certificate.crt ssl/129.159.139.0/private.key
        # ...
}

So it seems it is not possible to use TLS on a raw NAT'ed IP address, with custom certs (from ZeroSSL for example). It fails to handshake, unless there is another non-NATed IP address anywhere in the caddyfile, but then internal tls is always used regardless.

I also reproduced this issue at my house where my public IP is forwarded to my server. And the VPS that has public IP directly, does not have this issue.

I thought I've done this before, so I tried caddy 2.6.4, 2.5.2, and 2.4.6, but it's the same. I must have used nginx last time.

I did try adding the public IP to the host's interface but it did not change it.

[mholt]

What's in the Caddy logs though? (Enable debug mode just in case.) That should reveal the reason.

[ledlamp]

Enabled debug mode and loaded the site.

Jan 31 01:59:26 instance-20220613-2248 caddy[4169334]: {"level":"debug","ts":1738288766.1099389,"logger":"events","msg":"event","name":"tls_get_certificate","id":"74edd539-91a7-4968-8369-7a8b67f16ebe","origin":"tls","data":{"client_hello":{"CipherSuites":[47802,4865,4866,4867,49195,49199,49196,49200,52393,52392,49171,49172,156,157,47,53],"ServerName":"","SupportedCurves":[19018,4588,29,23,24],"SupportedPoints":"AA==","SignatureSchemes":[1027,2052,1025,1283,2053,1281,2054,1537],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[47802,772,771],"RemoteAddr":{"IP":"71.38.127.202","Port":59396,"Zone":""},"LocalAddr":{"IP":"10.0.0.79","Port":443,"Zone":""}}}}
Jan 31 01:59:26 instance-20220613-2248 caddy[4169334]: {"level":"debug","ts":1738288766.1100163,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"10.0.0.79"}
Jan 31 01:59:26 instance-20220613-2248 caddy[4169334]: {"level":"debug","ts":1738288766.1100287,"logger":"tls.handshake","msg":"no certificate matching TLS ClientHello","remote_ip":"71.38.127.202","remote_port":"59396","server_name":"","remote":"71.38.127.202:59396","identifier":"10.0.0.79","cipher_suites":[47802,4865,4866,4867,49195,49199,49196,49200,52393,52392,49171,49172,156,157,47,53],"cert_cache_fill":0.0002,"load_or_obtain_if_necessary":true,"on_demand":false}
Jan 31 01:59:26 instance-20220613-2248 caddy[4169334]: {"level":"debug","ts":1738288766.1102006,"logger":"http.stdlib","msg":"http: TLS handshake error from 71.38.127.202:59396: no certificate available for '10.0.0.79'"}
Jan 31 01:59:26 instance-20220613-2248 caddy[4169334]: {"level":"debug","ts":1738288766.5464563,"logger":"events","msg":"event","name":"tls_get_certificate","id":"25c204dc-c0a6-43b8-811a-1864cb00a2a1","origin":"tls","data":{"client_hello":{"CipherSuites":[10794,4865,4866,4867,49195,49199,49196,49200,52393,52392,49171,49172,156,157,47,53],"ServerName":"","SupportedCurves":[39578,4588,29,23,24],"SupportedPoints":"AA==","SignatureSchemes":[1027,2052,1025,1283,2053,1281,2054,1537],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[56026,772,771],"RemoteAddr":{"IP":"71.38.127.202","Port":59397,"Zone":""},"LocalAddr":{"IP":"10.0.0.79","Port":443,"Zone":""}}}}
Jan 31 01:59:26 instance-20220613-2248 caddy[4169334]: {"level":"debug","ts":1738288766.5465217,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"10.0.0.79"}
Jan 31 01:59:26 instance-20220613-2248 caddy[4169334]: {"level":"debug","ts":1738288766.546536,"logger":"tls.handshake","msg":"no certificate matching TLS ClientHello","remote_ip":"71.38.127.202","remote_port":"59397","server_name":"","remote":"71.38.127.202:59397","identifier":"10.0.0.79","cipher_suites":[10794,4865,4866,4867,49195,49199,49196,49200,52393,52392,49171,49172,156,157,47,53],"cert_cache_fill":0.0002,"load_or_obtain_if_necessary":true,"on_demand":false}
Jan 31 01:59:26 instance-20220613-2248 caddy[4169334]: {"level":"debug","ts":1738288766.5466733,"logger":"http.stdlib","msg":"http: TLS handshake error from 71.38.127.202:59397: no certificate available for '10.0.0.79'"}

It looks like the problem is that when connecting to an IP address with HTTPS, the browser is not including a server name in the client hello: "ServerName":"". So I guess caddy is assuming the server name ~~is the IP of the server~~ (it must be dst addr of the ip packets, this server has multiple ip and i connect on secondary one.) But when there is DNAT this does not match the IP the browser would connect to.

The solution is to set the default_sni in the global options (i.e. default_sni 129.159.139.0). It's not a bug that caddy can fix.