GET http://localhost:9993/status X-ZT1-Auth:... fails over IPv4 but works over IPv6

[dch]

Please let us know

There's a regression between 1.12.0 and 1.12.1 that breaks ipv4 localhost query via zerotier-cli info. I've only just noticed while adding a new node that this is broken.

Under v1.12.1:

# sudo zerotier-cli status
401 status {}

# zerotier-cli dump
Error connecting to the ZeroTier service: {}

Please check that the service is running and that
TCP port 9993 can be contacted via 127.0.0.1.

# sockstat -46lp 9993
USER     COMMAND    PID   FD  PROTO  LOCAL ADDRESS         FOREIGN ADDRESS
root     zerotier-o 85664 7   tcp4   *:9993                *:*
root     zerotier-o 85664 8   tcp46  *:9993                *:*
root     zerotier-o 85664 14  udp4   10.0.0.112:9993       *:*

But clearly the daemon is up, and listening.

We'd like to see, as under previous versions, the info.

Under v1.12.0:

> sudo zerotier-cli info
200 info 7c5005f22b 1.12.0 ONLINE

• environment is ansible-managed
• runs ZT since over a decade probably
• OS: FreeBSD (various versions)
• ZT: v1.12.2 (everywhere)
• authtoken.secret hasn't changed in years on nodes

Running the same thing using tcpdump/ngrep, we see that this is basically the same as running an HTTP GET to http://localhost:9993/status with header X-ZT1-Auth:my_secret so let's do that with curl:

$ curl -v http://localhost:9993/status -4HX-ZT1-Auth:...
< HTTP/1.1 401 Unauthorized
...
{}

$ curl -v http://localhost:9993/status -6HX-ZT1-Auth:...
< HTTP/1.1 200 OK
...
{"address":"9bbbdbfdd2","clock":1697648307359,"config":{"settings": ....}

The -v4 version should be authorized correctly, but isn't, whether via zerotier-cli or curl.

[laduke]

Hello! Thanks for reporting. This isn't happening on my mac; both curl -4 and curl -6 work. So that's strange.

Are there any error messages when zerotier is starting up?

"authCheck" is around here: https://github.com/zerotier/ZeroTierOne/blob/9ae8b0b3b60b27cf06d7e74629c17e4a0f248364/service/OneService.cpp#L1607

[dch]

No, there's no error messages that I see running a gmake -j debug flavoured build. This only occurs on FreeBSD ofc. I'll throw some printfs in and see what happens. the only change beween 1.12.0 and 1.12.1 is this splitting of address functionality for linux.

[dch]

ok I think we can see why the auth fails :D but do the numbers here mean anything to you?

if (remoteAddr.ipScope() == InetAddress::IP_SCOPE_LOOPBACK) {

zt1flo98dm17np8
authCheck: /status
remoteAddr.ipScope() = 4
IP_SCOPE_LOOPBACK() = 2

[dch]

ifconfig output if that's relevant

lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
	options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
	inet 127.0.0.1 netmask 0xff000000
	inet6 ::1 prefixlen 128
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
	groups: lo
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
wlan0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=0
	ether 00:28:f8:d0:91:52
	inet 172.16.2.21 netmask 0xffffff00 broadcast 172.16.2.255
	inet6 fe80::228:f8ff:fed0:9152%wlan0 prefixlen 64 scopeid 0x2
	groups: wlan
	ssid skunkwerks channel 48 (5240 MHz 11a) bssid 80:2a:a8:85:e2:a3
	regdomain ETSI2 country AT authmode WPA2/802.11i privacy ON
	deftxkey UNDEF AES-CCM 2:128-bit AES-CCM 3:128-bit txpower 17 bmiss 10
	mcastrate 6 mgmtrate 6 scanvalid 60 wme roaming MANUAL
	parent interface: iwm0
	media: IEEE 802.11 Wireless Ethernet OFDM/54Mbps mode 11a
	status: associated
	nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
lo1: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
	options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
	inet 100.64.0.0 netmask 0xffff8000
	inet 100.64.0.1 netmask 0xffffffff
	inet 100.64.0.2 netmask 0xffffffff
	inet 100.64.0.3 netmask 0xffffffff
	inet 100.64.0.4 netmask 0xffffffff
	inet 100.64.0.5 netmask 0xffffffff
	inet 100.64.0.6 netmask 0xffffffff
	inet 100.64.0.7 netmask 0xffffffff
	inet 100.64.0.8 netmask 0xffffffff
	inet 100.64.0.9 netmask 0xffffffff
	inet 100.64.0.10 netmask 0xffffffff
	inet 100.64.0.11 netmask 0xffffffff
	inet 100.64.0.12 netmask 0xffffffff
	inet 100.64.0.13 netmask 0xffffffff
	inet 100.64.0.14 netmask 0xffffffff
	inet 100.64.0.15 netmask 0xffffffff
	inet6 fe80::1%lo1 prefixlen 64 scopeid 0x3
	groups: lo
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ztagim5o45dhe4c: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 5000 mtu 2800
	options=80000<LINKSTATE>
	ether 8e:23:63:d1:3c:17
	hwaddr 58:9c:fc:10:ff:d3
	inet6 fca2:927d:4d9b:bbdb:fdd2::1 prefixlen 40
	inet6 fe80::8c23:63ff:fed1:3c17%ztagim5o45dhe4c prefixlen 64 scopeid 0x4
	groups: tap
	media: Ethernet 1000baseT <full-duplex>
	status: active
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	Opened by PID 5486
zt1flo98dm17np8: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 5000 mtu 2800
	options=80000<LINKSTATE>
	ether 2a:44:a8:b7:be:db
	hwaddr 58:9c:fc:00:0f:27
	inet6 fc7b:c4d6:6b9b:bbdb:fdd2::1 prefixlen 40
	inet6 fe80::2844:a8ff:feb7:bedb%zt1flo98dm17np8 prefixlen 64 scopeid 0x5
	groups: tap
	media: Ethernet 1000baseT <full-duplex>
	status: active
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	Opened by PID 5486

[laduke]

but do the numbers here mean anything to you?

Not really but they're in here https://github.com/zerotier/ZeroTierOne/blob/9ae8b0b3b60b27cf06d7e74629c17e4a0f248364/node/InetAddress.cpp#L29

4 is IP_SCOPE_GLOBAL. When the ip doesn't match anything else in that switch

I'm curious what remoteAddr actually is? I think you can print it like this:

char buf[64];
fprintf(stderr, "remoteAddr %s\n", remoteAddr.toIpString(buf));

[dch]

interesting:

$ doas ngrep -qid lo0 -W byline port 9993
interface: lo0 (127.0.0.0/255.0.0.0)
filter: (ip or ip6) and ( port 9993 )

T 127.0.0.1:10178 -> 127.0.0.1:9993 [AP]
GET /status HTTP/1.1.
X-ZT1-Auth: qrasbyedsabltduqypok8lqo.
.


T 127.0.0.1:9993 -> 127.0.0.1:10178 [AP]
HTTP/1.1 401 Unauthorized.
Content-Length: 2.
Content-Type: application/json.
Keep-Alive: timeout=5, max=5.
.


T 127.0.0.1:9993 -> 127.0.0.1:10178 [AP]
{}

$  doas ./zerotier-one /var/db/zerotier-one/
Starting Control Plane...
Starting V6 Control Plane...
ztagim5o45dhe4c
zt1flo98dm17np8
authCheck: /status
remoteAddr.ipScope() = 4
IP_SCOPE_LOOPBACK() = 2
remoteAddr ::ffff:127.0.0.1

something in between the IP stack (as seen by ngrep output) and zerotier turns this into a v6v4 sort of thing.

[laduke]

That's really strange. I don't think we'll be able to do a fix and release in a timely fashion. You might be able to put ::ffff:127.0.0.1 in your local.conf allowManagementFrom as a work around.

[dch]

That specific pattern didn't work, in the end I used :: and rely on the per-node authsecret and local firewall rules for security. Is that sufficient? It's certainly not ideal :-(

[laduke]

Seems like it to me, but I don't know the threat model, your requirements, etc...

remove 127.0.0.1 localhost from /etc/hosts :)

[dch]

This basically broke new zerotier installs on all FreeBSD and TrueNAS systems, you can't add nodes unless find this issue.

Any chance this could get addressed for 1.14? The fix is likely to be quite trivial, to add the missing match to InetAddress.cpp.

In OneService.cpp#L1601-L1610 if (remoteAddr.ipScope() == InetAddress::IP_SCOPE_LOOPBACK) {

is called with these values:

zt1flo98dm17np8
authCheck: /status
remoteAddr.ipScope() = 4
IP_SCOPE_LOOPBACK() = 2
remoteAddr ::ffff:127.0.0.1

According to InetAddress.hpp#L67C1-L67C75 ::ffff:127.0.0.1 is identified as IP_SCOPE_GLOBAL which seems incorrect.

AFAICT this is an error in InetAddress.cpp which decides that 0xff_ff_7f_00_00_01 or something similar to that in whatever internal representation is used, is not local.

I will test on FreeBSD 13.x and 14.x again to see what I can track down, in particular what the memory representation of ::ffff:127.0.0.1 is, which is what is needed to check against.

[laduke]

yeah lets take a look

[dch]

OK it aint pretty, but it does fix the issue... over to you c++ gurus to make it pretty/correct:

diff --git a/node/InetAddress.cpp b/node/InetAddress.cpp
index da1c7294..7a786e49 100644
--- a/node/InetAddress.cpp
+++ b/node/InetAddress.cpp
@@ -112,6 +112,20 @@ InetAddress::IpScope InetAddress::ipScope() const
                }       break;

                case AF_INET6: {
+         // Check for IPv4-mapped IPv6 addresses (::ffff::/96 prefix)
+            const struct sockaddr_in6* sa6 = reinterpret_cast<const struct sockaddr_in6*>(this);
+            if (sa6->sin6_addr.s6_addr[0] == 0 && sa6->sin6_addr.s6_addr[1] == 0 &&
+                sa6->sin6_addr.s6_addr[2] == 0 && sa6->sin6_addr.s6_addr[3] == 0 &&
+                sa6->sin6_addr.s6_addr[4] == 0 && sa6->sin6_addr.s6_addr[5] == 0 &&
+                sa6->sin6_addr.s6_addr[6] == 0 && sa6->sin6_addr.s6_addr[7] == 0 &&
+                sa6->sin6_addr.s6_addr[8] == 0 && sa6->sin6_addr.s6_addr[9] == 0 &&
+                sa6->sin6_addr.s6_addr[10] == 0xff && sa6->sin6_addr.s6_addr[11] == 0xff) {
+                // next 4 bytes should be IPv4 address 0x7f000001 / 127.0.0.1
+                uint32_t ipv4Part = *(reinterpret_cast<const uint32_t*>(&sa6->sin6_addr.s6_addr[12]));
+                if (ipv4Part == htonl(0x7f000001)) {
+                    return IP_SCOPE_LOOPBACK;
+                }
+            }
                        const unsigned char *ip = reinterpret_cast<const unsigned char *>(reinterpret_cast<const struct sockaddr_in6 *>(this)->sin6_addr.s6_addr);
                        if ((ip[0] & 0xf0) == 0xf0) {
                                if (ip[0] == 0xff) {
diff --git a/service/OneService.cpp b/service/OneService.cpp
index b06bcb9b..8f52ce58 100644
--- a/service/OneService.cpp
+++ b/service/OneService.cpp
@@ -1697,8 +1697,14 @@ public:
                                bool ipAllowed = false;
                                bool isAuth = false;
                                // If localhost, allow
+                               char buf[64];
+                               fprintf(stderr, "remoteAddr %s\n", remoteAddr.toIpString(buf));
                                if (remoteAddr.ipScope() == InetAddress::IP_SCOPE_LOOPBACK) {
+                                       fprintf(stderr, "ALLOWED\n");
                                        ipAllowed = true;
+                               } else
+                               {
+                                       fprintf(stderr, "DENIED\n");
                                }

                                if (!ipAllowed) {

[dch]

was this perhaps a change in the underlying httplib that passes a different form of remote address through?

[laduke]

Not sure yet. Thanks for working on it. Even curl -v -4 "http://127.0.0.1:9993/status pops out as ::ffff:127.0.0.1 inside of zerotier-one, but not on other operating systems.

[dch]

once there's a more appropriate patch I'll backport it to the current FreeBSD port.

[laduke]

The httplib unsets the IPV6_V6ONLY socket option. https://github.com/yhirose/cpp-httplib/commit/b2203bb05aa241a3dd00719c8afd07d82900ba3d

This seems to make all IPv4 requests come in with the ::ffff:, as a v6 to 4 mapping thing.

Making it parse ::ffff:127.0.0.1 as loopback works.

But it also does the 6 to 4 thing to other ip addresses:

curl -4 "http://45.32.69.185:9993/ -> ::ffff:45.32.69.185 which might make configuring things like "allowManagementFrom" a little weird.

this patch also makes it work. Maybe we can make that a FreeBSD specific patch or something.

[dch]

Your improved version of my ugly hack looks great, let's roll - thanks!

BTW I tested this patch and wasn't able to run info or join ....

Perhaps, for a dual-stack address like localhost the AF_INET6 branch never actually gets reached?

[dch]

thanks, works a treat. I pulled this into FreeBSD 1.12.2 as an upstream patch.