[Feature Request] Option to prefer Moon as relay server even if higher ping than public relay servers

[CaptainXLAB]

Recently got my first VPS and set it up as a zerotier "moon". Was easy to set up and it shows up as "moon" for all clients on my network who have added the vps as moon.

The reason why I did this is because my ISP recently started doing something to their internal network where no direct connections were working whatsoever over IPv4. I also tried alternatives (to zerotier) and they had the same result. Relay server was the only option.

The closest public relay server in my region has a 35 to 40ms ping from where I am which is great, but it is extremely slow and laggy. Friends simply time out in minecraft (the zt network is mostly to play minecraft with a few irl friends in the same country/state, sometimes SMB for quick and easy file sharing)

It took me over 1 month of triple checking everyone's systems, settings, firewalls, etc to figure out why no one could stay connected to the minecraft server (kept timing out) all of a sudden (friends could only connect to me over public relay)

Would be nice to be able to see which IP (or zt addr, anything identifiable) is being used for relay traffic.

Now, the VPS I set up has around 33ms ping but depending on network usage, it sometimes goes over 40ms for me, and this happens to other friends too, resulting in random stuttery ping (going from 35 to over 200 if a lot of data is being transferred), sudden and random unexpected lags, etc.

I know for sure that nothing is wrong with anyone's system config/etc on the network, since when we were all in different cities where ping was low enough that everyone was using the same moon (VPS) as relay, nothing was wrong.

I did my homework, and I know that using private root servers (moons) exclusively isn't an easy option yet (and I'd have to try and rebuild from source without public planet server definitions).

Since public relay servers are under heavy load most of the time (I assume - won't lag so much otherwise) and a major reason behind setting up a moon is to use it as a faster relay -

Can an option be added where moons are always preferred over public relay servers upon config or by default? Is there some way to configure this somehow without having to rebuild from source? (I have full access to the VPS and I can easily ask a few friends to follow some instructions if that's all it would take)

[laduke]

Hi, thanks for writing. Interesting question. I'm not sure if there's anything right now to help with that, but here is an alternative setup:

Just have zerotier on your VPS and your (Minecraft) server. Don't make it a moon. Use iptables to forward minecraft traffic from the VPS to your server over zerotier. Your users would connect to the public IP of the VPS.

Basically: iptables -t nat -A PREROUTING -i eno1 -p tcp --dport 25565 -j DNAT --to 10.147.20.5

There may be more ports needed. This may be a pain. I had to mess with iptables for a bit.

[CaptainXLAB]

I don't allow access to minecraft server from external IPs for security.

I also use Parsec over zerotier, and a lot more things like teamviewer LAN-only connections, SMB/SFTP/FTP, etc and plans to set up a plex server and nextcloud server(s) which will completely break randomly with these unnecessary relay switches to much slower and sluggish public relay servers.

I did find a workaround that involves an alternate solution. I set up Tailscale on all client systems, AND setup tailscale's relay server (DERP server) which requires a secondary VNIC on the VPS since it needs to bind on port 80 and 443 and it's own separate hostname to work, ZeroTier can connect through tailscale (but tailscale does not connect through zerotier, luckily).

Tailscale has the option to "omit default regions" for their relay servers on the entire network, which can completely isolate that tailscale network from normal tailscale servers (exactly what I wanted in zerotier). Now since zerotier thinks Tailscale path is a "direct" connection but tailscale is working over relay, it all works out exactly how I want.

But it's much slower, since it's going through a secondary encrypted tunnel, reducing performance on everything that uses it and limiting file transfer / speed throughput, and it does not work for android since only one VPN is allowed at a time and tailscale + zerotier are two tunnels.

Although I found a slow but functional workaround for now... would still love to have the feature built into zerotier (to let users disable public relay servers and use moon only, whether it's per-client or managed centrally)

[laduke]

Fair enough!

Yeah, doing VPN over VPN is slow.

Add latency to the zerotier roots with tc :D

How are you measuring or seeing that it's switching between roots/moons?

[CaptainXLAB]

How are you measuring or seeing that it's switching between roots/moons?

I simply ping one of my friends indefinitely over zerotier. I know exactly how much ping my own VPS would have - around 60-68ms RTT since everyone usually has 30-34ms ping to the VPS. Connection to the VPS is stable and never stutters since it's not loaded much and has a dedicated 1Gbps VNIC to zerotier moon.

Then, to make absolutely sure - I also use NetLimiter 4. The best network management tool for windows so far that I've seen, and I can clearly see the IPs to/from zerotier service is transferring data. I do see my own VPS's external IPv4 in that list of IPs, usually on top when a friend's connected to the minecraft server or if I'm connected to them over parsec.

If total transfer rate is say 380 KB/s, out of that, 15 KB/s is to other IPs other than my VPS which I assume are other relay servers. Since the view updates per second it's not easy to pinpoint and quickly confirm the other IPs to be zerotier relay servers (I'll have to screen record that view and analyze it later).

Here's a screenshot of how random the ping gets when zerotier is set to RELAY (without tailscale):

Also - these relay server switches usually happen when something bandwidth intensive is being done (parsec streaming at 50Mbps bandwidth as I shake some window on my friend's laptop, transferring some files over, or when all friends move to an area in minecraft which is full of mobs like a friendly mob farm). Some sort of congestion algorithm maybe? I'm not sure.

[laduke]

Thanks. Will have to try to reproduce. Wireshark might be able to help with the recording, but it might be tricky to get it to filter only relevant traffic so you can actually see.

Try setting up your machines with "active-backup" just for fun https://docs.zerotier.com/zerotier/multipath https://github.com/zerotier/ZeroTierOne/tree/master/service On windows, create local.conf in \ProgramData\ZeroTier\One

[CaptainXLAB]

Just realized your reply is edited with more info...

I tried the whole bond thing, created local.conf on my laptop and desktop (on the same physical LAN at the moment). Configured them both as "active-backup" and if I test the output, it does show active-backup for all connected peers.

Problem is - it doesn't list any peer which can't be connected to directly in the bond list. My laptop is shown but my friend somewhere else who is on RELAY isn't even listed in the bond list. My 2nd desktop on the same LAN not configured for active-backup bond type yet, is also listed in bond list, so I don't think it's a configuration issue.

Same behavior continues with relays - randomly jumping around from 60 to 278 to 500+ ms ping, dropping random packets all over the place and back.

It'll still be much simpler and obvious to have an option that prefers moon servers over planet servers for relaying traffic.

[arfianadam]

This. I also have a closer and faster server that I set as a moon, on cable it has around 15ms faster latency. It should be correctly set up already, zerotier-cli listpeers correctly lists the server as MOON. However for some reason, connecting to other peer in different network always uses ZT's relay server. After adding your own moon, ZT's relay should be optional. At least there should be a way to block ZT relay servers from OS level, like using host file. I already tried that but it somehow keeps connecting.

[ChinaNuke]

I have a similar problem. I cannot use p2p connection due to my NAT type so I use my self-hosted moon. Sometimes I feel the connection is very smooth but most of the time the connection is very slow, just like I'm using the planet.

I don't know if the client actually uses my moon. Is there any way I can check if my traffic is going through my moon? Or is there any way to force to use my moon instead of the planets? (I don't want to modify and compile the source code)

[Logiase]

same problem. But I have a very stable connection at 400+ ms, sometimes it use my selfhosted moon server at 30+ ms. In my case I prefer to use my selfhosted moon server for better connections.

[fahimalizain]

Same issue here. My devices are behind different kinds of NAT and DIRECT connections are not possible. I have my VPS connected as a moon. Most of the time I get a smooth RTT of 64ms. But sometimes, for a duration of 5-10 minutes, it goes via public roots with RTT of ~400ms.

Confirmed this in Wireshark by filtering packets by ip.dst = vps.ip.

[laduke]

Is anyone here still interested in this and able and willing to compile this repo themselves, and also running a moon? I can make a quick patch and if it helps do a more proper implementation.

[arfianadam]

Hi @laduke , sure, I can try your patch on weekends.

[nioroso-x3]

I'd also like to try this, some devices can only be relayed and I'd prefer them to use my own moons.

[laduke]

put a QoS setting in a your router that adds 100ms latency to root.zerotier.com 😈

[erikh]

Travis is clearly having a case of the mondays

------- Original Message ------- On Monday, April 10th, 2023 at 8:23 AM, Travis LaDuke @.***> wrote:

put a QoS setting in a your router that adds 100ms latency to root.zerotier.com 😈

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you are subscribed to this thread.Message ID: @.***>

[ninthseason]

I'm currently experiencing the same issue... Is there any progress on this issue？

[ChinaNuke]

Is this problem being noticed? I still have the problem and I can confirm it using wireshark to monitor the connected IPv4 address when I feel laggy. I have a moon server with a latency of about 40ms. I reproduce this by pinging from my computer A to computer B, which are connected via zerotier, and can only have relayed connection. Some times the latency printed by the ping command is around 30-40ms, and I can check in wireshark at the same time and find that my moon server is being connected. But some times the latency printed by the ping command becomes 300-400ms, and at the same time I find in wireshark that one or two the planets are connected.

I'm very confused that I have a closer and faster moon server, why the zerotier use the planet occasionally? It breaks up my experience, it's annoying. Is this a bug?

[laduke]

Hey everyone. Sorry to say, this will probably not get worked on. A tcp relay is a better choice in many situations:

https://github.com/zerotier/pylon?tab=readme-ov-file#reflect-dumb-tcp-relay

[CaptainXLAB]

Hey all, I ended up removing ZeroTier moons and also configuring Tailscale along with a "DERP" server on my vps instead, which uses a server written in Go to automatically handle tcp relaying between tailscale peers, and it supports disabling all other relays (omit default).

ZeroTier can connect through tailscale so it always gets a "DIRECT" connection for all other peers on the same zt and ts network(s).

The opposite of this workaround doesn't happen - Tailscale never connects through ZeroTier. ZT can also connect through Wire guard and any other path it can find between 2 peers, so setting up a wire guard server is another alternative for windows and Linux. (sadly not for Android, only one VPN at a time and wg-kernel didn't help much with this on Android12+)

Cost of this workaround - double encryption along with encapsulating UDP packets into TCP, then the opposite of that is taking place which isn't the most bandwidth efficient and will definitely be slower.

Not fully aware of the exact details but this is what I guess is happening with this workaround:

Comp 1 -> ZT Encryption -> TS Encryption -> UDP to TCP encapsulation -> Decapsulation -> TS Decryption -> ZT Decryption -> Comp 2

Along with that, a downside is that tailscale (without your own relay) might connect through its own public relays which are also very slow because of being public relays, and ZeroTier will consider that as direct and most efficient and not find another direct route (assuming one that takes more hops to reach)

I only needed to play Minecraft with my friends and do some secure file sharing so this much (TS with Relay) is a simple and easy workaround for me.

I'll try ZT pylon TCP relay someday when I have more time, this workaround is all I needed for now.