ZeroTierOne
ZeroTierOne copied to clipboard
zerotier
Segfaults every 10 minutes when built against miniupnpc 2.2.8
Happens on at least 2 different machines (one bare-metal, one VM).
OS & version: Zerotier 1.14.0 on Archlinux Package recently rebuilt against miniupnpc 2.2.8 https://gitlab.archlinux.org/archlinux/packaging/packages/zerotier-one/-/commit/cb502ed73643bc9bbf26f9f08f92620508f492f4
Journal log with stacktrace
Jul 06 09:09:47 archlinux kernel: zerotier-one[35799]: segfault at fffffffb ip 000074d25dacec46 sp 000074d25dffbc18 error 6 in libc.so.6[74d25da38000+16c000] likely on CPU 0 (core 0, socket 0)
Jul 06 09:09:47 archlinux kernel: Code: c3 0f 1f 00 48 8b 0e 8b 56 07 48 89 0f 89 57 07 49 83 e8 0b 48 8d 7f 0b 0f 85 66 07 00 00 c3 0f 1f 44 00 00 48 8b 0e 8b 56 08 <48> 89 0f 89 57 08 49 83 e8 0c 48 8d 7f 0c 0f 85 46 07 00 00 c3 0f
Jul 06 09:09:47 archlinux systemd-coredump[36532]: Process 35797 (zerotier-one) of user 971 terminated abnormally with signal 11/SEGV, processing...
Jul 06 09:09:47 archlinux systemd[1]: Started Process Core Dump (PID 36532/UID 0).
Jul 06 09:09:47 archlinux systemd-coredump[36533]: [🡕] Process 35797 (zerotier-one) of user 971 dumped core.
Stack trace of thread 35799:
#0 0x000074d25dacec46 n/a (libc.so.6 + 0xbac46)
#1 0x000074d25e7afc36 UPNP_GetValidIGD (libminiupnpc.so.18 + 0x9c36)
#2 0x0000598847e84a4e n/a (zerotier-one + 0x1daa4e)
#3 0x0000598847e848cd n/a (zerotier-one + 0x1da8cd)
#4 0x000074d25daa6ded n/a (libc.so.6 + 0x92ded)
#5 0x000074d25db2a0dc n/a (libc.so.6 + 0x1160dc)
Stack trace of thread 35797:
#0 0x000074d25db278f7 __select (libc.so.6 + 0x1138f7)
#1 0x0000598847e3c1c3 n/a (zerotier-one + 0x1921c3)
#2 0x0000598847e54bfd n/a (zerotier-one + 0x1aabfd)
#3 0x0000598847e94f0c n/a (zerotier-one + 0x1eaf0c)
#4 0x0000598847cf4861 n/a (zerotier-one + 0x4a861)
#5 0x000074d25da39c88 n/a (libc.so.6 + 0x25c88)
#6 0x000074d25da39d4c __libc_start_main (libc.so.6 + 0x25d4c)
#7 0x0000598847cf4d45 n/a (zerotier-one + 0x4ad45)
Stack trace of thread 35820:
#0 0x000074d25db2c05a recv (libc.so.6 + 0x11805a)
#1 0x0000598847e81c1c n/a (zerotier-one + 0x1d7c1c)
#2 0x0000598847e83b44 n/a (zerotier-one + 0x1d9b44)
#3 0x000074d25daa6ded n/a (libc.so.6 + 0x92ded)
#4 0x000074d25db2a0dc n/a (libc.so.6 + 0x1160dc)
Stack trace of thread 35798:
#0 0x000074d25daf2f43 clock_nanosleep (libc.so.6 + 0xdef43)
#1 0x000074d25dafed77 __nanosleep (libc.so.6 + 0xead77)
#2 0x0000598847d297b8 n/a (zerotier-one + 0x7f7b8)
#3 0x000074d25dce0c84 execute_native_thread_routine (libstdc++.so.6 + 0xe0c84)
#4 0x000074d25daa6ded n/a (libc.so.6 + 0x92ded)
#5 0x000074d25db2a0dc n/a (libc.so.6 + 0x1160dc)
Stack trace of thread 35801:
#0 0x000074d25daf2f43 clock_nanosleep (libc.so.6 + 0xdef43)
#1 0x000074d25dafed77 __nanosleep (libc.so.6 + 0xead77)
#2 0x0000598847d99f1b n/a (zerotier-one + 0xeff1b)
#3 0x000074d25dce0c84 execute_native_thread_routine (libstdc++.so.6 + 0xe0c84)
#4 0x000074d25daa6ded n/a (libc.so.6 + 0x92ded)
#5 0x000074d25db2a0dc n/a (libc.so.6 + 0x1160dc)
Stack trace of thread 35822:
#0 0x000074d25db278f7 __select (libc.so.6 + 0x1138f7)
#1 0x0000598847e7c6aa n/a (zerotier-one + 0x1d26aa)
#2 0x000074d25dce0c84 execute_native_thread_routine (libstdc++.so.6 + 0xe0c84)
#3 0x000074d25daa6ded n/a (libc.so.6 + 0x92ded)
#4 0x000074d25db2a0dc n/a (libc.so.6 + 0x1160dc)
Stack trace of thread 35802:
#0 0x000074d25db2bc7d accept (libc.so.6 + 0x117c7d)
#1 0x0000598847e5caf1 n/a (zerotier-one + 0x1b2af1)
#2 0x0000598847e36624 n/a (zerotier-one + 0x18c624)
#3 0x000074d25dce0c84 execute_native_thread_routine (libstdc++.so.6 + 0xe0c84)
#4 0x000074d25daa6ded n/a (libc.so.6 + 0x92ded)
#5 0x000074d25db2a0dc n/a (libc.so.6 + 0x1160dc)
Stack trace of thread 35812:
#0 0x000074d25daa34e9 n/a (libc.so.6 + 0x8f4e9)
#1 0x000074d25daa5ed9 pthread_cond_wait (libc.so.6 + 0x91ed9)
#2 0x000074d25dcd5e61 __gthread_cond_wait (libstdc++.so.6 + 0xd5e61)
#3 0x0000598847e5899b n/a (zerotier-one + 0x1ae99b)
#4 0x000074d25dce0c84 execute_native_thread_routine (libstdc++.so.6 + 0xe0c84)
#5 0x000074d25daa6ded n/a (libc.so.6 + 0x92ded)
#6 0x000074d25db2a0dc n/a (libc.so.6 + 0x1160dc)
Stack trace of thread 35800:
#0 0x000074d25daf2f43 clock_nanosleep (libc.so.6 + 0xdef43)
#1 0x000074d25dafed77 __nanosleep (libc.so.6 + 0xead77)
#2 0x0000598847df1dc3 n/a (zerotier-one + 0x147dc3)
#3 0x000074d25dce0c84 execute_native_thread_routine (libstdc++.so.6 + 0xe0c84)
#4 0x000074d25daa6ded n/a (libc.so.6 + 0x92ded)
#5 0x000074d25db2a0dc n/a (libc.so.6 + 0x1160dc)
Stack trace of thread 35804:
#0 0x000074d25daa34e9 n/a (libc.so.6 + 0x8f4e9)
#1 0x000074d25daa5ed9 pthread_cond_wait (libc.so.6 + 0x91ed9)
#2 0x000074d25dcd5e61 __gthread_cond_wait (libstdc++.so.6 + 0xd5e61)
#3 0x0000598847e5899b n/a (zerotier-one + 0x1ae99b)
#4 0x000074d25dce0c84 execute_native_thread_routine (libstdc++.so.6 + 0xe0c84)
#5 0x000074d25daa6ded n/a (libc.so.6 + 0x92ded)
#6 0x000074d25db2a0dc n/a (libc.so.6 + 0x1160dc)
Stack trace of thread 35805:
#0 0x000074d25daa34e9 n/a (libc.so.6 + 0x8f4e9)
#1 0x000074d25daa5ed9 pthread_cond_wait (libc.so.6 + 0x91ed9)
#2 0x000074d25dcd5e61 __gthread_cond_wait (libstdc++.so.6 + 0xd5e61)
#3 0x0000598847e5899b n/a (zerotier-one + 0x1ae99b)
#4 0x000074d25dce0c84 execute_native_thread_routine (libstdc++.so.6 + 0xe0c84)
#5 0x000074d25daa6ded n/a (libc.so.6 + 0x92ded)
#6 0x000074d25db2a0dc n/a (libc.so.6 + 0x1160dc)
Stack trace of thread 35803:
#0 0x000074d25db2bc7d accept (libc.so.6 + 0x117c7d)
#1 0x0000598847e5caf1 n/a (zerotier-one + 0x1b2af1)
#2 0x0000598847e364e4 n/a (zerotier-one + 0x18c4e4)
#3 0x000074d25dce0c84 execute_native_thread_routine (libstdc++.so.6 + 0xe0c84)
#4 0x000074d25daa6ded n/a (libc.so.6 + 0x92ded)
#5 0x000074d25db2a0dc n/a (libc.so.6 + 0x1160dc)
Stack trace of thread 35806:
#0 0x000074d25daa34e9 n/a (libc.so.6 + 0x8f4e9)
#1 0x000074d25daa5ed9 pthread_cond_wait (libc.so.6 + 0x91ed9)
#2 0x000074d25dcd5e61 __gthread_cond_wait (libstdc++.so.6 + 0xd5e61)
#3 0x0000598847e5899b n/a (zerotier-one + 0x1ae99b)
#4 0x000074d25dce0c84 execute_native_thread_routine (libstdc++.so.6 + 0xe0c84)
#5 0x000074d25daa6ded n/a (libc.so.6 + 0x92ded)
#6 0x000074d25db2a0dc n/a (libc.so.6 + 0x1160dc)
Stack trace of thread 35807:
#0 0x000074d25daa34e9 n/a (libc.so.6 + 0x8f4e9)
#1 0x000074d25daa5ed9 pthread_cond_wait (libc.so.6 + 0x91ed9)
#2 0x000074d25dcd5e61 __gthread_cond_wait (libstdc++.so.6 + 0xd5e61)
#3 0x0000598847e5899b n/a (zerotier-one + 0x1ae99b)
#4 0x000074d25dce0c84 execute_native_thread_routine (libstdc++.so.6 + 0xe0c84)
#5 0x000074d25daa6ded n/a (libc.so.6 + 0x92ded)
#6 0x000074d25db2a0dc n/a (libc.so.6 + 0x1160dc)
Stack trace of thread 35808:
#0 0x000074d25daa34e9 n/a (libc.so.6 + 0x8f4e9)
#1 0x000074d25daa5ed9 pthread_cond_wait (libc.so.6 + 0x91ed9)
#2 0x000074d25dcd5e61 __gthread_cond_wait (libstdc++.so.6 + 0xd5e61)
#3 0x0000598847e5899b n/a (zerotier-one + 0x1ae99b)
#4 0x000074d25dce0c84 execute_native_thread_routine (libstdc++.so.6 + 0xe0c84)
#5 0x000074d25daa6ded n/a (libc.so.6 + 0x92ded)
#6 0x000074d25db2a0dc n/a (libc.so.6 + 0x1160dc)
Stack trace of thread 35813:
#0 0x000074d25daa34e9 n/a (libc.so.6 + 0x8f4e9)
#1 0x000074d25daa5ed9 pthread_cond_wait (libc.so.6 + 0x91ed9)
#2 0x000074d25dcd5e61 __gthread_cond_wait (libstdc++.so.6 + 0xd5e61)
#3 0x0000598847e5899b n/a (zerotier-one + 0x1ae99b)
#4 0x000074d25dce0c84 execute_native_thread_routine (libstdc++.so.6 + 0xe0c84)
#5 0x000074d25daa6ded n/a (libc.so.6 + 0x92ded)
#6 0x000074d25db2a0dc n/a (libc.so.6 + 0x1160dc)
Stack trace of thread 35814:
#0 0x000074d25daa34e9 n/a (libc.so.6 + 0x8f4e9)
#1 0x000074d25daa5ed9 pthread_cond_wait (libc.so.6 + 0x91ed9)
#2 0x000074d25dcd5e61 __gthread_cond_wait (libstdc++.so.6 + 0xd5e61)
#3 0x0000598847e5899b n/a (zerotier-one + 0x1ae99b)
#4 0x000074d25dce0c84 execute_native_thread_routine (libstdc++.so.6 + 0xe0c84)
#5 0x000074d25daa6ded n/a (libc.so.6 + 0x92ded)
#6 0x000074d25db2a0dc n/a (libc.so.6 + 0x1160dc)
Stack trace of thread 35809:
#0 0x000074d25daa34e9 n/a (libc.so.6 + 0x8f4e9)
#1 0x000074d25daa5ed9 pthread_cond_wait (libc.so.6 + 0x91ed9)
#2 0x000074d25dcd5e61 __gthread_cond_wait (libstdc++.so.6 + 0xd5e61)
#3 0x0000598847e5899b n/a (zerotier-one + 0x1ae99b)
#4 0x000074d25dce0c84 execute_native_thread_routine (libstdc++.so.6 + 0xe0c84)
#5 0x000074d25daa6ded n/a (libc.so.6 + 0x92ded)
#6 0x000074d25db2a0dc n/a (libc.so.6 + 0x1160dc)
Stack trace of thread 35815:
#0 0x000074d25daa34e9 n/a (libc.so.6 + 0x8f4e9)
#1 0x000074d25daa5ed9 pthread_cond_wait (libc.so.6 + 0x91ed9)
#2 0x000074d25dcd5e61 __gthread_cond_wait (libstdc++.so.6 + 0xd5e61)
#3 0x0000598847e5899b n/a (zerotier-one + 0x1ae99b)
#4 0x000074d25dce0c84 execute_native_thread_routine (libstdc++.so.6 + 0xe0c84)
#5 0x000074d25daa6ded n/a (libc.so.6 + 0x92ded)
#6 0x000074d25db2a0dc n/a (libc.so.6 + 0x1160dc)
Stack trace of thread 35810:
#0 0x000074d25daa34e9 n/a (libc.so.6 + 0x8f4e9)
#1 0x000074d25daa5ed9 pthread_cond_wait (libc.so.6 + 0x91ed9)
#2 0x000074d25dcd5e61 __gthread_cond_wait (libstdc++.so.6 + 0xd5e61)
#3 0x0000598847e5899b n/a (zerotier-one + 0x1ae99b)
#4 0x000074d25dce0c84 execute_native_thread_routine (libstdc++.so.6 + 0xe0c84)
#5 0x000074d25daa6ded n/a (libc.so.6 + 0x92ded)
#6 0x000074d25db2a0dc n/a (libc.so.6 + 0x1160dc)
Stack trace of thread 35811:
#0 0x000074d25daa34e9 n/a (libc.so.6 + 0x8f4e9)
#1 0x000074d25daa5ed9 pthread_cond_wait (libc.so.6 + 0x91ed9)
#2 0x000074d25dcd5e61 __gthread_cond_wait (libstdc++.so.6 + 0xd5e61)
#3 0x0000598847e5899b n/a (zerotier-one + 0x1ae99b)
#4 0x000074d25dce0c84 execute_native_thread_routine (libstdc++.so.6 + 0xe0c84)
#5 0x000074d25daa6ded n/a (libc.so.6 + 0x92ded)
#6 0x000074d25db2a0dc n/a (libc.so.6 + 0x1160dc)
Stack trace of thread 35816:
#0 0x000074d25daa34e9 n/a (libc.so.6 + 0x8f4e9)
#1 0x000074d25daa5ed9 pthread_cond_wait (libc.so.6 + 0x91ed9)
#2 0x000074d25dcd5e61 __gthread_cond_wait (libstdc++.so.6 + 0xd5e61)
#3 0x0000598847e5899b n/a (zerotier-one + 0x1ae99b)
#4 0x000074d25dce0c84 execute_native_thread_routine (libstdc++.so.6 + 0xe0c84)
#5 0x000074d25daa6ded n/a (libc.so.6 + 0x92ded)
#6 0x000074d25db2a0dc n/a (libc.so.6 + 0x1160dc)
Stack trace of thread 35817:
#0 0x000074d25daa34e9 n/a (libc.so.6 + 0x8f4e9)
#1 0x000074d25daa5ed9 pthread_cond_wait (libc.so.6 + 0x91ed9)
#2 0x000074d25dcd5e61 __gthread_cond_wait (libstdc++.so.6 + 0xd5e61)
#3 0x0000598847e5899b n/a (zerotier-one + 0x1ae99b)
#4 0x000074d25dce0c84 execute_native_thread_routine (libstdc++.so.6 + 0xe0c84)
#5 0x000074d25daa6ded n/a (libc.so.6 + 0x92ded)
#6 0x000074d25db2a0dc n/a (libc.so.6 + 0x1160dc)
Stack trace of thread 35818:
#0 0x000074d25daa34e9 n/a (libc.so.6 + 0x8f4e9)
#1 0x000074d25daa5ed9 pthread_cond_wait (libc.so.6 + 0x91ed9)
#2 0x000074d25dcd5e61 __gthread_cond_wait (libstdc++.so.6 + 0xd5e61)
#3 0x0000598847e5899b n/a (zerotier-one + 0x1ae99b)
#4 0x000074d25dce0c84 execute_native_thread_routine (libstdc++.so.6 + 0xe0c84)
#5 0x000074d25daa6ded n/a (libc.so.6 + 0x92ded)
#6 0x000074d25db2a0dc n/a (libc.so.6 + 0x1160dc)
Stack trace of thread 35819:
#0 0x000074d25daa34e9 n/a (libc.so.6 + 0x8f4e9)
#1 0x000074d25daa5ed9 pthread_cond_wait (libc.so.6 + 0x91ed9)
#2 0x000074d25dcd5e61 __gthread_cond_wait (libstdc++.so.6 + 0xd5e61)
#3 0x0000598847e5899b n/a (zerotier-one + 0x1ae99b)
#4 0x000074d25dce0c84 execute_native_thread_routine (libstdc++.so.6 + 0xe0c84)
#5 0x000074d25daa6ded n/a (libc.so.6 + 0x92ded)
#6 0x000074d25db2a0dc n/a (libc.so.6 + 0x1160dc)
ELF object binary architecture: AMD x86-64
Jul 06 09:09:47 archlinux systemd[1]: [email protected]: Deactivated successfully.
Jul 06 09:09:47 archlinux systemd-networkd[410]: zt*********: Link DOWN
Jul 06 09:09:47 archlinux systemd-networkd[410]: zt*********: Lost carrier
Jul 06 09:09:47 archlinux systemd[1]: zerotier-one.service: Main process exited, code=dumped, status=11/SEGV
Jul 06 09:09:47 archlinux systemd[1]: zerotier-one.service: Failed with result 'core-dump'.
Jul 06 09:09:47 archlinux systemd[1]: zerotier-one.service: Triggering OnFailure= dependencies.
Jul 06 09:09:47 archlinux systemd[1]: Started Send a notification about a failed systemd unit.
Jul 06 09:09:47 archlinux systemd[1]: zerotier-one.service: Scheduled restart job, restart counter is at 53.
I never had to do this workaround, and I never got the "error loading shared libraries" problem. I updated my systems with both the latest miniupnpc and zerotier-one packages, which had the fix already. The problem is that somehow this version causes the zerotier-one process to crash with a segfault every 10 minutes or so.
Sorry about that. We include a specific version of miniupnpc in our builds. Arch must be doing something special.
You can disable upnp in local.conf portMappingEnabled
Doesn't look like it. The miniupnpc package doesn't have any patches, nor any special build commands: https://gitlab.archlinux.org/archlinux/packaging/packages/miniupnpc/-/blob/main/PKGBUILD?ref_type=heads
zerotier-one package also looks bog-standard: https://gitlab.archlinux.org/archlinux/packaging/packages/zerotier-one/-/blob/main/PKGBUILD?ref_type=heads
Nevertheless, I will try to disable upnp in config (which I guess is probably broken anyway, since segfaults).
We include a specific version of miniupnpc in our builds
Which version are you including, specifically? Can you include the latest?
It's part of the default build: https://github.com/zerotier/ZeroTierOne/tree/dev/ext/miniupnpc
I can see it's old (last commit a year ago, and VERSION file lists "2.0").
Looking at the latest changelog from the project, this seems significant: http://miniupnp.free.fr/files/changelog.php?file=miniupnpc-2.2.8.tar.gz
2024/05/08:
upnpc.c: Add -f option to upnpc program (delete multiple port redirections)
UPNP_GetValidIGD(): distinguish between not connected and connected to a
"private" network (with a reserved IP address).
Increments API_VERSION to 18
My segfault stacktrace lists this exact function:
Stack trace of thread 35799:
#0 0x000074d25dacec46 n/a (libc.so.6 + 0xbac46)
#1 0x000074d25e7afc36 UPNP_GetValidIGD (libminiupnpc.so.18 + 0x9c36)
#2 0x0000598847e84a4e n/a (zerotier-one + 0x1daa4e)
#3 0x0000598847e848cd n/a (zerotier-one + 0x1da8cd)
#4 0x000074d25daa6ded n/a (libc.so.6 + 0x92ded)
#5 0x000074d25db2a0dc n/a (libc.so.6 + 0x1160dc)
Yes indeed, sorry for not responding sooner. It helped with the crashes. However, I haven't tested the impact on connectivity between non-port-forwarded hosts (I use manual port forwarding on at least one of the endpoints in any of my configurations).
Anybody wants to try zerotier-one 1.14.2-2 form Arch Linux extra-testing repository?
@eworm-de Good news: it's working for me. Thank you very much!
A bittersweet realization for me is that I finally know that this timeout message that's constantly spamming my journal:
lis 06 17:22:50 overlord zerotier-one[534]: Starting V6 Control Plane...
lis 06 17:23:05 overlord zerotier-one[534]: connect: Connection timed out
(...)
lis 06 18:18:01 overlord zerotier-one[534]: connect: Connection timed out
lis 06 18:18:07 overlord zerotier-one[534]: connect: Connection timed out
lis 06 18:18:13 overlord zerotier-one[534]: connect: Connection timed out
lis 06 18:18:19 overlord zerotier-one[534]: connect: Connection timed out
must be because the UPnP is enabled, because it's not generated otherwise. However, this was happening (on multiple different Archlinux hosts) for a long time, long before this issue, and I couldn't find anything on the web about it. Maybe if someone here has any inkling as to why this could be happening, they could drop a dime. I didn't bother to create an issue for this, because I don't know any details, nor how to debug this.
yeah that message comes from miniupnpc. I don't 100% remember why. It probably can't talk to the router for either upnp or nat-pmp