ZeroTierOne icon indicating copy to clipboard operation
ZeroTierOne copied to clipboard
verified publisher icon zerotier

Informational commands segfault

Open mcondarelli opened this issue 4 years ago • 5 comments

Required

  • What you expect to be happening.
    • I expect to be able to use zerotier-cli
  • What is actually happening?
    • most commands exit with "Segmentation fault"
  • Any steps to reproduce the error.
    • It is fully reproducible in my environment
  • Any screenshots that would help us out.
    • I did a complete strace (see below)

Additional information

Embedded & NAS (please complete the following information):

  • Device: Custom board based on i.MX6 Quad/Dual Lite
  • OS/Distribution: Powered by LuCI for-15.05 branch (git-21.110.51727-2846234) / ITS-OB4-M Chaos Calmer v2.2.1 unknown
  • ZeroTier Version 1.6.6 from git (19f3a97a8de0dd9a4d5b29b57147e41c46672de3)

Additional context

  • Nothing relevant here

Specifics:

  • This is self built using custom (OpenWRT -generated) toolchain.
  • System works perfectly, routing seems absolutely OK.
  • Commands to join/leave network work as expected: 
    # /var/lib/zerotier-one/zerotier-cli join xxxxxxxxxxxxx
200 join OK
  • Informational commands fail: 
    # /var/lib/zerotier-one/zerotier-cli status
Segmentation fault
  • Also rather complex "no output" commands work (output file looks good): 
    # /var/lib/zerotier-one/zerotier-cli dump
Writing dump to: /root/zerotier_dump.txt
  • This may well be a problem in my compilation process, but I'm unable to divine where to look, any hint welcome.
  • I did a complete strace, but I see nothing suspicious:
strace /var/lib/zerotier-one/zerotier-cli info
execve("/var/lib/zerotier-one/zerotier-cli", ["/var/lib/zerotier-one/zerotier-c"..., "info"], [/* 12 vars */]) = 0
brk(0)                                  = 0x18de000
uname({sys="Linux", node="ITS-OB4-M-1002225", ...}) = 0
mmap2(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x76f8c000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/lib/tls/v7l/neon/vfp/libstdc++.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/lib/tls/v7l/neon/vfp", 0x7e9276d8) = -1 ENOENT (No such file or directory)
open("/lib/tls/v7l/neon/libstdc++.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/lib/tls/v7l/neon", 0x7e9276d8) = -1 ENOENT (No such file or directory)
open("/lib/tls/v7l/vfp/libstdc++.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/lib/tls/v7l/vfp", 0x7e9276d8)  = -1 ENOENT (No such file or directory)
open("/lib/tls/v7l/libstdc++.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/lib/tls/v7l", 0x7e9276d8)      = -1 ENOENT (No such file or directory)
open("/lib/tls/neon/vfp/libstdc++.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/lib/tls/neon/vfp", 0x7e9276d8) = -1 ENOENT (No such file or directory)
open("/lib/tls/neon/libstdc++.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/lib/tls/neon", 0x7e9276d8)     = -1 ENOENT (No such file or directory)
open("/lib/tls/vfp/libstdc++.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/lib/tls/vfp", 0x7e9276d8)      = -1 ENOENT (No such file or directory)
open("/lib/tls/libstdc++.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/lib/tls", 0x7e9276d8)          = -1 ENOENT (No such file or directory)
open("/lib/v7l/neon/vfp/libstdc++.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/lib/v7l/neon/vfp", 0x7e9276d8) = -1 ENOENT (No such file or directory)
open("/lib/v7l/neon/libstdc++.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/lib/v7l/neon", 0x7e9276d8)     = -1 ENOENT (No such file or directory)
open("/lib/v7l/vfp/libstdc++.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/lib/v7l/vfp", 0x7e9276d8)      = -1 ENOENT (No such file or directory)
open("/lib/v7l/libstdc++.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/lib/v7l", 0x7e9276d8)          = -1 ENOENT (No such file or directory)
open("/lib/neon/vfp/libstdc++.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/lib/neon/vfp", 0x7e9276d8)     = -1 ENOENT (No such file or directory)
open("/lib/neon/libstdc++.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/lib/neon", 0x7e9276d8)         = -1 ENOENT (No such file or directory)
open("/lib/vfp/libstdc++.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/lib/vfp", 0x7e9276d8)          = -1 ENOENT (No such file or directory)
open("/lib/libstdc++.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/lib", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
open("/usr/lib/tls/v7l/neon/vfp/libstdc++.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/tls/v7l/neon/vfp", 0x7e9276d8) = -1 ENOENT (No such file or directory)
open("/usr/lib/tls/v7l/neon/libstdc++.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/tls/v7l/neon", 0x7e9276d8) = -1 ENOENT (No such file or directory)
open("/usr/lib/tls/v7l/vfp/libstdc++.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/tls/v7l/vfp", 0x7e9276d8) = -1 ENOENT (No such file or directory)
open("/usr/lib/tls/v7l/libstdc++.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/tls/v7l", 0x7e9276d8)  = -1 ENOENT (No such file or directory)
open("/usr/lib/tls/neon/vfp/libstdc++.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/tls/neon/vfp", 0x7e9276d8) = -1 ENOENT (No such file or directory)
open("/usr/lib/tls/neon/libstdc++.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/tls/neon", 0x7e9276d8) = -1 ENOENT (No such file or directory)
open("/usr/lib/tls/vfp/libstdc++.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/tls/vfp", 0x7e9276d8)  = -1 ENOENT (No such file or directory)
open("/usr/lib/tls/libstdc++.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/tls", 0x7e9276d8)      = -1 ENOENT (No such file or directory)
open("/usr/lib/v7l/neon/vfp/libstdc++.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/v7l/neon/vfp", 0x7e9276d8) = -1 ENOENT (No such file or directory)
open("/usr/lib/v7l/neon/libstdc++.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/v7l/neon", 0x7e9276d8) = -1 ENOENT (No such file or directory)
open("/usr/lib/v7l/vfp/libstdc++.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/v7l/vfp", 0x7e9276d8)  = -1 ENOENT (No such file or directory)
open("/usr/lib/v7l/libstdc++.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/v7l", 0x7e9276d8)      = -1 ENOENT (No such file or directory)
open("/usr/lib/neon/vfp/libstdc++.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/neon/vfp", 0x7e9276d8) = -1 ENOENT (No such file or directory)
open("/usr/lib/neon/libstdc++.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/neon", 0x7e9276d8)     = -1 ENOENT (No such file or directory)
open("/usr/lib/vfp/libstdc++.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/vfp", 0x7e9276d8)      = -1 ENOENT (No such file or directory)
open("/usr/lib/libstdc++.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\1\1\1\3\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0\260\341\6\0004\0\0\0"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=1265196, ...}) = 0
mmap2(NULL, 1337044, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x76e18000
mprotect(0x76f46000, 65536, PROT_NONE)  = 0
mmap2(0x76f56000, 28672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x12e000) = 0x76f56000
mmap2(0x76f5d000, 5844, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x76f5d000
close(3)                                = 0
open("/lib/libm.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0\240<\0\0004\0\0\0"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=447924, ...}) = 0
mmap2(NULL, 512112, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x76d9a000
mprotect(0x76e06000, 65536, PROT_NONE)  = 0
mmap2(0x76e16000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x6c000) = 0x76e16000
close(3)                                = 0
open("/lib/libgcc_s.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0@;\0\0004\0\0\0"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0644, st_size=38956, ...}) = 0
mmap2(NULL, 103240, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x76d80000
mprotect(0x76d8a000, 61440, PROT_NONE)  = 0
mmap2(0x76d99000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x9000) = 0x76d99000
close(3)                                = 0
open("/lib/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0\230G\0\0004\0\0\0"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=88544, ...}) = 0
mmap2(NULL, 160320, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x76d58000
mprotect(0x76d6d000, 61440, PROT_NONE)  = 0
mmap2(0x76d7c000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x14000) = 0x76d7c000
mmap2(0x76d7e000, 4672, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x76d7e000
close(3)                                = 0
open("/lib/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0008h\1\0004\0\0\0"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=1226500, ...}) = 0
mmap2(NULL, 1295696, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x76c1b000
mprotect(0x76d42000, 65536, PROT_NONE)  = 0
mmap2(0x76d52000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x127000) = 0x76d52000
mmap2(0x76d55000, 9552, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x76d55000
close(3)                                = 0
mmap2(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x76f8a000
set_tls(0x76f8a4c0, 0x76f8abb8, 0x76f8f050, 0x76f8a4c0, 0x76f8f050) = 0
mprotect(0x76d52000, 8192, PROT_READ)   = 0
mprotect(0x76d7c000, 4096, PROT_READ)   = 0
mprotect(0x76e16000, 4096, PROT_READ)   = 0
mprotect(0x76f56000, 20480, PROT_READ)  = 0
mprotect(0x5f4000, 8192, PROT_READ)     = 0
mprotect(0x76f8e000, 4096, PROT_READ)   = 0
set_tid_address(0x76f8a068)             = 5562
set_robust_list(0x76f8a070, 12)         = 0
rt_sigaction(SIGRTMIN, {0x76d5c0b4, [], SA_SIGINFO|0x4000000}, NULL, 8) = 0
rt_sigaction(SIGRT_1, {0x76d5c198, [], SA_RESTART|SA_SIGINFO|0x4000000}, NULL, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0
getrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM_INFINITY}) = 0
brk(0)                                  = 0x18de000
brk(0x1903000)                          = 0x1903000
futex(0x76f5d2a8, FUTEX_WAKE_PRIVATE, 2147483647) = 0
futex(0x76f5d2ac, FUTEX_WAKE_PRIVATE, 2147483647) = 0
rt_sigaction(SIGHUP, {0x5a7f00, [HUP], SA_RESTART|0x4000000}, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGPIPE, {SIG_IGN, [PIPE], SA_RESTART|0x4000000}, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGIO, {SIG_IGN, [IO], SA_RESTART|0x4000000}, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGUSR1, {SIG_IGN, [USR1], SA_RESTART|0x4000000}, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGUSR2, {SIG_IGN, [USR2], SA_RESTART|0x4000000}, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGALRM, {0x5a7f38, [ALRM], SA_RESTART|0x4000000}, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGINT, {0x5a7f04, [INT], SA_RESTART|0x4000000}, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGTERM, {0x5a7f04, [TERM], SA_RESTART|0x4000000}, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGQUIT, {0x5a7f04, [QUIT], SA_RESTART|0x4000000}, {SIG_IGN, [], 0}, 8) = 0
rt_sigaction(SIGINT, {0x5a7f04, [INT], SA_RESTART|0x4000000}, {0x5a7f04, [INT], SA_RESTART|0x4000000}, 8) = 0
close(3)                                = -1 EBADF (Bad file descriptor)
close(4)                                = -1 EBADF (Bad file descriptor)
close(5)                                = -1 EBADF (Bad file descriptor)
close(6)                                = -1 EBADF (Bad file descriptor)
close(7)                                = 0
close(8)                                = -1 EBADF (Bad file descriptor)
...
close(1022)                             = -1 EBADF (Bad file descriptor)
close(1023)                             = -1 EBADF (Bad file descriptor)
open("/var/lib/zerotier-one/zerotier-one.port", O_RDONLY) = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=4, ...}) = 0
read(3, "9993", 16384)                  = 4
read(3, "", 12288)                      = 0
read(3, "", 16384)                      = 0
close(3)                                = 0
open("/var/lib/zerotier-one/authtoken.secret", O_RDONLY) = 3
fstat64(3, {st_mode=S_IFREG|0600, st_size=24, ...}) = 0
read(3, "2i6ampdp0y7ayssji2q53e7l", 16384) = 24
read(3, "", 12288)                      = 0
read(3, "", 16384)                      = 0
close(3)                                = 0
gettimeofday({1632841818, 962982}, NULL) = 0
pipe([3, 4])                            = 0
socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 5
setsockopt(5, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
setsockopt(5, SOL_TCP, TCP_NODELAY, [1], 4) = 0
fcntl64(5, F_SETFL, O_RDONLY|O_NONBLOCK) = 0
connect(5, {sa_family=AF_INET, sin_port=htons(9993), sin_addr=inet_addr("127.0.0.1")}, 16) = -1 EINPROGRESS (Operation now in progress)
select(6, [], [5], [], {30, 0})         = 1 (out [5], left {29, 999991})
getpeername(5, {sa_family=AF_INET, sin_port=htons(9993), sin_addr=inet_addr("127.0.0.1")}, [16]) = 0
gettimeofday({1632841818, 965732}, NULL) = 0
select(6, [5], [5], [], {30, 0})        = 1 (out [5], left {29, 999993})
send(5, "GET /status HTTP/1.1\r\nX-ZT1-Auth"..., 62, 0) = 62
gettimeofday({1632841818, 967499}, NULL) = 0
select(6, [5], [], [], {30, 0})         = 1 (in [5], left {29, 999992})
recv(5, "HTTP/1.1 200 OK\r\nCache-Control: "..., 131072, 0) = 781
gettimeofday({1632841818, 968359}, NULL) = 0
close(5)                                = 0
gettimeofday({1632841818, 968979}, NULL) = 0
close(3)                                = 0
close(4)                                = 0
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x84} ---
+++ killed by SIGSEGV +++
Segmentation fault

mcondarelli avatar Sep 28 '21 15:09 mcondarelli

Is there any way you could build with ZT_DEBUG=1 and run it through a debugger like lldb to get a stack trace? Unfortunately nothing in the strace log stands out as out of the ordinary.

joseph-henry avatar Oct 04 '21 17:10 joseph-henry

Curious mauro, are you building against musl here? I don't know how well that works, thought I'd throw it out there as a potential avenue for investigation later.

erikh avatar Oct 19 '21 09:10 erikh

also, reset your authtoken as it's in the paste above. :)

erikh avatar Oct 19 '21 09:10 erikh

Sorry for the late comeback. I built with ZT_DEBUG, but there's no debugger available on my target.

This was built with an OpenWRT Toolchain advertised as: "OpenWrt-SDK-v2.2.0-imx6_gcc-5.5.0_glibc-2.24_eabi.Linux-x86_64" so it should be "glibc", not "musl".

mcondarelli avatar Oct 19 '21 12:10 mcondarelli

alright. if you could ldd $(which zerotier-one) it may assist folks who are watching this thread for later.

Thanks btw.

erikh avatar Oct 22 '21 07:10 erikh