crashes with segmentation fault on aarch64 Android Termux when USE_MEMORY_PROFILING=1 and USE_THREAD=1

[timwm]

trafficstars

Detailed description of the problem

haproxy recieves a segmentation fault when launched even with no arguments, when built with USE_MEMORY_PROFILING=1 and USE_THREAD=1

Steps to reproduce the behavior

just run haproxy after compiling with USE_MEMORY_PROFILING=1 and USE_THREAD=1

Do you have any idea what may have caused this?

Tried a gdb debug, but seems the segmentation fault is due to a stackoverflow in the calloc function in src/activity.c, not pretty sure-didn't spend much time. Anyway here is my gdb debug session output:

u0_a201:haproxy-2.5-dev0$ gdb ./haproxy
GNU gdb (GDB) 9.2
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "aarch64-linux-android".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.                                                
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./haproxy...
(gdb) run
Starting program: /data/data/com.termux/files/home/.011/test/011/clang/c/haproxy-2.5-dev0/haproxy                                                                         
Program received signal SIGSEGV, Segmentation fault.
__emutls_get_address (obj=0x55557d5838 <__emutls_v.in_memprof>)
    at /usr/local/google/buildbot/src/android/gcc/toolchain/build/../gcc/gcc-4.9/libgcc/emutls.c:172                                                                           
172     /usr/local/google/buildbot/src/android/gcc/toolchain/build/../gcc/gcc-4.9/libgcc/emutls.c: (undocumented errno 2).
(gdb)
(gdb) bt
#0  __emutls_get_address (obj=0x55557d5838<__emutls_v.in_memprof>) at /usr/local/google/buildbot/src/android/gcc/toolchain/build/../gcc/gcc-4.9/libgcc/emutls.c:172
#1  0x000000555572da90 in memprof_calloc_initial_handler (nmemb=35, size=8) at src/activity.c:182
#2  0x000000555572d37c in calloc (nmemb=366506498104, nmemb@entry=35, size=366506426368, size@entry=8) at src/activity.c:275 
#3  0x000000555575dcb0 in __emutls_get_address (obj=0x55557d5838 <__emutls_v.in_memprof>) at /usr/local/google/buildbot/src/android/gcc/toolchain/build/../gcc/gcc-4.9/libgcc/emutls.c:204
#4  0x000000555572da90 in memprof_calloc_initial_handler (nmemb=35, size=8) at src/activity.c:182
#5  0x000000555572d37c in calloc (nmemb=366506498104, nmemb@entry=35, size=366506426368, size@entry=8) at src/activity.c:275
#6  0x000000555575dcb0 in __emutls_get_address (obj=0x55557d5838 <__emutls_v.in_memprof>) at /usr/local/google/buildbot/src/android/gcc/toolchain/build/../gcc/gcc-4.9/libgcc/emutls.c:204                                                                            
#7  0x000000555572da90 in memprof_calloc_initial_handler (nmemb=35, size=8) at src/activity.c:182
#8  0x000000555572d37c in calloc (nmemb=366506498104, nmemb@entry=35, size=366506426368, size@entry=8) at src/activity.c:275
#9  0x000000555575dcb0 in __emutls_get_address (obj=0x55557d5838 <__emutls_v.in_memprof>) at /usr/local/google/buildbot/src/android/gcc/toolchain/build/../gcc/gcc-4.9/libgcc/emutls.c:204
#10 0x000000555572da90 in memprof_calloc_initial_handler (nmemb=35, size=8) at src/activity.c:182
#11 0x000000555572d37c in calloc (nmemb=366506498104, nmemb@entry=35, size=366506426368, size@entry=8) at src/activity.c:275                              
#12 0x000000555575dcb0 in __emutls_get_address (obj=0x55557d5838 <__emutls_v.in_memprof>) at /usr/local/google/buildbot/src/android/gcc/toolchain/build/../gcc/gcc-4.9/libgcc/emutls.c:204
#13 0x000000555572da90 in memprof_calloc_initial_handler (nmemb=35, size=8) at src/activity.c:182
#14 0x000000555572d37c in calloc (nmemb=366506498104, nmemb@entry=35, size=366506426368, size@entry=8) at src/activity.c:275
#15 0x000000555575dcb0 in __emutls_get_address (obj=0x55557d5838 <__emutls_v.in_memprof>) at /usr/local/google/buildbot/src/android/gcc/toolchain/build/../gcc/gcc-4.9/libgcc/emutls.c:204
#16 0x000000555572da90 in memprof_calloc_initial_handler (nmemb=35, size=8) at src/activity.c:182
#17 0x000000555572d37c in calloc (nmemb=366506498104, nmemb@entry=35, size=366506426368, size@entry=8) at src/activity.c:275
#18 0x000000555575dcb0 in __emutls_get_address (obj=0x55557d5838 <__emutls_v.in_memprof>) at /usr/local/google/buildbot/src/android/gcc/toolchain/build/../gcc/gcc-4.9/libgcc/emutls.c:204
#19 0x000000555572da90 in memprof_calloc_initial_handler (nmemb=35, size=8) at src/activity.c:182
#20 0x000000555572d37c in calloc (nmemb=366506498104, nmemb@entry=35, size=366506426368, size@entry=8) at src/activity.c:275
#21 0x000000555575dcb0 in __emutls_get_address (obj=0x55557d5838 <__emutls_v.in_memprof>) at /usr/local/google/buildbot/src/android/gcc/toolchain/build/../gcc/gcc-4.9/libgcc/emutls.c:204
#22 0x000000555572da90 in memprof_calloc_initial_handler (nmemb=35, size=8)
--Type <RET> for more, q to quit, c to continue without paging--q
Quit
(gdb)

Do you have an idea how to solve the issue?

Tried compiling with -pthread but still with no success

If static THREAD_LOCAL int in_memprof = 0; is changed to static int in_memprof = 0; in src/activity.c:82 or if compiled with USE_MEMORY_PROFILING disabled and USE_THREAD=1, haproxy executes ok

What is your configuration?

N/A, since haproxy can't even start

Output of haproxy -vv and uname -a

u0_a201:haproxy-2.5-dev0$ uname -a
Linux localhost 4.9.190+ #1 SMP PREEMPT Sat Nov 14 08:59:58 CST 2020 aarch64 Android

Can't produce haproxy -vv since it can't even run, but for a previous successful build with USE_MEMORY_PROFILING disabled, here is the output:

u0_a201:haproxy-2.5-dev0$ haproxy -vv
HAProxy version 2.5-dev0-1f97306 2021/05/14 - https://haproxy.org/
Status: development branch - not safe for use in production.
Known bugs: https://github.com/haproxy/haproxy/issues?q=is:issue+is:open
Running on: Linux 4.9.190+ #1 SMP PREEMPT Sat Nov 14 08:59:58 CST 2020 aarch64
Build options :
  TARGET  = linux-glibc
  CPU     = generic
  CC      = cc
  CFLAGS  = -O2 -g -Wall -Wextra -Wdeclaration-after-statement -fwrapv -Wno-address-of-p
acked-member -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-missing-fiel
d-initializers -Wno-string-plus-int -Wtype-limits -Wshift-negative-value -Wnull-derefere
nce
  OPTIONS =
  DEBUG   =

Feature list : +EPOLL -KQUEUE +NETFILTER +PCRE +PCRE_JIT -PCRE2 -PCRE2_JIT +POLL +PRIVAT
E_CACHE +THREAD +PTHREAD_PSHARED +BACKTRACE -STATIC_PCRE -STATIC_PCRE2 +TPROXY +LINUX_TPROXY
+LINUX_SPLICE +LIBCRYPT +CRYPT_H +GETADDRINFO +OPENSSL -LUA +FUTEX +ACCEPT4 -CLOSEF
ROM -ZLIB +SLZ +CPU_AFFINITY +TFO +NS +DL +RT -DEVICEATLAS -51DEGREES -WURFL -SYSTEMD +O
BSOLETE_LINKER +PRCTL +THREAD_DUMP -EVPORTS -OT -QUIC +PROMEX -MEMORY_PROFILING

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_THREADS=64, default=8).
Built with clang compiler version 11.0.0 (https://github.com/termux/termux-packages 39de
c01e591687a324c84205de6c9713165c4802)
Encrypted password support via crypt(3): yes
Built with PCRE version : 8.44 2020-02-12
Running on PCRE version : 8.44 2020-02-12
PCRE library supports JIT : yes
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with libslz for stateless compression.
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate
("deflate"), gzip("gzip")
Built with network namespace support.
Built with the Prometheus exporter as a service
Built with OpenSSL version : OpenSSL 1.1.1i  8 Dec 2020
Running on OpenSSL version : OpenSSL 1.1.1i  8 Dec 2020
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
       <default> : mode=TCP        side=FE|BE     mux=PASS     flags=
            none : mode=TCP        side=FE|BE     mux=PASS     flags=NO_UPG
       <default> : mode=HTTP       side=FE|BE     mux=H1       flags=HTX
              h1 : mode=HTTP       side=FE|BE     mux=H1       flags=HTX|NO_UPG
            fcgi : mode=HTTP       side=BE        mux=FCGI     flags=HTX|HOL_RISK|NO_UPG
              h2 : mode=HTTP       side=FE|BE     mux=H2       flags=HTX|CLEAN_ABRT|HOL_RISK|NO_UPG

Available services : prometheus-exporter
Available filters :
        [TRACE] trace
        [COMP] compression
        [FCGI] fcgi-app
        [CACHE] cache
        [SPOE] spoe

Additional information (if helpful)

Building on Android Termux, here is also my lscpu

u0_a201:haproxy-2.5-dev0$ lscpu
Architecture:        aarch64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):             8
On-line CPU(s) list: 0-7
Thread(s) per core:  1
Core(s) per socket:  4
Socket(s):           2
Vendor ID:           ARM
Model:               4
Model name:          Cortex-A53
Stepping:            r0p4
CPU max MHz:         2301.0000
CPU min MHz:         400.0000
BogoMIPS:            26.00
Flags:               fp asimd evtstrm aes pmull sha1 sha2 crc32

[wtarreau]

So what seems to be happening is that your libc's calloc() function is not found by the linker, so the only function that pops up is the current one, hence the infinite recursion. I've met this situation once, I think it was with libmusl, but I'm not 100% sure anymore. I've suspected that maybe their symbol was marked weak and that ours was detected first (but it's just a guess).

Do you know what libc your distro is using ? Also the linker might play a role there. If it's too tricky and you really want the memory profiling, one workaround could be to simply link with an external allocator like jemalloc (which by far shows the best performance on haproxy anyway). In this case the dynamic linker will properly find its calloc library and will not be confused.

[wtarreau]

I'm almost sure it was uClibc and not musl in fact.

[TimWolla]

Do you know what libc your distro is using ?

This is Android, so most likely this is Bionic which is based on OpenBSD: https://en.wikipedia.org/wiki/Bionic_(software)

[wtarreau]

I hadn't thought about that one, but it's then indeed possible that the same issues happen. And maybe dlsym() and friends do not work exactly the same there. @timwm better try directly with jemalloc. I just don't know if it will work on android (I never used it natively, always with a chroot), but if it does, problem solved and we'll learn something.

[wtarreau]

Maybe what we can do for this one is to modify memprof_init() to check whether the next symbol is the same as the current one, and fail to start, saying that the linker isn't compatible with memory profiling. That's always better than crashing.