haproxy
haproxy copied to clipboard
Published
20 hours ago •
haproxy
haproxy
haproxy crashes when built using clang and thread sanitizer
Tool Name and Version
clang and thread sanitizer
Code Report
[Thread 0x7ffff52fe700 (LWP 9112) exited]
HAProxy version 2.7-dev5-e3e312-97 2022/09/16 - https://haproxy.org/
Status: development branch - not safe for use in production.
Known bugs: https://github.com/haproxy/haproxy/issues?q=is:issue+is:open
Running on: Linux 5.15.0-1019-azure #24~20.04.1-Ubuntu SMP Tue Aug 23 15:52:52 UTC 2022 x86_64
Build options :
TARGET = linux-glibc
CPU = generic
CC = clang
CFLAGS = -O2 -fsanitize=thread -ggdb -Wall -Wextra -Wundef -Wdeclaration-after-statement -Wfatal-errors -Wtype-limits -Wshift-negative-value -Wnull-dereference -fwrapv -Wno-unknown-warning-option -Wno-address-of-packed-member -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered -Wno-missing-field-initializers -Wno-cast-function-type -Wno-string-plus-int -Wno-atomic-alignment -Werror
OPTIONS = USE_OPENSSL=1
DEBUG = -DDEBUG_STRICT -DDEBUG_MEMORY_POOLS
Feature list : +EPOLL -KQUEUE +NETFILTER -PCRE -PCRE_JIT -PCRE2 -PCRE2_JIT +POLL +THREAD -PTHREAD_EMULATION +BACKTRACE -STATIC_PCRE -STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE +LIBCRYPT +CRYPT_H -ENGINE +GETADDRINFO +OPENSSL -LUA +ACCEPT4 -CLOSEFROM -ZLIB +SLZ +CPU_AFFINITY +TFO +NS +DL +RT -DEVICEATLAS -51DEGREES -WURFL -SYSTEMD -OBSOLETE_LINKER +PRCTL -PROCCTL +THREAD_DUMP -EVPORTS -OT -QUIC -PROMEX -MEMORY_PROFILING
Default settings :
bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with multi-threading support (MAX_TGROUPS=16, MAX_THREADS=256, default=2).
Built with OpenSSL version : OpenSSL 1.1.1f 31 Mar 2020
Running on OpenSSL version : OpenSSL 1.1.1f 31 Mar 2020
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with network namespace support.
Built with libslz for stateless compression.
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built without PCRE or PCRE2 support (using libc's regex instead)
Encrypted password support via crypt(3): yes
Built with clang compiler version 10.0.0
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
h2 : mode=HTTP side=FE|BE mux=H2 flags=HTX|HOL_RISK|NO_UPG
fcgi : mode=HTTP side=BE mux=FCGI flags=HTX|HOL_RISK|NO_UPG
h1 : mode=HTTP side=FE|BE mux=H1 flags=HTX|NO_UPG
<default> : mode=HTTP side=FE|BE mux=H1 flags=HTX
none : mode=TCP side=FE|BE mux=PASS flags=NO_UPG
<default> : mode=TCP side=FE|BE mux=PASS flags=
Available services : none
Available filters :
[BWLIM] bwlim-in
[BWLIM] bwlim-out
[CACHE] cache
[COMP] compression
[FCGI] fcgi-app
[SPOE] spoe
[TRACE] trace
Thread 1 "haproxy" received signal SIGSEGV, Segmentation fault.
0xffffffffffffffff in ?? ()
(gdb) bt full
#0 0xffffffffffffffff in ?? ()
No symbol table info available.
#1 0x0000000000476309 in sigaction ()
No symbol table info available.
#2 0x000000000047615e in signal ()
No symbol table info available.
#3 0x0000000000775991 in deinit_signals () at src/signal.c:158
sig = 93
sh = <optimized out>
shb = <optimized out>
#4 0x00000000006952b5 in deinit () at src/haproxy.c:2624
ua = 0x0
p = 0x0
cur_fd = <optimized out>
p0 = <optimized out>
uap = <optimized out>
pdf = <optimized out>
logb = <optimized out>
log = <optimized out>
wlb = <optimized out>
wl = <optimized out>
bolb = <optimized out>
bol = <optimized out>
pxdfb = <optimized out>
pxdf = <optimized out>
pdfb = <optimized out>
srvdfb = <optimized out>
srvdf = <optimized out>
pcfb = <optimized out>
pcf = <optimized out>
pscfb = <optimized out>
pscf = <optimized out>
ppcfb = <optimized out>
ppcf = <optimized out>
prcfb = <optimized out>
prcf = <optimized out>
tifb = <optimized out>
tif = <optimized out>
tdfb = <optimized out>
--Type <RET> for more, q to quit, c to continue without paging-- c
tdf = <optimized out>
tafb = <optimized out>
taf = <optimized out>
tffb = <optimized out>
tff = <optimized out>
pprsb = <optimized out>
pprs = <optimized out>
#5 0x000000000069616f in deinit_and_exit (status=0) at src/haproxy.c:2786
No locals.
#6 0x000000000069b51a in init_args (argc=<optimized out>, argv=<optimized out>) at src/haproxy.c:1602
flag = 0x7fffffffe749 "vv"
err_msg = <optimized out>
progname = 0x7b0400000060 "haproxy"
flag = <optimized out>
endptr = <optimized out>
c = <optimized out>
ret = <optimized out>
__x = <optimized out>
__x = <optimized out>
#7 main (argc=-6327, argv=0x7fffffffe4b8) at src/haproxy.c:3109
limit = {rlim_cur = 1024, rlim_max = 1048576}
intovf = <optimized out>
pidfd = -1
err = <optimized out>
retry = <optimized out>
(gdb)
### Additional Information
_No response_
### Output of `haproxy -vv`
```plain
HAProxy version 2.7-dev5-e3e312-97 2022/09/16 - https://haproxy.org/
Status: development branch - not safe for use in production.
Known bugs: https://github.com/haproxy/haproxy/issues?q=is:issue+is:open
Running on: Linux 5.15.0-1019-azure #24~20.04.1-Ubuntu SMP Tue Aug 23 15:52:52 UTC 2022 x86_64
Build options :
TARGET = linux-glibc
CPU = generic
CC = clang
CFLAGS = -O2 -fsanitize=thread -ggdb -Wall -Wextra -Wundef -Wdeclaration-after-statement -Wfatal-errors -Wtype-limits -Wshift-negative-value -Wnull-dereference -fwrapv -Wno-unknown-warning-option -Wno-address-of-packed-member -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered -Wno-missing-field-initializers -Wno-cast-function-type -Wno-string-plus-int -Wno-atomic-alignment -Werror
OPTIONS = USE_OPENSSL=1
DEBUG = -DDEBUG_STRICT -DDEBUG_MEMORY_POOLS
Feature list : +EPOLL -KQUEUE +NETFILTER -PCRE -PCRE_JIT -PCRE2 -PCRE2_JIT +POLL +THREAD -PTHREAD_EMULATION +BACKTRACE -STATIC_PCRE -STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE +LIBCRYPT +CRYPT_H -ENGINE +GETADDRINFO +OPENSSL -LUA +ACCEPT4 -CLOSEFROM -ZLIB +SLZ +CPU_AFFINITY +TFO +NS +DL +RT -DEVICEATLAS -51DEGREES -WURFL -SYSTEMD -OBSOLETE_LINKER +PRCTL -PROCCTL +THREAD_DUMP -EVPORTS -OT -QUIC -PROMEX -MEMORY_PROFILING
Default settings :
bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with multi-threading support (MAX_TGROUPS=16, MAX_THREADS=256, default=2).
Built with OpenSSL version : OpenSSL 1.1.1f 31 Mar 2020
Running on OpenSSL version : OpenSSL 1.1.1f 31 Mar 2020
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with network namespace support.
Built with libslz for stateless compression.
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built without PCRE or PCRE2 support (using libc's regex instead)
Encrypted password support via crypt(3): yes
Built with clang compiler version 10.0.0
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
h2 : mode=HTTP side=FE|BE mux=H2 flags=HTX|HOL_RISK|NO_UPG
fcgi : mode=HTTP side=BE mux=FCGI flags=HTX|HOL_RISK|NO_UPG
h1 : mode=HTTP side=FE|BE mux=H1 flags=HTX|NO_UPG
<default> : mode=HTTP side=FE|BE mux=H1 flags=HTX
none : mode=TCP side=FE|BE mux=PASS flags=NO_UPG
<default> : mode=TCP side=FE|BE mux=PASS flags=
Available services : none
Available filters :
[BWLIM] bwlim-in
[BWLIM] bwlim-out
[CACHE] cache
[COMP] compression
[FCGI] fcgi-app
[SPOE] spoe
[TRACE] trace
Thread 1 "haproxy" received signal SIGSEGV, Segmentation fault.
I'm not entirely sure yet but I managed to reproduce a crash and it seems to occur in the thread sanitizer library itself. The crash happens in our deinit_signals function which simply iterates over all signals from 0 to MAX_SIGNAL (256) and calls signal(sig, SIG_DFL); It then gets caught by tsan which ends up crashing :
Thread 1 "haproxy" received signal SIGSEGV, Segmentation fault.
0x00007ffff750a5c5 in __sanitizer::atomic_fetch_add<__sanitizer::atomic_uint64_t> (mo=__sanitizer::memory_order_relaxed, v=1, a=0x100037) at ../../../../src/libsanitizer/sanitizer_common/sanitizer_atomic_clang.h:46
46 ../../../../src/libsanitizer/sanitizer_common/sanitizer_atomic_clang.h: No such file or directory.
(gdb) bt
#0 0x00007ffff750a5c5 in __sanitizer::atomic_fetch_add<__sanitizer::atomic_uint64_t> (
mo=__sanitizer::memory_order_relaxed, v=1, a=0x100037)
at ../../../../src/libsanitizer/sanitizer_common/sanitizer_atomic_clang.h:46
#1 __tsan::MetaMap::GetAndLock (this=0x7, thr=thr@entry=0x7ffff6b0a7c0, pc=pc@entry=140737342330292,
addr=addr@entry=140737343197824, write_lock=write_lock@entry=true, create=create@entry=true)
at ../../../../src/libsanitizer/tsan/tsan_sync.cpp:241
#2 0x00007ffff750a9c3 in __tsan::MetaMap::GetOrCreateAndLock (this=<optimized out>,
thr=thr@entry=0x7ffff6b0a7c0, pc=pc@entry=140737342330292, addr=addr@entry=140737343197824,
write_lock=write_lock@entry=true) at ../../../../src/libsanitizer/tsan/tsan_sync.cpp:198
#3 0x00007ffff7501a1b in __tsan::ReleaseStore (thr=thr@entry=0x7ffff6b0a7c0,
pc=pc@entry=140737342330292, addr=addr@entry=140737343197824)
at ../../../../src/libsanitizer/tsan/tsan_rtl_mutex.cpp:462
#4 0x00007ffff74bd6a7 in sigaction_impl (sig=75, act=0x7fffffffc7f0, old=0x7fffffffc890)
at ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:2402
#5 0x00007ffff74bd817 in signal_impl (sig=75, h=<optimized out>)
at ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:2422
#6 0x0000555555b6bd0e in deinit_signals () at src/signal.c:158
#7 0x000055555599fa76 in deinit () at src/haproxy.c:2629
#8 0x00005555559a1c56 in deinit_and_exit (status=0) at src/haproxy.c:2791
#9 0x00005555559a5e27 in init_args (argc=1, argv=0x7fffffffe060) at src/haproxy.c:1602
#10 0x00005555559a31cf in main (argc=2, argv=0x7fffffffe058) at src/haproxy.c:3114
I'm closing, the 2.7 is EOL now. Feel free to reopen it if it happens on more recent versions. Thanks !
