ndk
ndk copied to clipboard
[FR] Missing TSAN runtime support?
Description
Add -fsanitize=thread to compile and link flags, and get error: undefined reference to '__tsan_func_entry' (and other tsan functions) link errors.
Environment Details
- NDK Version: 20.0.5594570
- Build system: CMake
- Host OS: Ubuntu 18.04
- ABI: arm64-v8a
- NDK API level: 21
- Device API level: 28
i know @yabinc had this working (for the platform) at one point, but it may have bitrotted. @eugenis might also know something useful...
That's basically all I know. Upstream tests are disabled (and AFAIK have never been enabled), so it is likely broken at this point.
TSan was never released in the NDK, so this is more of a feature request. @yabinc did have it working in the platform, but there are other components that would be needed in the NDK to actually let app developers enable it.
TSAN uses a TLS slot to store thread specific state. Bionic added that support in Android N as slot 8, but changed to slot 6 in Android Q. The tsan library still uses slot 8. So currently the platform tsan is broken. I think we need to add dynamic detection of android version in tsan library.
To use tsan on android, we need a tsan shared library linked and available on device, as https://android-review.googlesource.com/c/platform/prebuilts/clang/host/linux-x86/+/516421/1/Android.bp#253. static linking libtsan might also be possible, I didn't remember why we chose to use shared library.
llvm-symbolizer is also needed on device to report data race locations.
asan and ubsan have shared libraries in ndk, which can be bundled with apps. tsan for some reason has only static library.
aarch64-linux-android-clang++ -fsanitize=thread -O0 -fno-omit-frame-pointer -g -pie -fPIE -L/home/zhl/mycode/tmp_aosp_for_tsan/aosp/prebuilts/clang/host/./linux-x86/clang-r349610b/lib64/clang/8.0.9/lib/linux/ -lclang_rt.tsan-aarch64-android main.cc -o android_test_tsan
tsan_interceptors.cc:1747: undefined reference to `fileno_unlocked'
extern "C" int fileno_unlocked(void *stream); at tsan_interceptors.cc
but fileno_unlocked defined in libc: int fileno_unlocked(FILE* __fp)
issue same libclang_rt.tsan-aarch64-android.a link a different version of libc with NDK default libc.so?
int fileno_unlocked(FILE* __fp) int fileno_unlocked(void *stream)
different args ?
any idea about this? rebuild NDK?
Rebuilding will just rebuild the bug. TSan needs to be fixed to work for Android. This isn't currently supported.
@DanAlbert @equeim find a tmp solution to run tsan:
1: build tsan.so: aosp: external/compiler-rt/lib/tsan @c3c4d5dcbe329224ae09d0765f185a239a4d73da diff info: idiff --git a/lib/tsan/Android.bp b/lib/tsan/Android.bp index 13eb7da..f897dc4 100644 --- a/lib/tsan/Android.bp +++ b/lib/tsan/Android.bp @@ -141,3 +141,29 @@ cc_test_host { }, }, } + +cc_library_shared { + name: "libclang_rt.tsan", + + include_dirs: ["external/compiler-rt/lib"], + cflags: tsan_rtl_cflags, + cppflags: tsan_rtl_cppflags, + srcs: [ + "rtl/*.cc", + ], + stl: "none", + sanitize: { + never: true, + }, + compile_multilib: "64", + whole_static_libs: [ + "libinterception", + "libsan", + "libubsan", + ], + target: { + darwin: { + enabled: false, + }, + },
2: put step 1 build so: libclang_rt.tsan.so to android-ndk-r20/toolchains/llvm/prebuilt/linux-x86_64/lib64/clang/8.0.7/lib/linux/aarch64/libclang_rt.tsan.so
3: build you test file: android-ndk-r20/toolchains/llvm/prebuilt/linux-x86_64/bin/aarch64-linux-android29-clang++ -fsanitize=thread -O0 -fno-omit-frame-pointer -g -pie -fPIE -lclang_rt.tsan main.cc -o android_test_tsan
TMP test ok but may test more case:
xiaomi_9:~/zhl$ ./android_test_tsan
WARNING: ThreadSanitizer: data race (pid=23331)
Write of size 4 at 0x00555556b018 by thread T2:
#0
Previous write of size 4 at 0x00555556b018 by thread T1:
#0
Location is global '
Thread T2 (tid=23334, running) created by main thread at:
#0
Thread T1 (tid=23333, finished) created by main thread at:
#0
SUMMARY: ThreadSanitizer: data race (/data/data/com.termux/files/home/zhl/android_test_tsan+0xd63)
@DanAlbert @equeim find a tmp solution to run tsan:
1: build tsan.so: aosp: external/compiler-rt/lib/tsan @c3c4d5dcbe329224ae09d0765f185a239a4d73da diff info: idiff --git a/lib/tsan/Android.bp b/lib/tsan/Android.bp index 13eb7da..f897dc4 100644 --- a/lib/tsan/Android.bp +++ b/lib/tsan/Android.bp @@ -141,3 +141,29 @@ cc_test_host { }, }, } + +cc_library_shared {
- name: "libclang_rt.tsan",
- include_dirs: ["external/compiler-rt/lib"],
- cflags: tsan_rtl_cflags,
- cppflags: tsan_rtl_cppflags,
- srcs: [
- "rtl/*.cc",
- ],
- stl: "none",
- sanitize: {
- never: true,
- },
- compile_multilib: "64",
- whole_static_libs: [
- "libinterception",
- "libsan",
- "libubsan",
- ],
- target: {
- darwin: {
- enabled: false,
- },
- },
2: put step 1 build so: libclang_rt.tsan.so to android-ndk-r20/toolchains/llvm/prebuilt/linux-x86_64/lib64/clang/8.0.7/lib/linux/aarch64/libclang_rt.tsan.so
3: build you test file: android-ndk-r20/toolchains/llvm/prebuilt/linux-x86_64/bin/aarch64-linux-android29-clang++ -fsanitize=thread -O0 -fno-omit-frame-pointer -g -pie -fPIE -lclang_rt.tsan main.cc -o android_test_tsan
TMP test ok but may test more case:
xiaomi_9:~/zhl$ ./android_test_tsan
WARNING: ThreadSanitizer: data race (pid=23331) Write of size 4 at 0x00555556b018 by thread T2: #0 (android_test_tsan+0x000000000d63) #1 (libclang_rt.tsan.so+0x0000000487bb)
Previous write of size 4 at 0x00555556b018 by thread T1: #0 (android_test_tsan+0x000000000d03) #1 (libclang_rt.tsan.so+0x0000000487bb)
Location is global '' at 0x000000000000 (android_test_tsan+0x000000016018)
Thread T2 (tid=23334, running) created by main thread at: #0 (libclang_rt.tsan.so+0x00000004884b) #1 (android_test_tsan+0x000000000ddb) #2 (libc.so+0x0000000c888f)
Thread T1 (tid=23333, finished) created by main thread at: #0 (libclang_rt.tsan.so+0x00000004884b) #1 (android_test_tsan+0x000000000db3) #2 (libc.so+0x0000000c888f)
SUMMARY: ThreadSanitizer: data race (/data/data/com.termux/files/home/zhl/android_test_tsan+0xd63)
would you share the detail that how to build tsan.so? thank you very much
when i build a sample demo like @haolongzhangm done and run it ,I got crash, details as follows: Program received signal SIGSEGV, Segmentation fault. 0x0000007fb7ab28a4 in __tsan::in_symbolizer () at D:\work\Basiclib\llvm_build/compiler-rt\lib\tsan\rtl\tsan_interceptors.h:27 27 return UNLIKELY(cur_thread()->in_symbolizer); 0x0000007fb7ab2898 <__interceptor___cxa_atexit(void ()(void), void*, void*)+56>: f7 36 ff 97 bl 0x7fb7a80474 <__tsan::cur_thread()> 0x0000007fb7ab289c <__interceptor___cxa_atexit(void ()(void), void*, void*)+60>: 88 d7 9f 52 mov w8, #0xfebc // #65212 0x0000007fb7ab28a0 <__interceptor___cxa_atexit(void ()(void), void*, void*)+64>: 28 00 a0 72 movk w8, #0x1, lsl #16 => 0x0000007fb7ab28a4 <__interceptor___cxa_atexit(void ()(void), void*, void*)+68>: 08 68 68 38 ldrb w8, [x0, x8] (gdb) bt #0 0x0000007fb7ab28a4 in __tsan::in_symbolizer () at D:\work\Basiclib\llvm_build/compiler-rt\lib\tsan\rtl\tsan_interceptors.h:27 #1 __interceptor___cxa_atexit (f=0x55568a7cc0 <finalize()>, arg=0x0, dso=0x0) at D:\work\Basiclib\llvm_build\compiler-rt\lib\tsan\rtl\tsan_interceptors_posix.cpp:412 #2 0x00000055568a7c34 in InitializeInterceptors () at /out/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:2785 #3 0x00000055568c1688 in Initialize () at /out/llvm-project/compiler-rt/lib/tsan/rtl/tsan_rtl.cpp:384 #4 0x000000555687d7fc in ScopedInterceptor () at /out/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:252 #5 0x00000055568881e8 in __interceptor_strcmp () at /out/llvm-project/compiler-rt/lib/tsan/../sanitizer_common/sanitizer_common_interceptors.inc:440 #6 0x0000007fb79a6388 in __libc_init_vdso(libc_globals*, KernelArgumentBlock&) () from target:/system/lib64/libc.so #7 0x0000007fb79a9bd4 in __libc_init_globals(KernelArgumentBlock&) () from target:/system/lib64/libc.so #8 0x0000007fb79a44b4 in __libc_preinit() () from target:/system/lib64/libc.so #9 0x0000007fb7f57ac8 in __dl__ZN6soinfo10call_arrayEPKcPPFvvEmb () from target:/system/bin/linker64 #10 0x0000007fb7f55c48 in __dl__ZN6soinfo17call_constructorsEv () from target:/system/bin/linker64 #11 0x0000007fb7f55c48 in __dl__ZN6soinfo17call_constructorsEv () from target:/system/bin/linker64 #12 0x0000007fb7f55c48 in __dl__ZN6soinfo17call_constructorsEv () from target:/system/bin/linker64 #13 0x0000007fb7f5b380 in __dl__ZL29__linker_init_post_relocationR19KernelArgumentBlocky () from target:/system/bin/linker64 #14 0x0000007fb7f5a4c8 in __dl___linker_init () from target:/system/bin/linker64 #15 0x0000007fb7f51c6c in __dl__start () from target:/system/bin/linker64 Backtrace stopped: previous frame identical to this frame (corrupt stack?)
can anyone tell me why? thanks
@DanAlbert any idea about my problem? thanks
until we have someone actually look at what's wrong, no, you probably have more idea than we do since you've actually tried it! fixing tsan is in our plan between now and the Android T release though. (i don't know exactly when we'll get round to it, but it will be released in the next NDK after it's fixed --- we won't hold it hostage until T if it's ready sooner!)
You can't really use aosp: external/compiler-rt/lib/tsan to build a tsan runtime, as external/compiler-rt in AOSP is actually the compiler-rt used for RenderScript. It is also several years out of date. You might try using one of the libraries from prebuilts/clang/host/linux-x86/clang-r416183b/runtimes_ndk_cxx instead if you have an AOSP tree. If you're using the NDK only, you can try to find one in toolchains/llvm/prebuilt/linux-x86_64/lib64/clang/*/lib/linux/libclang_rt.tsan-*. As mentioned before, we're trying to work on this in the near future, but if you want to try this part out, you might be able to get ahead.
Will be great to have this in place at some point. Indeed threading issues need some good tools to help troubleshooting.
Rest assured, it's being actively worked on. No timeline because we're still in the porting phase and we won't know if we've found the last problem until after we've already fixed it.
This is actively being worked on but because we can't predict how long porting will take, I'm taking this off the release tracking to make this bug less misleading. Will update here when we can see the light at the end of the tunnel :)
I saw that r25 has TSan runtime libs, so jumped on it to test it. Unfortunately it crashed immediately:
07-17 19:48:16.955 p:23002 t:23002 F DEBUG signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x0
07-17 19:48:16.955 p:23002 t:23002 F DEBUG Cause: null pointer dereference
07-17 19:48:16.955 p:23002 t:23002 F DEBUG x0 000000763532354a x1 00000076415791c8 x2 0000000000000003 x3 000000763535347c
07-17 19:48:16.955 p:23002 t:23002 F DEBUG x4 ffffffffffffffff x5 0000000000000000 x6 6f632f786570612f x7 696f72646e612e6d
07-17 19:48:16.955 p:23002 t:23002 F DEBUG x8 0000000000000000 x9 0000000000000000 x10 0000000000000000 x11 0000007641579128
07-17 19:48:16.955 p:23002 t:23002 F DEBUG x12 000000006474e550 x13 0000000000000000 x14 0000000000000000 x15 0000000000000000
07-17 19:48:16.955 p:23002 t:23002 F DEBUG x16 00000076353c7060 x17 000000762712e9f0 x18 00000076414c4000 x19 00000076415791c8
07-17 19:48:16.955 p:23002 t:23002 F DEBUG x20 000000763532354a x21 000000763535347c x22 0000007641576000 x23 0000007635327bc7
07-17 19:48:16.955 p:23002 t:23002 F DEBUG x24 0000000000000005 x25 0000007641579000 x26 00000076415791c8 x27 0000000000000000
07-17 19:48:16.955 p:23002 t:23002 F DEBUG x28 0000007641579158 x29 0000007fc18ec460
07-17 19:48:16.955 p:23002 t:23002 F DEBUG lr 000000762712ea6c sp 0000007fc18ec3f0 pc 0000000000000000 pst 0000000080001000
07-17 19:48:16.955 p:23002 t:23002 F DEBUG backtrace:
07-17 19:48:16.955 p:23002 t:23002 F DEBUG #00 pc 0000000000000000 <unknown>
07-17 19:48:16.955 p:23002 t:23002 F DEBUG #01 pc 0000000000072a68 /data/app/~~9wJuvaNwR8JAYPuNLH2VbA==/com.epicgames.TP_FirstPerson-5Fsz6ZLUofv8g-3Q2C6p5A==/lib/arm64/libclang_rt.tsan-aarch64-android.so (strcmp+120) (BuildId: a563b2c927a34c480d7b6ba00060d50f3272c83a)
07-17 19:48:16.955 p:23002 t:23002 F DEBUG #02 pc 000000000004b478 /apex/com.android.runtime/lib64/bionic/libc.so (__libc_init_vdso(libc_globals*)+424) (BuildId: 53a228529316d67f22e241dd17ea9b9e)
07-17 19:48:16.955 p:23002 t:23002 F DEBUG #03 pc 0000000000059330 /apex/com.android.runtime/lib64/bionic/libc.so (__libc_init_globals()+100) (BuildId: 53a228529316d67f22e241dd17ea9b9e)
07-17 19:48:16.955 p:23002 t:23002 F DEBUG #04 pc 00000000000483d4 /apex/com.android.runtime/lib64/bionic/libc.so (__libc_preinit_impl()+40) (BuildId: 53a228529316d67f22e241dd17ea9b9e)
07-17 19:48:16.955 p:23002 t:23002 F DEBUG #05 pc 00000000000b5cd8 /apex/com.android.runtime/bin/linker64 (__dl__ZL13call_functionPKcPFviPPcS2_ES0_+136) (BuildId: b542a635b31e7c4fe0ac851ba377d3e6)
07-17 19:48:16.955 p:23002 t:23002 F DEBUG #06 pc 00000000000b5bf0 /apex/com.android.runtime/bin/linker64 (__dl__ZL10call_arrayIPFviPPcS1_EEvPKcPT_mbS5_+168) (BuildId: b542a635b31e7c4fe0ac851ba377d3e6)
07-17 19:48:16.955 p:23002 t:23002 F DEBUG #07 pc 00000000000a108c /apex/com.android.runtime/bin/linker64 (__dl__ZN6soinfo17call_constructorsEv+368) (BuildId: b542a635b31e7c4fe0ac851ba377d3e6)
07-17 19:48:16.955 p:23002 t:23002 F DEBUG #08 pc 00000000000a0fac /apex/com.android.runtime/bin/linker64 (__dl__ZN6soinfo17call_constructorsEv+144) (BuildId: b542a635b31e7c4fe0ac851ba377d3e6)
07-17 19:48:16.955 p:23002 t:23002 F DEBUG #09 pc 00000000000a0fac /apex/com.android.runtime/bin/linker64 (__dl__ZN6soinfo17call_constructorsEv+144) (BuildId: b542a635b31e7c4fe0ac851ba377d3e6)
07-17 19:48:16.955 p:23002 t:23002 F DEBUG #10 pc 00000000000b8e7c /apex/com.android.runtime/bin/linker64 (__dl__ZL29__linker_init_post_relocationR19KernelArgumentBlockR6soinfo+4176) (BuildId: b542a635b31e7c4fe0ac851ba377d3e6)
07-17 19:48:16.955 p:23002 t:23002 F DEBUG #11 pc 00000000000b7ddc /apex/com.android.runtime/bin/linker64 (__dl___linker_init+500) (BuildId: b542a635b31e7c4fe0ac851ba377d3e6)
07-17 19:48:16.955 p:23002 t:23002 F DEBUG #12 pc 0000000000039074 /apex/com.android.runtime/bin/linker64 (__dl__start+4) (BuildId: b542a635b31e7c4fe0ac851ba377d3e6)
release notes didn't mention TSan support, so I guess it's not fixed yet and we have to wait more?
release notes didn't mention TSan support, so I guess it's not fixed yet and we have to wait more?
yeah, not finished yet but still working on it. (and you're right: it'll be in the release notes when we think it's finished :-) )
I saw that r25 has TSan runtime libs, so jumped on it to test it. Unfortunately it crashed immediately:
07-17 19:48:16.955 p:23002 t:23002 F DEBUG signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x0 07-17 19:48:16.955 p:23002 t:23002 F DEBUG Cause: null pointer dereference 07-17 19:48:16.955 p:23002 t:23002 F DEBUG x0 000000763532354a x1 00000076415791c8 x2 0000000000000003 x3 000000763535347c 07-17 19:48:16.955 p:23002 t:23002 F DEBUG x4 ffffffffffffffff x5 0000000000000000 x6 6f632f786570612f x7 696f72646e612e6d 07-17 19:48:16.955 p:23002 t:23002 F DEBUG x8 0000000000000000 x9 0000000000000000 x10 0000000000000000 x11 0000007641579128 07-17 19:48:16.955 p:23002 t:23002 F DEBUG x12 000000006474e550 x13 0000000000000000 x14 0000000000000000 x15 0000000000000000 07-17 19:48:16.955 p:23002 t:23002 F DEBUG x16 00000076353c7060 x17 000000762712e9f0 x18 00000076414c4000 x19 00000076415791c8 07-17 19:48:16.955 p:23002 t:23002 F DEBUG x20 000000763532354a x21 000000763535347c x22 0000007641576000 x23 0000007635327bc7 07-17 19:48:16.955 p:23002 t:23002 F DEBUG x24 0000000000000005 x25 0000007641579000 x26 00000076415791c8 x27 0000000000000000 07-17 19:48:16.955 p:23002 t:23002 F DEBUG x28 0000007641579158 x29 0000007fc18ec460 07-17 19:48:16.955 p:23002 t:23002 F DEBUG lr 000000762712ea6c sp 0000007fc18ec3f0 pc 0000000000000000 pst 0000000080001000 07-17 19:48:16.955 p:23002 t:23002 F DEBUG backtrace: 07-17 19:48:16.955 p:23002 t:23002 F DEBUG #00 pc 0000000000000000 <unknown> 07-17 19:48:16.955 p:23002 t:23002 F DEBUG #01 pc 0000000000072a68 /data/app/~~9wJuvaNwR8JAYPuNLH2VbA==/com.epicgames.TP_FirstPerson-5Fsz6ZLUofv8g-3Q2C6p5A==/lib/arm64/libclang_rt.tsan-aarch64-android.so (strcmp+120) (BuildId: a563b2c927a34c480d7b6ba00060d50f3272c83a) 07-17 19:48:16.955 p:23002 t:23002 F DEBUG #02 pc 000000000004b478 /apex/com.android.runtime/lib64/bionic/libc.so (__libc_init_vdso(libc_globals*)+424) (BuildId: 53a228529316d67f22e241dd17ea9b9e) 07-17 19:48:16.955 p:23002 t:23002 F DEBUG #03 pc 0000000000059330 /apex/com.android.runtime/lib64/bionic/libc.so (__libc_init_globals()+100) (BuildId: 53a228529316d67f22e241dd17ea9b9e) 07-17 19:48:16.955 p:23002 t:23002 F DEBUG #04 pc 00000000000483d4 /apex/com.android.runtime/lib64/bionic/libc.so (__libc_preinit_impl()+40) (BuildId: 53a228529316d67f22e241dd17ea9b9e) 07-17 19:48:16.955 p:23002 t:23002 F DEBUG #05 pc 00000000000b5cd8 /apex/com.android.runtime/bin/linker64 (__dl__ZL13call_functionPKcPFviPPcS2_ES0_+136) (BuildId: b542a635b31e7c4fe0ac851ba377d3e6) 07-17 19:48:16.955 p:23002 t:23002 F DEBUG #06 pc 00000000000b5bf0 /apex/com.android.runtime/bin/linker64 (__dl__ZL10call_arrayIPFviPPcS1_EEvPKcPT_mbS5_+168) (BuildId: b542a635b31e7c4fe0ac851ba377d3e6) 07-17 19:48:16.955 p:23002 t:23002 F DEBUG #07 pc 00000000000a108c /apex/com.android.runtime/bin/linker64 (__dl__ZN6soinfo17call_constructorsEv+368) (BuildId: b542a635b31e7c4fe0ac851ba377d3e6) 07-17 19:48:16.955 p:23002 t:23002 F DEBUG #08 pc 00000000000a0fac /apex/com.android.runtime/bin/linker64 (__dl__ZN6soinfo17call_constructorsEv+144) (BuildId: b542a635b31e7c4fe0ac851ba377d3e6) 07-17 19:48:16.955 p:23002 t:23002 F DEBUG #09 pc 00000000000a0fac /apex/com.android.runtime/bin/linker64 (__dl__ZN6soinfo17call_constructorsEv+144) (BuildId: b542a635b31e7c4fe0ac851ba377d3e6) 07-17 19:48:16.955 p:23002 t:23002 F DEBUG #10 pc 00000000000b8e7c /apex/com.android.runtime/bin/linker64 (__dl__ZL29__linker_init_post_relocationR19KernelArgumentBlockR6soinfo+4176) (BuildId: b542a635b31e7c4fe0ac851ba377d3e6) 07-17 19:48:16.955 p:23002 t:23002 F DEBUG #11 pc 00000000000b7ddc /apex/com.android.runtime/bin/linker64 (__dl___linker_init+500) (BuildId: b542a635b31e7c4fe0ac851ba377d3e6) 07-17 19:48:16.955 p:23002 t:23002 F DEBUG #12 pc 0000000000039074 /apex/com.android.runtime/bin/linker64 (__dl__start+4) (BuildId: b542a635b31e7c4fe0ac851ba377d3e6)release notes didn't mention TSan support, so I guess it's not fixed yet and we have to wait more?
same result with testing on NDK25b, @enh-google Hi~ is there some latest news about this issue? :)
no, no news. it's still on the to-do list, but relatively low priority because pretty much everyone actually wants this for their apps (a problem we don't know how to solve, because of the ART side of things) rather than for their stand-alone unit test binaries (a problem we probably could solve).
(it was higher priority when we thought we could at least enable unit tests relatively cheaply, but we've hit a phase that requires some pretty significant investigation and potentially a rework of how sanitizer internals work on android that that's no longer the case)
@DanAlbert Thank you for all your hard work. I noticed the presence of libclang_rt.tsan-aarch64-android.so in NDK r26, and naturally assumed that TSan was supported. With this in mind, I proceeded with a multi-threaded project using the NDK, thinking that any data race issues in multi-threading could be easily resolved with TSan later on. Now that the time has come to address these issues, I find myself needing to use TSan. Is there any alternative method available? Also, could you provide any information on when TSan might be supported?
Nothing's changed since our last update.