[FR] UBSAN print_stacktrace=1

[arturbac]

trafficstars

ubsan has a feature originaly to print stack trace of UB. however on android port this seems to not work print_stacktrace=1 to ASAN and UBSAN but this doesn't seem to produce a stack trace like on linux in logcat

I added to wrap.sh

export UBSAN_OPTIONS=print_stacktrace=1

This sometimes is required to fix a bug for example when UB is detected in libc++ but source of ub happens much earlier in code calls. Example of useles information without stack trace.

SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /opt/android-ndk-r20/toolchains/llvm/prebuilt/linux-x86_64/sysroot/usr/include/c++/v1/algorithm:1636:21

NDK tested r20 beta 1 platform i686

[enh]

@eugenis

[eugenis]

Could you reproduce it with a standalone binary? Works fine for me with ndk r19.

[DanAlbert]

Is the problem maybe that the trace only goes to stderr and not logcat?

[arturbac]

exactly @DanAlbert logcat contains only SUMARY and last line of stack where the error was finaly went into action. Does stderr can be intercepted for ordinary java with native native android app ? how ?

[DanAlbert]

What device? Probably more interesting, what OS version? I could have sworn we switched the common sanitizer logging over to syslog ages ago (L or M). syslog as of about that time should print to logcat.

[arturbac]

I tried asan/ubsan on xiaomi redmi 4x + lineageos 15.1 (8.1 android) - no luck , it fails with null ptr in loadlibrary

it works for me only on emulator i686 with android 9.0 goggle apis and i686 build

i tried also emualtor x86_64 with 64 build but it fails to run with Shadow memory range interleaves with an existing memory mapping. ASan cannot proceed correctly. ABORTING. As i read it is known error and i cannot do anything , just have to wait for new kernel ..

[arturbac]

My wrap.sh (i build with cmake/gradle) app/src/main/resources/x86/wrap.sh

HERE="$(cd "$(dirname "$0")" && pwd)"
export ASAN_OPTIONS=log_to_syslog=false,allow_user_segv_handler=1,detect_stack_use_after_return=1,print_stacktrace=1
export UBSAN_OPTIONS=print_stacktrace=1
export LD_PRELOAD="$HERE/libclang_rt.asan-i686-android.so"
"$@"

[arturbac]

@DanAlbert You mentioned "syslog as of about that time should print to logcat." But example wrap.sh in ndk contains line log_to_syslog=false, is it correct ?

cat /opt/android-ndk-r20/wrap.sh/asan.x86.sh

HERE="$(cd "$(dirname "$0")" && pwd)"
export ASAN_OPTIONS=log_to_syslog=false,allow_user_segv_handler=1
export LD_PRELOAD=$HERE/libclang_rt.asan-i686-android.so
"$@"

[arturbac]

Could you reproduce it with a standalone binary? Works fine for me with ndk r19.

I use ASAN & UBSAN only for java android app with native library

[enh]

the log_to_syslog=false is in the canonical docs too: https://github.com/google/sanitizers/wiki/AddressSanitizerOnAndroid

@eugenis --- do you remember the reason for this? should we just delete this from both places, since it seems like it's not useful for the typical [app] developer. (i guess that option only really makes sense if you're running a gtest directly?)

[eugenis]

log_to_syslog=false is because wrap.sh gets a logwrapper around it. Without that every line will be written twice.

On Wed, Apr 10, 2019 at 11:47 AM Elliott Hughes [email protected] wrote:

the log_to_syslog=false is in the canonical docs too: https://github.com/google/sanitizers/wiki/AddressSanitizerOnAndroid

@eugenis https://github.com/eugenis --- do you remember the reason for this? should we just delete this from both places, since it seems like it's not useful for the typical [app] developer. (i guess that option only really makes sense if you're running a gtest directly?)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/android-ndk/ndk/issues/954#issuecomment-481815567, or mute the thread https://github.com/notifications/unsubscribe-auth/AAZuSgs3DtCLlV__Db3KbonQDXE6LJOrks5vfjHWgaJpZM4chCNq .

[arturbac]

Maybe the problem with logcat stack trace is that it is to long to push it at once, and get croped to single line. When i put custom stacktrace with boost::stacktrace to logcat I need to iterate line by line when pushing to logcat.

sout << " " << boost::stacktrace::stacktrace();
for (std::string line; std::getline(input, line); )
  LOG("%s", line);

adb logcat -g main: ring buffer is 4Mb (3Mb consumed), max entry is 5120b, max payload is 4068b system: ring buffer is 4Mb (0b consumed), max entry is 5120b, max payload is 4068b crash: ring buffer is 4Mb (15Kb consumed), max entry is 5120b, max payload is 4068b

[DanAlbert]

I didn't realize that logwrapper was used by default with wrap.sh, but the code agrees with you.

I can't speak to why this doesn't work on LineageOS, but given that it does work on Android I think this is something you'll need to take up with LineageOS.

[arturbac]

No w8. UBSAN doesn't produce full satcktrace on emulator i686, it produces only 1 stack trace line as i show. ASAN doesn't work at all on emulator x86_64, but stick to i686, i just need at least one device like emulator 686 to produce USBAN errors with stack traces.

[arturbac]

I think i know why it doesn't work. The method described placing wrap.sh with gradle/AS in resources/lib.ARCH_name at https://developer.android.com/ndk/guides/wrap-script simply doesn't work at all on device and emulator, the wrap.sh placed in apk is placed in folder name as folder name in resources, not in lib/ARCH-name/ but side to it in lib.ARCH_name. When I did asan_device_setup everything works as expected on x86 and x86_64 emulators. But I had to fix outdated asan_device_script

• add missing x86_64
• rename arch i386 to i686
• extend internal script creation and add UBSAN_OPTIONS

Concluding whats the proper way of placing wrap.sh with gradle/android studio ?

[enh]

what version of Studio are you using?

[enh]

the Studio folks tell me i got the docs wrong... apparently although it will show up as "libs.x86" under resources, it should actually be resources/libs/x86 on disk.

here's my suggested corrected text for the docs:

Android Studio only packages `.so` files from the `lib/` directories, so if
you're an Android Studio user, you'll need to place your `wrap.sh` files in the
`src/main/resources/lib/*` directories instead, so that they'll be packaged
correctly. Note that `resources/lib/x86` will be displayed in the UI as
`lib.x86`, but it should actually be a subdirectory:
[[existing image]]

can you confirm that if you create libs/x86 rather than libs.x86 (etc) that it works?

[arturbac]

Thats was the point on filesystem this must be located app/src/main/resources/lib/ARCH/wrap.sh then wrap.sh in apk is packaged with output native libs and I can confirm that i see UBSAN stacktraces bit different from asan_prepare_device but it looks liek it works ok.

the differences in logcat asan prepared device:

• taged with application tag x.y.z
• no escape color codes

wrap.sh

• taged with wrap.sh tag instead of app tag
• color escape codes

Do You maybe know how to disable escape color codes ? I dont see any option here https://github.com/google/sanitizers/wiki/AddressSanitizerFlags

04-15 21:47:24.798  4463  4463 I wrap.sh :   0xe6d20690: [1m[31mfa[1m[0m [1m[31mfa[1m[0m [1m[35mfd[1m[0m [1m[31mfa[1m[0m [1m[31mfa[1m[0m [1m[31mfa[1m[0m [1m[35mfd[1m[0m [1m[31mfa[1m[0m [1m[31mfa[1m[0m [1m[31mfa[1m[0m [1m[35mfd[1m[0m [1m[31mfa[1m[0m [1m[31mfa[1m[0m [1m[31mfa[1m[0m [1m[35mfd[1m[0m [1m[31mfa[1m[0m
04-15 21:47:24.798  4463  4463 I wrap.sh :   0xe6d206a0: [1m[31mfa[1m[0m [1m[31mfa[1m[0m [1m[35mfd[1m[0m [1m[31mfa[1m[0m [1m[31mfa[1m[0m [1m[31mfa[1m[0m [1m[35mfd[1m[0m [1m[35mfd[1m[0m [1m[31mfa[1m[0m [1m[31mfa[1m[0m [1m[35mfd[1m[0m [1m[31mfa[1m[0m [1m[31mfa[1m[0m [1m[31mfa[1m[0m [1m[35mfd[1m[0m [1m[31mfa[1m[0m
04-15 21:47:24.798  4463  4463 I wrap.sh :   0xe6d206b0: [1m[31mfa[1m[0m [1m[31mfa[1m[0m [1m[35mfd[1m[0m [1m[31mfa[1m[0m [1m[31mfa[1m[0m [1m[31mfa[1m[0m [1m[35mfd[1m[0m [1m[31mfa[1m[0m [1m[31mfa[1m[0m [1m[31mfa[1m[0m [1m[35mfd[1m[0m [1m[31mfa[1m[0m [1m[31mfa[1m[0m [1m[31mfa[1m[0m [1m[35mfd[1m[0m [1m[35mfd[1m[0m
04-15 21:47:24.798  4463  4463 I wrap.sh :   0xe6d206c0: [1m[31mfa[1m[0m [1m[31mfa[1m[0m [1m[35mfd[1m[0m [1m[35mfd[1m[0m [1m[31mfa[1m[0m [1m[31mfa[1m[0m [1m[35mfd[1m[0m [1m[31mfa[1m[0m [1m[31mfa[1m[0m [1m[31mfa[1m[0m [1m[35mfd[1m[0m [1m[31mfa[1m[0m [1m[31mfa[1m[0m [1m[31mfa[1m[0m [1m[35mfd[1m[0m [1m[35mfd[1m[0m
04-15 21:47:24.798  4463  4463 I wrap.sh :   0xe6d206d0: [1m[31mfa[1m[0m [1m[31mfa[1m[0m [1m[35mfd[1m[0m [1m[35mfd[1m[0m [1m[31mfa[1m[0m [1m[31mfa[1m[0m [1m[35mfd[1m[0m [1m[35mfd[1m[0m [1m[31mfa[1m[0m [1m[31mfa[1m[0m [1m[0m04[1m[0m [1m[31mfa[1m[0m [1m[31mfa[1m[0m [1m[31mfa[1m[0m [1m[35mfd[1m[0m [1m[35mfd[1m[0m
04-15 21:47:24.798  4463  4463 I wrap.sh : =>0xe6d206e0: [1m[31mfa[1m[0m [1m[31mfa[1m[0m [1m[0m04[1m[0m [1m[31mfa[1m[0m [1m[31mfa[1m[0m [1m[31mfa[1m[0m[[1m[0m03[1m[0m][1m[31mfa[1m[0m [1m[31mfa[1m[0m [1m[31mfa[1m[0m [1m[35mfd[1m[0m [1m[35mfd[1m[0m [1m[31mfa[1m[0m [1m[31mfa[1m[0m [1m[35mfd[1m[0m [1m[35mfd[1m[0m
04-15 21:47:24.798  4463  4463 I wrap.sh :   0xe6d206f0: [1m[31mfa[1m[0m [1m[31mfa[1m[0m [1m[35mfd[1m[0m [1m[35mfd[1m[0m [1m[31mfa[1m[0m [1m[31mfa[1m[0m [1m[35mfd[1m[0m [1m[35mfd[1m[0m [1m[31mfa[1m[0m [1m[31mfa[1m[0m [1m[31mfa[1m[0m [1m[31mfa[1m[0m [1m[31mfa[1m[0m [1m[31mfa[1m[0m [1m[35mfd[1m[0m [1m[35mfd[1m[0m
04-15 21:47:24.798  4463  4463 I wrap.sh :   0xe6d20700: [1m[31mfa[1m[0m [1m[31mfa[1m[0m [1m[0m00[1m[0m [1m[0m00[1m[0m [1m[31mfa[1m[0m [1m[31mfa[1m[0m [1m[35mfd[1m[0m [1m[35mfd[1m[0m [1m[31mfa[1m[0m [1m[31mfa[1m[0m [1m[35mfd[1m[0m [1m[31mfa[1m[0m [1m[31mfa[1m[0m [1m[31mfa[1m[0m [1m[35mfd[1m[0m [1m[35mfd[1m[0m
04-15 21:47:24.798  4463  4463 I wrap.sh :   0xe6d20710: [1m[31mfa[1m[0m [1m[31mfa[1m[0m [1m[31mfa[1m[0m [1m[31mfa[1m[0m [1m[31mfa[1m[0m [1m[31mfa[1m[0m [1m[35mfd[1m[0m [1m[35mfd[1m[0m [1m[31mfa[1m[0m [1m[31mfa[1m[0m [1m[35mfd[1m[0m [1m[31mfa[1m[0m [1m[31mfa[1m[0m [1m[31mfa[1m[0m [1m[31mfa[1m[0m [1m[31mfa[1m[0m
04-15 21:47:24.798  4463  4463 I wrap.sh :   0xe6d20720: [1m[31mfa[1m[0m [1m[31mfa[1m[0m [1m[31mfa[1m[0m [1m[31mfa[1m[0m [1m[31mfa[1m[0m [1m[31mfa[1m[0m [1m[35mfd[1m[0m [1m[35mfd[1m[0m [1m[31mfa[1m[0m [1m[31mfa[1m[0m [1m[35mfd[1m[0m [1m[35mfd[1m[0m [1m[31mfa[1m[0m [1m[31mfa[1m[0m [1m[35mfd[1m[0m [1m[35mfd[1m[0m
04-15 21:47:24.798  4463  4463 I wrap.sh :   0xe6d20730: [1m[31mfa[1m[0m [1m[31mfa[1m[0m [1m[35mfd[1m[0m [1m[31mfa[1m[0m [1m[31mfa[1m[0m [1m[31mfa[1m[0m [1m[35mfd[1m[0m [1m[35mfd[1m[0m [1m[31mfa[1m[0m [1m[31mfa[1m[0m [1m[35mfd[1m[0m [1m[35mfd[1m[0m [1m[31mfa[1m[0m [1m[31mfa[1m[0m [1m[35mfd[1m[0m [1m[35mfd[1m[0m
04-15 21:47:24.798  4463  4463 I wrap.sh : Shadow byte legend (one shadow byte represents 8 application bytes):

[enh]

does

LD_PRELOAD="$HERE/libclang_rt.asan-i686-android.so" exec "$@"

instead of

LD_PRELOAD="$HERE/libclang_rt.asan-i686-android.so" "$@"

fix the tag?

@eugenis should be able to answer the question about the escape sequences...

[eugenis]

It's in https://github.com/google/sanitizers/wiki/SanitizerCommonFlags, color = (always|never|auto). "auto" means only if isatty(stderr)==true. I don't understand why wrap.sh's stderr would be a tty device.

On Mon, Apr 15, 2019 at 1:08 PM Elliott Hughes [email protected] wrote:

does

LD_PRELOAD="$HERE/libclang_rt.asan-i686-android.so" exec "$@"

instead of

LD_PRELOAD="$HERE/libclang_rt.asan-i686-android.so" "$@"

fix the tag?

@eugenis https://github.com/eugenis should be able to answer the question about the escape sequences...

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/android-ndk/ndk/issues/954#issuecomment-483399322, or mute the thread https://github.com/notifications/unsubscribe-auth/AAZuSh53QNIGZIIqPkFark2QSZHVwl6-ks5vhNxMgaJpZM4chCNq .

[arturbac]

the exec doesn't change tag from wrap.sh however its not a problem for me.

To be more precise for futre readers the wrap.sh must look like below with exec in new line

LD_PRELOAD="$HERE/libclang_rt.asan-i686-android.so"
exec "$@"

Thank You very much @enh @eugenis @DanAlbert for help.

My final wrap.sh for future readers

#!/system/bin/sh
HERE="$(cd "$(dirname "$0")" && pwd)"
export ASAN_OPTIONS=log_to_syslog=false,allow_user_segv_handler=1,detect_stack_use_after_return=1,check_initialization_order=true,quarantine_size_mb=64,color=never
export UBSAN_OPTIONS=print_stacktrace=1,log_to_syslog=false,color=never
export LD_PRELOAD="$HERE/libclang_rt.asan-i686-android.so"
exec "$@"

[arturbac]

From my point of view we can close this FR Conclusion to source problem For gradle/cmake projects the wrap.sh must must be located on disk at app/src/main/resources/lib/ABI/wrap.sh where ABI means one of x86,x86_64,arm64-v8a depending for which ABi is script (LD_PRELOAD library)

[DanAlbert]

I'll leave it open so we remember to fix the docs

[arturbac]

and toolchains/llvm/prebuilt/linux-x86_64/lib64/clang/8.0.7/bin/asan_device_setup

• fixi386 ->i686
• add missing x86_64

[DanAlbert]

I'm not sure there's much value in keeping asan_device_setup in a world where we have wrap.sh.

[arturbac]

From a an app developer who prepares tools for team. asan_device_setup has and advantage that sanitized run in emualtor is easier to prepare, doesn't need any changes to gradle config and build. only simple cmake parameter of configuration for -fsanitize flags.

wrap.sh scripts are problematic as they cannot be a static resources. depending on build type i must dynamicaly change source tree and modify resources during build. I hope maybe someday gradle/jave will alllow out of source building. :-) with cmake + asan_device_setup , there is no source tree change and asan_device_setup needs oly once to be run for emulator.

wrap.sh wouldn't be so problematic if wrap.sh could be palced inside direcotory of native libs , but with gradle it cannot, gradle ignores everything except.so

I'm afraid (paranoic) with wrap.sh and dynamic soruce tree modification from gradle cofnig that with some error in gradle config of app public release build will contain wrap.sh when no one will remember in team about wrap.sh and do some change .. Of course there is layer of testers but I like to avoid errors by design and prefer dyamic code in build dir instead of soruce tree modification.

[DanAlbert]

Fair enough. I suppose at least until gradle makes this less of a PITA there's value in keeping it around.

@eugenis: do you want to make the asan_device_setup fixes, or should I send you a patch?

[eugenis]

Do you mind sending me a patch?

On Wed, Apr 17, 2019 at 12:42 PM Dan Albert [email protected] wrote:

Fair enough. I suppose at least until gradle makes this less of a PITA there's value in keeping it around.

@eugenis https://github.com/eugenis: do you want to make the asan_device_setup fixes, or should I send you a patch?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/android-ndk/ndk/issues/954#issuecomment-484233178, or mute the thread https://github.com/notifications/unsubscribe-auth/AAZuShtK6oxRnJ9X3JBp2RYuX1JpbrTPks5vh3ktgaJpZM4chCNq .

[DanAlbert]

The asan_device_setup patch that I never got around to sending should be less helpful now that hwasan is the thing that people should be using, right? Should I close this?

[enh-google]

The asan_device_setup patch that I never got around to sending should be less helpful now that hwasan is the thing that people should be using, right? Should I close this?

seems like the original submitter was using x86 and x86-64? no hwasan for them :-(

(but this is a pretty niche script that most people can't run, and we haven't heard anyone else notice it was broken, so probably not worth much effort.)