Caliper Segmentation fault on Xeon Phi Knight landing

[twang15]

I annotate LULESH with caliper macros. LULESH experienced segmentation fault with the following call stack information (More details, please refer to this caliper issue) :

#0 0x00002b06b0cc005f in ?? () #1 0x00002b06b11c8041 in (anonymous namespace)::cali_pthread_create_wrapper (thread=0x7ffff8a1d0d8, attr=0x7ffff8a1d338, fn=0x2b06b0d74eb0 <_INTERNAL_26_______src_z_Linux_util_cpp_313effc4::__kmp_launch_worker(void*)>, arg=0x2b06b490e400) at /work/03915/taoncsu/stampede2/Caliper/src/services/pthread/PthreadService.cpp:83 #2 0x00002b06b0d759c9 in __kmp_create_worker (gtid=-123612968, th=0x7ffff8a1d338, stack_size=47307741232816) at ../../src/z_Linux_util.cpp:878 #3 0x00002b06b0d42579 in __kmp_allocate_thread (root=0x7ffff8a1d0d8, team=0x7ffff8a1d338, new_tid=-1323532624) at ../../src/kmp_runtime.cpp:4521 #4 0x00002b06b0d472a7 in __kmp_allocate_team (root=0x7ffff8a1d0d8, new_nproc=-123612360, max_nproc=-1323532624, new_proc_bind=30692880, new_icvs=0x0, argc=16, master=0x2b06b490f200) at ../../src/kmp_runtime.cpp:5138 #5 0x00002b06b0d45a18 in __kmp_fork_call (loc=0x7ffff8a1d0d8, gtid=-123612360, call_context=(unknown: 2971434672), argc=30692880, microtask=0x0, invoker=0x10, ap=0x7ffff8a1d7b0) at ../../src/kmp_runtime.cpp:2150 #6 0x00002b06b0d1be2a in __kmpc_fork_call (loc=0x7ffff8a1d0d8, argc=-123612360, microtask=0x2b06b11c7eb0 <(anonymous namespace)::thread_wrapper(void*)>) at ../../src/kmp_csupport.cpp:328 #7 0x000000000040434d in main ()

[DavidPoliakoff]

Hey @twang15 , much appreciated. Any chance you could rerun it with GOTCHA_DEBUG set to 3? In the current state, I can't do much to test this (no KNL access). No stress if you can't, I have a request for KNL access in and will get you ready to go ASAP, but if you want quicker results it would be neat to be able to see the debug info.

Best,

David

[twang15]

Hey David,

I have set that environment variable. But I did not see any debug information. Please let me know where I should look for it? Is it dumped to a file or on the screen?

I am actually at B315 now. If you want to debug it here, I can grant you access to KNL.

On Mon, Jul 3, 2017 at 12:53 David Poliakoff [email protected] wrote:

Hey @twang15 https://github.com/twang15 , much appreciated. Any chance you could rerun it with GOTCHA_DEBUG set to 3? In the current state, I can't do much to test this (no KNL access). No stress if you can't, I have a request for KNL access in and will get you ready to go ASAP, but if you want quicker results it would be neat to be able to see the debug info.

Best,

David

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/LLNL/GOTCHA/issues/40#issuecomment-312722672, or mute the thread https://github.com/notifications/unsubscribe-auth/APd91oFB-SPweGkxghKGmxdwYzsyO0Loks5sKUaogaJpZM4OMm3p .

[DavidPoliakoff]

Sure, have an office number? I'll be right over

[twang15]

Sorry for the delay. But, here it is

c455-032.stampede2(17)$ GOTCHA_DEBUG=3 CALI_SERVICES_ENABLE=pthread ./a.out 5 [103544/103544][gotcha_utils.c:58] - Gotcha debug initialized at level 3 [103544/103544][gotcha.c:130] - GOTCHA attempted to mark the GOT table as writable and was unable to do so, calls to wrapped functions will likely fail [103544/103544][gotcha.c:130] - GOTCHA attempted to mark the GOT table as writable and was unable to do so, calls to wrapped functions will likely fail [103544/103544][gotcha.c:130] - GOTCHA attempted to mark the GOT table as writable and was unable to do so, calls to wrapped functions will likely fail [103544/103544][gotcha.c:130] - GOTCHA attempted to mark the GOT table as writable and was unable to do so, calls to wrapped functions will likely fail [103544/103544][gotcha.c:130] - GOTCHA attempted to mark the GOT table as writable and was unable to do so, calls to wrapped functions will likely fail [103544/103544][gotcha.c:130] - GOTCHA attempted to mark the GOT table as writable and was unable to do so, calls to wrapped functions will likely fail [103544/103544][gotcha.c:130] - GOTCHA attempted to mark the GOT table as writable and was unable to do so, calls to wrapped functions will likely fail [103544/103544][gotcha.c:130] - GOTCHA attempted to mark the GOT table as writable and was unable to do so, calls to wrapped functions will likely fail [103544/103544][gotcha.c:130] - GOTCHA attempted to mark the GOT table as writable and was unable to do so, calls to wrapped functions will likely fail [103544/103544][gotcha.c:130] - GOTCHA attempted to mark the GOT table as writable and was unable to do so, calls to wrapped functions will likely fail [103544/103544][gotcha.c:130] - GOTCHA attempted to mark the GOT table as writable and was unable to do so, calls to wrapped functions will likely fail [103544/103544][gotcha.c:130] - GOTCHA attempted to mark the GOT table as writable and was unable to do so, calls to wrapped functions will likely fail [103544/103544][gotcha.c:130] - GOTCHA attempted to mark the GOT table as writable and was unable to do so, calls to wrapped functions will likely fail [103544/103544][gotcha.c:130] - GOTCHA attempted to mark the GOT table as writable and was unable to do so, calls to wrapped functions will likely fail [103544/103544][gotcha.c:130] - GOTCHA attempted to mark the GOT table as writable and was unable to do so, calls to wrapped functions will likely fail [103544/103544][gotcha.c:130] - GOTCHA attempted to mark the GOT table as writable and was unable to do so, calls to wrapped functions will likely fail [103544/103544][gotcha.c:130] - GOTCHA attempted to mark the GOT table as writable and was unable to do so, calls to wrapped functions will likely fail [103544/103544][gotcha.c:130] - GOTCHA attempted to mark the GOT table as writable and was unable to do so, calls to wrapped functions will likely fail [103544/103544][gotcha.c:130] - GOTCHA attempted to mark the GOT table as writable and was unable to do so, calls to wrapped functions will likely fail [103544/103544][gotcha.c:135] - User called gotcha_wrap for tool Caliper with 1 bindings 0: pthread_create will map to 0x2ba9e9fa4850 [103544/103544][tool.c:34] - Created new tool Caliper [103544/103544][tool.c:85] - Created new binding table of size 1 for tool Caliper [103544/103544][gotcha.c:31] - Looking up exported symbols for 1 table entries [103544/103544][gotcha.c:38] - Searching for exported symbols in [EMPTY] [103544/103544][gotcha.c:58] - Checking ELF hash for pthread_create in [EMPTY] [103544/103544][gotcha.c:65] - pthread_create not found in [EMPTY] [103544/103544][gotcha.c:35] - Skipping VDSO library at 0x7ffc3a4e4000 with name [EMPTY] [103544/103544][gotcha.c:38] - Searching for exported symbols in /opt/apps/xalt/1.7/lib64/libxalt_init.so [103544/103544][gotcha.c:58] - Checking ELF hash for pthread_create in /opt/apps/xalt/1.7/lib64/libxalt_init.so [103544/103544][gotcha.c:65] - pthread_create not found in /opt/apps/xalt/1.7/lib64/libxalt_init.so [103544/103544][gotcha.c:38] - Searching for exported symbols in /work/03915/taoncsu/tools/caliper/lib64/libcaliper.so [103544/103544][gotcha.c:52] - Checking GNU hash for pthread_create in /work/03915/taoncsu/tools/caliper/lib64/libcaliper.so [103544/103544][gotcha.c:58] - Checking ELF hash for pthread_create in /work/03915/taoncsu/tools/caliper/lib64/libcaliper.so [103544/103544][gotcha.c:65] - pthread_create not found in /work/03915/taoncsu/tools/caliper/lib64/libcaliper.so [103544/103544][gotcha.c:38] - Searching for exported symbols in /opt/apps/gcc/5.4.0/lib64/libstdc++.so.6 [103544/103544][gotcha.c:58] - Checking ELF hash for pthread_create in /opt/apps/gcc/5.4.0/lib64/libstdc++.so.6 [103544/103544][gotcha.c:71] - Symbol pthread_create found but not exported in /opt/apps/gcc/5.4.0/lib64/libstdc++.so.6 [103544/103544][gotcha.c:38] - Searching for exported symbols in /usr/lib64/libm.so.6 [103544/103544][gotcha.c:52] - Checking GNU hash for pthread_create in /usr/lib64/libm.so.6 [103544/103544][gotcha.c:58] - Checking ELF hash for pthread_create in /usr/lib64/libm.so.6 [103544/103544][gotcha.c:65] - pthread_create not found in /usr/lib64/libm.so.6 [103544/103544][gotcha.c:38] - Searching for exported symbols in /opt/intel/compilers_and_libraries/linux/lib/intel64/libiomp5.so [103544/103544][gotcha.c:58] - Checking ELF hash for pthread_create in /opt/intel/compilers_and_libraries/linux/lib/intel64/libiomp5.so [103544/103544][gotcha.c:77] - Symbol pthread_create found in /opt/intel/compilers_and_libraries/linux/lib/intel64/libiomp5.so at 0x2ba9ea84c000 [103544/103544][gotcha.c:38] - Searching for exported symbols in /opt/apps/gcc/5.4.0/lib64/libgcc_s.so.1 [103544/103544][gotcha.c:38] - Searching for exported symbols in /usr/lib64/libpthread.so.0 [103544/103544][gotcha.c:38] - Searching for exported symbols in /usr/lib64/libc.so.6 [103544/103544][gotcha.c:38] - Searching for exported symbols in /usr/lib64/libdl.so.2 [103544/103544][gotcha.c:38] - Searching for exported symbols in /usr/lib64/libuuid.so.1 [103544/103544][gotcha.c:38] - Searching for exported symbols in /work/03915/taoncsu/tools/caliper/lib64/libgotcha.so.0 [103544/103544][gotcha.c:38] - Searching for exported symbols in /usr/lib64/librt.so.1 [103544/103544][gotcha.c:38] - Searching for exported symbols in /work/03915/taoncsu/tools/caliper/lib64/libcaliper-reader.so [103544/103544][gotcha.c:38] - Searching for exported symbols in /work/03915/taoncsu/tools/caliper/lib64/libcaliper-common.so [103544/103544][gotcha.c:38] - Searching for exported symbols in /opt/intel/compilers_and_libraries/linux/lib/intel64/libimf.so [103544/103544][gotcha.c:38] - Searching for exported symbols in /opt/intel/compilers_and_libraries/linux/lib/intel64/libsvml.so [103544/103544][gotcha.c:38] - Searching for exported symbols in /opt/intel/compilers_and_libraries/linux/lib/intel64/libirng.so [103544/103544][gotcha.c:38] - Searching for exported symbols in /opt/intel/compilers_and_libraries/linux/lib/intel64/libintlc.so.5 [103544/103544][gotcha.c:38] - Searching for exported symbols in /lib64/ld-linux-x86-64.so.2 [103544/103544][gotcha.c:85] - Found 1 / 1 during exported symbol lookup [103544/103544][gotcha.c:163] - Looking for wrapped callsites in [EMPTY] [103544/103544][gotcha.c:163] - Looking for wrapped callsites in [EMPTY] [103544/103544][gotcha.c:163] - Looking for wrapped callsites in /opt/apps/xalt/1.7/lib64/libxalt_init.so [103544/103544][gotcha.c:163] - Looking for wrapped callsites in /work/03915/taoncsu/tools/caliper/lib64/libcaliper.so [103544/103544][gotcha.c:163] - Looking for wrapped callsites in /opt/apps/gcc/5.4.0/lib64/libstdc++.so.6 [103544/103544][gotcha.c:104] - Remapped call to pthread_create at 0x2ba9ea544d10 in /opt/apps/gcc/5.4.0/lib64/libstdc++.so.6 to wrapper at 0x0x2ba9e9fa4850 [103544/103544][gotcha.c:163] - Looking for wrapped callsites in /usr/lib64/libm.so.6 [103544/103544][gotcha.c:163] - Looking for wrapped callsites in /opt/intel/compilers_and_libraries/linux/lib/intel64/libiomp5.so [103544/103544][gotcha.c:104] - Remapped call to pthread_create at 0x2ba9eabb6a48 in /opt/intel/compilers_and_libraries/linux/lib/intel64/libiomp5.so to wrapper at 0x0x2ba9e9fa4850 [103544/103544][gotcha.c:163] - Looking for wrapped callsites in /opt/apps/gcc/5.4.0/lib64/libgcc_s.so.1 [103544/103544][gotcha.c:163] - Looking for wrapped callsites in /usr/lib64/libpthread.so.0 [103544/103544][gotcha.c:163] - Looking for wrapped callsites in /usr/lib64/libc.so.6 [103544/103544][gotcha.c:163] - Looking for wrapped callsites in /usr/lib64/libdl.so.2 [103544/103544][gotcha.c:163] - Looking for wrapped callsites in /usr/lib64/libuuid.so.1 [103544/103544][gotcha.c:163] - Looking for wrapped callsites in /work/03915/taoncsu/tools/caliper/lib64/libgotcha.so.0 [103544/103544][gotcha.c:163] - Looking for wrapped callsites in /usr/lib64/librt.so.1 [103544/103544][gotcha.c:104] - Remapped call to pthread_create at 0x2ba9ebc02030 in /usr/lib64/librt.so.1 to wrapper at 0x0x2ba9e9fa4850 [103544/103544][gotcha.c:163] - Looking for wrapped callsites in /work/03915/taoncsu/tools/caliper/lib64/libcaliper-reader.so [103544/103544][gotcha.c:163] - Looking for wrapped callsites in /work/03915/taoncsu/tools/caliper/lib64/libcaliper-common.so [103544/103544][gotcha.c:163] - Looking for wrapped callsites in /opt/intel/compilers_and_libraries/linux/lib/intel64/libimf.so [103544/103544][gotcha.c:163] - Looking for wrapped callsites in /opt/intel/compilers_and_libraries/linux/lib/intel64/libsvml.so [103544/103544][gotcha.c:163] - Looking for wrapped callsites in /opt/intel/compilers_and_libraries/linux/lib/intel64/libirng.so [103544/103544][gotcha.c:163] - Looking for wrapped callsites in /opt/intel/compilers_and_libraries/linux/lib/intel64/libintlc.so.5 [103544/103544][gotcha.c:163] - Looking for wrapped callsites in /lib64/ld-linux-x86-64.so.2 [103544/103544][gotcha.c:176] - Returning code 0 from gotcha_wrap == CALIPER: Registered pthread service == CALIPER: Initialized Segmentation fault

[DavidPoliakoff]

Hey @twang15 : please excuse the slow fix. We're caught up on #26, I have a candidate fix to try for your bug, but that needs to be merged first. I believe @daboehme may be hiding some of the GOTCHA'isms from you so your Caliper stuff works, but if it doesn't I'd expect a fix in a few weeks.