dynamorio icon indicating copy to clipboard operation
dynamorio copied to clipboard

Published 20 hours ago • verified publisher icon DynamoRIO

fix failing tests in debug build long suite

Open derekbruening opened this issue 10 years ago • 49 comments

Xref #1793. There are many, many tests failing in the long suite. This issue covers fixing them (or removing sets of runs if we decide not to support them anymore) and getting the long suite clean so we can re-enable it.

debug-internal-32: 2022 tests passed, **** 81 tests failed, of which 15 were flaky: ****
    code_api|client.thread 
    code_api,opt_memory|linux.child-blacklist 
    code_api,opt_memory|security-common.selfmod =>  Application /work/dr/nightly/run/build_debug-internal-32/suite/tests/bin/security-common.selfmod (26116).  Internal Error: DynamoRIO debug check failure: /work/dr/git/src/core/vmareas.c:2999 (is_coarse  coarse != NULL) || (!is_coarse  coarse == NULL) 
    code_api,opt_memory|client.drwrap-test 
    code_api|linux.signest 
    code_api,opt_speed|linux.persist-use_FLAKY 
    code_api,opt_speed|client.call-retarget 
    code_api,opt_speed|client.count-ctis 
    code_api,opt_speed|client.count-ctis-noopt 
    code_api,opt_speed|client.syscall 
    code_api,opt_speed|client.flush 
    code_api,opt_speed|client.inline 
    code_api,opt_speed|client.drreg-test 
    code_api,opt_speed|client.drwrap-test 
    code_api,opt_speed|client.pcache-use 
    code_api,opt_speed|sample.bbbuf 
    code_api,opt_speed|sample.modxfer 
    code_api,opt_speed|sample.modxfer_app2lib 
    code_api,opt_speed|sample.instrcalls 
    code_api,opt_speed|tool.drcov.fib 
    code_api,thread_private|linux.child-whitelist 
    code_api,thread_private|linux.child-blacklist 
    code_api,thread_private|linux.freeze_FLAKY 
    code_api,thread_private|linux.persist_FLAKY 
    code_api,thread_private|linux.persist-use_FLAKY 
    code_api,thread_private|client.syscall =>  Application /work/dr/nightly/run/build_debug-internal-32/suite/tests/bin/client.syscall (28252).  Internal Error: DynamoRIO debug check failure: /work/dr/git/src/core/fragment.c:5648 thread_owns_no_locks(dcontext) || thread_owns_one_lock(dcontext, thread_initexit_lock) || thread_owns_two_locks(dcontext, thread_initexit_lock, all_threads_synch_lock) 
    code_api,opt_speed|linux.fib-pie 
    code_api,thread_private|client.thread 
    code_api,thread_private|client.inline 
    code_api,thread_private|client.pcache =>  Application /work/dr/nightly/run/build_debug-internal-32/suite/tests/bin/client.pcache (28746).  Internal Error: DynamoRIO debug check failure: /work/dr/git/src/core/options.c:1775 false 
    code_api,thread_private|client.pcache-use =>  Application /work/dr/nightly/run/build_debug-internal-32/suite/tests/bin/client.pcache (28747).  Internal Error: DynamoRIO debug check failure: /work/dr/git/src/core/options.c:1775 false 
    code_api,disable_traces|linux.child-whitelist 
    code_api,disable_traces|linux.child-blacklist 
    code_api,disable_traces|linux.freeze_FLAKY 
    code_api,disable_traces|linux.persist-use_FLAKY 
    code_api,disable_traces|client.pcache-use 
    code_api,thread_private,disable_traces|linux.child-blacklist 
    code_api,thread_private,disable_traces|linux.child-whitelist 
    code_api,thread_private,disable_traces|linux.freeze_FLAKY 
    code_api,thread_private,disable_traces|linux.persist_FLAKY 
    code_api,thread_private,disable_traces|linux.persist-use_FLAKY 
    code_api,thread_private,disable_traces|client.syscall =>  Application /work/dr/nightly/run/build_debug-internal-32/suite/tests/bin/client.syscall (30260).  Internal Error: DynamoRIO debug check failure: /work/dr/git/src/core/fragment.c:5648 thread_owns_no_locks(dcontext) || thread_owns_one_lock(dcontext, thread_initexit_lock) || thread_owns_two_locks(dcontext, thread_initexit_lock, all_threads_synch_lock) 
    code_api,thread_private,disable_traces|client.thread 
    code_api,thread_private,disable_traces|client.inline 
    code_api,thread_private,disable_traces|client.pcache =>  Application /work/dr/nightly/run/build_debug-internal-32/suite/tests/bin/client.pcache (30747).  Internal Error: DynamoRIO debug check failure: /work/dr/git/src/core/options.c:1775 false 
    code_api,thread_private,disable_traces|client.pcache-use =>  Application /work/dr/nightly/run/build_debug-internal-32/suite/tests/bin/client.pcache (30753).  Internal Error: DynamoRIO debug check failure: /work/dr/git/src/core/options.c:1775 false 
    code_api,no_early_inject|linux.child-blacklist 
    code_api,no_early_inject|client.drmgr-test 
    code_api,no_early_inject|client.pcache-use 
    code_api,no_early_inject|tool.drcachesim.simple 
    code_api,no_early_inject|tool.drcachesim.TLB-simple 
    code_api,no_early_inject|tool.drcachesim.phys 
    code_api,no_early_inject|tool.drcpusim.mmx 
    code_api,no_early_inject|tool.drcpusim.sse 
    code_api,no_early_inject|tool.drcpusim.sse2 
    code_api,no_early_inject|tool.drcpusim.sse3 
    code_api,no_early_inject|tool.drcpusim.ssse3 
    code_api,no_early_inject|tool.drcpusim.sse41 
    code_api,no_early_inject|tool.drcpusim.sse42 
    code_api,stack_size,loglevel|client.thread 
    code_api,sysenter_is_int80|linux.child-blacklist 
    code_api,sysenter_is_int80|linux.child-whitelist 
    code_api,sysenter_is_int80|linux.persist-use_FLAKY 
    code_api,thread_private,tracedump_binary|common.broadfun =>  Application /work/dr/nightly/run/build_debug-internal-32/suite/tests/bin/common.broadfun (537).  Internal Error: DynamoRIO debug check failure: /work/dr/git/src/core/vmareas.c:824 SHARED_FRAGMENTS_ENABLED() 
    code_api,thread_private,tracedump_binary|common.fib =>  Application /work/dr/nightly/run/build_debug-internal-32/suite/tests/bin/common.fib (546).  Internal Error: DynamoRIO debug check failure: /work/dr/git/src/core/vmareas.c:824 SHARED_FRAGMENTS_ENABLED() 
    code_api,thread_private,tracedump_binary|common.decode-bad =>  Application /work/dr/nightly/run/build_debug-internal-32/suite/tests/bin/common.decode-bad (548).  Internal Error: DynamoRIO debug check failure: /work/dr/git/src/core/vmareas.c:824 SHARED_FRAGMENTS_ENABLED() 
    code_api,thread_private,tracedump_binary|common.eflags =>  Application /work/dr/nightly/run/build_debug-internal-32/suite/tests/bin/common.eflags (550).  Internal Error: DynamoRIO debug check failure: /work/dr/git/src/core/vmareas.c:824 SHARED_FRAGMENTS_ENABLED() 
    code_api,thread_private,tracedump_binary|common.floatpc =>  Application /work/dr/nightly/run/build_debug-internal-32/suite/tests/bin/common.floatpc (552).  Internal Error: DynamoRIO debug check failure: /work/dr/git/src/core/vmareas.c:824 SHARED_FRAGMENTS_ENABLED() 
    code_api,thread_private,tracedump_binary|common.floatpc_xl8all =>  Application /work/dr/nightly/run/build_debug-internal-32/suite/tests/bin/common.floatpc (557).  Internal Error: DynamoRIO debug check failure: /work/dr/git/src/core/vmareas.c:824 SHARED_FRAGMENTS_ENABLED() 
    code_api,thread_private,tracedump_binary|common.getretaddr =>  Application /work/dr/nightly/run/build_debug-internal-32/suite/tests/bin/common.getretaddr (558).  Internal Error: DynamoRIO debug check failure: /work/dr/git/src/core/vmareas.c:824 SHARED_FRAGMENTS_ENABLED() 
    code_api,thread_private,tracedump_binary|common.nativeexec_exe =>  Application /work/dr/nightly/run/build_debug-internal-32/suite/tests/bin/common.nativeexec (565).  Internal Error: DynamoRIO debug check failure: /work/dr/git/src/core/vmareas.c:824 SHARED_FRAGMENTS_ENABLED() 
    code_api,thread_private,tracedump_binary|common.nativeexec_bindnow =>  Application /work/dr/nightly/run/build_debug-internal-32/suite/tests/bin/common.nativeexec (567).  Internal Error: DynamoRIO debug check failure: /work/dr/git/src/core/vmareas.c:824 SHARED_FRAGMENTS_ENABLED() 
    code_api,thread_private,tracedump_binary|common.segfault =>  Application /work/dr/nightly/run/build_debug-internal-32/suite/tests/bin/common.segfault (579).  Internal Error: DynamoRIO debug check failure: /work/dr/git/src/core/vmareas.c:824 SHARED_FRAGMENTS_ENABLED() 
    checklevel|linux.child-whitelist 
    checklevel|pthreads.ptsig_FLAKY 
    checklevel|linux.persist_FLAKY 
    code_api,sysenter_is_int80|linux.signest 
    checklevel|api.startstop_FLAKY 
    finite_shared_trace_cache,cache_shared_trace_regen|linux.child-whitelist 
    finite_shared_bb_cache,cache_shared_bb_regen|linux.persist_FLAKY 
    finite_shared_trace_cache,cache_shared_trace_regen|linux.persist-use_FLAKY 

debug-internal-64: 1991 tests passed, **** 68 tests failed, of which 2 were flaky: ****
    code_api,opt_memory|linux.child-blacklist 
    code_api,opt_memory|linux.child-whitelist 
    code_api,opt_memory|security-common.selfmod-big =>  Application /work/dr/nightly/run/build_debug-internal-64/suite/tests/bin/security-common.selfmod-big (13440).  Internal Error: DynamoRIO debug check failure: /work/dr/git/src/core/vmareas.c:2999 (is_coarse  coarse != NULL) || (!is_coarse  coarse == NULL) 
    code_api,opt_memory|float_DR =>  Application /work/dr/nightly/run/build_debug-internal-64/suite/tests/bin/security-common.selfmod (13444).  Internal Error: DynamoRIO debug check failure: /work/dr/git/src/core/vmareas.c:2999 (is_coarse  coarse != NULL) || (!is_coarse  coarse == NULL) 
    code_api,opt_memory|security-common.selfmod =>  Application /work/dr/nightly/run/build_debug-internal-64/suite/tests/bin/security-common.selfmod (13443).  Internal Error: DynamoRIO debug check failure: /work/dr/git/src/core/vmareas.c:2999 (is_coarse  coarse != NULL) || (!is_coarse  coarse == NULL) 
    code_api,opt_memory|float_vmbase =>  Application /work/dr/nightly/run/build_debug-internal-64/suite/tests/bin/security-common.selfmod (13454).  Internal Error: DynamoRIO debug check failure: /work/dr/git/src/core/vmareas.c:2999 (is_coarse  coarse != NULL) || (!is_coarse  coarse == NULL) 
    code_api,opt_memory|client.drwrap-test 
    code_api,opt_speed|client.call-retarget 
    code_api,opt_speed|client.count-ctis 
    code_api,opt_speed|client.count-ctis-noopt 
    code_api,opt_speed|client.syscall 
    code_api,opt_speed|client.flush 
    code_api,opt_speed|client.cbr-retarget 
    code_api,opt_speed|client.inline 
    code_api,opt_speed|client.drreg-test 
    code_api,opt_speed|sample.modxfer 
    code_api,opt_speed|sample.modxfer_app2lib 
    code_api,opt_speed|sample.instrcalls 
    code_api,opt_speed|tool.drcov.fib 
    code_api,thread_private|linux.child-whitelist 
    code_api,thread_private|client.syscall =>  Application /work/dr/nightly/run/build_debug-internal-64/suite/tests/bin/client.syscall (15506).  Internal Error: DynamoRIO debug check failure: /work/dr/git/src/core/fragment.c:5648 thread_owns_no_locks(dcontext) || thread_owns_one_lock(dcontext, thread_initexit_lock) || thread_owns_two_locks(dcontext, thread_initexit_lock, all_threads_synch_lock) 
    code_api,thread_private|client.pcache =>  Application /work/dr/nightly/run/build_debug-internal-64/suite/tests/bin/client.pcache (15986).  Internal Error: DynamoRIO debug check failure: /work/dr/git/src/core/options.c:1775 false 
    code_api,thread_private|client.pcache-use =>  Application /work/dr/nightly/run/build_debug-internal-64/suite/tests/bin/client.pcache (15987).  Internal Error: DynamoRIO debug check failure: /work/dr/git/src/core/options.c:1775 false 
    code_api,thread_private,disable_traces|client.syscall =>  Application /work/dr/nightly/run/build_debug-internal-64/suite/tests/bin/client.syscall (18399).  Internal Error: DynamoRIO debug check failure: /work/dr/git/src/core/fragment.c:5648 thread_owns_no_locks(dcontext) || thread_owns_one_lock(dcontext, thread_initexit_lock) || thread_owns_two_locks(dcontext, thread_initexit_lock, all_threads_synch_lock) 
    code_api,thread_private,disable_traces|client.pcache =>  Application /work/dr/nightly/run/build_debug-internal-64/suite/tests/bin/client.pcache (19030).  Internal Error: DynamoRIO debug check failure: /work/dr/git/src/core/options.c:1775 false 
    code_api,thread_private,disable_traces|client.pcache-use =>  Application /work/dr/nightly/run/build_debug-internal-64/suite/tests/bin/client.pcache (19055).  Internal Error: DynamoRIO debug check failure: /work/dr/git/src/core/options.c:1775 false 
    code_api,no_early_inject|linux.child-blacklist 
    code_api,no_early_inject|linux.readlink 
    code_api,no_early_inject|client.drmgr-test 
    code_api,no_early_inject|tool.drcachesim.simple 
    code_api,no_early_inject|tool.drcachesim.TLB-simple 
    code_api,no_early_inject|tool.drcachesim.phys 
    code_api,no_early_inject|tool.drcpusim.ssse3 
    code_api,no_early_inject|tool.drcpusim.sse41 
    code_api,no_early_inject|tool.drcpusim.sse42 
    code_api,sysenter_is_int80|linux.child-blacklist 
    code_api,sysenter_is_int80|linux.child-whitelist 
    code_api,tracedump_text,tracedump_origins|common.decode-bad =>  Application /work/dr/nightly/run/build_debug-internal-64/suite/tests/bin/common.decode-bad (21265).  Internal Error: DynamoRIO debug check failure: /work/dr/git/src/core/annotations.c:359 (dcontext) != NULL  (dcontext) != GLOBAL_DCONTEXT 
    code_api,tracedump_text,tracedump_origins|common.decode =>  Application /work/dr/nightly/run/build_debug-internal-64/suite/tests/bin/common.decode (21267).  Internal Error: DynamoRIO debug check failure: /work/dr/git/src/core/annotations.c:359 (dcontext) != NULL  (dcontext) != GLOBAL_DCONTEXT 
    code_api,tracedump_text,tracedump_origins|common.broadfun =>  Application /work/dr/nightly/run/build_debug-internal-64/suite/tests/bin/common.broadfun (21261).  Internal Error: DynamoRIO debug check failure: /work/dr/git/src/core/annotations.c:359 (dcontext) != NULL  (dcontext) != GLOBAL_DCONTEXT 
    code_api,tracedump_text,tracedump_origins|common.floatpc_xl8all =>  Application /work/dr/nightly/run/build_debug-internal-64/suite/tests/bin/common.floatpc (21275).  Internal Error: DynamoRIO debug check failure: /work/dr/git/src/core/annotations.c:359 (dcontext) != NULL  (dcontext) != GLOBAL_DCONTEXT 
    code_api,tracedump_text,tracedump_origins|common.getretaddr =>  Application /work/dr/nightly/run/build_debug-internal-64/suite/tests/bin/common.getretaddr (21277).  Internal Error: DynamoRIO debug check failure: /work/dr/git/src/core/annotations.c:359 (dcontext) != NULL  (dcontext) != GLOBAL_DCONTEXT 
    code_api,tracedump_text,tracedump_origins|common.floatpc =>  Application /work/dr/nightly/run/build_debug-internal-64/suite/tests/bin/common.floatpc (21273).  Internal Error: DynamoRIO debug check failure: /work/dr/git/src/core/annotations.c:359 (dcontext) != NULL  (dcontext) != GLOBAL_DCONTEXT 
    code_api,tracedump_text,tracedump_origins|common.eflags =>  Application /work/dr/nightly/run/build_debug-internal-64/suite/tests/bin/common.eflags (21271).  Internal Error: DynamoRIO debug check failure: /work/dr/git/src/core/annotations.c:359 (dcontext) != NULL  (dcontext) != GLOBAL_DCONTEXT 
    code_api,tracedump_text,tracedump_origins,syntax_intel|common.eflags =>  Application /work/dr/nightly/run/build_debug-internal-64/suite/tests/bin/common.eflags (21304).  Internal Error: DynamoRIO debug check failure: /work/dr/git/src/core/annotations.c:359 (dcontext) != NULL  (dcontext) != GLOBAL_DCONTEXT 
    code_api,tracedump_text,tracedump_origins,syntax_intel|common.broadfun =>  Application /work/dr/nightly/run/build_debug-internal-64/suite/tests/bin/common.broadfun (21296).  Internal Error: DynamoRIO debug check failure: /work/dr/git/src/core/annotations.c:359 (dcontext) != NULL  (dcontext) != GLOBAL_DCONTEXT 
    code_api,tracedump_text,tracedump_origins,syntax_intel|common.decode-bad =>  Application /work/dr/nightly/run/build_debug-internal-64/suite/tests/bin/common.decode-bad (21300).  Internal Error: DynamoRIO debug check failure: /work/dr/git/src/core/annotations.c:359 (dcontext) != NULL  (dcontext) != GLOBAL_DCONTEXT 
    code_api,tracedump_text,tracedump_origins,syntax_intel|common.decode =>  Application /work/dr/nightly/run/build_debug-internal-64/suite/tests/bin/common.decode (21303).  Internal Error: DynamoRIO debug check failure: /work/dr/git/src/core/annotations.c:359 (dcontext) != NULL  (dcontext) != GLOBAL_DCONTEXT 
    code_api,tracedump_text,tracedump_origins|common.segfault =>  Application /work/dr/nightly/run/build_debug-internal-64/suite/tests/bin/common.segfault (21294).  Internal Error: DynamoRIO debug check failure: /work/dr/git/src/core/annotations.c:359 (dcontext) != NULL  (dcontext) != GLOBAL_DCONTEXT 
    code_api,tracedump_text,tracedump_origins,syntax_intel|common.floatpc_xl8all =>  Application /work/dr/nightly/run/build_debug-internal-64/suite/tests/bin/common.floatpc (21308).  Internal Error: DynamoRIO debug check failure: /work/dr/git/src/core/annotations.c:359 (dcontext) != NULL  (dcontext) != GLOBAL_DCONTEXT 
    code_api,tracedump_text,tracedump_origins,syntax_intel|common.floatpc =>  Application /work/dr/nightly/run/build_debug-internal-64/suite/tests/bin/common.floatpc (21306).  Internal Error: DynamoRIO debug check failure: /work/dr/git/src/core/annotations.c:359 (dcontext) != NULL  (dcontext) != GLOBAL_DCONTEXT 
    code_api,tracedump_text,tracedump_origins,syntax_intel|common.getretaddr =>  Application /work/dr/nightly/run/build_debug-internal-64/suite/tests/bin/common.getretaddr (21310).  Internal Error: DynamoRIO debug check failure: /work/dr/git/src/core/annotations.c:359 (dcontext) != NULL  (dcontext) != GLOBAL_DCONTEXT 
    code_api,thread_private,tracedump_binary|common.fib =>  Application /work/dr/nightly/run/build_debug-internal-64/suite/tests/bin/common.fib (21330).  Internal Error: DynamoRIO debug check failure: /work/dr/git/src/core/vmareas.c:824 SHARED_FRAGMENTS_ENABLED() 
    code_api,thread_private,tracedump_binary|common.broadfun =>  Application /work/dr/nightly/run/build_debug-internal-64/suite/tests/bin/common.broadfun (21328).  Internal Error: DynamoRIO debug check failure: /work/dr/git/src/core/vmareas.c:824 SHARED_FRAGMENTS_ENABLED() 
    code_api,tracedump_text,tracedump_origins,syntax_intel|common.segfault =>  Application /work/dr/nightly/run/build_debug-internal-64/suite/tests/bin/common.segfault (21326).  Internal Error: DynamoRIO debug check failure: /work/dr/git/src/core/annotations.c:359 (dcontext) != NULL  (dcontext) != GLOBAL_DCONTEXT 
    code_api,thread_private,tracedump_binary|common.floatpc =>  Application /work/dr/nightly/run/build_debug-internal-64/suite/tests/bin/common.floatpc (21338).  Internal Error: DynamoRIO debug check failure: /work/dr/git/src/core/vmareas.c:824 SHARED_FRAGMENTS_ENABLED() 
    code_api,thread_private,tracedump_binary|common.eflags =>  Application /work/dr/nightly/run/build_debug-internal-64/suite/tests/bin/common.eflags (21336).  Internal Error: DynamoRIO debug check failure: /work/dr/git/src/core/vmareas.c:824 SHARED_FRAGMENTS_ENABLED() 
    code_api,thread_private,tracedump_binary|common.floatpc_xl8all =>  Application /work/dr/nightly/run/build_debug-internal-64/suite/tests/bin/common.floatpc (21340).  Internal Error: DynamoRIO debug check failure: /work/dr/git/src/core/vmareas.c:824 SHARED_FRAGMENTS_ENABLED() 
    code_api,thread_private,tracedump_binary|common.getretaddr =>  Application /work/dr/nightly/run/build_debug-internal-64/suite/tests/bin/common.getretaddr (21342).  Internal Error: DynamoRIO debug check failure: /work/dr/git/src/core/vmareas.c:824 SHARED_FRAGMENTS_ENABLED() 
    code_api,thread_private,tracedump_binary|common.decode-bad =>  Application /work/dr/nightly/run/build_debug-internal-64/suite/tests/bin/common.decode-bad (21332).  Internal Error: DynamoRIO debug check failure: /work/dr/git/src/core/vmareas.c:824 SHARED_FRAGMENTS_ENABLED() 
    code_api,thread_private,tracedump_binary|common.nativeexec_retakeover =>  Application /work/dr/nightly/run/build_debug-internal-64/suite/tests/bin/common.nativeexec (21346).  Internal Error: DynamoRIO debug check failure: /work/dr/git/src/core/vmareas.c:824 SHARED_FRAGMENTS_ENABLED() 
    code_api,thread_private,tracedump_binary|common.decode =>  Application /work/dr/nightly/run/build_debug-internal-64/suite/tests/bin/common.decode (21334).  Internal Error: DynamoRIO debug check failure: /work/dr/git/src/core/vmareas.c:824 SHARED_FRAGMENTS_ENABLED() 
    code_api,thread_private,tracedump_binary|common.nativeexec_bindnow =>  Application /work/dr/nightly/run/build_debug-internal-64/suite/tests/bin/common.nativeexec (21350).  Internal Error: DynamoRIO debug check failure: /work/dr/git/src/core/vmareas.c:824 SHARED_FRAGMENTS_ENABLED() 
    code_api,thread_private,tracedump_binary|common.nativeexec_exe =>  Application /work/dr/nightly/run/build_debug-internal-64/suite/tests/bin/common.nativeexec (21348).  Internal Error: DynamoRIO debug check failure: /work/dr/git/src/core/vmareas.c:824 SHARED_FRAGMENTS_ENABLED() 
    code_api,thread_private,tracedump_binary|common.segfault =>  Application /work/dr/nightly/run/build_debug-internal-64/suite/tests/bin/common.segfault (21358).  Internal Error: DynamoRIO debug check failure: /work/dr/git/src/core/vmareas.c:824 SHARED_FRAGMENTS_ENABLED() 
    checklevel|linux.child-whitelist 
    checklevel|pthreads.pthreads_fork_FLAKY =>  Application /work/dr/nightly/run/build_debug-internal-64/suite/tests/bin/pthreads.pthreads_fork_FLAKY (21584).  Internal Error: DynamoRIO debug check failure: /work/dr/git/src/core/unix/memcache.c:408 found 
    checklevel|api.startstop_FLAKY

derekbruening avatar Oct 13 '15 02:10 derekbruening

I fixed the tracedump failures in f54e24ae7b053676616e599f7606bca7a2c384d6

derekbruening avatar Oct 13 '15 18:10 derekbruening

For ptsig_FLAKY:

<(1+x) Handling our fault in a TRY at 0x71215b3c>
<dropping nested signal>
<Application tried to execute from unreadable memory 0x00000000.
This may be a result of an unsuccessful attack or a potential application vulnerability.>

In a different run:

<Initial options = -no_dynamic_options -code_api -msgbox_mask 12 -stack_size 56K -max_elide_jmp 0 -max_elide_call 0 -early_inject -emulate_brk -no_inline_ignored_syscalls -native_exec_default_list '' -no_native_exec_managed_code -no_indcall2direct >
<(1+x) Handling our fault in a TRY at 0x71215b3c>
<ERROR: master_signal_handler with no siginfo (i#26?): tid=5878, sig=10>
<press enter to continue>

Attaching gdb does not help, no symbol info:

(gdb) add-symbol-file '/home/zhaoqin/Workspace/DynamoRIO/builds/build_a32_dbg.git/lib32/debug/libdynamorio.so' 0x710228d8
...
(gdb) where
#0  0x4ec8e3a4 in ?? ()
#1  0xb6fa5ec2 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb) info threads
  Id   Target Id         Frame 
* 1    process 5876 "pthreads.ptsig_" 0x4ec8e3a4 in ?? ()

GDB did not see thread 5878. signal number 10 is SIGBUS

zhaoqin avatar Nov 27 '16 01:11 zhaoqin

Saw some more failures on a recent master merge (https://github.com/DynamoRIO/dynamorio/runs/2355003904). May need to add them to ignore list if failures persist:

prof_pcs|common.nativeexec_exe: <Application /home/runner/work/dynamorio/dynamorio/build_debug-internal-64/suite/tests/bin/common.nativeexec (5262). Internal Error: DynamoRIO debug check failure: /home/runner/work/dynamorio/dynamorio/core/native_exec.c:272 i < MAX_NATIVE_RETSTACK

|common.decode-stress: <Application /home/runner/work/dynamorio/dynamorio/build_debug-internal-32/suite/tests/bin/common.decode (5650). Internal Error: DynamoRIO debug check failure: /home/runner/work/dynamorio/dynamorio/core/translate.c:1966 spill_ibreg_outstanding_offs == UINT_MAX || mc.IF_X86_ELSE(xcx, r2) == (reg_t)d_r_get_tls(spill_ibreg_outstanding_offs)

abhinav92003 avatar Apr 15 '21 17:04 abhinav92003

More: code_api,loglevel|common.decode https://github.com/DynamoRIO/dynamorio/actions/runs/753520501 Output mismtach: Bad instruction, instance 64 early on which is weird b/c the counts only go to 19!

derekbruening avatar Apr 15 '21 21:04 derekbruening

code_api,sysenter_is_int80|common.nativeexec_exe https://github.com/DynamoRIO/dynamorio/runs/2357051106 <Application /home/runner/work/dynamorio/dynamorio/build_debug-internal-64/suite/tests/bin/common.nativeexec (5829). Internal Error: DynamoRIO debug check failure: /home/runner/work/dynamorio/dynamorio/core/native_exec.c:272 i < MAX_NATIVE_RETSTACK

derekbruening avatar Apr 15 '21 22:04 derekbruening

code_api,tracedump_text,tracedump_origins,syntax_intel|common.fib failed in github.com/DynamoRIO/dynamorio/actions/runs/753359537. Nothing useful in logs though.

Another common.fib failure was seen with different options (code_api,thread_private,disable_traces|common.fib), documented in #4077.

abhinav92003 avatar Apr 15 '21 22:04 abhinav92003

code_api,loglevel|common.decode-stress failed again in 32-bit x86 but it looks like a different assert: https://github.com/DynamoRIO/dynamorio/runs/2357236389

497: Start
497: Jumping to a sequence of every addr16 modrm byte
497: <Application /home/runner/work/dynamorio/dynamorio/build_debug-internal-32/suite/tests/bin/common.decode (7005).  Internal Error: DynamoRIO debug check failure: /home/runner/work/dynamorio/dynamorio/core/translate.c:1960 success_so_far || (!res && (instr_is_DR_reg_spill_or_restore(dcontext, in, NULL, NULL, NULL, NULL) || (!instr_reads_memory(in) && !instr_writes_memory(in))))

derekbruening avatar Apr 15 '21 23:04 derekbruening

prof_pcs|common.nativeexec_bindnow_opt failed in 64-bit x86: https://github.com/DynamoRIO/dynamorio/runs/2357236237 <Application /home/runner/work/dynamorio/dynamorio/build_debug-internal-64/suite/tests/bin/common.nativeexec (6130). Internal Error: DynamoRIO debug check failure: /home/runner/work/dynamorio/dynamorio/core/unix/signal.c:5207 tr == NULL || tr->under_dynamo_control || IS_CLIENT_THREAD(dcontext) || sig == SUSPEND_SIGNAL

derekbruening avatar Apr 15 '21 23:04 derekbruening

code_api|tool.drcpusim.cpuid-Merom failed in vs2017-64: https://github.com/DynamoRIO/dynamorio/runs/2364302489?check_suite_focus=true

abhinav92003 avatar Apr 16 '21 17:04 abhinav92003

code_api,thread_private|common.floatpc_xl8all failed in vs2017-64: https://github.com/DynamoRIO/dynamorio/runs/2366269668?check_suite_focus=true

abhinav92003 avatar Apr 17 '21 01:04 abhinav92003

x86 32-bit failed. BTW I think we should drop support for -no_early_inject. However this same assert might be listed up above so it might not remove this problem.

code_api,no_early_inject|common.decode-stress https://github.com/DynamoRIO/dynamorio/runs/2366269598 common.decode (25739). Internal Error: DynamoRIO debug check failure: /home/runner/work/dynamorio/dynamorio/core/translate.c:1114 false

derekbruening avatar Apr 17 '21 15:04 derekbruening

Every merge is red now -- I think some of these need to be put on the flaky list.

x86-32: code_api,stack_size,loglevel|common.decode-stress common.decode (5579). Internal Error: DynamoRIO debug check failure: /home/runner/work/dynamorio/dynamorio/core/translate.c:1114 false https://github.com/DynamoRIO/dynamorio/runs/2399752840 Is that #4723?

x86-64: code_api,tracedump_text,tracedump_origins,syntax_intel|common.nativeexec_bindnow common.nativeexec (4531). Internal Error: DynamoRIO debug check failure: /home/runner/work/dynamorio/dynamorio/core/native_exec.c:272 i < MAX_NATIVE_RETSTACK https://github.com/DynamoRIO/dynamorio/runs/2399752677

x86-64: code_api,bbdump_tags|common.nativeexec_bindnow common.nativeexec (10150). Internal Error: DynamoRIO debug check failure: /home/runner/work/dynamorio/dynamorio/core/native_exec.c:272 i < MAX_NATIVE_RETSTACK https://github.com/DynamoRIO/dynamorio/runs/2393892498

derekbruening avatar Apr 21 '21 17:04 derekbruening

x86-64: code_api,tracedump_text,tracedump_origins,syntax_intel|common.nativeexec_bindnow common.nativeexec (7841). Internal Error: DynamoRIO debug check failure: /home/runner/work/dynamorio/dynamorio/core/native_exec.c:272 i < MAX_NATIVE_RETSTACK https://github.com/DynamoRIO/dynamorio/runs/2370074937

vs2017-32: code_api|tool.drcachesim.threads-with-config-file Assertion failed: type_is_instr(cur_ref_.instr.type) || cur_ref_.instr.type == TRACE_TYPE_INSTR_NO_FETCH, file ..\clients\drcachesim\reader\reader.cpp, line 155 https://github.com/DynamoRIO/dynamorio/runs/2370074865 That looks like #3320

derekbruening avatar Apr 21 '21 17:04 derekbruening

code_api,tracedump_text,tracedump_origins,syntax_intel|common.decode-stress common.decode (26717). Internal Error: DynamoRIO debug check failure: /home/runner/work/dynamorio/dynamorio/core/translate.c:1114 false https://github.com/DynamoRIO/dynamorio/runs/2402763737

derekbruening avatar Apr 21 '21 18:04 derekbruening

code_api,opt_speed|common.floatpc_xl8all failed on vs2017-64: https://github.com/DynamoRIO/dynamorio/runs/2417344630. ~It's already in the ignore list, but with different options. Better to ignore it completely.~ Oh, I was mistaken. There's only one other variation on the ignore list code_api|common.floatpc_xl8all (from the regular suite). Maybe should wait a bit before ignoring this test completely.

abhinav92003 avatar Apr 23 '21 15:04 abhinav92003

finite_shared_trace_cache,cache_shared_trace_regen|common.nativeexec_exe failed on x86-64: <Application /home/runner/work/dynamorio/dynamorio/build_debug-internal-64/suite/tests/bin/common.nativeexec (7148). Internal Error: DynamoRIO debug check failure: /home/runner/work/dynamorio/dynamorio/core/native_exec.c:272 i < MAX_NATIVE_RETSTACK https://github.com/DynamoRIO/dynamorio/runs/2560910709

code_api|common.nativeexec_exe failed on x86-64 with the same assert: <Application /home/runner/work/dynamorio/dynamorio/build_debug-internal-64/suite/tests/bin/common.nativeexec (22533). Internal Error: DynamoRIO debug check failure: /home/runner/work/dynamorio/dynamorio/core/native_exec.c:272 i < MAX_NATIVE_RETSTACK https://github.com/DynamoRIO/dynamorio/runs/2561303391

abhinav92003 avatar May 12 '21 14:05 abhinav92003

enable_full_api|common.broadfun failed in vs2017-64: https://github.com/DynamoRIO/dynamorio/runs/2560909170 Seems like it died printing midway:

Test #350: enable_full_api|common.broadfun ................................................***Failed  Required regular expression not found. Regex=[^sort\(\) = >
2021-05-11T23:33:27.5642198Z ?
2021-05-11T23:33:27.5643845Z done
2021-05-11T23:33:27.5644457Z ?
2021-05-11T23:33:27.5644908Z $
2021-05-11T23:33:27.5645388Z ] 90.02 sec
2021-05-11T23:33:27.5645863Z sor

abhinav92003 avatar May 12 '21 14:05 abhinav92003

x86-32: code_api,stack_size,loglevel|common.decode: https://github.com/DynamoRIO/dynamorio/runs/2497247306 code_api,thread_private|common.decode: https://github.com/DynamoRIO/dynamorio/runs/2505963422 seems to crash

code_api,disable_traces|common.decode: https://github.com/DynamoRIO/dynamorio/runs/2497247306 regex mismatch -- there's an extra Bad instruction, instance 64 in the output

abhinav92003 avatar May 12 '21 15:05 abhinav92003

x86-64: code_api,loglevel|common.nativeexec_exe failed again with core/native_exec.c:272 i < MAX_NATIVE_RETSTACK: https://github.com/DynamoRIO/dynamorio/runs/2576605421?check_suite_focus=true

abhinav92003 avatar May 13 '21 15:05 abhinav92003

code_api,stack_size,loglevel|common.decode failed again on x86-32: https://github.com/DynamoRIO/dynamorio/runs/2622959469 Seems due to AVX-512 is not fully supported yet. 32-bit support is only partially implemented for scatter/gather. https://github.com/DynamoRIO/dynamorio/issues/2985#issuecomment-826247798

This test doesn't seem to be failing all the time though (no failure in last three master merges). So maybe it depends on the GA runner. AVX checks are enabled only if __AVX512F__ is defined.

common.decode doesn't use drcachesim, or I'd have thought it is related to the recently submitted PR#4912.

abhinav92003 avatar May 19 '21 18:05 abhinav92003

Oh actually the failure in previous comment is due to some other regex mismatch (the AVX message was in the expected template, not output by the test run): an extra Bad instruction, instance 64. Seems to have happened in https://github.com/DynamoRIO/dynamorio/issues/1807#issuecomment-839867318 too.

abhinav92003 avatar May 19 '21 19:05 abhinav92003

code_api,thread_private|common.decode failed on x86-32 Internal Error: DynamoRIO debug check failure: /home/runner/work/dynamorio/dynamorio/core/vmareas.c:10664 TEST(FRAG_WAS_DELETED, f->flags) || info.overlap https://github.com/DynamoRIO/dynamorio/runs/2823222611 The test failed once before as documented above, but with a different message.

prof_pcs,thread_private|common.nativeexec_retakeover_opt failed on x86-64 due to Internal Error: DynamoRIO debug check failure: /home/runner/work/dynamorio/dynamorio/core/unix/signal.c:5225 tr == NULL || tr->under_dynamo_control || IS_CLIENT_THREAD(dcontext) || sig == SUSPEND_SIGNAL https://github.com/DynamoRIO/dynamorio/runs/2823222433 Seems its first failure.

abhinav92003 avatar Jun 14 '21 20:06 abhinav92003

code_api,thread_private,tracedump_binary|common.fib failed in vs2017-32 https://github.com/DynamoRIO/dynamorio/runs/2897150100

Internal Error: DynamoRIO debug check failure: ..\core\vmareas.c:8312 !ok || !TEST(MEMPROT_WRITE, prot2) || TEST(FRAG_SELFMOD_SANDBOXED, *flags) || !INTERNAL_OPTION(hw_cache_consistency)

abhinav92003 avatar Jun 23 '21 17:06 abhinav92003

code_api,thread_private,tracedump_binary|common.fib failed in vs2017-32 again: https://github.com/DynamoRIO/dynamorio/runs/2955116184

abhinav92003 avatar Jun 30 '21 18:06 abhinav92003

Another case of prof_pcs|common.nativeexec_bindnow_opt failing in 64-bit x86: https://github.com/DynamoRIO/dynamorio/runs/3015061579?check_suite_focus=true

<Application /home/runner/work/dynamorio/dynamorio/build_debug-internal-32/suite/tests/bin/common.nativeexec (16894).  Internal Error: DynamoRIO debug check failure: /home/runner/work/dynamorio/dynamorio/core/unix/signal.c:5225 tr == NULL || tr->under_dynamo_control || IS_CLIENT_THREAD(dcontext) || sig == SUSPEND_SIGNAL

derekbruening avatar Jul 08 '21 14:07 derekbruening

code_api,thread_private,tracedump_binary|common.floatpc_xl8all failed on vs2017-64: https://github.com/DynamoRIO/dynamorio/runs/3023943403

FXSAVE64 inter is **incorrectly** handled

abhinav92003 avatar Jul 09 '21 16:07 abhinav92003

|common.nativeexec_exe failed on x86-64: https://github.com/DynamoRIO/dynamorio/runs/2964238871

Internal Error: DynamoRIO debug check failure: /home/runner/work/dynamorio/dynamorio/core/native_exec.c:272 i < MAX_NATIVE_RETSTACK

abhinav92003 avatar Jul 09 '21 16:07 abhinav92003

|common.nativeexec_exe failed on x86-64: https://github.com/DynamoRIO/dynamorio/runs/2964238871

Internal Error: DynamoRIO debug check failure: /home/runner/work/dynamorio/dynamorio/core/native_exec.c:272 i < MAX_NATIVE_RETSTACK

That is #4868

derekbruening avatar Jul 09 '21 16:07 derekbruening

code_api,thread_private|common.fib failed in vs2017-32: https://github.com/DynamoRIO/dynamorio/runs/3242195135 code_api,tracedump_text,tracedump_origins|common.fib failed in vs2017-64: https://github.com/DynamoRIO/dynamorio/runs/3242195000

abhinav92003 avatar Aug 05 '21 15:08 abhinav92003

code_api,thread_private|common.fib failed in vs2017-32: https://github.com/DynamoRIO/dynamorio/runs/3242195135 code_api,tracedump_text,tracedump_origins|common.fib failed in vs2017-64: https://github.com/DynamoRIO/dynamorio/runs/3242195000

Those are fairly basic tests so the failures are disturbing. I would request that we record the failure reason as well, to make it easier to compare failures across tests and distinguish different failures for the same test and evaluate severity of failures.

The 32-bit one is:

314: fib(5)=8
314: fib(15)=987
314: <Application D:\a\dynamorio\dynamorio\build_debug-internal-32\suite\tests\bin\common.fib.exe (5096).  Internal Error: DynamoRIO debug check failure: D:\a\dynamorio\dynamorio\core\vmareas.c:8312 !ok || !TEST(MEMPROT_WRITE, prot2) || TEST(FRAG_SELFMOD_SANDBOXED, *flags) || !INTERNAL_OPTION(hw_cache_consistency)
314: (Error occurred @34 frags in tid 3932)

The 64-bit looks like a timeout which could just be the tracedump being slow -- except there's no output at all for the test which makes it back to being disturbing:

] 90.01 sec

derekbruening avatar Aug 05 '21 16:08 derekbruening