dynamorio icon indicating copy to clipboard operation
dynamorio copied to clipboard

x86-32 passing tests regression

Open edeiana opened this issue 2 months ago • 3 comments

Several tests failed at the same time for PR #7691 (https://github.com/DynamoRIO/dynamorio/actions/runs/18724532585/job/53405532740?pr=7691)

code_api|tool.drcacheoff.burst_replaceall 
code_api|tool.drcacheoff.burst_syscall_inject 
code_api|tool.drcacheoff.legacy 
code_api|tool.drcacheoff.func_view_noret 
code_api|tool.drcacheoff.altbindir 

On top of:

code_api|client.drx-scattergather 
code_api|client.drx-scattergather-bbdup 
code_api|sample.memval_simple_scattergather 
code_api|tool.drcachesim.scattergather-x86 

Already reported in #7678 .

code_api|tool.drcacheoff.burst_syscall_inject: trace invariant failure.

2025-10-22T17:36:10.9025126Z 373: CMake Error at /home/runner/work/dynamorio/dynamorio/suite/tests/process_cmdline.cmake:109 (message):
2025-10-22T17:36:10.9026092Z 373:   *** cmd failed (Subprocess aborted): Collecting a trace...
2025-10-22T17:36:10.9026590Z 373: 
2025-10-22T17:36:10.9026884Z 373:   Done collecting trace
2025-10-22T17:36:10.9027242Z 373: 
2025-10-22T17:36:10.9027605Z 373:   Testing system call trace template injection...
2025-10-22T17:36:10.9028077Z 373: 
2025-10-22T17:36:10.9028529Z 373:   Trace invariant failure in T27411 at ref # 100 at instruction # 49 (1
2025-10-22T17:36:10.9029335Z 373:   instrs since timestamp 152329120): Kernel trace-end branch marker does not
2025-10-22T17:36:10.9029956Z 373:   match next pc
2025-10-22T17:36:10.9030272Z 373: 
2025-10-22T17:36:10.9030528Z 373:   ***
2025-10-22T17:36:10.9030800Z 373: 
2025-10-22T17:36:10.9031102Z 373: Call Stack (most recent call first):
2025-10-22T17:36:10.9031879Z 373:   /home/runner/work/dynamorio/dynamorio/suite/tests/runmulti.cmake:57 (process_cmdline)
2025-10-22T17:36:10.9032619Z 373: 
2025-10-22T17:36:10.9032950Z 373: 
2025-10-22T17:36:10.9033570Z 371/469 Test #373: code_api|tool.drcacheoff.burst_syscall_inject ....................***Failed    0.35 sec

code_api|tool.drcacheoff.legacy: couldn't open trace file.

2025-10-22T17:37:43.4870066Z 459: Test timeout computed to be: 120
2025-10-22T17:37:43.4870438Z 459: Unexpected trace sequence
2025-10-22T17:37:43.4871456Z 459: Failed to initialize scheduler: Failed to open /home/runner/work/dynamorio/dynamorio/clients/drcachesim/tests/offline-legacy-trace.gz (was RLIMIT_NOFILE exceeded?)
2025-10-22T17:37:43.4872512Z 459: ERROR: failed to initialize analyzer
2025-10-22T17:37:43.4873396Z       Test #459: code_api|tool.drcacheoff.legacy ..................................***Failed  Required regular expression not found. Regex=[^

tool.drcacheoff.func_view_noret: couldn't open trace file.

2025-10-22T17:37:43.5172008Z         Start 460: code_api|tool.drcacheoff.func_view_noret
2025-10-22T17:37:43.5172317Z 
2025-10-22T17:37:43.5188607Z 460: Test command: /home/runner/work/dynamorio/dynamorio/build_debug-internal-32/bin32/runstats "-s" "90" "-killpg" "-silent" "-env" "LD_LIBRARY_PATH" "/home/runner/work/dynamorio/dynamorio/build_debug-internal-32/lib32/debug:/home/runner/work/dynamorio/dynamorio/build_debug-internal-32/ext/lib32/debug:" "-env" "DYNAMORIO_OPTIONS" "-stderr_mask 0xC -dumpcore_mask 0 -code_api" "/home/runner/work/dynamorio/dynamorio/build_debug-internal-32/clients/bin32/drmemtrace_launcher" "-indir" "/home/runner/work/dynamorio/dynamorio/clients/drcachesim/tests/offline-func-trace" "-tool" "func_view"
2025-10-22T17:37:43.5191871Z 460: Test timeout computed to be: 120
2025-10-22T17:37:43.5192244Z 460: Unexpected trace sequence
2025-10-22T17:37:43.5193644Z 460: Failed to initialize scheduler: Failed to open /home/runner/work/dynamorio/dynamorio/clients/drcachesim/tests/offline-func-trace/trace/drmemtrace.tool.fib_plus.125254.3691.trace.gz (was RLIMIT_NOFILE exceeded?)
2025-10-22T17:37:43.5194956Z 460: ERROR: failed to initialize analyzer
2025-10-22T17:37:43.5195887Z       Test #460: code_api|tool.drcacheoff.func_view_noret .........................***Failed  Required regular expression not found.

code_api|tool.drcacheoff.altbindir: trying to read an AARCH64 trace in x86 tests???

2025-10-22T17:37:43.5606796Z 461: CMake Error at /home/runner/work/dynamorio/dynamorio/suite/tests/runmulti.cmake:72 (message):
2025-10-22T17:37:43.5607758Z 461:   output |ERROR: failed to initialize analyzer: Directory setup failed:
2025-10-22T17:37:43.5608416Z 461:   Failed sanity checks for thread log file
2025-10-22T17:37:43.5609638Z 461:   /home/runner/work/dynamorio/dynamorio/build_debug-internal-32/suite/tests/drmemtrace.altbindir.aarch64/raw/drmemtrace.threadsig.16274.6003.raw.gz:
2025-10-22T17:37:43.5611069Z 461:   Architecture mismatch: trace recorded on aarch64 but tools built for i386
2025-10-22T17:37:43.5611699Z 461: 
2025-10-22T17:37:43.5612169Z 461:   | failed to match expected output |ERROR: failed to initialize analyzer:
2025-10-22T17:37:43.5613140Z 461:   Failed to create analysis tool: Tool failed to initialize: Failed to load
2025-10-22T17:37:43.5613915Z 461:   binaries: Failed to map module /tmp/nonexistent/threadsig
2025-10-22T17:37:43.5614459Z 461: 
2025-10-22T17:37:43.5614714Z 461:   |
2025-10-22T17:37:43.5614986Z 461: 
2025-10-22T17:37:43.5615248Z 461: 
2025-10-22T17:37:43.5616032Z       Test #461: code_api|tool.drcacheoff.altbindir ...............................***Failed    0.01 sec

edeiana avatar Oct 23 '25 00:10 edeiana

I am baffled. All tests passed on second re-try.

edeiana avatar Oct 23 '25 00:10 edeiana

Same tests failed on master merge: https://github.com/DynamoRIO/dynamorio/commit/0caf3965ad9c3099e18568a387d9c4a2d4b63d9f.

2025-12-03T01:15:02.3683820Z ====> FAILURE in debug-internal-32 <====
2025-12-03T01:15:02.3684145Z debug-internal-32: 457 tests passed, **** 13 tests failed, of which 2 were flaky, but ignoring 2 for i#2941: ****
2025-12-03T01:15:02.3684281Z 	(ignore: i#2941) 	code_api|common.decode-stress 
2025-12-03T01:15:02.3684399Z 	(ignore: i#2941) 	code_api|client.exception 
2025-12-03T01:15:02.3684517Z 	code_api|client.drx-scattergather 
2025-12-03T01:15:02.3684633Z 	code_api|client.drx-scattergather-bbdup 
2025-12-03T01:15:02.3684756Z 	code_api|sample.memval_simple_scattergather 
2025-12-03T01:15:02.3684882Z 	code_api|tool.drcachesim.scattergather-x86 
2025-12-03T01:15:02.3685011Z 	code_api|tool.drcacheoff.burst_replaceall 
2025-12-03T01:15:02.3685139Z 	code_api|tool.drcacheoff.burst_syscall_inject 
2025-12-03T01:15:02.3685245Z 	code_api|api.detach_spawn_stress_FLAKY 
2025-12-03T01:15:02.3685378Z 	code_api|api.static_maps_mixup_novars_FLAKY 
2025-12-03T01:15:02.3685484Z 	code_api|tool.drcacheoff.legacy 
2025-12-03T01:15:02.3685598Z 	code_api|tool.drcacheoff.func_view_noret 
2025-12-03T01:15:02.3685708Z 	code_api|tool.drcacheoff.altbindir 
2025-12-03T01:15:02.3685912Z WARNING: maximum warning/error limit hit for release-external-32!

https://github.com/DynamoRIO/dynamorio/actions/runs/19878728699/job/56971949068 Raw log: https://productionresultssa9.blob.core.windows.net/actions-results/92c76fb9-89cd-4d32-8d5d-bf30c220323c/workflow-job-run-5fb50165-ab09-56b3-b3d6-afe4c6200403/logs/job/job-logs.txt?rsct=text%2Fplain&se=2025-12-03T01%3A38%3A01Z&sig=nM%2FFw1l9NXSbDEiY8kpHmEeA1SlkLiUmNshrI931IAg%3D&ske=2025-12-03T12%3A46%3A22Z&skoid=ca7593d4-ee42-46cd-af88-8b886a2f84eb&sks=b&skt=2025-12-03T00%3A46%3A22Z&sktid=398a6654-997b-47e9-b12b-9515b896b4de&skv=2025-11-05&sp=r&spr=https&sr=b&st=2025-12-03T01%3A27%3A56Z&sv=2025-11-05

edeiana avatar Dec 03 '25 01:12 edeiana

The tests listed in the first comment above also failed in the x86-32 test suite in the master merge for #7724.

2025-11-24T16:05:53.1183906Z 	code_api|tool.drcacheoff.burst_replaceall 
2025-11-24T16:05:53.1184299Z 	code_api|tool.drcacheoff.burst_syscall_inject 
...
2025-11-24T16:05:53.1185076Z 	code_api|tool.drcacheoff.legacy 
2025-11-24T16:05:53.1185309Z 	code_api|tool.drcacheoff.func_view_noret 
2025-11-24T16:05:53.1185568Z 	code_api|tool.drcacheoff.altbindir

https://github.com/DynamoRIO/dynamorio/actions/runs/19640496152/job/56242338756

abhinav92003 avatar Dec 03 '25 14:12 abhinav92003