CI jobs sometimes keep running

[maskit]

Some unit test may have a dead lock issue or something. I feel like log output always stops here when CI job is stuck.

make[4]: Entering directory '/home/jenkins/workspace/master/os_build/src/iocore/cache'
PASS: test_Alternate_S_to_L
PASS: test_Cache
PASS: test_Alternate_L_to_S
PASS: test_RWW
PASS: test_Alternate_L_to_S_remove_L
PASS: test_Alternate_L_to_S_remove_S
PASS: test_Alternate_S_to_L_remove_S
PASS: test_Alternate_S_to_L_remove_L
PASS: test_Update_L_to_S
PASS: test_Update_header

[maskit]

Build #18747 master ubuntu:23.04 gcc debug

[JosiahWI]

CTest caught it on #10097. fedora38 gcc debug

The test output was very long, so I've included here only a snippet of a few lines right before the hang.

[Jul 26 18:10:26.682] Alternate_S_to_ DEBUG: <CacheRead.cc:413 (openReadFromWriter)> (cache_read_agg) 0x7f18b2bbd740: key: 72AF8021 http passed stage 1, closed: 1, frag: 10
[Jul 26 18:10:26.682] Alternate_S_to_ DEBUG: <CacheRead.cc:468 (openReadFromWriter)> (cache_read_agg) 0x7f18b2bbd740: key: 72AF8021 closed: 1, fragment: 10, len: 10485760 starting first fragment
[Jul 26 18:10:28.388] [ET_AIO 1:7] DEBUG: <CacheWrite.cc:391 (aggWriteDone)> (cache_agg) Dir /tmp/cachetest.QlOWjw/var/trafficserver/cache.db 16384:32766, Write: 11223040, last Write: 9125888
[Jul 26 18:10:28.388] [ET_AIO 1:7] DEBUG: <CacheWrite.cc:1059 (aggWrite)> (agg_read) copying: 0, 11223040, key: -1266361926
WARNING: waiting on lock UnixEThread.cc:134 (process_event) for UNKNOWN
WARNING: waiting on lock UnixEThread.cc:134 (process_event) for UNKNOWN
WARNING: waiting on lock UnixEThread.cc:134 (process_event) for UNKNOWN
WARNING: waiting on lock UnixEThread.cc:134 (process_event) for UNKNOWN
[Jul 26 18:10:28.427] [ET_NET 0] DEBUG: <CacheWrite.cc:1059 (aggWrite)> (agg_read) copying: 4096, 11227136, key: -1266361926
[Jul 26 18:10:28.427] [ET_NET 0] DEBUG: <CacheWrite.cc:1200 (openWriteCloseDir)> (cache_stats) Fragment = 11
[Jul 26 18:10:28.427] [ET_NET 0] DEBUG: <P_CacheInternal.h:580 (free_CacheVC)> (cache_free) free 0x7f18b2bbdb10
[Jul 26 18:11:25.814] [ET_AIO 7:7] DEBUG: <CacheWrite.cc:391 (aggWriteDone)> (cache_agg) Dir /tmp/cachetest.QlOWjw/var/trafficserver/cache.db 16384:32766, Write: 11235328, last Write: 11223040


97% tests passed, 1 tests failed out of 32

Total Test time (real) = 1503.13 sec

The following tests FAILED:
	  7 - test_cache_Alternate_S_to_L (Timeout)
Errors while running CTest

Full log: https://ci.trafficserver.apache.org/job/Github_Builds/job/cmake/1075/consoleFull

[maskit]

Build #19461 master rockylinux:8 clang debug asan

[maskit]

Build 22076 master ubuntu:22.04 gcc debug

RPRINT Cache_dir: corrupt_bucket test
    REGRESSION_RESULT Cache_dir:                                PASSED
REGRESSION TEST SDK_API_TSHttpConnectServerIntercept started

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. Marking it stale to flag it for further consideration by the community.