mpich
mpich copied to clipboard
pmodels
Jenkins/bug: pt2pt/rqfreeb with ch4-ucx am-only
not ok 1698 - ./pt2pt/rqfreeb 4
---
Directory: ./pt2pt
File: rqfreeb
Num-procs: 4
Timeout: 180
Date: "Thu Jan 26 16:10:57 2023"
...
## Test output (expected 'No Errors'):
## [pmrs-centos64-240-04:436143:0:436143] ucp_worker.c:2781 Assertion `worker->inprogress++ == 0' failed
## No Errors
##
## /var/lib/jenkins-slave/workspace/mpich-review-ch4-ucx/jenkins_configure/am-only/label/centos64_review/modules/ucx/src/ucp/core/ucp_worker.c: [ ucp_worker_progress() ]
## ...
## 2778 UCP_WORKER_THREAD_CS_ENTER_CONDITIONAL(worker);
## 2779
## 2780 /* check that ucp_worker_progress is not called from within ucp_worker_progress */
## ==> 2781 ucs_assert(worker->inprogress++ == 0);
## 2782 count = uct_worker_progress(worker->uct);
## 2783 ucs_async_check_miss(&worker->async);
## 2784
##
## ==== backtrace (tid: 436143) ====
## 0 0x0000000000054c29 ucp_worker_progress() /var/lib/jenkins-slave/workspace/mpich-review-ch4-ucx/jenkins_configure/am-only/label/centos64_review/modules/ucx/src/ucp/core/ucp_worker.c:2781
## 1 0x00000000001eb77e MPID_Progress_test.constprop.40() commutil.c:0
## 2 0x00000000001ef715 MPIR_Comm_delete_internal() :0
## 3 0x00000000002a4ddd recv_target_cmpl_cb() mpidig_pt2pt_callbacks.c:0
## 4 0x00000000002a729e MPIDIG_send_target_msg_cb() :0
## 5 0x00000000002c4132 MPIDI_UCX_am_handler() :0
## 6 0x00000000000298da ucp_am_invoke_cb() /var/lib/jenkins-slave/workspace/mpich-review-ch4-ucx/jenkins_configure/am-only/label/centos64_review/modules/ucx/src/ucp/core/ucp_am.c:1234
## 7 0x00000000000298da ucp_am_handler_common() /var/lib/jenkins-slave/workspace/mpich-review-ch4-ucx/jenkins_configure/am-only/label/centos64_review/modules/ucx/src/ucp/core/ucp_am.c:1289
## 8 0x00000000000298da ucp_am_handler() /var/lib/jenkins-slave/workspace/mpich-review-ch4-ucx/jenkins_configure/am-only/label/centos64_review/modules/ucx/src/ucp/core/ucp_am.c:1340
## 9 0x0000000000019729 uct_iface_invoke_am() /var/lib/jenkins-slave/workspace/mpich-review-ch4-ucx/jenkins_configure/am-only/label/centos64_review/modules/ucx/src/uct/base/uct_iface.h:861
## 10 0x0000000000019729 uct_mm_iface_invoke_am() /var/lib/jenkins-slave/workspace/mpich-review-ch4-ucx/jenkins_configure/am-only/label/centos64_review/modules/ucx/src/uct/sm/mm/base/mm_iface.h:256
## 11 0x0000000000019729 uct_mm_iface_process_recv() /var/lib/jenkins-slave/workspace/mpich-review-ch4-ucx/jenkins_configure/am-only/label/centos64_review/modules/ucx/src/uct/sm/mm/base/mm_iface.c:272
## 12 0x0000000000019729 uct_mm_iface_poll_fifo() /var/lib/jenkins-slave/workspace/mpich-review-ch4-ucx/jenkins_configure/am-only/label/centos64_review/modules/ucx/src/uct/sm/mm/base/mm_iface.c:304
## 13 0x0000000000019729 uct_mm_iface_progress() /var/lib/jenkins-slave/workspace/mpich-review-ch4-ucx/jenkins_configure/am-only/label/centos64_review/modules/ucx/src/uct/sm/mm/base/mm_iface.c:357
## 14 0x0000000000054bca ucs_callbackq_dispatch() /var/lib/jenkins-slave/workspace/mpich-review-ch4-ucx/jenkins_configure/am-only/label/centos64_review/modules/ucx/src/ucs/datastruct/callbackq.h:211
## 15 0x0000000000054bca uct_worker_progress() /var/lib/jenkins-slave/workspace/mpich-review-ch4-ucx/jenkins_configure/am-only/label/centos64_review/modules/ucx/src/uct/api/uct.h:2638
## 16 0x0000000000054bca ucp_worker_progress() /var/lib/jenkins-slave/workspace/mpich-review-ch4-ucx/jenkins_configure/am-only/label/centos64_review/modules/ucx/src/ucp/core/ucp_worker.c:2782
## 17 0x00000000002c2c64 MPIDI_UCX_mpi_finalize_hook() :0
## 18 0x000000000028fa3f MPID_Finalize() :0
## 19 0x000000000021b62c MPII_Finalize() :0
## 20 0x00000000000d0a96 PMPI_Finalize() ???:0
## 21 0x000000000040272c MTest_Finalize() /var/lib/jenkins-slave/workspace/mpich-review-ch4-ucx/jenkins_configure/am-only/label/centos64_review/test/mpi/util/mtest.c:208
## 22 0x0000000000401c4a main() /var/lib/jenkins-slave/workspace/mpich-review-ch4-ucx/jenkins_configure/am-only/label/centos64_review/test/mpi/pt2pt/rqfreeb.c:117
## 23 0x0000000000022505 __libc_start_main() ???:0
## 24 0x0000000000402001 _start() ???:0
## =================================
##
## ===================================================================================
## = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
## = PID 436143 RUNNING AT pmrs-centos64-240-04.cels.anl.gov
## = EXIT CODE: 6
## = CLEANING UP REMAINING PROCESSES
## = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
## ===================================================================================
## YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Terminated (signal 15)
## This typically refers to a problem with your application.
## Please see the FAQ page for debugging suggestions
This may not be the same. In testing PR #6510, we hit on ch4-ucx-asan:
not ok 1837 - ./pt2pt/rqfreeb 4
---
Directory: ./pt2pt
File: rqfreeb
Num-procs: 4
Timeout: 180
Date: "Tue Jul 11 15:59:55 2023"
...
## Test output (expected 'No Errors'):
## No Errors
##
## =================================================================
## ==442297==ERROR: LeakSanitizer: detected memory leaks
##
## Direct leak of 16432 byte(s) in 1 object(s) allocated from:
## #0 0x7f8809639ae8 in __interceptor_malloc ../../../../libsanitizer/asan/asan_malloc_linux.cc:144
## #1 0x7f8806ae7702 in MPL_malloc /var/lib/jenkins-slave/workspace/mpich-review-ch4-ofi/jenkins_configure/asan/label/centos64_review/src/mpl/include/mpl_trmem.h:373
## #2 0x7f8806ae7702 in MPIDIU_avt_init src/mpid/ch4/src/ch4_proc.c:168
## #3 0x7f8806ac6c55 in MPID_Init src/mpid/ch4/src/ch4_init.c:457
## #4 0x7f8806913043 in MPII_Init_thread src/mpi/init/mpir_init.c:239
## #5 0x7f88065c0539 in internal_Init_thread src/binding/c/c_binding.c:47162
## #6 0x7f88065c0539 in PMPI_Init_thread src/binding/c/c_binding.c:47217
## #7 0x4033a3 in MTest_Init_thread /var/lib/jenkins-slave/workspace/mpich-review-ch4-ofi/jenkins_configure/asan/label/centos64_review/test/mpi/util/mtest.c:84
## #8 0x403701 in MTest_Init /var/lib/jenkins-slave/workspace/mpich-review-ch4-ofi/jenkins_configure/asan/label/centos64_review/test/mpi/util/mtest.c:169
## #9 0x4025cc in main /var/lib/jenkins-slave/workspace/mpich-review-ch4-ofi/jenkins_configure/asan/label/centos64_review/test/mpi/pt2pt/rqfreeb.c:22
## #10 0x7f8805e55504 in __libc_start_main (/lib64/libc.so.6+0x22504)
##
## SUMMARY: AddressSanitizer: 16432 byte(s) leaked in 1 allocation(s).