Running DAQmx system tests on Linux crashes when cleaning up session utilities

[bkeryan]

When I run gdb --args ./SystemTestsRunner --gtest_filter="*DAQmx*" on Linux with fixes for the task leaks described in #936, the SessionUtilitiesLibraryAndService destructor crashes when closing the NISysCfg handle.

Loading internal symbols shows that libmxRmCfg.so is asserting while acquiring a global mutex, which appears to have been destructed. See AB#2402182 for more info.

This crash doesn't occur when running the run_ubuntu_system_tests action. I suspect that's because some of the MI drivers keep the SysAPI libraries loaded, which could affect the order in which finalizers are called.

[----------] Global test environment tear-down
[==========] 63 tests from 2 test suites ran. (11392 ms total)
[  PASSED  ] 63 tests.
[Thread 0x7ffff692c640 (LWP 27368) exited]
[Thread 0x7ffff693e640 (LWP 27367) exited]

Thread 1 "SystemTestsRunn" received signal SIGABRT, Aborted.
__pthread_kill_implementation (no_tid=0, signo=6, threadid=140737348143168) at ./nptl/pthread_kill.c:44
Download failed: Invalid argument.  Continuing without source file ./nptl/./nptl/pthread_kill.c.
44	./nptl/pthread_kill.c: No such file or directory.
(gdb) bt
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=140737348143168) at ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal (signo=6, threadid=140737348143168) at ./nptl/pthread_kill.c:78
#2  __GI___pthread_kill (threadid=140737348143168, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3  0x00007ffff7a8f476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4  0x00007ffff7a757f3 in __GI_abort () at ./stdlib/abort.c:79
#5  0x00007fff83bba5e0 in ?? () from /usr/lib/x86_64-linux-gnu/libmxRmCfg.so
#6  0x00007fff83bba605 in ?? () from /usr/lib/x86_64-linux-gnu/libmxRmCfg.so
#7  0x00007fff83a963a8 in ?? () from /usr/lib/x86_64-linux-gnu/libmxRmCfg.so
#8  0x00007ffff7057bd9 in ?? () from /usr/lib/x86_64-linux-gnu/libnisyscfg.so
#9  0x00007ffff70408ad in ?? () from /usr/lib/x86_64-linux-gnu/libnisyscfg.so
#10 0x00007ffff7045fe9 in NISysCfgCloseHandle () from /usr/lib/x86_64-linux-gnu/libnisyscfg.so
#11 0x0000555555ae5e2e in nidevice_grpc::SysCfgLibrary::CloseHandle (this=0x5555592b8990, syscfg_handle=0x7fff84146210) at /home/bkeryan/grpc-device/source/server/syscfg_library.cpp:75
#12 0x0000555555ae7344 in nidevice_grpc::SysCfgSessionHandler::clear_syscfg_session (this=0x5555592b8a38) at /home/bkeryan/grpc-device/source/server/syscfg_session_handler.cpp:38
#13 0x0000555555ae2beb in nidevice_grpc::(anonymous namespace)::SessionUtilitiesLibraryAndService::~SessionUtilitiesLibraryAndService (this=0x5555592b8980, __in_chrg=<optimized out>)
    at /home/bkeryan/grpc-device/source/server/session_utilities_service_registrar.cpp:29
#14 0x0000555555ae39c8 in __gnu_cxx::new_allocator<nidevice_grpc::(anonymous namespace)::SessionUtilitiesLibraryAndService>::destroy<nidevice_grpc::(anonymous namespace)::SessionUtilitiesLibraryAndService> (this=0x5555592b8980, __p=0x5555592b8980) at /usr/include/c++/11/ext/new_allocator.h:168
#15 0x0000555555ae399b in std::allocator_traits<std::allocator<nidevice_grpc::(anonymous namespace)::SessionUtilitiesLibraryAndService> >::destroy<nidevice_grpc::(anonymous namespace)::SessionUtilitiesLibraryAndService> (__a=..., __p=0x5555592b8980) at /usr/include/c++/11/bits/alloc_traits.h:535
#16 0x0000555555ae385f in std::_Sp_counted_ptr_inplace<nidevice_grpc::(anonymous namespace)::SessionUtilitiesLibraryAndService, std::allocator<nidevice_grpc::(anonymous namespace)::SessionUtilitiesLibraryAndService>, (__gnu_cxx::_Lock_policy)2>::_M_dispose (this=0x5555592b8970) at /usr/include/c++/11/bits/shared_ptr_base.h:528
#17 0x000055555591e7d1 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x5555592b8970) at /usr/include/c++/11/bits/shared_ptr_base.h:168
#18 0x000055555591e6d9 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count (this=0x5555592aafd8, __in_chrg=<optimized out>) at /usr/include/c++/11/bits/shared_ptr_base.h:705
#19 0x000055555591e65c in std::__shared_ptr<void, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr (this=0x5555592aafd0, __in_chrg=<optimized out>) at /usr/include/c++/11/bits/shared_ptr_base.h:1154
#20 0x000055555591e67c in std::shared_ptr<void>::~shared_ptr (this=0x5555592aafd0, __in_chrg=<optimized out>) at /usr/include/c++/11/bits/shared_ptr.h:122
#21 0x000055555591f07a in __gnu_cxx::new_allocator<std::shared_ptr<void> >::destroy<std::shared_ptr<void> > (this=0x55555926a3c0, __p=0x5555592aafd0) at /usr/include/c++/11/ext/new_allocator.h:168
#22 0x000055555591eeb3 in std::allocator_traits<std::allocator<std::shared_ptr<void> > >::destroy<std::shared_ptr<void> > (__a=..., __p=0x5555592aafd0) at /usr/include/c++/11/bits/alloc_traits.h:535
#23 0x00005555559203f7 in std::vector<std::shared_ptr<void>, std::allocator<std::shared_ptr<void> > >::pop_back (this=0x55555926a3c0) at /usr/include/c++/11/bits/stl_vector.h:1229
#24 0x000055555591f78c in ni::tests::system::DeviceServer::~DeviceServer (this=0x5555591a5030, __in_chrg=<optimized out>) at /home/bkeryan/grpc-device/source/tests/system/device_server.cpp:52
#25 0x000055555591f830 in ni::tests::system::DeviceServer::~DeviceServer (this=0x5555591a5030, __in_chrg=<optimized out>) at /home/bkeryan/grpc-device/source/tests/system/device_server.cpp:54
#26 0x0000555555920d04 in std::default_delete<ni::tests::system::DeviceServer>::operator() (this=0x555559127420 <ni::tests::system::DeviceServerInterface::Singleton()::singleton>, __ptr=0x5555591a5030)
    at /usr/include/c++/11/bits/unique_ptr.h:85
#27 0x00005555559204d4 in std::unique_ptr<ni::tests::system::DeviceServer, std::default_delete<ni::tests::system::DeviceServer> >::~unique_ptr (
    this=0x555559127420 <ni::tests::system::DeviceServerInterface::Singleton()::singleton>, __in_chrg=<optimized out>) at /usr/include/c++/11/bits/unique_ptr.h:361
#28 0x00007ffff7a92495 in __run_exit_handlers (status=0, listp=0x7ffff7c66838 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true) at ./stdlib/exit.c:113
#29 0x00007ffff7a92610 in __GI_exit (status=<optimized out>) at ./stdlib/exit.c:143
#30 0x00007ffff7a76d97 in __libc_start_call_main (main=main@entry=0x55555591e357 <main(int, char**)>, argc=argc@entry=2, argv=argv@entry=0x7fffffffe088) at ../sysdeps/nptl/libc_start_call_main.h:74
#31 0x00007ffff7a76e40 in __libc_start_main_impl (main=0x55555591e357 <main(int, char**)>, argc=2, argv=0x7fffffffe088, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, 
    stack_end=0x7fffffffe078) at ../csu/libc-start.c:392
#32 0x000055555591e025 in _start ()
(gdb) 

OS: Ubuntu 22.04 installed via Hyper-V Quick Create.

Installed NI software: LabVIEW 2023 Pro, NI-DAQmx 23.3.

AB#2402182