google-cloud-cpp v3: investigate flaky `prepare-for-v3-0-0-cmake-install-pr` check

Noticed in https://github.com/googleapis/google-cloud-cpp/pull/15797/checks?check_run_id=56478992529. This check is occasionally failing with an exit 134 error code.
Dec 01 '25 17:12 mpeddada1
Different build, but same failure when running quickstart https://github.com/googleapis/google-cloud-cpp/pull/15808/checks?check_run_id=56825021178
Dec 01 '25 20:12 scotthart
Was able to reproduce the issue with asan enabled:
docker:demo-rockylinux-9$ /workspace/cmake-out/quickstart/cmake-bigquery/quickstart cloud-cpp-testing-resources projects/bigquery-public-data/datasets/usa_names/tables/usa_1910_current
6311504 rows read from table: projects/bigquery-public-data/datasets/usa_names/tables/usa_1910_current
=================================================================
==554528==ERROR: AddressSanitizer: attempting double-free on 0x602000058010 in thread T0:
    #0 0x7f69b28206b7 in free (/lib64/libasan.so.6+0xb46b7)
    #1 0x7f69ad01159f in init_thread_deregister.isra.0 crypto/initthread.c:462
    #2 0x7f69acf21f3c in ossl_cleanup_thread crypto/initthread.c:213
    #3 0x7f69acf21f3c in OPENSSL_cleanup crypto/init.c:442
    #4 0x7f69acf21f3c in OPENSSL_cleanup crypto/init.c:353
    #5 0x7f69ab0362dc in __run_exit_handlers /usr/src/debug/glibc-2.34-168.el9_6.24.x86_64/stdlib/exit.c:126
    #6 0x7f69ab03642f in __GI_exit /usr/src/debug/glibc-2.34-168.el9_6.24.x86_64/stdlib/exit.c:156
    #7 0x7f69ab01e5d6 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:74
    #8 0x7f69ab01e67f in __libc_start_main_impl ../csu/libc-start.c:389
    #9 0x45cc04 in _start (/workspace/cmake-out/quickstart/cmake-bigquery/quickstart+0x45cc04)
 
0x602000058010 is located 0 bytes inside of 8-byte region [0x602000058010,0x602000058018)
freed by thread T12 here:
    #0 0x7f69b28206b7 in free (/lib64/libasan.so.6+0xb46b7)
    #1 0x7f69ab07c7c0 in __GI___nptl_deallocate_tsd /usr/src/debug/glibc-2.34-168.el9_6.24.x86_64/nptl/nptl_deallocate_tsd.c:74
    #2 0x7f69ab07c7c0 in __GI___nptl_deallocate_tsd /usr/src/debug/glibc-2.34-168.el9_6.24.x86_64/nptl/nptl_deallocate_tsd.c:23
 
previously allocated by thread T12 here:
    #0 0x7f69b2820a07 in __interceptor_malloc (/lib64/libasan.so.6+0xb4a07)
    #1 0x7f69acf1b4b0 in CRYPTO_malloc (/lib64/libcrypto.so.3+0x1cc4b0)
 
Thread T12 created by T0 here:
    #0 0x7f69b27c47d5 in pthread_create (/lib64/libasan.so.6+0x587d5)
    #1 0x7f69ad3ab9a7 in ThreadInternalsPosix /var/tmp/build/grpc/src/core/util/posix/thd.cc:113
    #2 0x7f69ad3acb43 in grpc_core::Thread::Thread(char const*, void (*)(void*), void*, bool*, grpc_core::Thread::Options const&) /var/tmp/build/grpc/src/core/util/posix/thd.cc:220
    #3 0x7f69b02c1738 in grpc_event_engine::experimental::WorkStealingThreadPool::WorkStealingThreadPoolImpl::StartThread() /var/tmp/build/grpc/src/core/lib/event_engine/thread_pool/work_stealing_thread_pool.cc:255
    #4 0x7f69b02c0f12 in grpc_event_engine::experimental::WorkStealingThreadPool::WorkStealingThreadPoolImpl::Start() /var/tmp/build/grpc/src/core/lib/event_engine/thread_pool/work_stealing_thread_pool.cc:232
    #5 0x7f69b02c0081 in grpc_event_engine::experimental::WorkStealingThreadPool::WorkStealingThreadPool(unsigned long) /var/tmp/build/grpc/src/core/lib/event_engine/thread_pool/work_stealing_thread_pool.cc:177
    #6 0x7f69b02bf656 in void __gnu_cxx::new_allocator<grpc_event_engine::experimental::WorkStealingThreadPool>::construct<grpc_event_engine::experimental::WorkStealingThreadPool, unsigned long&>(grpc_event_engine::experimental::WorkStealingThreadPool*, unsigned long&) /usr/include/c++/11/ext/new_allocator.h:162
    #7 0x7f69b02bf4d9 in void std::allocator_traits<std::allocator<grpc_event_engine::experimental::WorkStealingThreadPool> >::construct<grpc_event_engine::experimental::WorkStealingThreadPool, unsigned long&>(std::allocator<grpc_event_engine::experimental::WorkStealingThreadPool>&, grpc_event_engine::experimental::WorkStealingThreadPool*, unsigned long&) /usr/include/c++/11/bits/alloc_traits.h:516
    #8 0x7f69b02bf29c in std::_Sp_counted_ptr_inplace<grpc_event_engine::experimental::WorkStealingThreadPool, std::allocator<grpc_event_engine::experimental::WorkStealingThreadPool>, (__gnu_cxx::_Lock_policy)2>::_Sp_counted_ptr_inplace<unsigned long&>(std::allocator<grpc_event_engine::experimental::WorkStealingThreadPool>, unsigned long&) /usr/include/c++/11/bits/shared_ptr_base.h:519
    #9 0x7f69b02bee22 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count<grpc_event_engine::experimental::WorkStealingThreadPool, std::allocator<grpc_event_engine::experimental::WorkStealingThreadPool>, unsigned long&>(grpc_event_engine::experimental::WorkStealingThreadPool*&, std::_Sp_alloc_shared_tag<std::allocator<grpc_event_engine::experimental::WorkStealingThreadPool> >, unsigned long&) /usr/include/c++/11/bits/shared_ptr_base.h:650
    #10 0x7f69b02bebe4 in std::__shared_ptr<grpc_event_engine::experimental::WorkStealingThreadPool, (__gnu_cxx::_Lock_policy)2>::__shared_ptr<std::allocator<grpc_event_engine::experimental::WorkStealingThreadPool>, unsigned long&>(std::_Sp_alloc_shared_tag<std::allocator<grpc_event_engine::experimental::WorkStealingThreadPool> >, unsigned long&) /usr/include/c++/11/bits/shared_ptr_base.h:1342
    #11 0x7f69b02beab8 in std::shared_ptr<grpc_event_engine::experimental::WorkStealingThreadPool>::shared_ptr<std::allocator<grpc_event_engine::experimental::WorkStealingThreadPool>, unsigned long&>(std::_Sp_alloc_shared_tag<std::allocator<grpc_event_engine::experimental::WorkStealingThreadPool> >, unsigned long&) /usr/include/c++/11/bits/shared_ptr.h:409
    #12 0x7f69b02be91c in std::shared_ptr<grpc_event_engine::experimental::WorkStealingThreadPool> std::allocate_shared<grpc_event_engine::experimental::WorkStealingThreadPool, std::allocator<grpc_event_engine::experimental::WorkStealingThreadPool>, unsigned long&>(std::allocator<grpc_event_engine::experimental::WorkStealingThreadPool> const&, unsigned long&) /usr/include/c++/11/bits/shared_ptr.h:863
    #13 0x7f69b02be808 in std::shared_ptr<grpc_event_engine::experimental::WorkStealingThreadPool> std::make_shared<grpc_event_engine::experimental::WorkStealingThreadPool, unsigned long&>(unsigned long&) /usr/include/c++/11/bits/shared_ptr.h:879
    #14 0x7f69b02be5bd in grpc_event_engine::experimental::MakeThreadPool(unsigned long) /var/tmp/build/grpc/src/core/lib/event_engine/thread_pool/thread_pool_factory.cc:25
    #15 0x7f69b022e456 in grpc_event_engine::experimental::PosixEventEngine::PosixEventEngine() /var/tmp/build/grpc/src/core/lib/event_engine/posix_engine/posix_engine.cc:579
    #16 0x7f69b022d9d7 in grpc_event_engine::experimental::PosixEventEngine::MakePosixEventEngine() /var/tmp/build/grpc/src/core/lib/event_engine/posix_engine/posix_engine.cc:553
    #17 0x7f69b01d3c25 in grpc_event_engine::experimental::DefaultEventEngineFactory() /var/tmp/build/grpc/src/core/lib/event_engine/default_event_engine_factory.cc:48
    #18 0x7f69b01cee60 in grpc_event_engine::experimental::CreateEventEngine() /var/tmp/build/grpc/src/core/lib/event_engine/default_event_engine.cc:84
    #19 0x7f69b01cf18c in grpc_event_engine::experimental::GetDefaultEventEngine() /var/tmp/build/grpc/src/core/lib/event_engine/default_event_engine.cc:106
    #20 0x7f69afa12d64 in grpc_core::TokenFetcherCredentials::TokenFetcherCredentials(std::shared_ptr<grpc_event_engine::experimental::EventEngine>, bool) /var/tmp/build/grpc/src/core/credentials/call/token_fetcher/token_fetcher_credentials.cc:226
    #21 0x7f69af9ffe45 in grpc_core::Oauth2TokenFetcherCredentials::Oauth2TokenFetcherCredentials() /var/tmp/build/grpc/src/core/credentials/call/oauth2/oauth2_credentials.h:93
    #22 0x7f69af9f624a in grpc_google_refresh_token_credentials::grpc_google_refresh_token_credentials(grpc_auth_refresh_token) /var/tmp/build/grpc/src/core/credentials/call/oauth2/oauth2_credentials.cc:350
    #23 0x7f69afa00ac2 in grpc_core::RefCountedPtr<grpc_google_refresh_token_credentials> grpc_core::MakeRefCounted<grpc_google_refresh_token_credentials, grpc_auth_refresh_token&>(grpc_auth_refresh_token&) /var/tmp/build/grpc/src/core/util/ref_counted_ptr.h:369
    #24 0x7f69af9f760d in grpc_refresh_token_credentials_create_from_auth_refresh_token(grpc_auth_refresh_token) /var/tmp/build/grpc/src/core/credentials/call/oauth2/oauth2_credentials.cc:395
    #25 0x7f69afa4d62c in create_default_creds_from_path /var/tmp/build/grpc/src/core/credentials/transport/google_default/google_default_credentials.cc:305
    #26 0x7f69afa4e5e5 in make_default_call_creds /var/tmp/build/grpc/src/core/credentials/transport/google_default/google_default_credentials.cc:415
    #27 0x7f69afa4f464 in grpc_google_default_credentials_create /var/tmp/build/grpc/src/core/credentials/transport/google_default/google_default_credentials.cc:448
    #28 0x7f69b258defb in grpc::GoogleDefaultCredentials() /var/tmp/build/grpc/src/cpp/client/secure_credentials.cc:88
    #29 0x50f4d6 in visit /workspace/google/cloud/internal/unified_grpc_credentials.cc:106
    #30 0x62a116 in google::cloud::v3_0::internal::GoogleDefaultCredentialsConfig::dispatch(google::cloud::v3_0::internal::CredentialsVisitor&) const /workspace/google/cloud/internal/credentials_impl.h:96
    #31 0x628ba8 in google::cloud::v3_0::internal::CredentialsVisitor::dispatch(google::cloud::v3_0::Credentials const&, google::cloud::v3_0::internal::CredentialsVisitor&) /workspace/google/cloud/internal/credentials_impl.cc:51
    #32 0x510224 in google::cloud::v3_0::internal::CreateAuthenticationStrategy(google::cloud::v3_0::Credentials const&, google::cloud::v3_0::CompletionQueue, google::cloud::v3_0::Options) /workspace/google/cloud/internal/unified_grpc_credentials.cc:134
    #33 0x50ef4b in google::cloud::v3_0::internal::CreateAuthenticationStrategy(google::cloud::v3_0::CompletionQueue, google::cloud::v3_0::Options const&) /workspace/google/cloud/internal/unified_grpc_credentials.cc:81
    #34 0x46ca65 in google::cloud::bigquery_storage_v1::v3_0::MakeBigQueryReadConnection(google::cloud::v3_0::Options) /workspace/google/cloud/bigquery/storage/v1/bigquery_read_connection.cc:72
    #35 0x45d376 in main /workspace/google/cloud/bigquery/quickstart/quickstart.cc:46
    #36 0x7f69ab01e5cf in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58
Dec 01 '25 23:12 scotthart