yugabyte-db
yugabyte-db copied to clipboard
Published
20 hours ago •
yugabyte
yugabyte
[TSAN Unit Test Failure] ClientTest.TestServerTooBusyRetry
Jira Link: DB-3456
Description
The unit test ClientTest.TestServerTooBusyRetry has TSAN issues.
Stack trace: https://gist.github.com/shamanthchandra-yb/158d67cde23bcd7b2d941bf7d8ee3152
cc: @adithya-yb since you made recent changes to table_handle.cc
Note: It has failed 102 times recently
WARNING: ThreadSanitizer: data race (pid=31208)
Write of size 4 at 0x7f2f4fc72290 by thread T615:
#0 google::LogMessage::Init(char const*, int, int, void (google::LogMessage::*)()) <null> (libglog.so.0+0x19dac)
#1 google::LogMessageFatal::LogMessageFatal(char const*, int) <null> (libglog.so.0+0x1faa7)
#2 yb::client::TableIterator::HandleError(yb::Status const&) /nfusr/centos-gcp-cloud/jenkins-worker-33a2dy/jenkins/jenkins-github-yugabyte-db-centos-master-clang12-tsan-896/yugabyte-db/build/tsan-clang12-dynamic-ninja/../../src/yb/client/table_handle.cc:371:5 (libyb_client.so+0xb7346e)
#3 yb::client::TableIterator::IsFlushStatusOkOrHandleErrors(yb::client::FlushStatus) /nfusr/centos-gcp-cloud/jenkins-worker-33a2dy/jenkins/jenkins-github-yugabyte-db-centos-master-clang12-tsan-896/yugabyte-db/build/tsan-clang12-dynamic-ninja/../../src/yb/client/table_handle.cc:357:3 (libyb_client.so+0xb73e1c)
#4 yb::client::TableIterator::ExecuteOps() /nfusr/centos-gcp-cloud/jenkins-worker-33a2dy/jenkins/jenkins-github-yugabyte-db-centos-master-clang12-tsan-896/yugabyte-db/build/tsan-clang12-dynamic-ninja/../../src/yb/client/table_handle.cc:280:8 (libyb_client.so+0xb735dc)
#5 yb::client::TableIterator::TableIterator(yb::client::TableHandle const*, yb::client::TableIteratorOptions const&) /nfusr/centos-gcp-cloud/jenkins-worker-33a2dy/jenkins/jenkins-github-yugabyte-db-centos-master-clang12-tsan-896/yugabyte-db/build/tsan-clang12-dynamic-ninja/../../src/yb/client/table_handle.cc:267:7 (libyb_client.so+0xb73332)
#6 yb::client::TableRange::begin() const /nfusr/centos-gcp-cloud/jenkins-worker-33a2dy/jenkins/jenkins-github-yugabyte-db-centos-master-clang12-tsan-896/yugabyte-db/build/tsan-clang12-dynamic-ninja/../../src/yb/client/table_handle.h:234:12 (client-test+0x3e1ddc)
#7 boost::range_iterator<yb::client::TableRange const, void>::type boost::range_detail::range_begin<yb::client::TableRange const>(yb::client::TableRange const&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181649-c1907426fc-centos7-x86_64-clang12/installed/tsan/include/boost/range/begin.hpp:45:18 (client-test+0x420290)
#8 boost::range_iterator<yb::client::TableRange const, void>::type boost::range_adl_barrier::begin<yb::client::TableRange>(yb::client::TableRange const&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181649-c1907426fc-centos7-x86_64-clang12/installed/tsan/include/boost/range/begin.hpp:115:12 (client-test+0x41e5c0)
#9 boost::disable_if<boost::range_detail::has_member_size<yb::client::TableRange>, boost::range_size<yb::client::TableRange const>::type>::type boost::range_detail::range_calculate_size<yb::client::TableRange>(yb::client::TableRange const&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181649-c1907426fc-centos7-x86_64-clang12/installed/tsan/include/boost/range/size.hpp:49:34 (client-test+0x41e466)
#10 boost::range_size<yb::client::TableRange const>::type boost::size<yb::client::TableRange>(yb::client::TableRange const&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181649-c1907426fc-centos7-x86_64-clang12/installed/tsan/include/boost/range/size.hpp:71:16 (client-test+0x3e19c9)
#11 yb::client::(anonymous namespace)::CountRowsFromClient(yb::client::TableHandle const&, yb::YBConsistencyLevel, int, int) /nfusr/centos-gcp-cloud/jenkins-worker-33a2dy/jenkins/jenkins-github-yugabyte-db-centos-master-clang12-tsan-896/yugabyte-db/build/tsan-clang12-dynamic-ninja/../../src/yb/client/client-test.cc:529:10 (client-test+0x3bd906)
#12 yb::client::(anonymous namespace)::CountRowsFromClient(yb::client::TableHandle const&, int, int) /nfusr/centos-gcp-cloud/jenkins-worker-33a2dy/jenkins/jenkins-github-yugabyte-db-centos-master-clang12-tsan-896/yugabyte-db/build/tsan-clang12-dynamic-ninja/../../src/yb/client/client-test.cc:533:10 (client-test+0x3ac77e)
#13 yb::client::(anonymous namespace)::CountRowsFromClient(yb::client::TableHandle const&) /nfusr/centos-gcp-cloud/jenkins-worker-33a2dy/jenkins/jenkins-github-yugabyte-db-centos-master-clang12-tsan-896/yugabyte-db/build/tsan-clang12-dynamic-ninja/../../src/yb/client/client-test.cc:537:10 (client-test+0x3ac223)
#14 yb::client::(anonymous namespace)::CheckRowCount(yb::client::TableHandle const&) /nfusr/centos-gcp-cloud/jenkins-worker-33a2dy/jenkins/jenkins-github-yugabyte-db-centos-master-clang12-tsan-896/yugabyte-db/build/tsan-clang12-dynamic-ninja/../../src/yb/client/client-test.cc:544:3 (client-test+0x3cd259)
#15 yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_6::operator()() const /nfusr/centos-gcp-cloud/jenkins-worker-33a2dy/jenkins/jenkins-github-yugabyte-db-centos-master-clang12-tsan-896/yugabyte-db/build/tsan-clang12-dynamic-ninja/../../src/yb/client/client-test.cc:2169:11 (client-test+0x3cd174)
#16 void yb::TestThreadHolder::AddThreadFunctor<yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_6>(yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_6 const&)::'lambda'()::operator()() const /nfusr/centos-gcp-cloud/jenkins-worker-33a2dy/jenkins/jenkins-github-yugabyte-db-centos-master-clang12-tsan-896/yugabyte-db/build/tsan-clang12-dynamic-ninja/../../src/yb/util/test_thread_holder.h:63:7 (client-test+0x3cd09d)
#17 decltype(std::__1::forward<yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_6>(fp)(std::__1::forward<void yb::TestThreadHolder::AddThreadFunctor<yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_6>(yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_6 const&)::'lambda'()>(fp0)...)) std::__1::__invoke<void yb::TestThreadHolder::AddThreadFunctor<yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_6>(yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_6 const&)::'lambda'()>(yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_6&&, void yb::TestThreadHolder::AddThreadFunctor<yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_6>(yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_6 const&)::'lambda'()&&...) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181649-c1907426fc-centos7-x86_64-clang12/installed/tsan/libcxx/include/c++/v1/type_traits:3694:1 (client-test+0x3cd009)
#18 void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void yb::TestThreadHolder::AddThreadFunctor<yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_6>(yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_6 const&)::'lambda'()>(std::__1::tuple<yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_6, void yb::TestThreadHolder::AddThreadFunctor<yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_6>(yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_6 const&)::'lambda'()>&, std::__1::__tuple_indices<>) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181649-c1907426fc-centos7-x86_64-clang12/installed/tsan/libcxx/include/c++/v1/thread:280:5 (client-test+0x3ccfd1)
#19 void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void yb::TestThreadHolder::AddThreadFunctor<yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_6>(yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_6 const&)::'lambda'()> >(void*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181649-c1907426fc-centos7-x86_64-clang12/installed/tsan/libcxx/include/c++/v1/thread:291:5 (client-test+0x3ccd9a)
Previous write of size 4 at 0x7f2f4fc72290 by thread T981:
#0 google::LogMessage::Init(char const*, int, int, void (google::LogMessage::*)()) <null> (libglog.so.0+0x1a3a0)
#1 google::LogMessageFatal::LogMessageFatal(char const*, int) <null> (libglog.so.0+0x1faa7)
#2 yb::client::TableIterator::HandleError(yb::Status const&) /nfusr/centos-gcp-cloud/jenkins-worker-33a2dy/jenkins/jenkins-github-yugabyte-db-centos-master-clang12-tsan-896/yugabyte-db/build/tsan-clang12-dynamic-ninja/../../src/yb/client/table_handle.cc:371:5 (libyb_client.so+0xb7346e)
#3 yb::client::TableIterator::IsFlushStatusOkOrHandleErrors(yb::client::FlushStatus) /nfusr/centos-gcp-cloud/jenkins-worker-33a2dy/jenkins/jenkins-github-yugabyte-db-centos-master-clang12-tsan-896/yugabyte-db/build/tsan-clang12-dynamic-ninja/../../src/yb/client/table_handle.cc:357:3 (libyb_client.so+0xb73e1c)
#4 yb::client::TableIterator::ExecuteOps() /nfusr/centos-gcp-cloud/jenkins-worker-33a2dy/jenkins/jenkins-github-yugabyte-db-centos-master-clang12-tsan-896/yugabyte-db/build/tsan-clang12-dynamic-ninja/../../src/yb/client/table_handle.cc:280:8 (libyb_client.so+0xb735dc)
#5 yb::client::TableIterator::TableIterator(yb::client::TableHandle const*, yb::client::TableIteratorOptions const&) /nfusr/centos-gcp-cloud/jenkins-worker-33a2dy/jenkins/jenkins-github-yugabyte-db-centos-master-clang12-tsan-896/yugabyte-db/build/tsan-clang12-dynamic-ninja/../../src/yb/client/table_handle.cc:267:7 (libyb_client.so+0xb73332)
#6 yb::client::TableRange::begin() const /nfusr/centos-gcp-cloud/jenkins-worker-33a2dy/jenkins/jenkins-github-yugabyte-db-centos-master-clang12-tsan-896/yugabyte-db/build/tsan-clang12-dynamic-ninja/../../src/yb/client/table_handle.h:234:12 (client-test+0x3e1ddc)
#7 boost::range_iterator<yb::client::TableRange const, void>::type boost::range_detail::range_begin<yb::client::TableRange const>(yb::client::TableRange const&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181649-c1907426fc-centos7-x86_64-clang12/installed/tsan/include/boost/range/begin.hpp:45:18 (client-test+0x420290)
#8 boost::range_iterator<yb::client::TableRange const, void>::type boost::range_adl_barrier::begin<yb::client::TableRange>(yb::client::TableRange const&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181649-c1907426fc-centos7-x86_64-clang12/installed/tsan/include/boost/range/begin.hpp:115:12 (client-test+0x41e5c0)
#9 boost::disable_if<boost::range_detail::has_member_size<yb::client::TableRange>, boost::range_size<yb::client::TableRange const>::type>::type boost::range_detail::range_calculate_size<yb::client::TableRange>(yb::client::TableRange const&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181649-c1907426fc-centos7-x86_64-clang12/installed/tsan/include/boost/range/size.hpp:49:34 (client-test+0x41e466)
#10 boost::range_size<yb::client::TableRange const>::type boost::size<yb::client::TableRange>(yb::client::TableRange const&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181649-c1907426fc-centos7-x86_64-clang12/installed/tsan/include/boost/range/size.hpp:71:16 (client-test+0x3e19c9)
#11 yb::client::(anonymous namespace)::CountRowsFromClient(yb::client::TableHandle const&, yb::YBConsistencyLevel, int, int) /nfusr/centos-gcp-cloud/jenkins-worker-33a2dy/jenkins/jenkins-github-yugabyte-db-centos-master-clang12-tsan-896/yugabyte-db/build/tsan-clang12-dynamic-ninja/../../src/yb/client/client-test.cc:529:10 (client-test+0x3bd906)
#12 yb::client::(anonymous namespace)::CountRowsFromClient(yb::client::TableHandle const&, int, int) /nfusr/centos-gcp-cloud/jenkins-worker-33a2dy/jenkins/jenkins-github-yugabyte-db-centos-master-clang12-tsan-896/yugabyte-db/build/tsan-clang12-dynamic-ninja/../../src/yb/client/client-test.cc:533:10 (client-test+0x3ac77e)
#13 yb::client::(anonymous namespace)::CountRowsFromClient(yb::client::TableHandle const&) /nfusr/centos-gcp-cloud/jenkins-worker-33a2dy/jenkins/jenkins-github-yugabyte-db-centos-master-clang12-tsan-896/yugabyte-db/build/tsan-clang12-dynamic-ninja/../../src/yb/client/client-test.cc:537:10 (client-test+0x3ac223)
#14 yb::client::(anonymous namespace)::CheckRowCount(yb::client::TableHandle const&) /nfusr/centos-gcp-cloud/jenkins-worker-33a2dy/jenkins/jenkins-github-yugabyte-db-centos-master-clang12-tsan-896/yugabyte-db/build/tsan-clang12-dynamic-ninja/../../src/yb/client/client-test.cc:544:3 (client-test+0x3cd259)
#15 yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_6::operator()() const /nfusr/centos-gcp-cloud/jenkins-worker-33a2dy/jenkins/jenkins-github-yugabyte-db-centos-master-clang12-tsan-896/yugabyte-db/build/tsan-clang12-dynamic-ninja/../../src/yb/client/client-test.cc:2169:11 (client-test+0x3cd174)
#16 void yb::TestThreadHolder::AddThreadFunctor<yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_6>(yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_6 const&)::'lambda'()::operator()() const /nfusr/centos-gcp-cloud/jenkins-worker-33a2dy/jenkins/jenkins-github-yugabyte-db-centos-master-clang12-tsan-896/yugabyte-db/build/tsan-clang12-dynamic-ninja/../../src/yb/util/test_thread_holder.h:63:7 (client-test+0x3cd09d)
#17 decltype(std::__1::forward<yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_6>(fp)(std::__1::forward<void yb::TestThreadHolder::AddThreadFunctor<yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_6>(yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_6 const&)::'lambda'()>(fp0)...)) std::__1::__invoke<void yb::TestThreadHolder::AddThreadFunctor<yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_6>(yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_6 const&)::'lambda'()>(yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_6&&, void yb::TestThreadHolder::AddThreadFunctor<yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_6>(yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_6 const&)::'lambda'()&&...) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181649-c1907426fc-centos7-x86_64-clang12/installed/tsan/libcxx/include/c++/v1/type_traits:3694:1 (client-test+0x3cd009)
#18 void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void yb::TestThreadHolder::AddThreadFunctor<yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_6>(yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_6 const&)::'lambda'()>(std::__1::tuple<yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_6, void yb::TestThreadHolder::AddThreadFunctor<yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_6>(yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_6 const&)::'lambda'()>&, std::__1::__tuple_indices<>) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181649-c1907426fc-centos7-x86_64-clang12/installed/tsan/libcxx/include/c++/v1/thread:280:5 (client-test+0x3ccfd1)
#19 void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void yb::TestThreadHolder::AddThreadFunctor<yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_6>(yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_6 const&)::'lambda'()> >(void*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181649-c1907426fc-centos7-x86_64-clang12/installed/tsan/libcxx/include/c++/v1/thread:291:5 (client-test+0x3ccd9a)
Location is global 'google::fatal_msg_data_shared' of size 30392 at 0x7f2f4fc6ac70 (libglog.so.0+0x000000041290)
```WARNING: ThreadSanitizer: data race (pid=31208)
Write of size 4 at 0x7f2f4fc72290 by thread T615:
#0 google::LogMessage::Init(char const*, int, int, void (google::LogMessage::*)()) <null> (libglog.so.0+0x19dac)
#1 google::LogMessageFatal::LogMessageFatal(char const*, int) <null> (libglog.so.0+0x1faa7)
#2 yb::client::TableIterator::HandleError(yb::Status const&) /nfusr/centos-gcp-cloud/jenkins-worker-33a2dy/jenkins/jenkins-github-yugabyte-db-centos-master-clang12-tsan-896/yugabyte-db/build/tsan-clang12-dynamic-ninja/../../src/yb/client/table_handle.cc:371:5 (libyb_client.so+0xb7346e)
#3 yb::client::TableIterator::IsFlushStatusOkOrHandleErrors(yb::client::FlushStatus) /nfusr/centos-gcp-cloud/jenkins-worker-33a2dy/jenkins/jenkins-github-yugabyte-db-centos-master-clang12-tsan-896/yugabyte-db/build/tsan-clang12-dynamic-ninja/../../src/yb/client/table_handle.cc:357:3 (libyb_client.so+0xb73e1c)
#4 yb::client::TableIterator::ExecuteOps() /nfusr/centos-gcp-cloud/jenkins-worker-33a2dy/jenkins/jenkins-github-yugabyte-db-centos-master-clang12-tsan-896/yugabyte-db/build/tsan-clang12-dynamic-ninja/../../src/yb/client/table_handle.cc:280:8 (libyb_client.so+0xb735dc)
#5 yb::client::TableIterator::TableIterator(yb::client::TableHandle const*, yb::client::TableIteratorOptions const&) /nfusr/centos-gcp-cloud/jenkins-worker-33a2dy/jenkins/jenkins-github-yugabyte-db-centos-master-clang12-tsan-896/yugabyte-db/build/tsan-clang12-dynamic-ninja/../../src/yb/client/table_handle.cc:267:7 (libyb_client.so+0xb73332)
#6 yb::client::TableRange::begin() const /nfusr/centos-gcp-cloud/jenkins-worker-33a2dy/jenkins/jenkins-github-yugabyte-db-centos-master-clang12-tsan-896/yugabyte-db/build/tsan-clang12-dynamic-ninja/../../src/yb/client/table_handle.h:234:12 (client-test+0x3e1ddc)
#7 boost::range_iterator<yb::client::TableRange const, void>::type boost::range_detail::range_begin<yb::client::TableRange const>(yb::client::TableRange const&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181649-c1907426fc-centos7-x86_64-clang12/installed/tsan/include/boost/range/begin.hpp:45:18 (client-test+0x420290)
#8 boost::range_iterator<yb::client::TableRange const, void>::type boost::range_adl_barrier::begin<yb::client::TableRange>(yb::client::TableRange const&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181649-c1907426fc-centos7-x86_64-clang12/installed/tsan/include/boost/range/begin.hpp:115:12 (client-test+0x41e5c0)
#9 boost::disable_if<boost::range_detail::has_member_size<yb::client::TableRange>, boost::range_size<yb::client::TableRange const>::type>::type boost::range_detail::range_calculate_size<yb::client::TableRange>(yb::client::TableRange const&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181649-c1907426fc-centos7-x86_64-clang12/installed/tsan/include/boost/range/size.hpp:49:34 (client-test+0x41e466)
#10 boost::range_size<yb::client::TableRange const>::type boost::size<yb::client::TableRange>(yb::client::TableRange const&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181649-c1907426fc-centos7-x86_64-clang12/installed/tsan/include/boost/range/size.hpp:71:16 (client-test+0x3e19c9)
#11 yb::client::(anonymous namespace)::CountRowsFromClient(yb::client::TableHandle const&, yb::YBConsistencyLevel, int, int) /nfusr/centos-gcp-cloud/jenkins-worker-33a2dy/jenkins/jenkins-github-yugabyte-db-centos-master-clang12-tsan-896/yugabyte-db/build/tsan-clang12-dynamic-ninja/../../src/yb/client/client-test.cc:529:10 (client-test+0x3bd906)
#12 yb::client::(anonymous namespace)::CountRowsFromClient(yb::client::TableHandle const&, int, int) /nfusr/centos-gcp-cloud/jenkins-worker-33a2dy/jenkins/jenkins-github-yugabyte-db-centos-master-clang12-tsan-896/yugabyte-db/build/tsan-clang12-dynamic-ninja/../../src/yb/client/client-test.cc:533:10 (client-test+0x3ac77e)
#13 yb::client::(anonymous namespace)::CountRowsFromClient(yb::client::TableHandle const&) /nfusr/centos-gcp-cloud/jenkins-worker-33a2dy/jenkins/jenkins-github-yugabyte-db-centos-master-clang12-tsan-896/yugabyte-db/build/tsan-clang12-dynamic-ninja/../../src/yb/client/client-test.cc:537:10 (client-test+0x3ac223)
#14 yb::client::(anonymous namespace)::CheckRowCount(yb::client::TableHandle const&) /nfusr/centos-gcp-cloud/jenkins-worker-33a2dy/jenkins/jenkins-github-yugabyte-db-centos-master-clang12-tsan-896/yugabyte-db/build/tsan-clang12-dynamic-ninja/../../src/yb/client/client-test.cc:544:3 (client-test+0x3cd259)
#15 yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_6::operator()() const /nfusr/centos-gcp-cloud/jenkins-worker-33a2dy/jenkins/jenkins-github-yugabyte-db-centos-master-clang12-tsan-896/yugabyte-db/build/tsan-clang12-dynamic-ninja/../../src/yb/client/client-test.cc:2169:11 (client-test+0x3cd174)
#16 void yb::TestThreadHolder::AddThreadFunctor<yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_6>(yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_6 const&)::'lambda'()::operator()() const /nfusr/centos-gcp-cloud/jenkins-worker-33a2dy/jenkins/jenkins-github-yugabyte-db-centos-master-clang12-tsan-896/yugabyte-db/build/tsan-clang12-dynamic-ninja/../../src/yb/util/test_thread_holder.h:63:7 (client-test+0x3cd09d)
#17 decltype(std::__1::forward<yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_6>(fp)(std::__1::forward<void yb::TestThreadHolder::AddThreadFunctor<yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_6>(yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_6 const&)::'lambda'()>(fp0)...)) std::__1::__invoke<void yb::TestThreadHolder::AddThreadFunctor<yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_6>(yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_6 const&)::'lambda'()>(yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_6&&, void yb::TestThreadHolder::AddThreadFunctor<yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_6>(yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_6 const&)::'lambda'()&&...) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181649-c1907426fc-centos7-x86_64-clang12/installed/tsan/libcxx/include/c++/v1/type_traits:3694:1 (client-test+0x3cd009)
#18 void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void yb::TestThreadHolder::AddThreadFunctor<yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_6>(yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_6 const&)::'lambda'()>(std::__1::tuple<yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_6, void yb::TestThreadHolder::AddThreadFunctor<yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_6>(yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_6 const&)::'lambda'()>&, std::__1::__tuple_indices<>) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181649-c1907426fc-centos7-x86_64-clang12/installed/tsan/libcxx/include/c++/v1/thread:280:5 (client-test+0x3ccfd1)
#19 void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void yb::TestThreadHolder::AddThreadFunctor<yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_6>(yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_6 const&)::'lambda'()> >(void*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181649-c1907426fc-centos7-x86_64-clang12/installed/tsan/libcxx/include/c++/v1/thread:291:5 (client-test+0x3ccd9a)
Previous write of size 4 at 0x7f2f4fc72290 by thread T981:
#0 google::LogMessage::Init(char const*, int, int, void (google::LogMessage::*)()) <null> (libglog.so.0+0x1a3a0)
#1 google::LogMessageFatal::LogMessageFatal(char const*, int) <null> (libglog.so.0+0x1faa7)
#2 yb::client::TableIterator::HandleError(yb::Status const&) /nfusr/centos-gcp-cloud/jenkins-worker-33a2dy/jenkins/jenkins-github-yugabyte-db-centos-master-clang12-tsan-896/yugabyte-db/build/tsan-clang12-dynamic-ninja/../../src/yb/client/table_handle.cc:371:5 (libyb_client.so+0xb7346e)
#3 yb::client::TableIterator::IsFlushStatusOkOrHandleErrors(yb::client::FlushStatus) /nfusr/centos-gcp-cloud/jenkins-worker-33a2dy/jenkins/jenkins-github-yugabyte-db-centos-master-clang12-tsan-896/yugabyte-db/build/tsan-clang12-dynamic-ninja/../../src/yb/client/table_handle.cc:357:3 (libyb_client.so+0xb73e1c)
#4 yb::client::TableIterator::ExecuteOps() /nfusr/centos-gcp-cloud/jenkins-worker-33a2dy/jenkins/jenkins-github-yugabyte-db-centos-master-clang12-tsan-896/yugabyte-db/build/tsan-clang12-dynamic-ninja/../../src/yb/client/table_handle.cc:280:8 (libyb_client.so+0xb735dc)
#5 yb::client::TableIterator::TableIterator(yb::client::TableHandle const*, yb::client::TableIteratorOptions const&) /nfusr/centos-gcp-cloud/jenkins-worker-33a2dy/jenkins/jenkins-github-yugabyte-db-centos-master-clang12-tsan-896/yugabyte-db/build/tsan-clang12-dynamic-ninja/../../src/yb/client/table_handle.cc:267:7 (libyb_client.so+0xb73332)
#6 yb::client::TableRange::begin() const /nfusr/centos-gcp-cloud/jenkins-worker-33a2dy/jenkins/jenkins-github-yugabyte-db-centos-master-clang12-tsan-896/yugabyte-db/build/tsan-clang12-dynamic-ninja/../../src/yb/client/table_handle.h:234:12 (client-test+0x3e1ddc)
#7 boost::range_iterator<yb::client::TableRange const, void>::type boost::range_detail::range_begin<yb::client::TableRange const>(yb::client::TableRange const&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181649-c1907426fc-centos7-x86_64-clang12/installed/tsan/include/boost/range/begin.hpp:45:18 (client-test+0x420290)
#8 boost::range_iterator<yb::client::TableRange const, void>::type boost::range_adl_barrier::begin<yb::client::TableRange>(yb::client::TableRange const&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181649-c1907426fc-centos7-x86_64-clang12/installed/tsan/include/boost/range/begin.hpp:115:12 (client-test+0x41e5c0)
#9 boost::disable_if<boost::range_detail::has_member_size<yb::client::TableRange>, boost::range_size<yb::client::TableRange const>::type>::type boost::range_detail::range_calculate_size<yb::client::TableRange>(yb::client::TableRange const&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181649-c1907426fc-centos7-x86_64-clang12/installed/tsan/include/boost/range/size.hpp:49:34 (client-test+0x41e466)
#10 boost::range_size<yb::client::TableRange const>::type boost::size<yb::client::TableRange>(yb::client::TableRange const&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181649-c1907426fc-centos7-x86_64-clang12/installed/tsan/include/boost/range/size.hpp:71:16 (client-test+0x3e19c9)
#11 yb::client::(anonymous namespace)::CountRowsFromClient(yb::client::TableHandle const&, yb::YBConsistencyLevel, int, int) /nfusr/centos-gcp-cloud/jenkins-worker-33a2dy/jenkins/jenkins-github-yugabyte-db-centos-master-clang12-tsan-896/yugabyte-db/build/tsan-clang12-dynamic-ninja/../../src/yb/client/client-test.cc:529:10 (client-test+0x3bd906)
#12 yb::client::(anonymous namespace)::CountRowsFromClient(yb::client::TableHandle const&, int, int) /nfusr/centos-gcp-cloud/jenkins-worker-33a2dy/jenkins/jenkins-github-yugabyte-db-centos-master-clang12-tsan-896/yugabyte-db/build/tsan-clang12-dynamic-ninja/../../src/yb/client/client-test.cc:533:10 (client-test+0x3ac77e)
#13 yb::client::(anonymous namespace)::CountRowsFromClient(yb::client::TableHandle const&) /nfusr/centos-gcp-cloud/jenkins-worker-33a2dy/jenkins/jenkins-github-yugabyte-db-centos-master-clang12-tsan-896/yugabyte-db/build/tsan-clang12-dynamic-ninja/../../src/yb/client/client-test.cc:537:10 (client-test+0x3ac223)
#14 yb::client::(anonymous namespace)::CheckRowCount(yb::client::TableHandle const&) /nfusr/centos-gcp-cloud/jenkins-worker-33a2dy/jenkins/jenkins-github-yugabyte-db-centos-master-clang12-tsan-896/yugabyte-db/build/tsan-clang12-dynamic-ninja/../../src/yb/client/client-test.cc:544:3 (client-test+0x3cd259)
#15 yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_6::operator()() const /nfusr/centos-gcp-cloud/jenkins-worker-33a2dy/jenkins/jenkins-github-yugabyte-db-centos-master-clang12-tsan-896/yugabyte-db/build/tsan-clang12-dynamic-ninja/../../src/yb/client/client-test.cc:2169:11 (client-test+0x3cd174)
#16 void yb::TestThreadHolder::AddThreadFunctor<yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_6>(yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_6 const&)::'lambda'()::operator()() const /nfusr/centos-gcp-cloud/jenkins-worker-33a2dy/jenkins/jenkins-github-yugabyte-db-centos-master-clang12-tsan-896/yugabyte-db/build/tsan-clang12-dynamic-ninja/../../src/yb/util/test_thread_holder.h:63:7 (client-test+0x3cd09d)
#17 decltype(std::__1::forward<yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_6>(fp)(std::__1::forward<void yb::TestThreadHolder::AddThreadFunctor<yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_6>(yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_6 const&)::'lambda'()>(fp0)...)) std::__1::__invoke<void yb::TestThreadHolder::AddThreadFunctor<yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_6>(yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_6 const&)::'lambda'()>(yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_6&&, void yb::TestThreadHolder::AddThreadFunctor<yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_6>(yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_6 const&)::'lambda'()&&...) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181649-c1907426fc-centos7-x86_64-clang12/installed/tsan/libcxx/include/c++/v1/type_traits:3694:1 (client-test+0x3cd009)
#18 void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void yb::TestThreadHolder::AddThreadFunctor<yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_6>(yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_6 const&)::'lambda'()>(std::__1::tuple<yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_6, void yb::TestThreadHolder::AddThreadFunctor<yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_6>(yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_6 const&)::'lambda'()>&, std::__1::__tuple_indices<>) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181649-c1907426fc-centos7-x86_64-clang12/installed/tsan/libcxx/include/c++/v1/thread:280:5 (client-test+0x3ccfd1)
#19 void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void yb::TestThreadHolder::AddThreadFunctor<yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_6>(yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_6 const&)::'lambda'()> >(void*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181649-c1907426fc-centos7-x86_64-clang12/installed/tsan/libcxx/include/c++/v1/thread:291:5 (client-test+0x3ccd9a)
Location is global 'google::fatal_msg_data_shared' of size 30392 at 0x7f2f4fc6ac70 (libglog.so.0+0x000000041290)
@adithya-yb , Is this related to the changes that you made to table_handle?
@rthallamko3 No, This is the PR in question: https://phabricator.dev.yugabyte.com/D19201 And I introduced two new functions in table_handle (AddMapColumnValue, AddMapEntryToColumn) , both of which aren't in the stack traces.
@adithya-yb , Shamanth's data indicates that the tsan failures could have increased after you commit landed. @shamanthchandra-yb , Can we schedule a loop of 100 executions of tsan for this test, with a build that doesn't have the changes mentioned? That can help provide better data on whether that change increased the flakiness or not?
@rthallamko3 this was raised as an observation, on the existing run. QA team doesn't have context on this, because of gap in understanding from UT framework. Can someone from development team, help here ? cc: @kripasreenivasan
There are other tests that run into the same error, See report
Failed: IO error (yb/client/batcher.cc:110)
seems like a deadlock of some sort. Pasting a small snippet below. Full log can be found here
table_handle.cc:371] Failed: IO error (yb/client/batcher.cc:110): Errors occurred while reaching out to the tablet servers
@ 0x7f8323aa29ca yb::LogFatalHandlerSink::send(int, char const*, char const*, int, tm const*, char const*, unsigned long) (src/yb/util/logging.cc:493)
@ 0x7f832342a27c
@ 0x7f832342194e
@ 0x7f8323422a8c
@ 0x7f8323426bdd
@ 0x7f8327b13d8a yb::client::TableIterator::HandleError(yb::Status const&) (src/yb/client/table_handle.cc:371)
@ 0x7f8327b1470c yb::client::TableIterator::IsFlushStatusOkOrHandleErrors(yb::client::FlushStatus) (src/yb/client/table_handle.cc:357)
@ 0x7f8327b141ea yb::client::TableIterator::Move() (src/yb/client/table_handle.cc:317)
@ 0x7f8327b14907 yb::client::TableIterator::operator++() (src/yb/client/table_handle.cc:301)
@ 0x425627 std::iterator_traits<yb::client::TableIterator>::difference_type std::__distance<yb::client::TableIterator>(yb::client::TableIterator, yb::client::TableIterator, std::input_iterator_tag) (/opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181649-c1907426fc-centos7-x86_64-clang12/installed/tsan/libcxx/include/c++/v1/iterator:678)
@ 0x42550d std::iterator_traits<yb::client::TableIterator>::difference_type std::distance<yb::client::TableIterator>(yb::client::TableIterator, yb::client::TableIterator) (/opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181649-c1907426fc-centos7-x86_64-clang12/installed/tsan/libcxx/include/c++/v1/iterator:696)
@ 0x425458 boost::disable_if<boost::range_detail::has_member_size<yb::client::TableRange>, boost::range_size<yb::client::TableRange const>::type>::type boost::range_detail::range_calculate_size<yb::client::TableRange>(yb::client::TableRange const&) (/opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181649-c1907426fc-centos7-x86_64-clang12/installed/tsan/include/boost/range/size.hpp:49)
@ 0x3e75f9 boost::range_size<yb::client::TableRange const>::type boost::size<yb::client::TableRange>(yb::client::TableRange const&) (/opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181649-c1907426fc-centos7-x86_64-clang12/installed/tsan/include/boost/range/size.hpp:71)
@ 0x3c2bb6 yb::client::(anonymous namespace)::CountRowsFromClient(yb::client::TableHandle const&, yb::YBConsistencyLevel, int, int) (src/yb/client/client-test.cc:529)
@ 0x3b1a2e yb::client::(anonymous namespace)::CountRowsFromClient(yb::client::TableHandle const&, int, int) (src/yb/client/client-test.cc:533)
@ 0x3b14d3 yb::client::(anonymous namespace)::CountRowsFromClient(yb::client::TableHandle const&) (src/yb/client/client-test.cc:537)
@ 0x3d2529 yb::client::(anonymous namespace)::CheckRowCount(yb::client::TableHandle const&) (src/yb/client/client-test.cc:544)
@ 0x3d2444 operator() (src/yb/client/client-test.cc:2255)
@ 0x3d236d operator() (src/yb/util/test_thread_holder.h:63)
@ 0x3d22d9 decltype(std::forward<yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_7>(fp)(std::forward<void yb::TestThreadHolder::AddThreadFunctor<yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_7>(yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_7 const&)::'lambda'()>(fp0)...)) std::__invoke<void yb::TestThreadHolder::AddThreadFunctor<yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_7>(yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_7 const&)::'lambda'()>(yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_7&&, void yb::TestThreadHolder::AddThreadFunctor<yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_7>(yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_7 const&)::'lambda'()&&...) (/opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181649-c1907426fc-centos7-x86_64-clang12/installed/tsan/libcxx/include/c++/v1/type_traits:3694)
@ 0x3d22a1 void std::__thread_execute<unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct> >, void yb::TestThreadHolder::AddThreadFunctor<yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_7>(yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_7 const&)::'lambda'()>(tuple<yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_7, void yb::TestThreadHolder::AddThreadFunctor<yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_7>(yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_7 const&)::'lambda'()>&, std::__tuple_indices<>) (/opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181649-c1907426fc-centos7-x86_64-clang12/installed/tsan/libcxx/include/c++/v1/thread:280)
@ 0x3d206a void* std::__thread_proxy<tuple<unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct> >, void yb::TestThreadHolder::AddThreadFunctor<yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_7>(yb::client::ClientTest_TestServerTooBusyRetry_Test::TestBody()::$_7 const&)::'lambda'()> >(void*) (/opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181649-c1907426fc-centos7-x86_64-clang12/installed/tsan/libcxx/include/c++/v1/thread:291)
@ 0x338c1c __tsan_thread_start_func (/opt/yb-build/llvm/yb-llvm-v12.0.1-yb-1-1633099823-bdb147e6-centos7-x86_64-build/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:955)
@ 0x7f8321a05ea4
@ 0x7f8321517b0c
@ 0x7f832c23917d yb::master::ScopedLeaderSharedLock::Unlock() (src/yb/master/scoped_leader_shared_lock.cc:174)
@ 0x7f832c238f4a ~ScopedLeaderSharedLock (src/yb/master/scoped_leader_shared_lock.cc:156)
@ 0x7f832c2e8f6a yb::master::(anonymous namespace)::MasterHeartbeatServiceImpl::TSHeartbeat(yb::master::TSHeartbeatRequestPB const*, yb::master::TSHeartbeatResponsePB*, yb::rpc::RpcContext) (src/yb/master/master_heartbeat_service.cc:204)
@ 0x7f83257fbc3f operator() (src/yb/master/master_heartbeat.service.cc:68)
@ 0x7f83257fbaf3 auto yb::rpc::HandleCall<yb::rpc::RpcCallPBParamsImpl<yb::master::TSHeartbeatRequestPB, yb::master::TSHeartbeatResponsePB>, yb::master::MasterHeartbeatIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_0::operator()(shared_ptr<yb::rpc::InboundCall>) const::'lambda'(yb::master::TSHeartbeatRequestPB const*, yb::master::TSHeartbeatResponsePB*, yb::rpc::RpcContext)>(shared_ptr<yb::rpc::InboundCall>, yb::master::MasterHeartbeatIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_0::operator()(shared_ptr<yb::rpc::InboundCall>) const::'lambda'(yb::master::TSHeartbeatRequestPB const*, yb::master::TSHeartbeatResponsePB*, yb::rpc::RpcContext)) (src/yb/rpc/local_call.h:124)
@ 0x7f83257fb87f operator() (src/yb/master/master_heartbeat.service.cc:66)
@ 0x7f83257fb7db decltype(std::forward<yb::master::MasterHeartbeatIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_0&>(fp)(std::forward<shared_ptr<yb::rpc::InboundCall> >(fp0))) std::__invoke<yb::master::MasterHeartbeatIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_0&, shared_ptr<yb::rpc::InboundCall> >(yb::master::MasterHeartbeatIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_0&, shared_ptr<yb::rpc::InboundCall>&&) (/opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181649-c1907426fc-centos7-x86_64-clang12/installed/tsan/libcxx/include/c++/v1/type_traits:3694)
@ 0x7f83257fb748 void std::__invoke_void_return_wrapper<void, true>::__call<yb::master::MasterHeartbeatIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_0&, shared_ptr<yb::rpc::InboundCall> >(yb::master::MasterHeartbeatIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_0&, shared_ptr<yb::rpc::InboundCall>&&) (/opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181649-c1907426fc-centos7-x86_64-clang12/installed/tsan/libcxx/include/c++/v1/__functional_base:348)
@ 0x7f83257fb703 std::__function::__alloc_func<yb::master::MasterHeartbeatIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_0, std::allocator<yb::master::MasterHeartbeatIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_0>, void (shared_ptr<yb::rpc::InboundCall>)>::operator()(shared_ptr<yb::rpc::InboundCall>&&) (/opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181649-c1907426fc-centos7-x86_64-clang12/installed/tsan/libcxx/include/c++/v1/functional:1558)
@ 0x7f83257fa99c std::__function::__func<yb::master::MasterHeartbeatIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_0, std::allocator<yb::master::MasterHeartbeatIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_0>, void (shared_ptr<yb::rpc::InboundCall>)>::operator()(shared_ptr<yb::rpc::InboundCall>&&) (/opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181649-c1907426fc-centos7-x86_64-clang12/installed/tsan/libcxx/include/c++/v1/functional:1732)
@ 0x7f832ca02416 std::__function::__value_func<void (shared_ptr<yb::rpc::InboundCall>)>::operator()(shared_ptr<yb::rpc::InboundCall>&&) const (/opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181649-c1907426fc-centos7-x86_64-clang12/installed/tsan/libcxx/include/c++/v1/functional:1885)
@ 0x7f832ca016e8 std::function<void (shared_ptr<yb::rpc::InboundCall>)>::operator()(shared_ptr<yb::rpc::InboundCall>) const (/opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181649-c1907426fc-centos7-x86_64-clang12/installed/tsan/libcxx/include/c++/v1/functional:2560)
@ 0x7f83257fa5ba yb::master::MasterHeartbeatIf::Handle(shared_ptr<yb::rpc::InboundCall>) (src/yb/master/master_heartbeat.service.cc:51)
@ 0x7f83241de72c yb::rpc::ServicePoolImpl::Handle(shared_ptr<yb::rpc::InboundCall>) (src/yb/rpc/service_pool.cc:270)
@ 0x7f83240ee5b6 yb::rpc::InboundCall::InboundCallTask::Run() (src/yb/rpc/inbound_call.cc:225)
@ 0x7f83241fc23c yb::rpc::(anonymous namespace)::Worker::Execute() (src/yb/rpc/thread_pool.cc:104)
@ 0x7f83241fd613 decltype(*(std::forward<yb::rpc::(anonymous namespace)::Worker*&>(fp0)).*fp()) std::__invoke<void (yb::rpc::(anonymous namespace)::Worker::*&)(), yb::rpc::(anonymous namespace)::Worker*&, void>(void (yb::rpc::(anonymous namespace)::Worker::*&)(), yb::rpc::(anonymous namespace)::Worker*&) (/opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181649-c1907426fc-centos7-x86_64-clang12/installed/tsan/libcxx/include/c++/v1/type_traits:3635)
@ 0x7f83241fd588 std::__bind_return<void (yb::rpc::(anonymous namespace)::Worker::*)(), tuple<yb::rpc::(anonymous namespace)::Worker*>, tuple<>, __is_valid_bind_return<void (yb::rpc::(anonymous namespace)::Worker::*)(), tuple<yb::rpc::(anonymous namespace)::Worker*>, tuple<> >::value>::type std::__apply_functor<void (yb::rpc::(anonymous namespace)::Worker::*)(), tuple<yb::rpc::(anonymous namespace)::Worker*>, 0ul, tuple<> >(void (yb::rpc::(anonymous namespace)::Worker::*&)(), tuple<yb::rpc::(anonymous namespace)::Worker*>&, std::__tuple_indices<0ul>, tuple<>&&) (/opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181649-c1907426fc-centos7-x86_64-clang12/installed/tsan/libcxx/include/c++/v1/functional:2857)
@ 0x7f83241fd53d std::__bind_return<void (yb::rpc::(anonymous namespace)::Worker::*)(), tuple<yb::rpc::(anonymous namespace)::Worker*>, tuple<>, __is_valid_bind_return<void (yb::rpc::(anonymous namespace)::Worker::*)(), tuple<yb::rpc::(anonymous namespace)::Worker*>, tuple<> >::value>::type std::__bind<void (yb::rpc::(anonymous namespace)::Worker::* const&)(), yb::rpc::(anonymous namespace)::Worker* const&>::operator()<>() (/opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181649-c1907426fc-centos7-x86_64-clang12/installed/tsan/libcxx/include/c++/v1/functional:2890)
@ 0x7f83241fd4f9 decltype(std::forward<std::__bind<void (yb::rpc::(anonymous namespace)::Worker::* const&)(), yb::rpc::(anonymous namespace)::Worker* const&>&>(fp)()) std::__invoke<std::__bind<void (yb::rpc::(anonymous namespace)::Worker::* const&)(), yb::rpc::(anonymous namespace)::Worker* const&>&>(std::__bind<void (yb::rpc::(anonymous namespace)::Worker::* const&)(), yb::rpc::(anonymous namespace)::Worker* const&>&) (/opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181649-c1907426fc-centos7-x86_64-clang12/installed/tsan/libcxx/include/c++/v1/type_traits:3694)
@ 0x7f83241fd489 void std::__invoke_void_return_wrapper<void, true>::__call<std::__bind<void (yb::rpc::(anonymous namespace)::Worker::* const&)(), yb::rpc::(anonymous namespace)::Worker* const&>&>(std::__bind<void (yb::rpc::(anonymous namespace)::Worker::* const&)(), yb::rpc::(anonymous namespace)::Worker* const&>&) (/opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181649-c1907426fc-centos7-x86_64-clang12/installed/tsan/libcxx/include/c++/v1/__functional_base:348)
@ 0x7f83241fd451 std::__function::__alloc_func<std::__bind<void (yb::rpc::(anonymous namespace)::Worker::* const&)(), yb::rpc::(anonymous namespace)::Worker* const&>, std::allocator<std::__bind<void (yb::rpc::(anonymous namespace)::Worker::* const&)(), yb::rpc::(anonymous namespace)::Worker* const&> >, void ()>::operator()() (/opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181649-c1907426fc-centos7-x86_64-clang12/installed/tsan/libcxx/include/c++/v1/functional:1558)
@ 0x7f83241fc6fd std::__function::__func<std::__bind<void (yb::rpc::(anonymous namespace)::Worker::* const&)(), yb::rpc::(anonymous namespace)::Worker* const&>, std::allocator<std::__bind<void (yb::rpc::(anonymous namespace)::Worker::* const&)(), yb::rpc::(anonymous namespace)::Worker* const&> >, void ()>::operator()() (/opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181649-c1907426fc-centos7-x86_64-clang12/installed/tsan/libcxx/include/c++/v1/functional:1732)
@ 0x7f832c4d7c54 std::__function::__value_func<void ()>::operator()() const (/opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181649-c1907426fc-centos7-x86_64-clang12/installed/tsan/libcxx/include/c++/v1/functional:1885)
@ 0x7f832c4c2bf9 std::function<void ()>::operator()() const (/opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181649-c1907426fc-centos7-x86_64-clang12/installed/tsan/libcxx/include/c++/v1/functional:2560)
@ 0x7f8323bfc0c8 yb::Thread::SuperviseThread(void*) (src/yb/util/thread.cc:774)
@ 0x338c1c __tsan_thread_start_func (/opt/yb-build/llvm/yb-llvm-v12.0.1-yb-1-1633099823-bdb147e6-centos7-x86_64-build/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:955)
@ 0x7f8321a05ea4
@ 0x7f8321517b0c
@ 0x7f832c23917d yb::master::ScopedLeaderSharedLock::Unlock() (src/yb/master/scoped_leader_shared_lock.cc:174)
@ 0x7f832c238f4a ~ScopedLeaderSharedLock (src/yb/master/scoped_leader_shared_lock.cc:156)
@ 0x7f832c2cbc3c void yb::master::MasterServiceBase::HandleOnLeader<yb::master::GetTableLocationsRequestPB, yb::master::GetTableLocationsResponsePB, yb::master::(anonymous namespace)::MasterClientServiceImpl::GetTableLocations(yb::master::GetTableLocationsRequestPB const*, yb::master::GetTableLocationsResponsePB*, yb::rpc::RpcContext)::'lambda'()>(yb::master::GetTableLocationsRequestPB const*, yb::master::GetTableLocationsResponsePB*, yb::rpc::RpcContext*, yb::master::(anonymous namespace)::MasterClientServiceImpl::GetTableLocations(yb::master::GetTableLocationsRequestPB const*, yb::master::GetTableLocationsResponsePB*, yb::rpc::RpcContext)::'lambda'(), char const*, int, char const*, yb::StronglyTypedBool<yb::master::HoldCatalogLock_Tag>) (src/yb/master/master_service_base-internal.h:87)
@ 0x7f832c2cb758 yb::master::(anonymous namespace)::MasterClientServiceImpl::GetTableLocations(yb::master::GetTableLocationsRequestPB const*, yb::master::GetTableLocationsResponsePB*, yb::rpc::RpcContext) (src/yb/master/master_client_service.cc:113)
@ 0x7f832571dd0f operator() (src/yb/master/master_client.service.cc:223)
@ 0x7f832571dbc3 auto yb::rpc::HandleCall<yb::rpc::RpcCallPBParamsImpl<yb::master::GetTableLocationsRequestPB, yb::master::GetTableLocationsResponsePB>, yb::master::MasterClientIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_1::operator()(shared_ptr<yb::rpc::InboundCall>) const::'lambda'(yb::master::GetTableLocationsRequestPB const*, yb::master::GetTableLocationsResponsePB*, yb::rpc::RpcContext)>(shared_ptr<yb::rpc::InboundCall>, yb::master::MasterClientIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_1::operator()(shared_ptr<yb::rpc::InboundCall>) const::'lambda'(yb::master::GetTableLocationsRequestPB const*, yb::master::GetTableLocationsResponsePB*, yb::rpc::RpcContext)) (src/yb/rpc/local_call.h:124)
@ 0x7f832571d94f operator() (src/yb/master/master_client.service.cc:221)
@ 0x7f832571d8ab decltype(std::forward<yb::master::MasterClientIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_1&>(fp)(std::forward<shared_ptr<yb::rpc::InboundCall> >(fp0))) std::__invoke<yb::master::MasterClientIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_1&, shared_ptr<yb::rpc::InboundCall> >(yb::master::MasterClientIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_1&, shared_ptr<yb::rpc::InboundCall>&&) (/opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181649-c1907426fc-centos7-x86_64-clang12/installed/tsan/libcxx/include/c++/v1/type_traits:3694)
@ 0x7f832571d818 void std::__invoke_void_return_wrapper<void, true>::__call<yb::master::MasterClientIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_1&, shared_ptr<yb::rpc::InboundCall> >(yb::master::MasterClientIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_1&, shared_ptr<yb::rpc::InboundCall>&&) (/opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181649-c1907426fc-centos7-x86_64-clang12/installed/tsan/libcxx/include/c++/v1/__functional_base:348)
@ 0x7f832571d7d3 std::__function::__alloc_func<yb::master::MasterClientIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_1, std::allocator<yb::master::MasterClientIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_1>, void (shared_ptr<yb::rpc::InboundCall>)>::operator()(shared_ptr<yb::rpc::InboundCall>&&) (/opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181649-c1907426fc-centos7-x86_64-clang12/installed/tsan/libcxx/include/c++/v1/functional:1558)
@ 0x7f832571ca6c std::__function::__func<yb::master::MasterClientIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_1, std::allocator<yb::master::MasterClientIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_1>, void (shared_ptr<yb::rpc::InboundCall>)>::operator()(shared_ptr<yb::rpc::InboundCall>&&) (/opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181649-c1907426fc-centos7-x86_64-clang12/installed/tsan/libcxx/include/c++/v1/functional:1732)
@ 0x7f832ca02416 std::__function::__value_func<void (shared_ptr<yb::rpc::InboundCall>)>::operator()(shared_ptr<yb::rpc::InboundCall>&&) const (/opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181649-c1907426fc-centos7-x86_64-clang12/installed/tsan/libcxx/include/c++/v1/functional:1885)
@ 0x7f832ca016e8 std::function<void (shared_ptr<yb::rpc::InboundCall>)>::operator()(shared_ptr<yb::rpc::InboundCall>) const (/opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181649-c1907426fc-centos7-x86_64-clang12/installed/tsan/libcxx/include/c++/v1/functional:2560)
@ 0x7f832571aeca yb::master::MasterClientIf::Handle(shared_ptr<yb::rpc::InboundCall>) (src/yb/master/master_client.service.cc:191)
@ 0x7f83241de72c yb::rpc::ServicePoolImpl::Handle(shared_ptr<yb::rpc::InboundCall>) (src/yb/rpc/service_pool.cc:270)
@ 0x7f83240ee5b6 yb::rpc::InboundCall::InboundCallTask::Run() (src/yb/rpc/inbound_call.cc:225)
@ 0x7f83241fc23c yb::rpc::(anonymous namespace)::Worker::Execute() (src/yb/rpc/thread_pool.cc:104)
@ 0x7f83241fd613 decltype(*(std::forward<yb::rpc::(anonymous namespace)::Worker*&>(fp0)).*fp()) std::__invoke<void (yb::rpc::(anonymous namespace)::Worker::*&)(), yb::rpc::(anonymous namespace)::Worker*&, void>(void (yb::rpc::(anonymous namespace)::Worker::*&)(), yb::rpc::(anonymous namespace)::Worker*&) (/opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220505181649-c1907426fc-centos7-x86_64-clang12/installed/tsan/libcxx/include/c++/v1/type_traits:3635)
cc @ttyusupov , @lingamsandeep , if this is a known deadlock in GetTableLocations space. cc @amitanandaiyer , @bmatican
