ProtocolIntegrationTest.HandleUpstreamSocketFail is flaky

[danzh2010]

The test is flaky in various ways:

Fails at:

  EXPECT_THAT(waitForAccessLog(access_log_name_),
              HasSubstr("upstream_reset_before_response_started{connection_termination}"));

with corresponding log observed:

[2022-08-12 18:49:00.193][14][critical][assert] [test/integration/base_integration_test.cc:502] assert failure: 0. Details: Timed out waiting for access log. Found: [2022-08-12 18:49:00.193][14][critical][backtrace] [./source/server/backtrace.h:104] Caught Aborted, suspect faulting address 0x50e8d0000000e [2022-08-12 18:49:00.193][14][critical][backtrace] [./source/server/backtrace.h:91] Backtrace (use tools/stack_decode.py to get line numbers): [2022-08-12 18:49:00.193][14][critical][backtrace] [./source/server/backtrace.h:92] Envoy version: 0/1.24.0-dev/test/DEBUG/BoringSSL [2022-08-12 18:49:00.217][14][critical][backtrace] [./source/server/backtrace.h:96] #0: Envoy::SignalAction::sigHandler() [0x46fab78] [2022-08-12 18:49:00.217][14][critical][backtrace] [./source/server/backtrace.h:96] #1: __restore_rt [0x7f06aad14200] [2022-08-12 18:49:00.237][14][critical][backtrace] [./source/server/backtrace.h:96] #2: Envoy::ProtocolIntegrationTest_HandleUpstreamSocketFail_Test::TestBody() [0x1ea0757] [2022-08-12 18:49:00.257][14][critical][backtrace] [./source/server/backtrace.h:96] #3: testing::internal::HandleSehExceptionsInMethodIfSupported<>() [0x53cfc4b] [2022-08-12 18:49:00.277][14][critical][backtrace] [./source/server/backtrace.h:96] #4: testing::internal::HandleExceptionsInMethodIfSupported<>() [0x53c05bd] [2022-08-12 18:49:00.296][14][critical][backtrace] [./source/server/backtrace.h:96] #5: testing::Test::Run() [0x53a8e93] [2022-08-12 18:49:00.315][14][critical][backtrace] [./source/server/backtrace.h:96] #6: testing::TestInfo::Run() [0x53a9a5a] [2022-08-12 18:49:00.334][14][critical][backtrace] [./source/server/backtrace.h:96] #7: testing::TestSuite::Run() [0x53aa2ab]

And it also fails with errors related to singleton::get() before initialize(). [ RUN ] UpstreamProtocols/ProtocolIntegrationTest.HandleUpstreamSocketFail/IPv6_Http2Downstream_Http3UpstreamWrappedNghttp2WithDeferredProcessing [2022-08-19 16:05:12.336][2862][error][quic] [external/com_github_google_quiche/quiche/quic/core/quic_connection.cc:3899] Client: Write failed with error: 9 (Bad file descriptor) [2022-08-19 16:05:12.370][2854][critical][assert] [./source/common/singleton/threadsafe_singleton.h:56] assert failure: loader_ != nullptr. Details: InjectableSingleton used prior to initialization [2022-08-19 16:05:12.370][2854][critical][backtrace] [./source/server/backtrace.h:104] Caught Aborted, suspect faulting address 0x50e8d0000000e [2022-08-19 16:05:12.370][2854][critical][backtrace] [./source/server/backtrace.h:91] Backtrace (use tools/stack_decode.py to get line numbers): [2022-08-19 16:05:12.370][2854][critical][backtrace] [./source/server/backtrace.h:92] Envoy version: 0/1.24.0-dev/test/DEBUG/BoringSSL [2022-08-19 16:05:12.394][2854][critical][backtrace] [./source/server/backtrace.h:96] #0: Envoy::SignalAction::sigHandler() [0x471f468] [2022-08-19 16:05:12.394][2854][critical][backtrace] [./source/server/backtrace.h:96] #1: __restore_rt [0x7fdb42e67200] [2022-08-19 16:05:12.426][2854][critical][backtrace] [./source/server/backtrace.h:96] #2: Envoy::Network::Address::(anonymous namespace)::sockInterfaceOrDefault() [0x46b291c] [2022-08-19 16:05:12.454][2854][critical][backtrace] [./source/server/backtrace.h:96] #3: Envoy::Network::Address::Ipv6Instance::Ipv6Instance() [0x46b5a40] [2022-08-19 16:05:12.488][2854][critical][backtrace] [./source/server/backtrace.h:96] #4: Envoy::Network::Address::InstanceFactory::createInstancePtr<>() [0x46b7765] [2022-08-19 16:05:12.516][2854][critical][backtrace] [./source/server/backtrace.h:96] #5: Envoy::Network::Address::addressFromSockAddr() [0x46b1056] [2022-08-19 16:05:12.550][2854][critical][backtrace] [./source/server/backtrace.h:96] #6: Envoy::Network::Address::addressFromSockAddrOrDie() [0x46b1ef8] [2022-08-19 16:05:12.578][2854][critical][backtrace] [./source/server/backtrace.h:96] #7: Envoy::Quic::quicAddressToEnvoyAddressInstance() [0x3fa1dc2] [2022-08-19 16:05:12.611][2854][critical][backtrace] [./source/server/backtrace.h:96] #8: Envoy::Quic::EnvoyQuicPacketWriter::GetNextWriteLocation() [0x3ddf037] [2022-08-19 16:05:12.637][2854][critical][backtrace] [./source/server/backtrace.h:96] #9: quic::QuicConnection::GetPacketBuffer() [0x416afa9] [2022-08-19 16:05:12.662][2854][critical][backtrace] [./source/server/backtrace.h:96] #10: q

repro: $bazel test test/integration:quic_protocol_integration_test --test_filter="HandleUpstreamSocketFail/IPv6_Http2Downstream_Http3UpstreamWrapp" --test_arg="-l trace" --runs_per_test=5000

[danzh2010]

Another similar error: [ RUN ] UpstreamProtocols/ProtocolIntegrationTest.HandleUpstreamSocketFail/IPv6_Http2Downstream_Http3UpstreamNghttp2NoDeferredProcessing [2022-08-19 16:28:18.694][2869][critical][assert] [./test/integration/socket_interface_swap.h:22] assert failure: matched_iohandle_ == nullptr || matched_iohandle_ == io_handle. Details: Matched multiple io_handles, expected at most one to match. [2022-08-19 16:28:18.694][2869][critical][backtrace] [./source/server/backtrace.h:104] Caught Aborted, suspect faulting address 0x50e8d0000000e [2022-08-19 16:28:18.694][2869][critical][backtrace] [./source/server/backtrace.h:91] Backtrace (use tools/stack_decode.py to get line numbers): [2022-08-19 16:28:18.694][2869][critical][backtrace] [./source/server/backtrace.h:92] Envoy version: 0/1.24.0-dev/test/DEBUG/BoringSSL [2022-08-19 16:28:18.728][2869][critical][backtrace] [./source/server/backtrace.h:96] #0: Envoy::SignalAction::sigHandler() [0x471f468] [2022-08-19 16:28:18.728][2869][critical][backtrace] [./source/server/backtrace.h:96] #1: __restore_rt [0x7ff635735200] [2022-08-19 16:28:18.761][2869][critical][backtrace] [./source/server/backtrace.h:96] #2: Envoy::SocketInterfaceSwap::SocketInterfaceSwap()::$_0::operator()() [0x3cfe802] [2022-08-19 16:28:18.789][2869][critical][backtrace] [./source/server/backtrace.h:96] #3: std::__invoke_impl<>() [0x3cfe7a5] [2022-08-19 16:28:18.818][2869][critical][backtrace] [./source/server/backtrace.h:96] #4: std::__invoke_r<>() [0x3cfe6ff] [2022-08-19 16:28:18.853][2869][critical][backtrace] [./source/server/backtrace.h:96] #5: std::_Function_handler<>::_M_invoke() [0x3cfe54f] [2022-08-19 16:28:18.883][2869][critical][backtrace] [./source/server/backtrace.h:96] #6: std::function<>::operator()() [0x3d02732] [2022-08-19 16:28:18.906][2869][critical][backtrace] [./source/server/backtrace.h:96] #7: Envoy::Network::TestIoSocketHandle::sendmsg() [0x3d01e1f] [2022-08-19 16:28:18.933][2869][critical][backtrace] [./source/server/backtrace.h:96] #8: Envoy::Network::Utility::wri

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.

[github-actions[bot]]

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted" or "no stalebot". Thank you for your contributions.