test flake: RebalancerTest.BindToPortUpdate

[kyessenov]

[ RUN      ] IpVersions/RebalancerTest.BindToPortUpdate/IPv4
[2022-08-11 16:48:57.147][566][critical][assert] [source/server/connection_handler_impl.cc:300] assert failure: absl::holds_alternative<std::reference_wrapper<ActiveTcpListener>>( active_listener->get().typed_listener_) && active_listener->get().listener_->listener() != nullptr.
[2022-08-11 16:48:57.147][566][critical][backtrace] [./source/server/backtrace.h:104] Caught Aborted, suspect faulting address 0xfffe00000011
[2022-08-11 16:48:57.148][566][critical][backtrace] [./source/server/backtrace.h:91] Backtrace (use tools/stack_decode.py to get line numbers):
[2022-08-11 16:48:57.148][566][critical][backtrace] [./source/server/backtrace.h:92] Envoy version: 0/1.22.4/test/DEBUG/BoringSSL
[2022-08-11 16:48:57.229][566][critical][backtrace] [./source/server/backtrace.h:96] #0: Envoy::SignalAction::sigHandler() [0x81d204e]
[2022-08-11 16:48:57.308][566][critical][backtrace] [./source/server/backtrace.h:96] #1: __tsan::CallUserSignalHandler() [0x2d8282c]
==================
WARNING: ThreadSanitizer: data race (pid=17)
  Read of size 8 at 0x7b540004a8f0 by thread T24:

https://dev.azure.com/cncf/envoy/_build/results?buildId=114803&view=logs&j=d1f76054-8f79-554b-6f4a-11d6a63b8e00&t=266e17e3-d213-54b5-deef-0dcee01da137

cc @soulxu might be related to your recent PRs