TCPTransport: multiple "dictionary changed size during iteration" issues

[rayrapetyan]

opensearch-benchmark 0.0.2 thespian 3.10.1

While running opensearch-benchmark (based on thespian) I'm sporadically getting following errors in logs:

2022-03-27 00:35:36.463023 p374 ERR xmit UNcaught exception dictionary changed size during iteration; aborting intent. Traceback (most recent call last): File "/opt/sm/python/lib/python3.9/site-packages/thespian/system/transport/TCPTransport.py", line 801, in _nextTransmitStep return getattr(self, 'next_XMIT%s' % intent.stage, File "/opt/sm/python/lib/python3.9/site-packages/thespian/system/transport/TCPTransport.py", line 830, in _next_XMIT_1 if any(T for T in self._transmitIntents.values() File "/opt/sm/python/lib/python3.9/site-packages/thespian/system/transport/TCPTransport.py", line 830, in if any(T for T in self._transmitIntents.values() RuntimeError: dictionary changed size during iteration

2022-03-26 23:28:41.846563 p844 ERR Actor osbenchmark.worker_coordinator.worker_coordinator.Worker @ ActorAddr-(T|:19163) transport run exception: Traceback (most recent call last): File "/opt/sm/python/lib/python3.9/site-packages/thespian/system/actorManager.py", line 87, in run r = self.transport.run(self.handleMessages) File "/opt/sm/python/lib/python3.9/site-packages/thespian/system/transport/wakeupTransportBase.py", line 74, in run rval = self._run_subtransport(incomingHandler, max_runtime) File "/opt/sm/python/lib/python3.9/site-packages/thespian/system/transport/wakeupTransportBase.py", line 80, in _run_subtransport rval = self._runWithExpiry(incomingHandler) File "/opt/sm/python/lib/python3.9/site-packages/thespian/system/transport/TCPTransport.py", line 1098, in _runWithExpiry partition(TCPTransport._waitForSendable, File "/opt/sm/python/lib/python3.9/site-packages/thespian/system/utilis.py", line 207, in partition return foldl(appendLeftOrRight, inp_iterable, (output_type(), output_type())) RuntimeError: dictionary changed size during iteration