deadlock in StringFamilyTest.MGetSet

[romange]

Full log is here: https://github.com/dragonflydb/dragonfly/actions/runs/6605113149/job/17939973096

specifically:

50: 0x560764532e64  boost::context::detail::fiber_ontop<>()
50: 0x5607645326ae  boost::context::fiber::resume_with<>()
50: 0x560764531bae  util::fb2::detail::FiberInterface::SwitchTo()
50: 0x5607645131ed  util::fb2::detail::Scheduler::Preempt()
50: 0x56076289b337  util::fb2::detail::FiberInterface::Suspend()
50: 0x56076289d086  util::fb2::EventCount::wait()
50: 0x560763a491cb  util::fb2::EventCount::await<>()
50: 0x560763a47740  dfly::Transaction::WaitForShardCallbacks()
50: 0x560763a217e1  dfly::Transaction::ScheduleSingleHop()
50: 0x560763218d93  _ZN4dfly11Transaction18ScheduleSingleHopTIZNS_12_GLOBAL__N_110SetGenericEPNS_17ConnectionContextERKNS_6SetCmd9SetParamsESt17basic_string_viewIcSt11char_traitsIcEESC_bEUlPS0_PNS_11EngineShardEE_EEDTclfp_fpTLDn0EEEOT_
50: 0x5607631fbcac  dfly::(anonymous namespace)::SetGeneric()
50: 0x560763201f4c  dfly::StringFamily::Set()

and

50: 0x560764531bae  util::fb2::detail::FiberInterface::SwitchTo()
50: 0x5607645131ed  util::fb2::detail::Scheduler::Preempt()
50: 0x56076289b337  util::fb2::detail::FiberInterface::Suspend()
50: 0x56076289d086  util::fb2::EventCount::wait()
50: 0x560763a491cb  util::fb2::EventCount::await<>()
50: 0x560763a47740  dfly::Transaction::WaitForShardCallbacks()
50: 0x560763a217e1  dfly::Transaction::ScheduleSingleHop()
50: 0x56076320e51c  dfly::StringFamily::MGet()
50: 0x560762e9ae9b  fu2::abi_400::detail::invocation::invoke<>()

meaning two single shard invocations deadlocked. TODO: to add more debugging to deadlock functions:

1. Print transactions in shard queues.
2. Print their state, id.

[romange]

@adiholden I did not know you assigned @chakaz and I did it myself (the debugging part).

[adiholden]

Sorry I did not know you were working on this @romange . Please assign on yourself when you are working on a issue

[romange]

Does not reproduce