shuffle route_setup_nodes array in visor config on failure to dial
For a long time a redcurrant issue has been that more often than not the default (first entry) route setup-node on prod cannot be dialed.
A simple optimization might be - when a setup-node cannot be dialed after the default number of dial attempts, move that setup-node key to the last position (len -1) and move all the other keys in that array forward
thus, the initial config:
"route_setup_nodes": [
"0324579f003e6b4048bae2def4365e634d8e0e3054a20fc7af49daf2a179658557",
"024fbd3997d4260f731b01abcfce60b8967a6d4c6a11d1008812810ea1437ce438",
"03b87c282f6e9f70d97aeea90b07cf09864a235ef718725632d067873431dd1015"
],
would be re-ordered as follows and flushed to the file:
"route_setup_nodes": [
"024fbd3997d4260f731b01abcfce60b8967a6d4c6a11d1008812810ea1437ce438",
"03b87c282f6e9f70d97aeea90b07cf09864a235ef718725632d067873431dd1015"
"0324579f003e6b4048bae2def4365e634d8e0e3054a20fc7af49daf2a179658557",
],
assuming that the second key was successfully dialed.
Additionally, we need some debug logging that indicates no setup-nodes could be dialed when all the setup-nodes in the list have been exhausted, before (I suppose) trying again to dial all of them in the original order.
I believe this is better than just randomizing the order of the keys as it's responsive to feedback of unsuccessful dial attempts.