Performance updates to DNS discovery

[davemec]

Description

• Modify DNS discovery code to not dump all of the results into discovery immediately but slowly add them in batches of roughly 25 at a time
• Modify to not always traverse the DNS tree in the same order, that puts pressure on “earlier” leaves. Perhaps adding in a random xor and then using that to sort the depth first traversals.
• Modify trie traversal logic to depth first and emitting nodes as trie leafs are encountered

[iamhsk]

@garyschulte Could you help provide a bit more context on what this issue is trying to achieve? Is this issue still relevant? If not, we'll close it.

[garyschulte]

This may be relevant to a prior DNS Discovery implementation, but I believe it is still a viable improvement to our current use of tuweni's DNSDaemon. We use DNSDaemon to periodically poll DNS txt records for ethereum nodes for the network we are on (if the dns entry exists).

I am skeptical of the first bullet item, batching. I believe a better batching behavior would be to batch the DNS queries in DNSDaemon rather than recursively hitting an upstream dns server for all the nodes found. This would be an improvement in tuweni, rather than besu (and may or may not already have been addressed)

The second item, if we randomize our access to the returned entries we could spread the peering load across all nodes rather than attempting to connect to the same public nodes in the same order for all besu nodes. This is good for both besu and for the p2p network

Third item is a variation of the second, but since tuweni returns nodes in a list rather than a tree structure it isn't relevant to besu.

[shemnon]

The biggest gain I see is quickly finding good peers. If we start in the middle of the trie, stream nodes one by one, and add the to the pool as we find them we can get good peering much quicker than waiting for a complete walk of the dns tree.

I did a spike while I was at Reddit and needed some stress relief. I'll see if I can find it and post it. I didn't use Tuweni code.

[shemnon]

Here's the proof of concept spike code. Not done. Probably not worth the effort to bring in compared to other priorites. Possibly a good entry issue for new contributors.

https://github.com/shemnon/besu/tree/streamdns

[non-fungible-nelson]

Can we close in favor of other peering fixes and DNS discovery outsourced to Tuweni library? Do we need to implement #4018 in light of this --> post-Merge we will be looking to move to discV5 discovery regardless, which will remove the need for v4 improvements.