Improve push sync success rate

[metacertain]

Summary

The chunk specific 5 minute expiring skiplist in push sync creates an environment in which already tried paths (next hop nodes) are avoided by originators and forwarders for a limited time frame for a specific chunk address. This increases the probability of working around malfunctioning forwarding nodes (by avoiding them), and creates a probability to avoid malfunctioning final destination (closest) nodes.

A malfunctioning destination node can be described as one that is refusing to store a chunk with a valid postage stamp that is closest to it's own address out of the connected addresses.

Working around such a node can be understood as when another in-neighborhood node already avoids this closest node by an expiring skiplist entry, and instead of pushing to this closest node, it proceeds to do act of immediate replication, storage receipt signing, and responding to the upstream with the created storage receipt.

This secondary in-neighborhood node who establishes storage, also needs to have any other closer address already on the expiring skiplist for the specific chunk.

One likely candidate for this position is the second closest node, however, this only works if this second closest node receives 2 requests for the same chunk within this 5 minutes of expiry.

As near destination nodes tend to be connected to most of the destination neighborhood, we can expect some bias towards forwarding from outside the neighborhood to immediately to the closest node within the neighborhood.

The probability that the last forwarder already has the closest node on expiring list (and forwards to second closest instead) also depends on whether the same forwarder was requested twice for the same chunk within a 5 minute timeframe.

The number of potential last forwarders (connected to the closest node) equals to the number of nodes within neighborhood (min 4), and the nodes included in the bins between it and ( "Closest node PO (~neighborhood depth + 2)" - "Average PO gain per hop" ) proximity order (those nodes from lower bins that are connected to the whole destination neighborhood because of balanced connections). Since average PO gain per hop is currently 5, we can expect this range of last forwarders currently to be at least 4 * 2 ^ 3 ~ 32 nodes. So since there's a relatively large number of last forwarders, hitting the same last forwarder with consecutive requests can be estimated as 1/32 chance at most.

And, we need the second closest node hit twice so that it actually proceeds to establishing storage, which can happen by chance, or when 2 different last forwarders are requested 2 times for the same chunk within 5 minutes.

The identified problem is, because of these probabilities, it might take a large number of retries within 5 minutes for these chances to be hit, however, the originator might only make a limited number of retries within 5 minutes, that might not be enough for this to succeed, because after that it has all the possible downstream peers on expiring skiplist itself, making the process reset and start out with mostly expired skiplist entries near the destination as well by the time it is tried again by the originator.

Motivation

Push sync reliability improvement

Implementation

Possible improvements over the situation

• Increasing originator retries by expiring skiplist reset on originators when originator is returned as the closest node while not being in neighborhood of the chunk
• Multiplexing proposal making the probability of establishing storage by some other in-neighborhood node higher (this would decrease the condition of success to only hitting 1 last forwarder 2 times, as the second closest node would proceed to establish storage for the first time it receives the request)

Drawbacks

[ldeffenb]

I actually have 4 chunks (out of 55 million) in one of my source nodes that steadfastly refuse to get back into the swarm via a PUT /stewardship API invocation. I'll see if I can determine exactly what the chunk IDs are that fail to get back into the swarm.

In other words, I concur that this is a really need fix, whatever the fix ends up being. I wonder if there's a way for a node to determine that it is "close enough" to the "closest" and simply store the pushed chunk, return the receipt, push it to the remainder of the neighborhood (the closest of which might still reject the chunk), and then rely on pullsync to finish getting the chunk to the peers that should have it.

I did have one pusher/kademlia hack during the original testnet era that would attempt to establish a direct connection to a "closer" known peer whenever a pushsync of a chunk failed. It did this regardless of the current depth and/or connected peer count and worked surprisingly well at getting a receipt when originally pushing chunks. In hindsight, I see that this would work because it would enhance the chances of hitting the "next closest" peer more frequently than relying on the multi-hops ordinarily required to get into a distant neighborhood.

[ldeffenb]

Sorry for the delay, but I've located the 4 /bytes uploads and the associated chunks that refuse to pushsync into the swarm right now. Not that it matters, but the 1st and 3rd are mantaray manifest nodes and the 2nd and 4th are file contents, all of which were originally uploaded with the /bytes API and locally pinned.

Ref: 154cae44b4e3fa65f36fa1a3fa41c97e074bca8127da6a62065952a54573b95f Chunk: 154cae44b4e3fa65f36fa1a3fa41c97e074bca8127da6a62065952a54573b95f

Ref: 30df1884f7af6f5760b005e23540d949e84719d3717ad881d2e60256f180d9fe Chunk: 259e1dc186336ebadc214793c237ec820eeaf457b4bc14f173a08241b5384fad

Ref: 2a30399b0452dc93af4a35339cd7aebf90f16469c453d9fdb762ae767141a513 Chunk: 2a30399b0452dc93af4a35339cd7aebf90f16469c453d9fdb762ae767141a513

Ref: 5a6f9306634417ee74aa0b8eca71ba2ee7fdea6cf50fd41cdbf83c1c29ded701 Chunk: cb65ecace31c68986ecc9b2e0f6f7e459ae9c94883c56fb1f0b55b9da65f6230

The log of the pushsync failures is attached showing the peers that my pusher node in neighborhood 0x6127891... attempted to use and their resulting error. It'd be wonderful if pushsync returned the actual error instead of simply resetting the stream! pushFail.txt