noosphere icon indicating copy to clipboard operation
noosphere copied to clipboard

`LinkRecord` can propagate before blocks are available

Open cdata opened this issue 2 years ago • 0 comments

In https://github.com/subconsciousnetwork/noosphere/pull/253 we introduced a worker thread to help the gateway interact with the Noosphere Name System. This worker thread runs in parallel with the IPFS syndication worker thread introduced in #177.

This arrangement has worked well for small, incremental updates to spheres over time. However, in the case of a large update with many blocks, this arrangement results in an undesirable phase mismatch where the name system will receive an updated LinkRecord for a new sphere version before all of the blocks have managed to be syndicated to IPFS.

The end-user experience that results from this is that a user will try to traverse to a new version of a peer that should exist, but they won't be able to get the blocks for that version until some time in the future.

In my tests, the latency difference between the two processes is mainly bottlenecked by Kubo: one Kubo instance is only able to service syndication requests up to the number of cores on the local machine before it starts to queue requests and everything slows down (empirically / as observed from the outside, at any rate).

Perhaps the best solution to this problem is for name system and IPFS syndication tasks to work cooperatively / in serial. First, a syndication job is dispatched to the IFPS worker, and then when that job is completed it forwards a name system publish job to the NNS worker. This would ensure that blocks are always available in IPFS before an updated name system record is published to peers. All other work to be done here is mainly tuning syndication and Kubo to reduce syndication latency as much as we can.

cdata avatar Aug 28 '23 17:08 cdata