Unreachable providers for popular CIDs

[yiannisbot]

We've recently started measuring the performance of PL websites over kubo. We've been presenting some of these results in our weekly reports and we're also now putting more results at probelab.io (e.g., https://probelab.io/websites/protocol.ai/ for protocol.ai). As a way to get more insight into why the performance is what it is, we have collected the number of providers for each one of them. That will enable us to see if, for instance, there are no providers for a site.

We've found an unexpected result, which might makes sense if one gives a deeper thought into it: there are a ton of unreachable providers for most of the websites we're monitoring as shown in the graph below for protocol.ai. Note that the stable providers for protocol.ai should be two, i.e., that's where we currently pin content.

This happens because clients fetch the site, re-provide it and then leave the network, leaving stale records behind. In turn, this means that popular content, which is supposed to be privileged due to the content addressing nature of IPFS, is basically disadvantaged because clients would have to contact tens of "would be" providers before they find one that is actually available.

I'm starting this issue to raise attention to the issue, which should be addressed asap, IMO. We've previously discussed in slack a couple of fixes, such as setting a TTL for provider records equal to the average uptime of the node publishing the provider record. However, this would be a breaking protocol change and would therefore not be easy to deploy before the Composable DHT is in place. Turning off reproviding (temporarily, until we have Composable DHT) could be another avenue to fix this issue.

Other ideas are more than welcome. Tagging people who contributed to the discussion earlier, or would likely have ideas, or be aware of previous discussion around this issue: @Jorropo @guillaumemichel @aschmahmann @lidel @dennis-tra