Revisit how Companion does preload at public gateways

[lidel]

Background

Right now Companion preloads uploaded content via HTTP HEAD requests to a public gateway. Public gateway can be customized by the user, and there is an option on Preferences screen to disable preload.

The js-ipfs and infra teams are working on a solution that will fix discoverability and restore "realtime feel" in browser contexts: https://github.com/ipfs/js-ipfs/issues/1459. In short there will be a set of "official" preload gateways that will be used for every write operation made by js-ipfs when running in web browser.

UPDATE: recursive preloads are now possible with:

https://node0.preload.ipfs.io/api/v0/refs?arg={CID}&r=true
https://node1.preload.ipfs.io/api/v0/refs?arg={CID}&r=true

TODO

Update preloadAtPublicGateway and in addition to existing HTTP HEAD request to user-specified public gateway, do additional asynchronous recursive preload via HTTP GET request to one of preload nodes. In case of "quick-upload" screen, the CID should be of the wrapping directory.

Open Questions

Given discussion in https://github.com/ipfs/js-ipfs/issues/1459, questions arise:

• [ ] (1): should we switch from HTTP HEAD per CID to /api/v0/refs?r=true&arg=<cid> ?
  • :+1: when adding N files you only need 1 preload request (for root) to /api/v0/refs instead of N+1 HTTP HEADs (one per file + wrapping dir).
  • :-1: refs requires /api/ to be exposed to the internet, some public gateways hide everything but /ipfs/.
    • This means we need to use specific gateway instead of the one picked by the user, or implement a fallback to HTTP GET if refs returns statuc code different than 200.
• (2): should we enable preload for all window.ipfs operations? (mirroring behaviour of js-ipfs)
  • initial answer is yes, the rationale for doing this is we want to ensure fantastic UX and DX and key to that is for dapp data to not disappear from the swarm immediately when user closes laptop.
  • Given js-ipfs will be doing preload by default starting with next release, should we limit it to state when running against non-js-ipfs backend?
  • If implemented, global "preload" checkbox in Companion should also control preload in window.ipfs
• [ ] (3): Should global "preload" checkbox in Companion disable implicit preload when running against embedded js-ipfs as the backend ?
  • initial answer is yes, otherwise user is confused why preload requests are still sent by the browser

[alanshaw]

FYI the HEAD request won't preload a file if it is bigger than max chunk size (unless you traverse the tree that's created and send a request for EVERY node that's created, as well as all it's children). This is why js-ipfs uses the refs API.