Allow Web Pages to query companion for the best available node

[whizzzkid]

Is your feature request related to a problem? Please describe. companion right now redirects requests based on regex for a select few domain. ipfs.io e.g. has been throwing 429 for me. So if a developer embeds cloudflare-ipfs.com companion does not redirect this request to local-node.

Describe the solution you'd like There should be a mechanism where the page can request the best available local node and use that to make requests. Something like page <-> extension communication, metamask does something similar.

[lidel]

companion right now redirects requests based on regex for a select few domain. ipfs.io e.g. has been throwing 429 for me. So if a developer embeds cloudflare-ipfs.com companion does not redirect this request to local-node.

AFAIK this is incorrect: ipfs-companion uses https://www.npmjs.com/package/is-ipfs and will redirect every gateway URL that follows conventions described in https://docs.ipfs.tech/how-to/address-ipfs-on-web. No specific hostname is hardcoded, redirection will work for any compatible URL, as long as CID or valid DNSLink is present.

If this is no longer happening, then we have a bug.

There should be a mechanism where the page can request the best available local node and use that to make requests. Something like page <-> extension communication, metamask does something similar.

There is a lot prior art aorund this, happy to discuss in January when I am back, but for now quick brain-dump:

• No, websites should be oblivious of my gateway choice.
  • When in doubt, remember who is the agent of user, and who is not. Websites are not user agents, browsers are.
  • Websites do not care about your good, may load arbitrary JS from third-party hosts. They may load ads and tracking JS – they can not be trusted by default, and adding another layer of popups for permissions is making UX of web platform worse, not better.
• What MetaMask does for blockchain and identity does not translate into a good pattern to follow for IPFS.
  • Reinventing "window.ipfs" won't help with 429 errors, nor will be a good UX/DX – been there, done that, we removed it in 2020 – some notes in https://github.com/ipfs/ipfs-companion/issues/589#issuecomment-712356448
  • Rule of thumb: it should not be a web page asking the browser for gateway URLs, but the other way around: browser (Companion) detecting IPFS resources and upgrading the transport based on user's preferences.
    • This is how our extension works today, and we should not regress in this matter.
    • This also follow existing paradigm where web browser automatically follows HTTP 301 redirects and these redirects are transparent for JS running on a page.

ipfs.io e.g. has been throwing 429 for me.

429 on a public gateway are something Companion can detect and recover from, but you need more than one Public Gateway to handle scenario when local gateway is absent and the default public gateway is down.

Changing Public Gateway setting from being a single URL to a list of public gateways sounds like a sensible step.

Long term, this problem will go away when we have native ipfs:// – abstracting backend away, allowing for seamless resume without changing Origin.

[whizzzkid]

Thanks for the context, for some reason redirection was spotty when I posted the issue, I need to get to the root of this.

Changing Public Gateway setting from being a single URL to a list of public gateways sounds like a sensible step.

Absolutely, in fact the gateway checker list and tests could be leveraged for this purpose, we can use the top-N public gateways closest to the user for this.

[SgtPooki]

@whizzzkid bifrost has some open issues around 429 issues. I believe they've nailed most of them down now, but we should ensure that companion works properly if a gateway is throwing errors. Based on the convo so far, it sounds like this task has turned into the following action items:

• [ ] Ensure gateway errors are handled appropriately by companion (supposed to already be done, may have a bug)
• [ ] Allow users to add multiple different gateways
• [ ] Allow users to automatically select top-N public gateways closest to them (checkbox for: use public-gateways?
  • [ ] This may require adding a user input for allow/block listing of public gateways
  • [ ] This task depends on https://github.com/ipfs/public-gateway-checker/issues/192

Do any of these tasks remain valid in the long-term scenario where ipfs:// is supported natively?

If not, do any of these tasks provide a decent QoL improvement with minimal dev-effort until then?

[lidel]

Allow users to add multiple different gateways Allow users to automatically select top-N public gateways closest to them (checkbox for: use public-gateways?

We need to be very careful what we suggest to users as default or fallback.

Public gateways are run by random people, and people who use public gateway for fetching already deserialized data delegate IPFS to a third-party that is other than Protocol Labs. (IPFS happens on remote server, and not your machine).

Unless we are able to guarantee user is fetching data as raw block or CAR and verify it matches expected CID, we should not assist user in delegating trust to some random person on the internet.

If not, do any of these tasks provide a decent QoL improvement with minimal dev-effort until then?

I'd say no. Only verifiable response types are relevant for native ipfs://, and Companion, given web and webextension APIs available today, does not have ability to serve verified bytes on its own (we delegate it to IPFS Desktop/Brave/Kubo). We should not make it easier for people to get MITM'd.

If we want to invest time towards native ipfs://, prioritize https://github.com/ipfs/ipfs-companion/issues/1128 to help Igalia with handler discussions (redirect handler today means it is easier to discuss native one tomorrow).

[whizzzkid]

@lidel we can defer this. This would require a portable version of the gateway checker. I'd say this is blocked till then.