Use recursive gateways (HTTPGatewayRouter) as a fallback when other routers fails

[2color]

Background

Right now the Routing class, which wraps the different router implementations (of which are 3) treats all routers as equal, and calls them in parallel (by using the merge from it-merge):

https://github.com/ipfs/helia/blob/b723a1d9910c69064eafba965d4d96d753b91297/packages/utils/src/routing.ts#L57-L60

This means that if you have both:

• delegatedHttpRouter (with delegated routing endpoint)
• HTTPGatewayRouter (for recursive gasteways),

There is no programmatic way for your to prioritise direct retrieval over a recursive gateway.

Suggestion

In the following discussion, the idea of using recursive gateways as a "last resort" fallback came up. This means that if direct retrieval fails (because there are no providers or because helia fails to retrieve directly from a provider), it falls back on the recursive gateway to fetch data.

[SgtPooki]

I don't think we want to always fallback to recursive gateways.. but a solution that could get us the desired behavior is extending the Routers interface to support a priority key and then sorting on that prior to https://github.com/ipfs/helia/blob/b723a1d9910c69064eafba965d4d96d753b91297/packages/utils/src/routing.ts#L60

[SgtPooki]

I don't think we want to always fallback to recursive gateways.. but a solution that could get us the desired behavior is extending the Routers interface to support a priority key and then sorting on that prior to

https://github.com/ipfs/helia/blob/b723a1d9910c69064eafba965d4d96d753b91297/packages/utils/src/routing.ts#L60

I think if we want to actually delay fallback routers we might need some changes other than just a priority key, since they’re all kicking off at essentially the same time.. we could do something to indicate that certain routers should be run in parallel/serial?

[achingbrain]

You could add a delay to the HTTPGatewayRouter before it started it's search, that'd have the effect of allowing other routers to respond first, but... does the user care or do they just want whatever method is fastest?

[SgtPooki]

I think users will want the fastest methods, but I think the motivation here is deprioritizing the use of recursive gateways, and trying to do all the work on the user side first.

@lidel could speak more to primary motivations

[lidel]

Primary motivation is to ensure users running with default settings (helia, verified-fetch) avoid hitting default recursive gateway when a direct provider is available.

If custom recursive gateways are set by user, such user should be free to disable delay and make all methods race to get the fastest result. So maybe make that delay configurable (if not already)?

I would go as far, as have constructor check if recursive gateway is one of public goods (ipfs.io/dweb.link/trustless-gateway.link) and force set a delay if true (overriding user setting). It would ensure that by default people don't hammer public infra, and also have incentive to run their own.