ipfs
feat: add active probing to HTTPGatewayRouter
Background
Arkadiy from the Internet Archive mentioned he wanted more control over the default HTTPGatewayRouter.
This demonstrates how you would add active probing to ensure CORS.
Description
This PR adds active CORS probing functionality to the HTTPGatewayRouter to ensure only CORS-compatible gateways are returned as providers. The implementation includes caching of probe results to avoid repeated checks.
Key changes:
- Introduces CORS probing with configurable cache duration
- Transforms gateway storage from simple PeerInfo array to GatewayStatus objects with metadata
- Filters providers based on CORS compatibility before yielding them
@parkan I'm missing a little context here - do we need this in the HTTP Gateway routing? I ask because the gateways it will return are from a hard-coded list supplied by the user (defaults), so if they aren't compatible with the client (due to lack of CORS support), they should be omitted from the list.
Checking CORS support for providers found via the delegated/libp2p routing makes more sense to me as these are arbitrary peers found by searching the network and not from a pre-supplied list?
@parkan I'm missing a little context here - do we need this in the HTTP Gateway routing? I ask because the gateways it will return are from a hard-coded list supplied by the user (defaults), so if they aren't compatible with the client (due to lack of CORS support), they should be omitted from the list.
Checking CORS support for providers found via the delegated/libp2p routing makes more sense to me as these are arbitrary peers found by searching the network and not from a pre-supplied list?
the specific situation that surfaced this feature was a transient misconfiguration of one of the specified http peers that broke CORS, which resulted in spamming them on every content load despite all requests failing
this happens periodically in our experience (likewise with TLS certificate expirations) and results in degradation from the user perspective, even when content is available with other peers
I'm not 100% sure specifically CORS probing (versus let's say denylisting a peer more broadly in case of fatal errors) is the right solution here, but it would have helped
FWIW https://github.com/ipfs/helia-verified-fetch/pull/261 has reduced the chattiness of ls operations significantly, which lowered the impact of events like the above
@achingbrain I should also note that I'm going to come back and review our delegated routing setup in the coming days/weeks, we're now running our own someguy which would in theory be where the error propagation ends (excepting cached entries)