Remove dependency on Kubo RPC and `node[0-2].delegate.ipfs.io`

[lidel]

IPNS resolution and raw signed records are fetched from Kubo RPC at node[0-2].delegate.ipfs.io nodes (details why: https://github.com/ipfs/bifrost-gateway/issues/5)

https://github.com/ipfs/bifrost-gateway/blob/49d8b1475fd4298f8f50a6557181810d542ddcac/routing.go#L91

Problem

PL wants to shut down delegate/preload nodes at some point: https://github.com/ipfs/ipfs/issues/499

Solution

Remove Kubo RPC code from bifrost-gateway.

Switch to support for /routing/v1/ipns and/or some external trustless gateway that supports application/vnd.ipfs.ipns-record responses (IPIP-351). Make the default behavior work out of the box when user is self-hosting with own trustless gateway.

Default: use the same backend as Block and CAR requests

When KUBO_RPC_URL is not set, bifrost-gateway should fetch IPNS record from PROXY_GATEWAY_URL via /ipns/id?format=ipns-record (IPIP-351)

This allows users to self-host using regular Kubo or other trustless-gateway based on boxo/gateway library as a backend, without having to run a dedicated service just for IPNS resolution.

There should be optional IPNS_RECORD_GATEWAY that overrides PROXY_GATEWAY_URL, in case someone wants to have a dedicate service for IPNS cache/lookups, separate from blocks and cars.

Or if we want to use this in Rhea, where PROXY_GATEWAY_URL is not used, and we have Saturn-specific orchestrator.

Opt-in: delegate to /routing/v1/ipns

Since /ipns/id?format=ipns-record described above depends on the backend gateway to support application/vnd.ipfs.ipns-record responses, this may not be possible to do in deployments like Rhea, because Saturn does not speak IPNS.

If IPNS_RECORD_GATEWAY ends with /routing/v1 HTTP APIs then things still work, as long we concatenate path. Namely, we would use https://cid.contact/routing/v1 and ask it for routing/v1/ipns/id from IPIP-379.

This depends on cid.contact implementing IPIP-379.

[lidel]

@mashi @willscott just for the sake of planning, do we have any ETA for cid.contact supporting IPNS lookups (IPIP-351)?

If it is not happening in next 1-2 months, we could still do this cleanup if we repurpose delegate nodes and expose /ipns/id?format=ipns-record there (they already run kubo, so it is only a nginx config change).

[lidel]

Filled https://github.com/protocol/bifrost-infra/pull/2701 to enable /ipns/id?format=ipns-record (Accept: application/vnd.ipfs.ipns-record) lookups at node[0-2].delegate.ipfs.io.

Once we have it in prod, we can remove Kubo RPC related code incl. KUBO_RPC_URL and implement IPNS_GATEWAY_URL

[lidel]

https://github.com/protocol/bifrost-infra/pull/2701 got merged, and we now can switch routing.go to fetch signed IPNS records via:

$ curl -s -H 'Accept: application/vnd.ipfs.ipns-record' "https://node2.delegate.ipfs.io/ipns/k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r" -o test.ipns-record

And then verify signature locally:

$ ipfs name inspect  --verify k51qzi5uqu5dgutdk6i1ynyzgkqngpha5xpgia3a5qqp4jsh0u4csozksxel2r ./test.ipns-record
Value:          "/ipfs/bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi"
[...]
Valid: true
[...]

With this, adding IPNS_RECORD_GATEWAY as a list of URLs will allow us to support both endpoints from IPIP-351 and IPIP-379.

[hacdias]

Remove Kubo RPC code from bifrost-gateway.

Maybe I'm overlooking something: does this mean we should also remove the proxy code to the RPC API? If we don't, we'll still depend on those delegation nodes.

https://github.com/ipfs/bifrost-gateway/blob/b852ebadb04011684b935a949c50de1e4c86e6c6/handlers.go#L166-L169

[lidel]

@hacdias you are right, we need to keep KUBO_RPC_URL for these legacy /api/v0 paths handled by newKuboRPCHandler – but these are only redirects, and we should only register /api/v0 listener when KUBO_RPC_URL is set (see https://github.com/ipfs/bifrost-gateway/pull/185#discussion_r1289266069)