Feature request: Add ipfs-search.com functionality into ipfs-webui

[SgtPooki]

Is your feature request related to a problem? Please describe.

Currently, IPFS Webui and Desktop to not allow for content discovery. The only way I know to discover content on ipfs is via ipfs-search.com. I would love be able to discover new ipfs content, or old interesting content, or any content on IPFS so I can better appreciate the value of all the data that exists.

Describe the solution you'd like

I want to be able to search for content on IPFS via ipfs-search's API and then see the results within the ipfs-webui on a results screen.

Rough plan we've discussed

V0

• [ ] Get API calls to ipfs-search working
• [ ] get input inside of webui calling ipfs-search
• [ ] render results from ipfs-search
  • [ ] Clicking on a result item will link you to that page (outside of webui) on ipfs-search, i.e. https://ipfs-search.com/#/search/detail/text/QmYByXDHpPJs6Q3V83YdDA9MiXHnHb1VCLCqSgGZ3FnqnU?q=user+design&page=1

V1

• [ ] Add cross-functionality between ipfs-search.com results and ipld-explore page functionality.
  • ideally, a user could "explore" an ipfs-search.com result within webui, or "ipfs-search" for similar content from the explore CID page. i.e. backlinks on both sides.
• [ ] Enhanced and simplified UX that handles EITHER explore(select & explore) CID submission, or search(discover) functionality within the SINGLE input at the top of webui screens.
• [ ] Results list
  • [ ] Items in the list allow you to access the content on ipfs-search (small target=blank icon) OR explore (explore CID) them easily, or VIEW them (videos/images/etc) within webui

Describe alternatives you've considered

• I've considered not adding this functionality and just redirecting users to ipfs-search.com, but I believe our large userbase could benefit from being exposed to ipfs-search.com through our familiar ipld-explore-cid functionality that already exists.

Additional context

• I've discussed this with the ipfs-search team face to face at #IPFSCamp22 and they're excited to collaborate on this and agree it would be a big win for both of us and our customers.
• One implementation we've talked about doing is overloading the CID Explorer input field, however, this may cause UX issues with users familiar with that input. We want to make sure that we do not break their current expectations, while also exposing them gently and encouragingly to the new functionality.

[femans]

@djmcquillan @SgtPooki Please don't hesitate to reach out to us (ipfs-search team) for any feedback, e.g. how to query or how we implemented something. You can also check all of our code and documentation on http://github.com/ipfs-search Looking forward to the result!

[lidel]

s/customers/users/ :)

I'll echo the usual concerns around using DNS-bound services in software that aims to be more robust than Web2.

What happens when DNS is down or ipfs-search.com gets censored on the default DNS resolver our desktop user have? Or when DNS + PKI cert are spoofed (MITM) to return malicious results?

If we hardcode https://ipfs-search.com these will be some problems introduced to places that use webui, e.g., IPFS Desktop, Brave.

@femans are you considering exposing service over libp2p? It would allow for securely querying search without reliance on DNS and PKI. fwiw, Kubo has support for exposing HTTP over libp2p, which should get us 80% there without changing anything on your end (/search could be used as-is)

[dokterbob]

@lidel libp2p sockets (as well as .onion) is something on the horizon and I do follow your arguments. I have hereby added it explicitly to our roadmap-in-progress.

However, I do have my reservations about the stability of these alternative sockets. Even if we do get a prototype working, I am uncertain whether making it the default for the 100k+ or so active daily users of the IPFS UI would provide an acceptable experience. (A challenge here would be how to scale the libp2p proxy service, apart from it generally not having received a lot of testing AFAIK.)

Here, I would also like to point out that IPFS implementations actively relay on DNS being available.

That having been said, I think using libp2p as a fallback could be a great idea. Is JS-IPFS available everywhere the UI is available?

[lidel]

I am uncertain whether making it the default for the 100k+ or so active daily users of the IPFS UI would provide an acceptable experience.

I was thinking about p2p-as-a-fallback (when DNS/HTTP is down) or if user explicitly opt-in to search over libp2p tunel via settings. This way, default traffic/load (from majority of users) would go over HTTPS.

IPFS implementations actively relay on DNS being available

Not for content-addressing.

DNS is used only for DNSLink and connecting to default bootstrappers via DNSaddr (but Kubo also has IP ones, so it still works without DNS).

IPFS Desktop node, including webui precached with each desktop node, works fine without regular HTTP and DNS:

• shut down down DNS completely and CIDs and signed IPNS records work fine
• block all HTTP and HTTPS traffic, install ipfs-desktop from pendrive, set up DNS over Tor, and even DNSLinks work again.

If we add hardcoded HTTP endpoint, this functionality will be broken in context where connectivity is limited or not existant (remote villages without WAN etc). So either we have a fallback, or need to hide such feature unless we confirm the HTTP endpoint is reachable.

I think using libp2p as a fallback could be a great idea. Is JS-IPFS available everywhere the UI is available?

No JS-IPFS. ipfs-webui works only with Kubo RPC: p2p http proxy can be controlled via /api/v0/p2p/* RPC commands.

[dokterbob]

I am uncertain whether making it the default for the 100k+ or so active daily users of the IPFS UI would provide an acceptable experience.

I was thinking about p2p-as-a-fallback (when DNS/HTTP is down) or if user explicitly opt-in to search over libp2p tunel via settings. This way, default traffic/load (from majority of users) would go over HTTPS.

I follow your point and libp2p tuneling is indeed on our roadmap, albeit not on the very top right now (in part because it is experimental and I see no sane solution to distribute a libp2p endpoint over multiple hosts).

• block all HTTP and HTTPS traffic, install ipfs-desktop from pendrive, set up DNS over Tor, and even DNSLinks work again.

If users can use DNS over Tor they can use Tor so they can use HTTPS.

If we add hardcoded HTTP endpoint, this functionality will be broken in context where connectivity is limited or not existant (remote villages without WAN etc). So either we have a fallback, or need to hide such feature unless we confirm the HTTP endpoint is reachable.

I see how you are worried about degraded UX and unmet user expectations where internet connections are degraded. My perspective is that where people have degraded internet to this point, they are used to stuff not always loading. Although I can imagine building a simple 'ping' check to see whether our API is available.

Would that address your concerns?

I think using libp2p as a fallback could be a great idea. Is JS-IPFS available everywhere the UI is available?

No JS-IPFS. ipfs-webui works only with Kubo RPC: p2p http proxy can be controlled via /api/v0/p2p/* RPC commands.

That's a pity, it would be great if we can provide non-WebUI users with a fallback as well. :/