Shared community indexer/crawler

[jflix89]

Requirements

• [X] Is this a feature request? For questions or discussions use https://lemmy.ml/c/lemmy_support
• [X] Did you check to see if this issue already exists?
• [X] Is this only a feature request? Do not put multiple feature requests in one issue.
• [x] Is this a UI / front end issue? Use the lemmy-ui repo.

Is your proposal related to a problem?

There is practically no discovery built into Lemmy instances for communities outside one's own, even when users share links for community suggestions they often break and make the Jerboa app crash.

Describe the solution you'd like.

Implementing https://lemmyverse.net/communities as a shared indexer for communities, even if instances don't automatically pull any data from the instances themselves. For the time being just having a link to this crawler at the top of the website would be an improvement.

Describe alternatives you've considered.

Implementing https://lemmyverse.net/communities as part of the backend of the search function.

Additional context

No response

[tgxn]

Hey, I'm the Dev of https://lemmyverse.net/ 👋.

I think that Lemmy needs some more discovery features, but I'm not sure that using an external website is the right way to implement it - That installs a centralized component to the system and also a single point of failure for discovery.

I think that Mastodon uses Relay servers to enable quicker/easier discovery of instances. Maybe a similar project would work for Lemmy. (I'm thinking a server that communicates with REST, and is used by the backend to initially poll for a list of all known communities (by that relay).

There's some discussion happening in this ticket too https://github.com/LemmyNet/lemmy/issues/2951

[pe1uca]

I have two proposals based on this endpoint lemmy.external/api/v3/community/list?type_=Local:

• Handle it in the UI with a page like lemmy.local/i/lemmy.external which will list the response and allow easy import of the communities (instead of manually typing in the search page).
  This will require the instances page to link to this new page first instead of the actual instance (similar to how a link to an external community is handled).
• Handle it in the server with a scheduled task every day or two which will query a list of instance and just import the community name to insert into the community table.

I'm leaning more for the second option.
It does require a bit more work to include a configuration of the time between pulls and to configure the list of instances to pull from (since maybe some admins won't want to pull from all the federated instances by default)
But this will put less pressure on other instances since the list will be stored in the instance's DB and won't require changes in the flow of the clients (lemmy-ui, jeroba, and any other) which users would need to learn about, just the backend which will handle it automatically.

One thing to consider is currently the discovery flow automatically imports 20 posts from that community, for this flow I think it'll be better to postpone this import until the first user subscribes or access the community.
AFAIK the current discovery flow is done through the endpoint lemmy.local/api/v3/resolve_object but I don't actually know how it works to know if it's possible to separate the import.

[argentcorvid]

I have no idea how this actually works internally but:

Each instance maintains a list of its own communities, so couldn't just that list be published/subscribed to for federation purposes? Would also probably want something to subscribe to for changes to the list so that you wouldn't have to get the whole list every time. Hold off on the actual importing of posts until a user actually subscribes, but maintain a list of 'possibles' locally.

[jflix89]

As for costings;

I feel it would be inevitable that the entire lemmyverse share an official indexer, even if instance Admins could technically opt for a third-party solution, or none at all like we already have. This matter could go hand in hand with creating a donations solution in the UI.

Lets say that the UI donations link is agreed and implemented, instances that had opted in could share 5% of their incoming funds to a LemmyNet indexer for running costs.

I know its not a truly decentralised solution, but with the ability to opt-out, it would be greater than no indexer and there could be a constitution of rules, voted by all participating instances.

[Nutomic]

Duplicate of https://github.com/LemmyNet/lemmy/issues/3105