Co-hosting communities across instances (e.g. "sharding")

[calculuschild]

• [x] Did you check to see if this issue already exists?
• [x] Is this only a single feature request? Do not put multiple feature requests in one issue.
• [ ] Is this a question or discussion? Don't use this, use https://lemmy.ml/c/lemmy_support.
• [ ] Is this a UI / front end issue? Use the lemmy-ui repo.

Describe the feature request below

There is some uncertainty from Reddit users coming to Lemmy that "my community could vanish without notice!" if an instance goes offline or gets blocked from federating, with some expressed desire to be able to migrate communities across instances #3057. There is also some concern about federation scaling eventually becoming burdensome as traffic increases #3062 .

I propose a form of co-hosting of a single community across multiple instances, similar to sharding a database. I'm not even sure this is possible, but since Federation already replicates data across instances, I don't think its too big a stretch. I'm picturing one instance designated as the "primary" instance, but then other instances in cooperation can be designated as co-hosts that will step up to handle requests while the owner is down, and perhaps eventually a cohost might be promoted to the new primary with permission from the original community mods.

it seems like it might provide a few benefits:

• Distributing data storage over many machines -- increases the total storage capacity of the system
• Balancing traffic load around different instances -- reduces the risk of overloading a single machine hosting a particularly large community
• Increasing availability and fault tolerance -- if an instance goes down, the community remains accessible.
• ~~Encourage or facilitate combining fractured communities. -- since /games on one instance can be the same community as /games on another, users won't feel the pressure to make duplicates "in case one goes down".~~ Edit: this is just another possible side effect. Let's not fixate on it as it's not the main point.

[Andonome]

Why not just subscribe to /c/games on multiple instances?

Having a single point of truth threatens to remove one of the joys of federation.

[calculuschild]

I can edit my post to clarify that combining communities was just one possible side effect. The main issue is am addressing is the vulnerability of communities at the mercy of an instance going offline.

My point with that bullet item was that people should be making duplicate communities because they are interested, not out of fear that instances might drop at any moment.

[jazir555]

I can edit my post to clarify that combining communities was just one possible side effect. The main issue is am addressing is the vulnerability of communities at the mercy of an instance going offline.

My point with that bullet item was that people should be making duplicate communities because they are interested, not out of fear that instances might drop at any moment.

I think a blockchain type model that is reminiscent of the way bitcoin and other cryptocurrencies are structured may be applicable here. A shared source of truth that all instances can access that are synced between them, but remains completely independent of/between all of them.

Sharding across the chain would probably be more complex than the current way that blockchain is implemented via bitcoin since other federated servers can willfully drop the instance from federating and appearing in their feed, but as a base model I think it may make sense.

Thoughts?

[Nutomic]

FEP-2100 has a proposal along these lines. Apparently it is implemented in a GNU social plugin (the link is not working for me). However it would take a lot of work to make this production ready for Lemmy.

[Artraxon]

If this was something where communities opt-into, you could organize the content storage with a DHT. I do not really know how lemmy looks like under the hood, but this could in theory at least allow to leverage the resources of multiple instances.

Of course, nodes could in addition always replicate the stored items for redundancy, but they wouldn't have to serve it.

Whether this would be worth it depends at the end on how much the bandwidth / processing posts and comments make up of the total bandwidth / processing and if the overhead of this would be worth it, because every node still needs a complete index.

So maybe this could be interesting for instances and communities that want to host pictures themselves?

[calculuschild]

Of course, nodes could in addition always replicate the stored items for redundancy, but they wouldn't have to serve it.

My understanding is this already happens when an instance subscribes to a remote community. If that's true, it seems we already have a good foundation for next steps.

[JustinLex]

I think that FEP is exactly what we want. I think it would relieve a lot of user's concerns about instances shutting down.

[etyarews]

I've started using Lemmy recently, and I've noticed that some communities in popular instances were created a while ago, but a number of them eventually grew enough and became whole instances. Or changed from a "general" instance to a more "themed" one (like for technology, or for a specific country).

The current model makes a change in instance to be a very difficult process, and that isn't even mentioning the actual issue of instances shutting down. Co-hosting or syncing would fix a bunch of those small issues.

[Die4Ever]

oh hey we were just talking about this https://programming.dev/comment/11585424

would definitely be a cool feature!