LemmyNet
Backfilling threads
Requirements
- [X] Is this a feature request? For questions or discussions use https://lemmy.ml/c/lemmy_support
- [X] Did you check to see if this issue already exists?
- [X] Is this only a feature request? Do not put multiple feature requests in one issue.
- [X] Is this a backend issue? Use the lemmy-ui repo for UI / frontend issues.
- [X] Do you agree to follow the rules in our Code of Conduct?
Is your proposal related to a problem?
It is not possible to backfill a thread after discovering a Page object (e.g. when fetching it by id).
Describe the solution you'd like.
Publish a collection containing all activities related to a thread: Create(Page), Create(Note), Like, Dislike and others. This collection of activities can be attached to a Page object using a property such as threadContext.
Describe alternatives you've considered.
Publish a collection containing all comments, which can be attached to a Page object using a different property (e.g. thread). This is probably easier to implement, but then it won't be possible to backfill upvotes / downvotes.
Additional context
Streams and Hubzilla already provide collections of conversation activities, although their implementation is slightly different from what I propose here (see FEP-171b).
https://github.com/LemmyNet/lemmy/issues/3448 is a similar issue but it seems to be focused on backfilling community content, which can be achieved by pulling from community outbox.
Ive been talking with @julianlam about the same topic. His idea is an implementation based on FEP 7888, similar to the one provided by NodeBB. To make it simple:
- Each post has a
contextfield which points to a collection of Note IDs (not activities) - Comments also include the same context url as the parent post
A collection of activities is not really possible because Lemmy doesnt store that information. For example for a vote, only the username, post id and date are stored, but not the activity id or full activity json (it would be too much data).
Initially Lemmy would only provide this data to other platforms, but not use it to do any backfilling on its own. Doing that would be much more complicated as it would have to be done in background threads, and ensure not to overload the server. Maybe it should even be handled by external tools via API.
A collection of activities is not really possible because Lemmy doesnt store that information.
This is an important distinction for some implementors. NodeBB also doesn't store full activity JSON. If we were to implement it, it would be a regeneration of an activity based on locally available data. That would be ok for an actor outbox, but not in situations like these where we'd need to also represent remote activities.
@Nutomic
A collection of activities is not really possible because Lemmy doesnt store that information. For example for a vote, only the username, post id and date are stored, but not the activity id or full activity json (it would be too much data).
Activity ID would be enough (activities don't need to be embedded in the collection).
Each post has a
contextfield which points to a collection of Note IDs (not activities)
That's fine, and will complement collection of activities, if you decide to provide it. I'm slightly in favor of the name thread because it is more specific but context is okay too.
We also dont store the IDs of activities associated with particular actions, so thats not possible.
A FEP that describes implemented thead backfilling mechanisms: https://codeberg.org/fediverse/fep/src/branch/main/fep/f228/fep-f228.md
Today we have 5 implementations that publish collection of posts (all of them use context property, not thread), and 2 implementations that publish collection of activities.