LemmyNet
Batch DB writes
There are various places in the code where a lot of similar data is written to the db. For example outgoing activities are all written to SentActivity one by one. It would be more efficient to store these in memory and then write a whole batch to the database at once. For example, wait until at least 5s elapsed or 100 items in memory. This way less db queries are used for actions that are not time-critical.
Some places where this would make sense:
- https://github.com/LemmyNet/lemmy/blob/9d98c82ec30e8f261fd9459a4e75dc0f979de0eb/crates/apub/activities/src/lib.rs#L140
- https://github.com/LemmyNet/lemmy/blob/9d98c82ec30e8f261fd9459a4e75dc0f979de0eb/crates/apub/send/src/worker.rs#L435
- https://github.com/LemmyNet/lemmy/blob/9d98c82ec30e8f261fd9459a4e75dc0f979de0eb/crates/apub/activities/src/voting/mod.rs#L80
- https://github.com/LemmyNet/lemmy/blob/9d98c82ec30e8f261fd9459a4e75dc0f979de0eb/crates/apub/activities/src/voting/mod.rs#L97
In general batching can be used for all background actions (particularly federation). However it cant be used for local user actions which should be reflected in the UI immediately.
Collecting up a lot of DB writes and creating transactions for them might add a lot of complication, and end up being no performance benefit.
You'll also potentially create memory leaks because you're adding yet another storage layer on top of the database (IE the memory queue that stores them). Also if you wait to collect up a certain amount before writing, that's unreliable because some instances might not have too much activity, and they'd never get written.
Its much simpler and safer to just write them immediately. Postgres can handle these atomic writes fine, I'm more worried about the #5854 which slow down the writes. The performance of creating comments / posts and liking them could benefit a lot by fixing that issue.
