Better handling of background jobs

[elegaanz]

Is your feature request related to a problem? Please describe.

For the moment, there is no way to retry a failed job. And if the app is stopped, all queued jobs are lost.

Describe the solution you'd like

Having a jobs table that stores queued jobs, and use a worker pool to regularly pull them and run them. Rustodon has a good implementation of background jobs that does that IIRC, and we could probably get inspiration from here (maybe it was even published as a crate that we could just pull?).

Describe alternatives you've considered

Storing these jobs elsewhere like in Redis or RabbitMQ as it is more usually done (and there are probably crates to handle most of the basic logic for that), but I don't really like adding another dependency, so going for a database table seems better (and also, installing Redis when you want to run a very small instance is… meh).

~~Additional context~~

[epsilon-phase]

What particular jobs would be useful to support?

[elegaanz]

For the moment, I can think of:

• media processing jobs
• federation jobs
• anything else that we currently do in background threads

[trinity-1686a]

at the moment, background threads are used for federation (broadcast and fetch), and key rotation upon deleting comments