[docs] Example integration with external persistance layer

[naz]

trafficstars

Problem

There's a need to track, discuss, and document best approaches around adding jobs persistence layer on top of bree. This issue is a focused continuation of this comment:

An example in the README for using sockets, or redis pubsub to communicate with Bree to add new jobs (?)

and another comment :

I would recommend keeping dynamic jobs actually stored in a queue, with a persistent database, and then have a job that runs every so often to flush the queue (with limited concurrency).

References

How job persistence is approached in other job processing/queuing libraries:

• Redis backed queue in node-resque
• MongoDB backed per-job persistence in agenda
• Redis backed queue in bull

Current use of bree along with MongoDB:

• forwardemail.net - here jobs are generated based on state of records in the database
• TBD: expand this list with more examples for future reference

Expected outcome

What I think would be the ideal outcome for this issue is documentation an example implementation of persisted jobs queue on top of bree. Ideally the example would be backend agnostic so that multiple storage approaches - NoSQL, SQL or key/value stores like Redis, could utilize it.

Maybe bree could even provide an adapter/plugin system to optionally add persistence for (1) scheduled one-off jobs and (2) failed jobs :thinking:

[Zireael]

Expected outcome

What I think would be the ideal outcome for this issue is documentation an example implementation of persisted jobs queue on top of bree. Ideally the example would be backend agnostic so that multiple storage approaches - NoSQL, SQL or key/value stores like Redis, could utilize it.

Maybe bree could even provide an adapter/plugin system to optionally add persistence for (1) scheduled one-off jobs and (2) failed jobs 🤔

How about using an abstraction layer like TypeORM (good support for SQL databases + MongoDB. Doesn't support Redis tho). You write job model/schema once and then it gets migrated into a database of user's choice by TypeORM.

[dturton]

Prisma can be another great option to TypeORM.