Migrate the backend database to postgres

[msm-code]

We're using redis for historic reasons, but we should probably move the metadata to postgres, to ensure data integrity and maybe even improve performance.

• [ ] Design a schema and update docker-compose files DB;
• [ ] Change all uses of redis in the code to postgres;
• [ ] Provide a migration scripts?
• [ ] Update documentation (including installation methods)

This is pretty low priority because it's a lot of work and don't add any immediate features though

Suggested by @BonusPlay (not directly, but related to schema problems)

[ITAYC0HEN]

On one hand you say,

and don't add any immediate features

OTOH you also mentioned

...and maybe even improve performance.

Can you measure or guess what is the expected performence improvements, if any? If it is meaningful, the priority rises, isn't it? It if it not meaningful, then yes... sure

[msm-code]

Yeah, we could measure performance after the changes. OTOH i expect improvements, if any, to be be pretty small. I think the huge wins from this change is making the system a bit more stable and easier to understand (for example, right now it's not obvious what exactly is stored in job:xxx or meta:xxx objects).

[msm-code]

And there's one more thing. We had to implement a GC for query results, because we need to store matched filenames somewhere (in the db = redis in our case), and redis wants to to fit all the data into memory. After some time this may exhaust the server memory, so right now we remove stale results after some time. With postgres, only disk space is the limit.

[msm-code]

I still like Postgres, but can't justify working on this issue:

• I don't know about any users that have performance problems because of this (if you're one, please let me know). There are bottleneck, but redis is not one.
• This will make upgrade way harder for current users.
• This is a lot of work, and my spare time is (unfortunately) finite. So I'd like to focus on issues with a better effort/benefit ratio.

[msm-code]

The current plan:

• [x] Create a MVP that will use a database for something simple (like configuration) and pass the e2e tests. Update the installation documentation. #367
• [x] Migrate all the persistent objects to this layer, while keeping the db.Database shim (don't make changes in random places in code yet)
• [ ] Refactor the inenvitable behemoth - with ORM we may consider moving away from the one central database interface.