[Database, Media store] Incorporate the official Synapse backup and restore guidelines and improve the docs

[krassle]

Is your feature request related to a problem? Please describe. No.

Describe the solution you'd like Align this playbook's backup and restore strategy with the newly added official recommendations from the Synapse developers and improve the docs.

1. Incorporate important hints about Synapse database backups and restores
2. Follow the official recommendation about backing up media store directories
3. Update and improve the docs (backing-up-postgres, importing-postgres, importing-synapse-media-store and others)

Database

• The e2e_one_time_keys_json table should not be backed up, or if it is backed up, should be TRUNCATEd after restoring the database before Synapse is started.

• Be very careful not to restore into a database that already has tables present. At best, this will error; at worst, it will lead to subtle database inconsistencies.

Media Store

• Include local_thumbnails to BorgBackup
• Exclude remote_content from BorgBackup
  • it's disk space heavy and is usually unnecessary to keep around forever, as it gets re-fetched upon request after purging the media cache (over the synapse API /_synapse/admin/v1/purge_media_cache?before_ts=[TIMESTAMP])

[spantaleev]

The explanation about e2e_one_time_keys_json (as seen in that document) is:

[Background: restoring the database to an older backup can cause used one-time-keys to be re-issued, causing subsequent message decryption errors. Clearing all one-time-keys from the database ensures that this cannot happen, and will prompt clients to generate and upload new one-time-keys.

These additional instructions only seem to apply if you:

• make a backup
• let your server continue running for a while
• try to restore the backup

If you're doing:

• stop services
• make a backup
• restore the backup onto a new server

..., then it's fine to ignore this advice.

I believe our docs mostly target the former though, so it's better to update them and possible make --tags=import-postgres do the right thing (exclude INSERTs to the e2e_one_time_keys_json table). Solving this is a bit tricky, because:

• you may or may not wish to do this (though.. always excluding these entries will probably not hurt)
• import-postgres is part of the Postgres Ansible role, which is also independent and bending it to do Matrix-specific things should be done carefully

For people wishing to migrate their server to another host, etc., we have the docs/maintenance-migrating.md guide which lets you transfer your setup with rsync (unless you're also switching to another CPU architecture - in which case, you'll do the latter and won't be bothered by any of this).