Tiered storage - data archiving

[spetz]

To support the different requirements for data archiving e.g. massive amount of records (TBs or even more), automatic backup in multiple environments and so on, it would be great to implement the so-called tiered storage - for example, instead of deleting the expired segments (as currently configurable via retention policy), it should be also possible to archive them, by storing somewhere else than the current server.

[seroze]

Should we start with AWS S3 here ? background process would pipe the expired segment files directly to s3

[spetz]

If you'd like to contribute, sure thing! I was thinking that instead of the background process (unless you have the actual application in mind), we could simply stick to the background task, similar to the existing message cleaner or saver.

[spetz]

Ok, after the latest release with a new storage implementation, it's high time to start working on this, so I'll focus on it :)

[spetz]

Disk archiver has been implemented in #1053, S3 is next on the list.

[spetz]

S3 archiver has been implemented in #1055 which closes this feature in its basic form.