Feature/PR proposal: s3 target support

[nward]

Hi there, I would like to have the ability to store backups on S3.

This is possible already in several ways:

• write locally, sync the dir with an s3 tool, requires lots of local storage
• write to stdout, pipe to an s3 upload tool which works with stdin, works but needs local metadata etc. and does not work with -l auto
• write to an s3 fuse mounted filesystem, which I have had reliability issues with

These options are OK (and I am using the stdout option right now, in fact), but each have drawbacks. Being able to store the entire backup (with metadata) directly on S3 would be great, as would be being able to restore from S3 directly.

I have looked at how this would be implemented, and it looks like we can add an additional target, and perhaps change how target.openfile works to support metadata files.

Would you be open to a PR implementing this functionality?

[abbbi]

thanks for the feedback! Ive thought about an s3 backend implementation but personally i don't use s3 as storage for backups (or lets say, s3 compatible ones like minio) because i prefer to keep my data locally. The main problem is that its hard for me to have such a feature and maintain its codebase if i don't have any usage for it.

Also, i would have opted for s3fs implementions first, but i dont know about their stability as you mention.

As for a PR, i dont know if i would accept, probably only if i want to use the feature myself and then it needs to get tested in the automatic CI tests to ensure its functionality.

[abbbi]

One idea might be to redesign the output writer stuff to be "plugin" capable, and use regular URI based notation:

 -o fs://dir / -o dir/-> loads module output/dir/
 -o - > loads module output/stdout
 -o s3://user@foo -> loads module output/s3

so the s3 implementation could be shipped as sort of a additional "plugin" maintained by the persons caring about it.

[nward]

Hah, that's exactly what I was thinking about when pondering your reply. I think that the same would be possible for the encryption. Perhaps both encryption and compression can be "filters" in a chain, or something like that.