Proposal: New command: protect snapshot

[martint17r]

Output of restic version

Not applicable

What should restic do differently? Which functionality do you think we should add?

I propose a new command "protect" which marks data for a given snapshot as immutable (to prevent deletion/modification)

restic protect --repo s3:https://s3.amazonaws.com/restic-demo --snapshot [snapshotid | latest if not given] 30d 

This would add an object lock for 30 days to all files in the repo needed to restore data from that snapshot. If no snapshot is given it would use the latest snapshot.

As far as I know only S3 and S3-compatible backends (backblaze, minio, etc) do support immutable data right now, so the command would only support these backends.

Additionally this could be added as an option to the backup command, so the protection is put directly in place.

If this is accepted, I would go forward and write the necessary pull request(s).

What are you trying to do? What problem would this solve?

We need to have immutable backups to protect the data from ransomware and I think using an explicit command makes the implementation much easier in the first step.

Right now we are using a second host running rclone with a forced ssh command, but that is clumsy at best and adds another dependency...

Did restic help you today? Did it make you happy in any way?

restic saved my day so many times - I really like it and I really appreciate the developments this year ;)

[fd0]

Hm, I'm not sure I understand what you're trying to do. Can you please explain your idea (what you're trying to do) without specifying the solution you have in mind?

For all feature requests for restic, we need to carefully balance initial implementation work, maintenance and also complexity added in relation to how many users will benefit. It may happen that we decide to not implement such a feature :)

[martint17r]

I need backups protected from ransomware. Restic supports this right now, using either the REST server in append mode or using rclone with --append-only. Both ways require a second server and I would like to get rid of that requirement.

[fd0]

A third way would be to use the sftp backend and a cron job which regularly changes the file and directory attributes so that the system user can only create new files and not remove any (besides in the locks/ subdir).

Supporting this use case would require significant complexity. For example, for each newly added snapshot, restic would need to mark all files containing blobs that are required to restore this snapshot as locked.

What does "immutable" mean for those backends? Can the files not be modified (which would be fine for restic, as it never modifies existing files)? Or can the files not be deleted? For backup, that's only required for lock files...

[martint17r]

Right, using SFTP or basically any backend which supports file based ACLs would work as well. Maybe I should add a documentation entry, stating the three existing solutions.

For S3 (and Backblaze) immutability is achieved by placing an object lock on an object (or the whole bucket) for a specific amount of time. Objects with a lock applied can't be deleted or modified (a new version is created on modification) until the specified point in time is reached.

The automatic expiry enables pruning of data, as long as the object lock for the data shared by newer snapshots is refreshed.

That's why I propose a command embedded into restic, as it needs to understand the repo format and the shared data in pack files

[aawsome]

I think that it might be possible to extend restic such that it can give you the information which pack files are needed to fully restore a snapshot, see #2796 . This functionality is also needed for other use cases (like cold storage) and would allow you to write a script that does this protection for you. I would vote against including a explicit protect possibility into restic as it is very specific to just some backends.

Pruning is another issue, as restic cannot know that it should not repack the pack files that are protected. Again, this is similar to some cold storage backends which have a minimum storage time and pack files younger than this should not be removed or repacked. Here it would be nice to give prune a list of pack files that are just kept. I can propose a PR to add this functionality into prune.

[martint17r]

If there is an official API to get the files required for a restore from a given snapshot I would be quite happy.

The proposed solution for pruning also works or me.

[rptaylor]

Related to https://github.com/restic/restic/issues/2202

I think supporting immutable backups is a good idea, I was just trying to figure out how to do it myself. S3 (incl Backblaze S3) supports immutable storage (for a defined period of time) via Object Lock: https://help.backblaze.com/hc/en-us/articles/360052973274 It would just be an extra option that is set in the API call when restic uploads a file, so it should not be that complicated hopefully? Alternatively it can be an extra API call to set the immutability period on an object after it is uploaded.

I would vote against including a explicit protect possibility into restic as it is very specific to just some backends.

Whether it is done via a separate 'protect' command or an option in 'backup' that specifies a duration of immutability does not matter to me. And it is true that the implementation details may vary between different backends, but I hope that restic will support immutable backups as much as possible on any backends that support it!

This is discussed in: https://github.com/restic/restic/issues/187 where the goal is to prevent an attacker from deleting your backups along with your data. I believe most of the work to handle this is done by the cloud storage APIs (i.e. "non-dumb storage") and hopefully restic can just use the features exposed by the APIs.

Here is another example of someone trying to achieve this: https://github.com/restic/restic/issues/1544 In that case the credential for the cloud storage doesn't have delete permission (another way to achieve immutability). This way, restic doesn't need to know about or implement immutability in terms of API calls, however it runs into the issue that the way restic does locking doesn't play well with immutable storage.

[aawsome]

@rptaylor While all of your point count into securing the repository, I think there are several issues:

• Making your storage append-only (protecting all snapshots): This should IMO already work if you managed to handle the locks which of course are to be removed.
• Protecting just a single snaphot: This is the topic of this issue.
• Sending a checksum of a file to be saved in the storage backend: This might be a prerequisite for some storages which secure your repository in one of the above mentioned ways. However, this is IMHO a completely independent issue.

It would just be an extra option that is set in the API call when restic uploads a file, so it should not be that complicated hopefully? Whether it is done via a separate 'protect' command or an option in 'backup' that specifies a duration of immutability does not matter to me.

It is not that easy. The point is, that during backup only new data is uploaded. Due to the deduplication there might be lots of blobs which are needed for the newly-generated snapshot but already exist in the repo. During backup restic does not even need to know, in which files those are located, it suffices to know they exist. So adding this to the backup command would require to rework the backup code.

So asking for a new command is IMO the right way. But restic has a very restricted interface to storage backends, basically one that every backend does support. Enlarging this would be much work which I think can be much better spent on other issues. The workaround (with only two comparably small future improvements) would be:

Backup with snapshot protection:

• normally run backupto get a new snapshot
• get the list of files needed for this snapshot (needs #2796 to be implemented)
• protect all those files + the snapshot file itself

Forgetting/pruning a repository with locked snapshots:

• first get a list of all locked files of your storage backend
• run forget --dry-run (maybe with --json) to see which snapshots would be removed
• remove the locked snapshots from that list
• run forget --prune with the resulting snapshots adding --keep-packs-from (once #3196 is finished) with the list of locked pack files

This is a little bit of scripting, but should be doable.

There might be another use-case which is a bit more tricky - how to get the files to "unlock" if you want to "unlock" (whatever that means..) a snapshot. This is a bit involved but could be done like this:

• get a list of all locked files (snapshots and pack files) of the repository
• for all locked snaphots (except the one to "unlock") get the list of files needed for the snapshot (needs #2796 to be implemented)
• remove the needed files from the list of all locked files
• the resulting list of files can be manually "unlocked" and later pruned by the above described pruning algorithm.

A second way would be to first run a prune as described before, than run forget --prune --max-unused=unlimited --dry-run <snapshot IDs to "unlock"> and get the list of files that would be removed.

[MichaelEischer]

Enabling Object Lock on S3 is currently not yet possible, see #2202, it requires sending file checksums to the storage backend. I've already opened PR #3178 for that.

[rawtaz]

While the idea is admirable I think it has a few flaws:

• The suggested protection is initiated in the wrong place. IMO your backups' immutability is not something that should be triggered/configured in the place where the backups are made from. Doing so exposes your backups to the risk that an adversary prepares their attacks by reconfiguring your backup runs such that your backups are no longer immutable for a period of time, and then when they finally hit the nail in the coffin you end up with no backups. I'm not saying the idea will never work, but that it's a scheme that is far from bullet proof. IMO backup protection should be configured/done by a system that is completely separate from the systems doing the backups. Ideally an attacker shouldn't even be able to find out that the protection is there (let them think they deleted your backups, while in reality you still have copies of them somewhere). But that's perhaps a slightly different thing than immutability, the latter point was more about snapshots or similar on the repository side. Anyway the main problem persists, an attacker can disable future immutability if it's done on the restic side.

• It's backend-specific, and in this case very few backends support this. I'd say we should keep making features in restic as cross-platform as possible, otherwise we'll end up with a rats nest of user-facing features applying to certain backends only - not great. I think @aawsome is talking about a more generic solution which is better and more widely usable, but it is of course complicated :)

[martint17r]

@rawtaz Thank you for your feedback.

I don't agree on your first point though, because restic also features an exclude option and the adversary can easily exclude important files from the backup as well. Still we want to have such a feature. Maybe there needs to be a way to lock the configuration using an immutable backend.

I agree that it is backend specific, but for me an immutable backup is as important as checksums and encryption. A backend which does not support immutable backups is simply not a viable solution in times like these.

I don't necessarily think that it is necessary to include a command for this with restic, but the code will depend on a lot of internals of the repository structure and configuration.

I created a PR to @aawsome #3303 PR to support JSON output and wrote a small program to add the object locks for a given snapshot: https://github.com/martint17r/restic-protect-snapshot

If you look at the ugly source, you will see lot of restic internals (regarding configuration and file paths). The code itself is about 150 lines long (including some copied lines), so it should not be much of a burden to maintain.

[rptaylor]

@rawtaz it is true restic can not be responsible for ensuring the backups are immutable, but restic should be able to support having backups that are immutable. In the issue discussed here , the immutability is guaranteed by the S3 storage provider when the object lock feature is activated via the S3 API. If restic supported this by being able to make the S3 API call, it would provide a way to have immutable backups. Restic would only need to activate the protection initially after uploading files, but after that restic is not responsible for maintaining it, and there is nothing an adversary could reconfigure to remove it (unless perhaps they acquired more privileged credentials to access your object storage - managing and securing the credentials is the user's responsibility). https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lock-overview.html

It is specific to the S3 protocol, but many backends are S3-compatible. Restic has to use the S3 protocol to talk to the object storage so this would only be an extension to the current use and support of the S3 protocol.

Similarly , using object storage with a token that has read and write privilege, but not delete, would be able to achieve immutable backups without restic knowing or having to do anything different - except that the restic locking model can not support this because it relies on deleting lock files. https://github.com/restic/restic/issues/1544

[virtualdxs]

Adding a little bit to this: If implemented as a flag rather than as a dedicated command, it should at minimum be supported on both backup and copy. My use case is a 2-tier backup system, where I have a group of devices that back up to a NAS, and then that NAS uses restic copy to back up to Wasabi.

I'm currently working on a workaround that after each sync to Wasabi recursively calls PutObjectRetention on everything in my restic bucket except the locks directory, but it would be nice to have restic do it more precisely.

[kapsiR]

I read through some issues here that should help solving the ransomware threat with an "object lock" (B2/S3 terms)

Just curious, but what about using a different bucket for the restic locks? (that's the only thing that needs to be changed/deleted in time?)

This would enforce object lock server side and still let restic do it's job as usual?

Am I wrong? 🤔

Edit: And yes, ransomware protection is really important in my opinion 😉

[aawsome]

@kapsiR If you have a backend which allows individual configuration for pathes it should suffice to exclude /lock from the the "object lock" or from ACLs which prevent removal or overwrites of existing repo data. If you can't do this but still want to do append-only backups, you have to play around with rest-server (or build something similar yourself) to self-implement a routing for lock files. Alternatively, have a look at rustic which works completely without locks.

[martint17r]

Please also note that the new hide-Files on B2 in restic 0.15 also solves the ransomware use case.

[AxisNL]

I also want to chip in and request this feature.

For a project I'm building a 1PB cluster. Actually, it's a primary 1PB cluster running Ceph+CephFS, and a backup 1PB cluster running Ceph+RGW, an s3-compatible on-prem object store, or perhaps Minio, which does the same. I think we'll start out with about 800TB of research data, with perhaps 5-10TB changes per day.

Of course we want to make backups to protect against system failures and user errors. But we also really want ransomware protection, and we need to make sure that when an adversary gets root access to the primary cluster and gets our s3 credentials, they can't delete the backups as well.

I was planning on using restic to backup file shares from the primary cluster to the backup cluster daily using s3, and I was hoping restic would be able to set the object lock-bit on new objects, so the s3 backend will make sure they can never be deleted within the configured retention time, like 30 days or so. But I guess that requires restic to keep track of which objects are locked and until when. Or perhaps, as described above, that I can tell restic to forget about snapshots after 30 days, and that the prune process will just ignore objects that is it not allowed to delete yet. Then it would not need to keep track of object locks itself.

I'm not married to s3 by the way, but it seems s3 with object locking is turning out to be the de-facto standard in ransomware protection (even if there are other viable options), so it would make sense for all of us to jump on that train.

Perhaps running restic-rest with append-only would work, but I can't find good resources on how to forget older snapshots after a few weeks on the receiving side without also deleting blocks the sending party is expecting for dedup? Confused here. If you have other great ideas, I'd love to hear them!

[martint17r]

Let me try to answer the different topics raised:

1. Ransomware safe backups

If you control the receiving server and are able to run software there, you don't need object lock features to support a ransomware safe backup.

First run rclone serve restic --append-only to receive backups via http from the production cluster. Second, run restic forget --prune using read/write file access methods, such as SFTP or locally mounted files.

2. Considerations on pruning repos with append only access

If a repo is pruned of some snapshots and associated data, the next restic backup run will notice the changed repo state and download the changed indexes. That is transparent (apart from taking some additional time) to the user and you don't need to do anything special to make that work.

3. Data size and run times

Please take into consideration, that it will take some time and a lot of memory to check 1PB for changes. Running restic forget on such a repo will interfere with your backup runs, as both need to hold the exclusive write lock and currently restic only supports coarse locking on an operation level.

[virtualdxs]

That is safe against ransomware on the backed up server, but not on the receiving server. Object lock means an attacker would have to compromise your S3 provider in order to encrypt/delete those backups.

[martint17r]

In AcisNLs case the receiving server would be the S3 provider, that's why I don't think there is a real advantage in using S3. Otherwise I would agree.