andrewchambers
Document how to safely work with append-only mode
Assumptions: because ransomware commonly tries to delete backups (especially of organisations), safe backup software must be able to prevent this. At the same time, storage space is not infinite so old backups must be removable. Of course each backup will be removed as it ages, but they should be unremovable for the configured time period. Especially organisations but also (nerdy) individuals will automate this maintenance process.
Various backup tools have different problems with ransomware protection:
- Restic has an optional
rest-servercomponent can run in append-only mode. However, the client fully controls the (meta)data of new backups, meaning that all an attacker has to do is create a far-future dummy backup such that the next timerestic forget --keep-within 1yruns, it will think all other backups are stale and delete them. - Borg has an append-only mode where the client can mark objects for deletion. The next time any operation is run by the administrator, the marked objects will be deleted.
- Tarsnap has write-only keys. Objects can't be marked for deletion and there is no command to prune old backups. You can only
-da snapshot by its ID, so to work around this limitation you either write your own code or use third-party scripts. From a quick peek, it seems to me that most of these have the same problem as restic, but I don't know for sure and either way, the author of Tarsnap simply leaves the problem as an exercise to the user. (And Tarsnap isn't cheap, people will definitely want to save those bytes.)
Seeing all this while looking for a solution for myself and friends, I was curious how bupstash solves it and I asked in the restic issue thread where the author mentioned bupstash's advantages, but the only response I got was that "it blocks any operations that would cause deletion". Woa, who'd have thought? :wink:
The relevant bupstash docs give detailed instructions on how to setup the remote server to protect against client-initiated deletion, but then deleting stale snapshots is left vague, concluding the page with:
The
bupstash servecommand [supports various permission options]. With these options we can create a backup plan where clients can create new backups, and an administrator is able to cycle old backups from the secure machine.
If an administrator just uses bupstash rm older-than 30d, will that also delete any objects marked by a hacked client for deletion? Who adds the timestamp to a snapshot, client or server? Might the attacker trick the age detector with new backups? (Looks like no for that particular question.) Can the garbage collector be tricked in some way, or is it safe to allow an attacker to run that (directly, or indirectly through a cron job that then operates on attacker-supplied data)? Does the put permission that the docs recommend to use not allow overwriting objects?
I think the docs could be improved:
- It would be good to have a short technical overview of how the solution is designed since there are all these pitfalls.
- Even more importantly: how does one securely delete older backups? For example, are there operators to avoid using in
bupstash-rmsimilar to how--keep-weeklyis not safe in restic and Tarsnap managers? - What circumstances does the deletion/corruption protection hold under? (E.g.: only for the duration configured in
bupstash-rm; so long as permissions x and y are not granted; so long as you have notifications turned on for when backups were not made for X amount of time; and perhaps more things that one might reasonably forget or not be aware of.)
As it is, it's quite unclear to me whether there are any pitfalls to be aware of when keeping bupstash backups safe for a while.
I think borg was poorly designed and you seem to be transferring knowledge of this poor design onto bupstash - To be honest, my dissatisfaction with borgs handling of this was part of my motivation to write bupstash.
will that also delete any objects marked by a hacked client for deletion?
A client cannot mark anything for deletion If the clients ssh access/'bupstash serve' command is in a restricted mode. Without rm permission or gc permission bupstash will simply refuse to do the operation, the client cannot pend anything for deletion, The client also cannot spoof timestamps as they are signed by the client but verified by the server side before being added to the repository, the two must be in agreement.
Does the put permission that the docs recommend to use not allow overwriting objects?
Bupstash never overwrites objects that already exist for this reason. A new backup cannot corrupt an older backup to preserve a forensic trail. A compromised client can still upload 'false' backups, but it cannot spoof the timestamp or alter previous good backups. This means if you know when a compromise occurred you are able to exclude any backups that happened after that and have confidence it is safe.
If an administrator runs a command without any restrictions imposed by his 'bupstash serve' command then bupstash will perform the action. For example the administrator logs into the storage server then runs the 'rm' command directly. I am definitely open to repository level configuration that further restricts such operations, but the administrator with direct access to the repository can easily edit these anyway.
All that being said, we can and should improve the documentation and provide further guides. In the meantime I think it might become more clear how it works if you create a test bupstash repository and try some combinations of
$ export BUPSTASH_REPOSITORY_COMMAND=bupstash serve --allow-put /path/to/repo
$ bupstash rm ...
I also encourage you to experiment with ssh force commands if you have not done so before (I just don't know if you are familiar with them, but i am assuming you probably are).
I am happy to clarify further, and would also accept any updates to the docs and can publish them to the website. Be assured that I take this issue very seriously.
Restic has an optional rest-server component can run in append-only mode. However, the client fully controls the (meta)data of new backups, meaning that all an attacker has to do is create a far-future dummy backup such that the next time restic forget --keep-within 1y runs, it will think all other backups are stale and delete them.
This is not the case for bupstash - a client cannot spoof timestamps, he can sign them and the client and server must agree on the time. If the client and server disagree on the current time, the server will reject the data preventing this issue entirely.
You can experiment with this by deliberately setting your clocks out of sync more than 15 minutes. You can see the code here https://github.com/andrewchambers/bupstash/blob/01116125a5decadff74b63d799b61868bdfba47f/src/server.rs#L176
I think I want to publish a blog post or some other document describing the exact threats bupstash addresses to reassure people and catch anything we have not accounted for.
Wow, thanks for the extensive answer! This is almost documentation in and of itself :). Cool that things are so strict, did you specifically design it with this use case and these pitfalls in mind, or is it just defensive programming and this is how it turned out?
I'm a bit hesitant to pick up the documentation task given how little I know the project and how, um, I want to say rusty but I never had rust language reading skills to begin with. Perhaps I could create a pull request just based on what you wrote here, though, and add a few sentences to the Remote Access Controls docs? And perhaps also reference this thread for people that want the in-depth information you posted (not sure if a link to an issue thread is something you'd want to have in the docs).
Cool that things are so strict, did you specifically design it with this use case and these pitfalls in mind, or is it just defensive programming and this is how it turned out?
I had a lot of this in mind, but some I had to go back and fix a few mistakes by tweaking the protocol, it has been a long process.
Perhaps I could create a pull request just based on what you wrote here, though, and add a few sentences to the Remote Access Controls docs?
Whatever you are comfortable with, I don't mind doing it myself either, it just might take me a while as I'm working on a few other aspects of bupstash at the moment.
We can leave this thread open until there is some sort of resolution.
haven't checked lately but I was surprised that a "list" key can also "rm", and a subsequent "gc" will lose the data thus removed. This seems only slightly different than borg's unfortunate handling of append-only (link in 2nd bullet of original post in this issue)
to my mind that needs not just documentation but a rethink of the name of that key. As a naive user, I see the phrase "list key" and I expect it can only do what bupstash list --help describes as the function of "list".
the decryption key types are related to decryption, they do not affect access control which is related to your ssh key and/or access configuration.
This seems only slightly different than borg's unfortunate handling of append-only
Another way to think about it - even though you have a list decryption key - you still had the ability to simply run 'rm -rf ./repository'.
We can improve the documentation further, I think this is a fair point of confusion, we can also consider renaming the key like you say... At the very least we can emphasize they are decryption key types.
Overall this behavior is quite important as it allows an administrator to prune a repository without having access to the contents of a backup.
I think the situation can be further improved by having a repository metadata option to disable gc and rm entirely that must be turned off manually.
I have created an issue #333 which will likely make it into one of the next few releases.