Audit an append-only mode repo to make sure the client was well behaved

[nadalle]

This idea is an attempt to detect/mitigate the following attack scenario involving append-only mode:

Say you're using append-only mode to protect your repo from an untrusted and hacked client deleting / changing your old archives. In addition, you periodically connect to the repo from a special trusted client in normal mode, and issue some prune commands.

Currently, if you do this, the trusted client will happily compact the repo as soon as it tries to write, happily applying all the untrusted client's bad changes permanently. So, the administrator would have to qualify the repo contents before compaction happens.

How could an automated qualification work?

The repo itself is a log-like structure, that means it is a sequence of operations like:

put(1, A), commit, put(1, B), commit

The current value of object 1 is now B, but we still can see it was A before. Until compaction happens and removes the superceded first put().

Untrusted clients are forced into append-only mode by the server, so we see their log of operations. The trusted admin client is not append-only and removes the log as soon as compaction runs.

PUT to id 0: these are manifest updates, we can check if archives are appended and previously present archives are still same (name, date, id). Special case: overwriting a .checkpoint archive of the same name.

PUT to other ids:

• id is new: likely legitimate, done by "borg create"
• id is not new: this means we overwrite already present data with new (different?) data. "borg recreate" legitimately does this when recompressing chunks, but other than that it should not happen. existing-id overwrite maybe could also be denied by server in append-only mode.

DEL: should not happen, only exceptions are .checkpoint archives (? CHECK THIS). A well behaved borg client should reject high level "borg delete" and "borg prune" commands in append-only mode.

This stuff could be a "borg qualify" ("borg audit"?) command and it would exit with success if all looks ok or emit warnings/errors and exit with warning/error status if not. If not, it should also give the last known-good transaction number.

Additionally, the trusted client likely wants to run "borg check" to assert the integrity of the archive (like discovering bad sectors or other corruption / inconsistency).

Optionally, the trusted client could make added archives pass a set of tests (like checking for expected content: is it there, can it be extracted, can it be used?).

This way, a user who worries about their untrusted clients being hacked can feel safe running a prune command automatically on their trusted client with a cron job: they'll still have old backups from before the hack available, so long as they detect it in time.

(by tw based on nadelle's post)