django-dbbackup icon indicating copy to clipboard operation
django-dbbackup copied to clipboard

Deduplicated backups (e.g. Borg or restic)

Open bittner opened this issue 5 years ago • 5 comments
trafficstars

IIUC, the current feature set requires you to handle rotating the backups yourself.

  • Whether you compress, encrypt and store to whatever backend, there will always be something like a tarball for every new execution of a backup process. Hence, you have to deal yourself with deleting old copies (and keeping selected ones, e.g. one every week for the last 3 months) to make sure you don't run out of disk space.
  • Also, if there are no changes (be it in the database or the media files) there will still be a backup, which will unnecessarily occupy disk space and/or incur additional cost on a storage service.

These problems are solved by backup applications that do deduplicated backups, such as Borg or Restic. The latter is a Go implementation, the former a Python tool.

Would it be realistic to try and integrate those tools in django-dbbackup? Calling them as external binaries would likely be sufficient.

bittner avatar Jul 03 '20 08:07 bittner

runrestic is one of the more promising attempts to provide restic for Python.

bittner avatar Oct 10 '20 22:10 bittner

@bittner What could this package do better than runrestic does?

jonathan-s avatar Nov 04 '20 21:11 jonathan-s

Integration with Django for both the database data and the static files?

Really, all the restic wrappers are just there because restic itself doesn't provide a way to maintain a configuration. The "promising" from above relates to merely the rather poor state of the other restic-related packages. :pensive: At least, for what I can see.

bittner avatar Nov 04 '20 22:11 bittner

If you'd like to provide a patch for this it would be welcome. If you take this on, please do describe the approach you'd like to take so that we can agree that it sounds sensible for both of us :).

jonathan-s avatar Nov 05 '20 08:11 jonathan-s

Hi @bittner dbbackup has a simple retention option: DBBACKUP_CLEANUP_KEEP and DBBACKUP_CLEANUP_KEEP_MEDIA In number of backups

If you add the --clean option to the commands, it will apply this retention by guessing the date from fliename.

https://django-dbbackup.readthedocs.io/en/master/configuration.html#dbbackup-tmp-file-max-size

ZuluPro avatar Dec 25 '20 22:12 ZuluPro