borgbackup
optimize to store LVM snapshot differences
Had an interesting discussion at 35c3 and the idea came up to add a fixed blocksize chunker for LVM LE size (4MiB? And configurable also to other block sizes, of course.). We have that now in borg 1.2, see #4238.
It would be possible then to ask LVM for the changed LEs between 2 snapshots and efficiently just add the changed LEs to a borg repo:
- do LVM snapshot lvm-ID1
- borg backup snapshot to archive repo::arch-ID1:lvm-device-ID1
- work work work
- do LVM snapshot lvm-ID2
- ask LVM for the snapshot difference D = {LEs ID2} - {LEs ID1}
- fetch chunk id list for arch-ID1 (could use the files cache or just fetch from the repo / archive metadata)
- update the repo, storing the changed LEs from D (nice and efficient: only has to read all changed LEs as all other are still the same as in ID1 archive).
- update the chunk id list with the changed LEs/chunks from D
- create an item with the updated chunk id list to repo::arch-ID2:lvm-device-ID2
Related:
https://github.com/QubesOS/qubes-issues/issues/858, especially https://github.com/QubesOS/qubes-issues/issues/858#issuecomment-262256101
https://github.com/tasket/sparsebak
update: https://github.com/tasket/wyng-backup
Have you ever come across the dm-era device mapper target? Looks relevant: https://web.archive.org/web/20160811124312/http://blog.rackcorp.com/2016/03/
I don't know if the output from era_invalidate is any saner....
On Wed, 7 Aug 2019 at 15:41, TW [email protected] wrote:
Interesting. But (OMG), arrays of bits (ints) as XML...
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/borgbackup/borg/issues/4363?email_source=notifications&email_token=AB4TJUR3E2ZRW7XTXSZRV2LQDLNJFA5CNFSM4GXMGDKKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3YUFMA#issuecomment-519127728, or mute the thread https://github.com/notifications/unsubscribe-auth/AB4TJUQ2YINN5MYTTDALRKDQDLNJFANCNFSM4GXMGDKA .
https://github.com/mpalmer/lvmsync might be worth a look too - seems to be focused on syncing images using snapshots..