digint
Snapshots without existing parent are not backuped
I have some snapshots the parents of which have been removed. They are not eligible for send/receive to the backup target for some reason. However I think that incremental backups should still be possible and also made by btrbk.
FWIW, if I issue a btrfs send -p <snapshot of yesterday> <current snapshot>, a sensibly small amount of data is produced, indicating an incremental backup. Can’t this be used to send the current snapshot to the backup medium?
Yes this can be used to send the current snapshot.
I'm not sure what is going wrong with btrbk here, could you please provide more information? Do you mean the parent defined in "subvolume xyz"? If this one is gone, you can still backup using "btrbk archive".
I use send/receive for sync’ing a large VirtualBox .vdi file between my home server and my laptop. Both computers share a snapshot, which is snapshotted on the laptop, and then I use this for my work in the VM. At the end of the day, it is incrementally send/received to the server.
After that, the old shared snapshots are deleted on both machines, and the send/received snapshot becomes the new basis. And so on and so forth.
Additionally, every night btrbk makes a snapshot for backup purposes. Due to the above rotating scheme, these snapshots regularly lose their parent. This seems to irritate btrbk: If I attach my external USB disk for backup (i.e. “btrbk resume”), it does not copy the snapshots.
And yes, I mean the parent in “subvolume xyz”.
By the way, I implemented a workaround by separating the sync’ing from the btrbk subvolume, and cp --reflink between both.
I'm also having this issue. btrbk seems to ignore sending snapshots which do not have a parent.
For example, I have the following series of snapshots:
./storage.20220518T000000+0800
./storage.20220619T205732+0800
./storage.20220704T220758+0800
./storage.20220907T221810+0800
./storage.20221031T210458+0800
./storage.20221110T210032+0800
./storage.20221130T010001+0800
./storage.20221130T020003+0800
The parent-child uuid relationship is broken from ./storage.20221130T010001+0800 onwards.
With btrbk resume:
*** /mnt/backup/storage/storage.20221130T010001+0800
>>> /mnt/backup/storage/storage.20221130T020003+0800
With btrbk archive:
*** /mnt/backup/storage.20220518T000000+0800
>>> /mnt/backup/storage.20220619T205732+0800
>>> /mnt/backup/storage.20220704T220758+0800
>>> /mnt/backup/storage.20220907T221810+0800
>>> /mnt/backup/storage.20221031T210458+0800
>>> /mnt/backup/storage.20221110T210032+0800
>>> /mnt/backup/storage.20221130T010001+0800
>>> /mnt/backup/storage.20221130T020003+0800
I have tried setting incremental_prefs defaults,sao,san,aro,arn but that gives the same result.
I suppose the workaround is to use btrbk archive when snapshots do not have their parent?
Are you sure there's no target_preserve_ rule restricting backups? Please check with `btrbk run -S -n".
If the problem persists, please paste you /etc/btrbk/btrbk.conf and used version.
Version: btrbk command line client, version 0.32.5
btrbk.conf:
transaction_syslog user
lockfile /var/lock/btrbk.lock
timestamp_format long-iso
archive_preserve_min latest
archive_preserve 24h 7d 4w 12m 3y
# Main storage
volume /mnt/storage-root
snapshot_dir snapshots
target /mnt/system-root
target_preserve_min latest
target_preserve 24h 7d 4w 12m 3y
subvolume root
snapshot_name storage
snapshot_preserve_min latest
snapshot_preserve 24h 7d 4w 12m 3y
The strange thing is that btrbk run is able to indicate that the snapshots should be preserved, but doesn't include them in the backup summary:
root@server ~ # btrbk run -n -S
SNAPSHOT SCHEDULE
-----------------
ACTION SUBVOLUME SCHEME REASON
- /mnt/storage-root/snapshots/storage.20220518T000000+0800 24h 7d 4w 12m 3y (sunday, 00:00) preserve yearly: first weekly of year 2022 (0 years ago, 3d after sunday 00:00)
- /mnt/storage-root/snapshots/storage.20220619T205732+0800 24h 7d 4w 12m 3y (sunday, 00:00) preserve monthly: first weekly of month 2022-06 (6 months ago, 20h after sunday 00:00)
- /mnt/storage-root/snapshots/storage.20220704T220758+0800 24h 7d 4w 12m 3y (sunday, 00:00) preserve monthly: first weekly of month 2022-07 (5 months ago, 1d 22h after sunday 00:00)
- /mnt/storage-root/snapshots/storage.20220907T221810+0800 24h 7d 4w 12m 3y (sunday, 00:00) preserve monthly: first weekly of month 2022-09 (3 months ago, 3d 22h after sunday 00:00)
- /mnt/storage-root/snapshots/storage.20221031T210458+0800 24h 7d 4w 12m 3y (sunday, 00:00) preserve monthly: first weekly of month 2022-10 (2 months ago, 1d 21h after sunday 00:00)
- /mnt/storage-root/snapshots/storage.20221110T210032+0800 24h 7d 4w 12m 3y (sunday, 00:00) preserve monthly: first weekly of month 2022-11 (1 months ago, 4d 21h after sunday 00:00)
- /mnt/storage-root/snapshots/storage.20221130T010001+0800 24h 7d 4w 12m 3y (sunday, 00:00) preserve weekly: 1 weeks ago, 3d 1h after sunday 00:00
- /mnt/storage-root/snapshots/storage.20221201T070000+0800 24h 7d 4w 12m 3y (sunday, 00:00) preserve daily: first of day, 3 days ago, 7h after 00:00
- /mnt/storage-root/snapshots/storage.20221202T000002+0800 24h 7d 4w 12m 3y (sunday, 00:00) preserve daily: first of day, 2 days ago, at 00:00
- /mnt/storage-root/snapshots/storage.20221203T000005+0800 24h 7d 4w 12m 3y (sunday, 00:00) preserve daily: first of day, 1 days ago, at 00:00
<truncated>
BACKUP SCHEDULE
---------------
ACTION SUBVOLUME SCHEME REASON
- /mnt/system-root/storage.20221130T010001+0800 24h 7d 4w 12m 3y (sunday, 00:00) preserve yearly: first weekly of year 2022 (0 years ago, 3d 1h after sunday 00:00)
- /mnt/system-root/storage.20221201T070000+0800 24h 7d 4w 12m 3y (sunday, 00:00) preserve daily: first of day, 3 days ago, 7h after 00:00
- /mnt/system-root/storage.20221202T000002+0800 24h 7d 4w 12m 3y (sunday, 00:00) preserve daily: first of day, 2 days ago, at 00:00
- /mnt/system-root/storage.20221203T000005+0800 24h 7d 4w 12m 3y (sunday, 00:00) preserve daily: first of day, 1 days ago, at 00:00
<truncated>
--------------------------------------------------------------------------------
Backup Summary (btrbk command line client, version 0.32.5)
Date: Sun Dec 4 06:41:50 2022
Config: /home/user/btrbk.conf
Dryrun: YES
Legend:
=== up-to-date subvolume (source snapshot)
+++ created subvolume (source snapshot)
--- deleted subvolume
*** received subvolume (non-incremental)
>>> received subvolume (incremental)
--------------------------------------------------------------------------------
/mnt/storage-root/root
+++ /mnt/storage-root/snapshots/storage.20221204T064150+0800
*** /mnt/system-root/storage.20221130T010001+0800
>>> /mnt/system-root/storage.20221201T070000+0800
>>> /mnt/system-root/storage.20221202T000002+0800
>>> /mnt/system-root/storage.20221203T000005+0800
<truncated>
As it is, the subvolume storage.20220518T000000+0800 is not backed-up, despite btrbk giving the preserve yearly reason as first weekly of year 2022 (0 years ago, 3d after sunday 00:00).
Oh, I stumbled onto this while working on some major refactoring a couple of months ago, and did not give it too much priority. I cherry-picked the fix in the fix-backup-all-snapshots-in-snapdir branch, please give it a quick test:
cd /tmp/
wget https://raw.githubusercontent.com/digint/btrbk/fix-backup-all-snapshots-in-snapdir/btrbk
chmod +x btrbk
./btrbk run -S -n
it should give you something like this:
SNAPSHOT SCHEDULE
-----------------
ACTION SUBVOLUME SCHEME REASON
- /tmp/btrbk_t/mnt_source/snapshots/storage.20220518T000000+0800 24h 7d 4w 12m 3y (sunday, 00:00) preserve yearly: first weekly of year 2022 (0 years ago, 2d 18h after sunday 00:00)
- /tmp/btrbk_t/mnt_source/snapshots/storage.20220619T205732+0800 24h 7d 4w 12m 3y (sunday, 00:00) preserve monthly: first weekly of month 2022-06 (6 months ago, 14h after sunday 00:00)
- /tmp/btrbk_t/mnt_source/snapshots/storage.20220704T220758+0800 24h 7d 4w 12m 3y (sunday, 00:00) preserve monthly: first weekly of month 2022-07 (5 months ago, 1d 16h after sunday 00:00)
- /tmp/btrbk_t/mnt_source/snapshots/storage.20220907T221810+0800 24h 7d 4w 12m 3y (sunday, 00:00) preserve monthly: first weekly of month 2022-09 (3 months ago, 3d 16h after sunday 00:00)
- /tmp/btrbk_t/mnt_source/snapshots/storage.20221031T210458+0800 24h 7d 4w 12m 3y (sunday, 00:00) preserve monthly: first weekly of month 2022-10 (2 months ago, 1d 14h after sunday 00:00)
- /tmp/btrbk_t/mnt_source/snapshots/storage.20221110T210032+0800 24h 7d 4w 12m 3y (sunday, 00:00) preserve monthly: first weekly of month 2022-11 (1 months ago, 4d 14h after sunday 00:00)
- /tmp/btrbk_t/mnt_source/snapshots/storage.20221130T010001+0800 24h 7d 4w 12m 3y (sunday, 00:00) preserve weekly: 1 weeks ago, 2d 18h after sunday 00:00
- /tmp/btrbk_t/mnt_source/snapshots/storage.20221201T070000+0800 24h 7d 4w 12m 3y (sunday, 00:00) preserve daily: first of day, 3 days ago, at 00:00
delete /tmp/btrbk_t/mnt_source/snapshots/storage.20221202T000002+0800 24h 7d 4w 12m 3y (sunday, 00:00) -
- /tmp/btrbk_t/mnt_source/snapshots/storage.20221203T000005+0800 24h 7d 4w 12m 3y (sunday, 00:00) preserve daily: first of day, 2 days ago, 17h after 00:00
- /tmp/btrbk_t/mnt_source/snapshots/storage.20221204T021553+0100 24h 7d 4w 12m 3y (sunday, 00:00) preserve monthly: first weekly of month 2022-12 (0 months ago, 2h after sunday 00:00)
BACKUP SCHEDULE
---------------
ACTION SUBVOLUME SCHEME REASON
- /tmp/btrbk_t/mnt_target/storage.20220518T000000+0800 24h 7d 4w 12m 3y (sunday, 00:00) preserve yearly: first weekly of year 2022 (0 years ago, 2d 18h after sunday 00:00)
- /tmp/btrbk_t/mnt_target/storage.20220619T205732+0800 24h 7d 4w 12m 3y (sunday, 00:00) preserve monthly: first weekly of month 2022-06 (6 months ago, 14h after sunday 00:00)
- /tmp/btrbk_t/mnt_target/storage.20220704T220758+0800 24h 7d 4w 12m 3y (sunday, 00:00) preserve monthly: first weekly of month 2022-07 (5 months ago, 1d 16h after sunday 00:00)
- /tmp/btrbk_t/mnt_target/storage.20220907T221810+0800 24h 7d 4w 12m 3y (sunday, 00:00) preserve monthly: first weekly of month 2022-09 (3 months ago, 3d 16h after sunday 00:00)
- /tmp/btrbk_t/mnt_target/storage.20221031T210458+0800 24h 7d 4w 12m 3y (sunday, 00:00) preserve monthly: first weekly of month 2022-10 (2 months ago, 1d 14h after sunday 00:00)
- /tmp/btrbk_t/mnt_target/storage.20221110T210032+0800 24h 7d 4w 12m 3y (sunday, 00:00) preserve monthly: first weekly of month 2022-11 (1 months ago, 4d 14h after sunday 00:00)
- /tmp/btrbk_t/mnt_target/storage.20221130T010001+0800 24h 7d 4w 12m 3y (sunday, 00:00) preserve weekly: 1 weeks ago, 2d 18h after sunday 00:00
- /tmp/btrbk_t/mnt_target/storage.20221201T070000+0800 24h 7d 4w 12m 3y (sunday, 00:00) preserve daily: first of day, 3 days ago, at 00:00
- /tmp/btrbk_t/mnt_target/storage.20221203T000005+0800 24h 7d 4w 12m 3y (sunday, 00:00) preserve daily: first of day, 2 days ago, 17h after 00:00
- /tmp/btrbk_t/mnt_target/storage.20221204T021553+0100 24h 7d 4w 12m 3y (sunday, 00:00) preserve monthly: first weekly of month 2022-12 (0 months ago, 2h after sunday 00:00)
btrbk will still make incremental backups (i.e. btrfs send -p), even if there is no common parent found, this can be controlled by removing "sao,san" from incremental_prefs, e.g. --override=incremental_prefs='sro:1 srn:1 aro:1 arn:1'.
After successful run, you can check the approximate disk usage with:
btrbk extents diff /mnt/system-root/storage.2022*
(please paste results)
Here is the output of btrbk run -S -n:
SNAPSHOT SCHEDULE
-----------------
ACTION SUBVOLUME SCHEME REASON
- /mnt/storage-root/snapshots/storage.20220518T000000+0800 24h 7d 4w 12m 3y (sunday, 00:00) preserve yearly: first weekly of year 2022 (0 years ago, 3d after sunday 00:00)
- /mnt/storage-root/snapshots/storage.20220619T205732+0800 24h 7d 4w 12m 3y (sunday, 00:00) preserve monthly: first weekly of month 2022-06 (6 months ago, 20h after sunday 00:00)
- /mnt/storage-root/snapshots/storage.20220704T220758+0800 24h 7d 4w 12m 3y (sunday, 00:00) preserve monthly: first weekly of month 2022-07 (5 months ago, 1d 22h after sunday 00:00)
- /mnt/storage-root/snapshots/storage.20220907T221810+0800 24h 7d 4w 12m 3y (sunday, 00:00) preserve monthly: first weekly of month 2022-09 (3 months ago, 3d 22h after sunday 00:00)
- /mnt/storage-root/snapshots/storage.20221031T210458+0800 24h 7d 4w 12m 3y (sunday, 00:00) preserve monthly: first weekly of month 2022-10 (2 months ago, 1d 21h after sunday 00:00)
- /mnt/storage-root/snapshots/storage.20221110T210032+0800 24h 7d 4w 12m 3y (sunday, 00:00) preserve monthly: first weekly of month 2022-11 (1 months ago, 4d 21h after sunday 00:00)
- /mnt/storage-root/snapshots/storage.20221130T010001+0800 24h 7d 4w 12m 3y (sunday, 00:00) preserve weekly: 1 weeks ago, 3d 1h after sunday 00:00
- /mnt/storage-root/snapshots/storage.20221201T070000+0800 24h 7d 4w 12m 3y (sunday, 00:00) preserve daily: first of day, 6 days ago, 7h after 00:00
<truncated>
BACKUP SCHEDULE
---------------
ACTION SUBVOLUME SCHEME REASON
- /mnt/system-root/storage.20220518T000000+0800 24h 7d 4w 12m 3y (sunday, 00:00) preserve yearly: first weekly of year 2022 (0 years ago, 3d after sunday 00:00)
- /mnt/system-root/storage.20220619T205732+0800 24h 7d 4w 12m 3y (sunday, 00:00) preserve monthly: first weekly of month 2022-06 (6 months ago, 20h after sunday 00:00)
- /mnt/system-root/storage.20220704T220758+0800 24h 7d 4w 12m 3y (sunday, 00:00) preserve monthly: first weekly of month 2022-07 (5 months ago, 1d 22h after sunday 00:00)
- /mnt/system-root/storage.20220907T221810+0800 24h 7d 4w 12m 3y (sunday, 00:00) preserve monthly: first weekly of month 2022-09 (3 months ago, 3d 22h after sunday 00:00)
- /mnt/system-root/storage.20221031T210458+0800 24h 7d 4w 12m 3y (sunday, 00:00) preserve monthly: first weekly of month 2022-10 (2 months ago, 1d 21h after sunday 00:00)
- /mnt/system-root/storage.20221110T210032+0800 24h 7d 4w 12m 3y (sunday, 00:00) preserve monthly: first weekly of month 2022-11 (1 months ago, 4d 21h after sunday 00:00)
- /mnt/system-root/storage.20221130T010001+0800 24h 7d 4w 12m 3y (sunday, 00:00) preserve weekly: 1 weeks ago, 3d 1h after sunday 00:00
- /mnt/system-root/storage.20221201T070000+0800 24h 7d 4w 12m 3y (sunday, 00:00) preserve daily: first of day, 6 days ago, 7h after 00:00
<truncated>
--------------------------------------------------------------------------------
Backup Summary (btrbk command line client, version 0.32.6-dev)
Date: Wed Dec 7 15:15:25 2022
Config: btrbk.conf
Dryrun: YES
Legend:
=== up-to-date subvolume (source snapshot)
+++ created subvolume (source snapshot)
--- deleted subvolume
*** received subvolume (non-incremental)
>>> received subvolume (incremental)
--------------------------------------------------------------------------------
/mnt/storage-root/root
+++ /mnt/storage-root/snapshots/storage.20221207T151525+0800
*** /mnt/system-root/storage.20221130T010001+0800
>>> /mnt/system-root/storage.20220619T205732+0800
>>> /mnt/system-root/storage.20220704T220758+0800
>>> /mnt/system-root/storage.20220907T221810+0800
>>> /mnt/system-root/storage.20221031T210458+0800
>>> /mnt/system-root/storage.20221110T210032+0800
>>> /mnt/system-root/storage.20220518T000000+0800
>>> /mnt/system-root/storage.20221201T070000+0800
<truncated>
storage.20220518T000000+0800 is in the backup summary, although I think it should be the first in the list (i.e. as a non-incremental snapshot)?
sorry for the late reply, was super busy with other things lately...
storage.20220518T000000+0800 is in the backup summary, although I think it should be the first in the list (i.e. as a non-incremental snapshot)?
The order is defined by the subvolume cgen (generation at creation), not by the subvolume name. The idea behind this is that this way it better reflects "the order on how things were created on the filesystem". The end result is the same. In your case this really looks a bit confusing, I will consider changing this later.
Just fyi, I've stuck to using btrbk archive for now, it works perfectly for my use case (copying snapshots to another drive every few months).
This fix (from branch "fix-backup-all-snapshots-in-snapdir") is included in btrbk-0.32.6, closing ticket