rsync-time-backup
rsync-time-backup copied to clipboard
Published
20 hours ago •
laurent22
laurent22
default expiration logic not work with hourly cron job
I use cron job to hourly back up my folder
# m h dom mon dow command
11 * * * * rsync_tmbackup.sh -p 22 ~ root@remote-backup:~
and the remote back folder looks like
drwxr-x--- 34 root root 4.0K Dec 24 01:25 2018-12-24-101101/
drwxr-x--- 34 root root 4.0K Dec 24 02:46 2018-12-24-111101/
drwxr-x--- 34 root root 4.0K Dec 24 03:25 2018-12-24-121102/
drwxr-x--- 34 root root 4.0K Dec 24 03:25 2018-12-24-131101/
drwxr-x--- 34 root root 4.0K Dec 24 05:51 2018-12-24-141101/
drwxr-x--- 34 root root 4.0K Dec 24 07:02 2018-12-24-151102/
drwxr-x--- 34 root root 4.0K Dec 24 07:02 2018-12-24-161101/
drwxr-x--- 34 root root 4.0K Dec 24 07:02 2018-12-24-171101/
drwxr-x--- 34 root root 4.0K Dec 24 07:02 2018-12-24-181101/
drwxr-x--- 34 root root 4.0K Dec 24 07:02 2018-12-24-191101/
drwxr-x--- 34 root root 4.0K Dec 24 07:02 2018-12-24-201101/
drwxr-x--- 34 root root 4.0K Dec 24 07:02 2018-12-24-211101/
drwxr-x--- 34 root root 4.0K Dec 24 07:02 2018-12-24-221101/
drwxr-x--- 34 root root 4.0K Dec 24 07:02 2018-12-24-231102/
drwxr-x--- 34 root root 4.0K Dec 24 07:02 2018-12-25-011102/
drwxr-x--- 34 root root 4.0K Dec 24 07:02 2018-12-25-021101/
drwxr-x--- 34 root root 4.0K Dec 24 07:02 2018-12-25-031102/
drwxr-x--- 34 root root 4.0K Dec 24 07:02 2018-12-25-041101/
drwxr-x--- 34 root root 4.0K Dec 24 07:02 2018-12-25-051101/
drwxr-x--- 34 root root 4.0K Dec 24 07:02 2018-12-25-061101/
drwxr-x--- 34 root root 4.0K Dec 24 07:02 2018-12-25-071101/
drwxr-x--- 34 root root 4.0K Dec 24 07:02 2018-12-25-081102/
drwxr-x--- 34 root root 4.0K Dec 24 07:02 2018-12-25-091101/
drwxr-x--- 34 root root 4.0K Dec 24 07:02 2018-12-25-101101/
according to readme :
The default strategy is 1:1 30:7 365:30, which means:
After 1 day, keep one backup every 1 day (1:1).
After 30 days, keep one backup every 7 days (30:7).
After 365 days, keep one backup every 30 days (365:30).
but if all the backups(more than one day) were deleted, how to keep the backups older than 30days?
I think the default backup strategy is similar as Grandfather-father-son(https://en.wikipedia.org/wiki/Backup_rotation_scheme), so if an error occur, but I do not noticed in a week, I still have chance to restore from monthly backup. How could I combined GFS with hourly backup ?
I have the same problem. It's only keeping the backups from the last 24h.
I have the same issue. I run the script on multiple directories with different schedules in cron (some 4 hours apart, some 6, some 12), but in all cases, only the past 24 hours of backups are kept.
I have attempted to leave --strategy unset, and to explicitly set it. The behavior does not seem to change.
I do make use of following rsync flags: --numeric-ids --links --hard-links --one-file-system --archive --no-perms --no-group --itemize-changes
My base OS is Ubuntu 18.04.01 LTS (Bionic Beaver). I'm using the git clone version of this software as of today's date.
I'm wondering if it's a log file issue. As far as I can tell, there are no files in the directory specified by the --log-file directory, which is the default.
Please let me know if I can provide any additional diagnostic information. I don't think I'm good enough with bash to puzzle this one out myself.
Same thing here, on OSX 10.14.2.
Using the default config and a cron every hour, the script is only keeping the 24 latest backups.
rsync_tmbackup.sh --log-dir /logdir [email protected]:/docPath /backupPath excludeFile.txt
I think this is actually the same issue as that reported in #121
Albert444 suggested a solution in that issue, and I have copied it into my local version to see if that will fix the issues.
Here's the diff of the version provided by Albert444:
60c60
< linux*) date -d "${1:0:10} ${1:11:2}:${1:13:2}:${1:15:2}" +%s ;;
---
> linux*) date -d "${1:0:10}" +%s ;;
129c129,133
< if [ "$interval_since_last_kept" -lt "$cut_off_interval" ]; then
---
>
> dayssincestart=$(((backup_timestamp-start_timestamp)/86400))
> modulo=$((dayssincestart % (cut_off_interval/86400)))
>
> if [ "$modulo" -ne "0" ]; then
133,134c137,142
< # No: Keep it
< last_kept_timestamp=$backup_timestamp
---
> if [ "$interval_since_last_kept" -lt "86400" ]; then #if diffenrence is less than one day
> fn_expire_backup "$backup_dir"
> else
> # No: Keep it
> last_kept_timestamp=$backup_timestamp
> fi
now I use borg backup instead.
it's prune example is more clear.
borg prune visualized
=====================
Assume it is 2016-01-01, today's backup has not yet been made and you have
created at least one backup on each day in 2015 except on 2015-12-19 (no
backup made on that day).
This is what borg prune --keep-daily 14 --keep-monthly 6 would keep.
Backups kept by the --keep-daily rule are marked by a "d" to the right,
backups kept by the --keep-monthly rule are marked by a "m" to the right.
Calendar view
-------------
2015
January February March
Mo Tu We Th Fr Sa Su Mo Tu We Th Fr Sa Su Mo Tu We Th Fr Sa Su
1 2 3 4 1 1
5 6 7 8 9 10 11 2 3 4 5 6 7 8 2 3 4 5 6 7 8
12 13 14 15 16 17 18 9 10 11 12 13 14 15 9 10 11 12 13 14 15
19 20 21 22 23 24 25 16 17 18 19 20 21 22 16 17 18 19 20 21 22
26 27 28 29 30 31 23 24 25 26 27 28 23 24 25 26 27 28 29
30 31
April May June
Mo Tu We Th Fr Sa Su Mo Tu We Th Fr Sa Su Mo Tu We Th Fr Sa Su
1 2 3 4 5 1 2 3 1 2 3 4 5 6 7
6 7 8 9 10 11 12 4 5 6 7 8 9 10 8 9 10 11 12 13 14
13 14 15 16 17 18 19 11 12 13 14 15 16 17 15 16 17 18 19 20 21
20 21 22 23 24 25 26 18 19 20 21 22 23 24 22 23 24 25 26 27 28
27 28 29 30 25 26 27 28 29 30 31 29 30m
July August September
Mo Tu We Th Fr Sa Su Mo Tu We Th Fr Sa Su Mo Tu We Th Fr Sa Su
1 2 3 4 5 1 2 1 2 3 4 5 6
6 7 8 9 10 11 12 3 4 5 6 7 8 9 7 8 9 10 11 12 13
13 14 15 16 17 18 19 10 11 12 13 14 15 16 14 15 16 17 18 19 20
20 21 22 23 24 25 26 17 18 19 20 21 22 23 21 22 23 24 25 26 27
27 28 29 30 31m 24 25 26 27 28 29 30 28 29 30m
31m
October November December
Mo Tu We Th Fr Sa Su Mo Tu We Th Fr Sa Su Mo Tu We Th Fr Sa Su
1 2 3 4 1 1 2 3 4 5 6
5 6 7 8 9 10 11 2 3 4 5 6 7 8 7 8 9 10 11 12 13
12 13 14 15 16 17 18 9 10 11 12 13 14 15 14 15 16 17d18d19 20d
19 20 21 22 23 24 25 16 17 18 19 20 21 22 21d22d23d24d25d26d27d
26 27 28 29 30 31m 23 24 25 26 27 28 29 28d29d30d31d
30m
List view
---------
--keep-daily 14 --keep-monthly 6
-------------------------------------------------
1. 2015-12-31 (2015-12-31 kept by daily rule)
2. 2015-12-30 1. 2015-11-30
3. 2015-12-29 2. 2015-10-31
4. 2015-12-28 3. 2015-09-30
5. 2015-12-27 4. 2015-08-31
6. 2015-12-26 5. 2015-07-31
7. 2015-12-25 6. 2015-06-30
8. 2015-12-24
9. 2015-12-23
10. 2015-12-22
11. 2015-12-21
12. 2015-12-20
(no backup made on 2015-12-19)
13. 2015-12-18
14. 2015-12-17
Notes
-----
2015-12-31 is kept due to the --keep-daily 14 rule (because it is applied
first), not due to the --keep-monthly rule.
Because of that, the --keep-monthly 6 rule keeps Nov, Oct, Sep, Aug, Jul and
Jun. December is not considered for this rule, because that backup was already
kept because of the daily rule.
2015-12-17 is kept to satisfy the --keep-daily 14 rule - because no backup was
made on 2015-12-19. If a backup had been made on that day, it would not keep
the one from 2015-12-17.
We did not include yearly, weekly, hourly, minutely or secondly rules to keep
this example simple. They all work in basically the same way.
The weekly rule is easy to understand roughly, but hard to understand in all
details. If interested, read "ISO 8601:2000 standard week-based year".
and from my test, it works as expected:
remote-2019-01-07T23:05:02 Mon, 2019-01-07 23:05:03 [b9e32f94e9a2a37c41df42548a42423b9d292b4fd5d923a176cedad511fc0bdd]
remote-2019-01-08T23:05:02 Tue, 2019-01-08 23:05:03 [7ee1988d17177c24d7d151deae500138d772ba2502f212a2644890d670f72525]
remote-2019-01-09T23:05:02 Wed, 2019-01-09 23:05:03 [e91d6c2453522af0d10f8cbe0eb1a8bb0116e4e8426a0597892480fc871a68c6]
remote-2019-01-10T23:05:02 Thu, 2019-01-10 23:05:03 [7f7f884715abd62bc91719b98ef074b89f57ff098a776d6c83d6d15961de86aa]
remote-2019-01-11T23:05:02 Fri, 2019-01-11 23:05:03 [c672daebf23fa8239a252797c23c74bf49f651eafc915dc4759df7bd0b15919a]
remote-2019-01-12T14:05:02 Sat, 2019-01-12 14:05:03 [73547871b921e80d9b920b8f0cf418e4d774327a8d975dacfa15bbbbe8473c2f]
remote-2019-01-12T15:05:01 Sat, 2019-01-12 15:05:03 [523fa148c60f44d97dee9963e6a5f08dd94f132c51e5d5752dc439785d2e3a26]
remote-2019-01-12T16:05:02 Sat, 2019-01-12 16:05:03 [719f16e12fbf10effa27eb62a3d7de4864bc400cb690967212fc56b11a774ec4]
remote-2019-01-12T17:05:02 Sat, 2019-01-12 17:05:03 [8cb564a20ec9b07de8ff10b776c83adead62421a9fbf40564cbd0e6881777f84]
remote-2019-01-12T18:05:01 Sat, 2019-01-12 18:05:03 [9e75f2f61a4f4a064ddb7be9a5135d0611e4e371fdab723a66d44ca47a7b9e5e]
remote-2019-01-12T19:05:02 Sat, 2019-01-12 19:05:03 [8668038c460f0f1d7e7c5ebfbccb15b903403715b519feea357b554523fe0e80]
remote-2019-01-12T20:05:02 Sat, 2019-01-12 20:05:03 [21c321b53c8e81dfef2d500e7ddfe15fd7358e1b46354f39e6685044991e5ce8]
remote-2019-01-12T21:05:02 Sat, 2019-01-12 21:05:03 [e33b2c34b6e7f51b6850d74bb7ef4e67271844e59cc3ae2065de775c8522c92e]
remote-2019-01-12T22:05:02 Sat, 2019-01-12 22:05:03 [e8a2771c62c5c418464ac44311fc1d196a3d2bb863ed08fb13bef9b02bd06179]
remote-2019-01-12T23:05:01 Sat, 2019-01-12 23:05:03 [82797bcb4d77b56fbfb404938682c34cf79242293bb54c8dbe0d994a339be9b2]
remote-2019-01-13T00:05:02 Sun, 2019-01-13 00:05:03 [3f2108ed7d176b0c35efbb23a4b4d118e38910fb6178a9a1c521082e986f53ff]
remote-2019-01-13T01:05:02 Sun, 2019-01-13 01:05:03 [e1fd3117529fa0f5a9f58dea151286394cf1bf6774ef5c0bff8f0ec2697d905e]
remote-2019-01-13T02:05:02 Sun, 2019-01-13 02:05:03 [31aca3bf46d5678920a3b1bb979c7e8672e00cf2a3f51228337018680dc06f8d]
remote-2019-01-13T03:05:02 Sun, 2019-01-13 03:05:03 [2b55a79a3b8b8c4b46f4c79d429e628c637b3839bc8d1aedb844b292c40b7149]
remote-2019-01-13T04:05:02 Sun, 2019-01-13 04:05:03 [5352485332c36de4249aaab02b0eaf292aa62af5cc4cc83ab8a90083aae747a6]
remote-2019-01-13T05:05:01 Sun, 2019-01-13 05:05:02 [4486d27a9b8b6eb915ad00ad700fe5e96e2d175cc31dbd7128015c4b7dd39098]
remote-2019-01-13T06:05:02 Sun, 2019-01-13 06:05:03 [a278295343d40a75e7aad8a16c46418ef146681929eb9748def5e44f42eabe00]
remote-2019-01-13T07:05:01 Sun, 2019-01-13 07:05:03 [b5bd9f8c7f371608ec1e45b115f500ae38b9c716ed64b710fb14892b4625ca0d]
remote-2019-01-13T08:05:01 Sun, 2019-01-13 08:05:03 [2698f1690967f40451f0c7c840b03e178ab1a2c8591461e0e9e2241b7e2f9f9d]
remote-2019-01-13T09:05:01 Sun, 2019-01-13 09:05:02 [70e20a5167deb7a0b8b6d64d97edac40b1e57598b78e20f5ff5907cbd483dcaf]
remote-2019-01-13T10:05:01 Sun, 2019-01-13 10:05:02 [6d69cea793fbb486b1417a3431d4d8b4c1f2fd9a2004dd95ab6582ea64d5de16]
remote-2019-01-13T11:05:02 Sun, 2019-01-13 11:05:03 [0ca8b8c500ee72bd23962959a55719fedbebbd33f84d4ba219b310dd6770abc4]
remote-2019-01-13T12:05:01 Sun, 2019-01-13 12:05:03 [38c50ec0b313fa1cc399eb4d343a7f99fc6422bfbe440e36d95cb3fa87bbb66c]
remote-2019-01-13T13:05:02 Sun, 2019-01-13 13:05:03 [18e18a874649a653d1a7d1a961ae87e897574fa0cae0411f766bc3cda468f31f]