Issue about Github backup utility

[scho-zenithn]

Hello team,

I have question about backup utility. Is there any suspicious part or, need corrections on my comment below? And is there any recommended troubleshooting method for solving this issue?

While I run backup manually, backup is stucked on coping audit logs. It was took an hour and half to make me aborted it.

After upgrading live instance, I checked previous backup history. Backup runs every morning at 02:10:01

20220420T021001: 761.98 GB / audit-log directory size: 1.82 GB 20220421T021001: 761.69 GB / audit-log directory size: 1.81 GB current -> 20220421T021001 (last modified time: 02:25:xx)

According to above data, usually it takes 10 to 15 minitues for making snapshot, because it started at 20220421T021001, and finished at 20220421T0225XX, when symbolic link was created (shot of 20220420 was also takes 10 to 15 minitues, last modified time of that directory was 02:2X:XX)

But snapshot that I tried last night(20220421T193003), it was only 1.78 GB with 1.54 GB audit-log directory. (I can't believe that it takes an hour an half to copy 1.78 GB of data!!)

Unfortunately, I didn't added "-v" switch with ghe-backup command. So, I couldn't figure out what was the cause of this issue last night.

In my guess, it supposed to be finished with in 30 minitues, because entire GHE storage usage was less then 800 GB

In my understanding, the utility creates incremental backup of GHE instance. Previous data was refered using hard link, and new data since the last snapshot will be added. (And it means, even though all snapshot is deleted except the last successful one, backup data is still valid. And If I want to make a fresh backup, make a snapshot on empty directory. Am I right for it?,)

But, I read againg documents, repository data backup with incremental data, but some of other data is created with full backup. If you don't mind, please help me to understand github backup utility mechanism.

==>The ghe-backup command creates incremental snapshots of repository data, along with full snapshots of all other pertinent data stores. (from: https://github.com/github/backup-utils/blob/master/docs/usage.md)

==>Backup snapshots are stored in rotating increment directories named after the date and time the snapshot was taken. Each snapshot directory contains a full backup snapshot of all relevant data stores. Repository, Search, and Pages data is stored efficiently via hard links. (from: https://github.com/github/backup-utils/blob/master/docs/backup-snapshot-file-structure.md)