Barman incremental backup does full backup

[Edureka-spandey]

the timing of incremental and full backup is same , the point of incremental backup is taking the last backup id and backing up changes. But the size and time of incremental and full backup is same. Using reuse_backup=link not working for incrementals

[didiermichel]

Hi @Edureka-spandey. Could you add more context to your problem so I can investigate?

• barman and postgres versions
• The failing command.
• barman logs

Thank you

[ghost]

barman and postgres versions : barman version 2.18 . postgres version 10.13 The failing command.: The command is not failing but incremental backup is taking same time as full backup barman backup --jobs 4 --reuse-backup=link server --wait Barman show-backup SO if take the incremental backup id (wal size 7.1GB) The copy time is

Copy time            : 3 hours, 45 minutes, 10 seconds + 2 seconds startup 
barman@Backup-server:~$ barman show-backup server 20220308T220003
Backup 20220308T220003:
  Server Name            : server
  System Id              : 6595745951660340829
  Status                 : DONE
  PostgreSQL Version     : 100010
  PGDATA directory       : /mnt/data/postgresql/10/main

  Base backup information:
    Disk usage           : 1.5 TiB (1.5 TiB with WALs)
    Incremental size     : 1.0 TiB (-30.51%)
    Timeline             : 1
    Begin WAL            : 000000010001162E000000A2
    End WAL              : 000000010001163C00000073
    WAL number           : 3538
    WAL compression ratio: 58.16%
    Begin time           : 2022-03-08 22:00:03.902821+00:00
    End time             : 2022-03-09 01:45:22.923195+00:00
    Copy time            : 3 hours, 45 minutes, 10 seconds + 2 seconds startup
    Estimated throughput : 79.9 MiB/s (2 jobs)
    Begin Offset         : 164128
    End Offset           : 2208976
    Begin LSN           : 1162E/A2028120
    End LSN             : 1163C/7321B4D0

  WAL information:
    No of files          : 1229
    Disk usage           : 7.1 GiB
    WAL rate             : 565.43/hour
    Compression ratio    : 63.16%
    Last available       : 000000010001164100000040

  Catalog information:
    Retention Policy     : VALID
    Previous Backup      : 20220308T010003
    Next Backup          : 20220309T030003

ANd Full BACkup

Copy time            : 3 hours, 28 minutes, 5 seconds + 2 seconds startup
barman@backup-server:~$ barman show-backup server 20220308T010003
Backup 20220308T010003:
Server Name            : server
System Id              : 6595745951660340829
Status                 : DONE
PostgreSQL Version     : 100010
PGDATA directory       : /mnt/data/postgresql/10/main

Base backup information:
  Disk usage           : 1.5 TiB (1.5 TiB with WALs)
  Incremental size     : 1.0 TiB (-30.60%)
  Timeline             : 1
  Begin WAL            : 000000010001161A00000031
  End WAL              : 000000010001161C00000058
  WAL number           : 552
  WAL compression ratio: 60.09%
  Begin time           : 2022-03-08 01:00:03.519559+00:00
  End time             : 2022-03-08 04:28:17.502676+00:00
  Copy time            : 3 hours, 28 minutes, 5 seconds + 2 seconds startup
  Estimated throughput : 86.3 MiB/s (2 jobs)
  Begin Offset         : 2223104
  End Offset           : 14919200
  Begin LSN           : 1161A/3121EC00
  End LSN             : 1161C/58E3A620

WAL information:
  No of files          : 8219
  Disk usage           : 49.0 GiB
  WAL rate             : 354.34/hour
  Compression ratio    : 61.87%
  Last available       : 000000010001163C00000073

Catalog information:
  Retention Policy     : VALID
  Previous Backup      : 20220306T220004

barman logs --- No error in logs

[mikewallace1979]

Hi @Edureka-spandey - thanks for providing more information.

Am I correct in saying that both backups were taken using the --reuse-backup=link option, and that you expected the more recent backup to have been quicker since it would only need to copy the data that changed since the older backup was taken?

Looking at the show-backup output for both backups indicates that file-level incremental backup is working - we can see this because the Disk usage and Incremental size fields differ for both backups, yielding a reduction of around 30% in both cases.

If both backups were taken with the --reuse-backup=link option then:

1. The older backup, 20220308T010003, would have only copied the files from the source which differed from those of the previous backup.
2. The newer backup, 20220308T220003, would have only copied the files from the source which differed from those in backup 20220308T010003.

To understand why there isn't much difference in copy times between these two backups it's worth considering the limitations of Barman's file-level incremental backup. When --reuse-backup=link is used it causes Barman to add the --link-dest option to rsync, where the --link-dest target is a location in the previous backup - this means that any files on the source which are identical to those in the previous backup are not copied and instead a hard link is created in the destination directory for the new backup.

Any difference between the files will cause the entire source file to be copied to the destination - given PostgreSQL tables and indexes are stored in files of up to 1GB this means that a single update to a table could cause up to 1GB to be copied during the backup.

The fact that both backups take a similar amount of time and have a similar incremental size compared to their total disk usage suggests that the PostgreSQL workload is causing many index and table files to change between the two backups. The rsync jobs executed by Barman then end up copying each of those files to the Barman server.

This is not particularly unusual and really just highlights the limitations of file-level incremental backups. Improvements to incremental backups (e.g. block-level incremental backup) are on the roadmap however there is currently no ETA.