Feature Request - expose environment variable (for hook scripts) for pg_basebackup files missed due to being outside PGDATA

[kcslb92]

Hi Guys,

I think there would be value in adding an environment variable (to be used in hook scripts) to capture the files that are referenced in the backup output (and log file) for files that need to be manually backed up as they're outside of PGDATA, eg:

WARNING: pg_basebackup does not copy the PostgreSQL configuration files that reside outside PGDATA. Please manually backup the following files:
        /etc/postgresql/12/main/postgresql.conf
        /etc/postgresql/12/main/pg_hba.conf
        /etc/postgresql/12/main/pg_ident.conf
        /etc/postgresql/12/main/conf.d/replication.conf

If this could be exposed as an environment variable like:

BARMAN_PG-BASEBACKUP_OUTSIDE_PGDATA_FILES = '/etc/postgresql/12/main/postgresql.conf;/etc/postgresql/12/main/pg_hba.conf;/etc/postgresql/12/main/pg_ident.conf;/etc/postgresql/12/main/conf.d/replication.conf'

Or something to that effect.

This would make it possible to then have a hook script run that parses the env var back into an array, and use rsync/scp/etc to pull those files to the barman backup server to be included as part of the backup.

Other options exist:

• scraping log files/console output
• using barman show-server <server> to retrieve values for:
  • ident_file
  • included_files
  • hba_file
  • config_file

It appears that not all files from the above output are included in the console output for missed files, ie:

barman -f json show-server <server-name>| jq '.[] | .ident_file, .included_files, .hba_file, .config_file'
"/etc/postgresql/12/main/pg_ident.conf"
[
  "/etc/postgresql/12/main/conf.d/replication.conf",
  "/var/lib/postgresql/12/main/postgresql.auto.conf"
]
"/etc/postgresql/12/main/pg_hba.conf"
"/etc/postgresql/12/main/postgresql.conf"

Note that /var/lib/postgresql/12/main/postgresql.auto.conf did not get included in the previous output to console/log.

Keen to hear your thoughts.

If this is not the appropriate place to raise a feature request, please let me know and I will put it elsewhere.

Cheers!

[barthisrael]

Hello @kcslb92,

I think there would be value in adding an environment variable (to be used in hook scripts) to capture the files that are referenced in the backup output (and log file) for files that need to be manually backed up as they're outside of PGDATA

Other options exist:

• using barman show-server <server> to retrieve values for:
  • ident_file
  • included_files
  • hba_file
  • config_file

That is a viable option.

When backup_method = postgres is present in the configuration of the Barman server, Barman uses pg_basebackup to take backups from the Postgres server.

As the WARNING message issued by Barman says, pg_basebackup does not copy configuration files that live outside of PGDATA. In that case, it's the user responsibility to back those files up.

As Barman does not have SSH connections to Postgres when backup_method = postgres, it has no way of copying those external configuration files for you.

On the other hand, when backup_method = rsync, Barman does copy the external configuration files for you.

In any case, as you mentioned, you can configure a post-backup hook to grab external configuration files and store them somewhere in your server, when using backup_method = postgres.

It appears that not all files from the above output are included in the console output for missed files Note that /var/lib/postgresql/12/main/postgresql.auto.conf did not get included in the previous output to console/log.

That's because the WARNING message doesn't mention files that are inside PGDATA. In this case, the postgresql.auto.conf is inside PGDATA, so it's not part of the WARNING message. That file is copied as part of the pg_basebackup call made by Barman.

If this is not the appropriate place to raise a feature request, please let me know and I will put it elsewhere.

Yes, it is an appropriate place to request features.

However, as you already found out by yourself, you have a viable way of performing that task through a post-backup hook that uses barman show-server.

On a side topic, we intend to add information to the docs about how to manage external configuration files when backing up or restoring backups taken with rsync or pg_basebackup.

Best regards, Israel.