Keep track of remote xlog and basebackup
The purpose of this PR is to keep tracks of remote files uploaded. As mentioned in #366, with a list of xlog on remote storage, we can compute several new metrics and check if there is gap in the xlog sequence. This PR is a based for another PR that will add new metrics as described in #366.
This PR mainly adds 2 attributes on PGHoard class:
remote_xlogremote_basebackup
and code to keep those attributes up to date:
- First run of
handle_site()will fetch xlogs and basebackups from remote storage and populatesremote_xlogandremote_basebackupattributes. - On successful tranfert, new xlog or basebackup will be added to
remote_xlogandremote_basebackupattributes. - When deleting xlog or basebackup,
remote_xlogorremote_basebackupattributes will be updated.
Additionally, this PR merge remote_basebackup attribute with state["backup_sites"][site]["basebackups"]:
self.state["backup_sites"][site]["basebackups"] = self.remote_basebackup[site]
@Ormod Can you take a look at this merge request ? As discussed in #366, I want to keep track of remote files uploaded. So I added some attributes to store remote files and expose metrics:
- total wal segments
- continious wal segments
- valid base backup count
- missing wal segments
- missing wal segments at end This PR might impact performances on database with high traffic so I would like to have your point of view.