pghoard icon indicating copy to clipboard operation
pghoard copied to clipboard

Keep track of remote xlog and basebackup

Open Vampouille opened this issue 6 years ago • 1 comments

The purpose of this PR is to keep tracks of remote files uploaded. As mentioned in #366, with a list of xlog on remote storage, we can compute several new metrics and check if there is gap in the xlog sequence. This PR is a based for another PR that will add new metrics as described in #366.

This PR mainly adds 2 attributes on PGHoard class:

  • remote_xlog
  • remote_basebackup

and code to keep those attributes up to date:

  • First run of handle_site() will fetch xlogs and basebackups from remote storage and populates remote_xlog and remote_basebackup attributes.
  • On successful tranfert, new xlog or basebackup will be added to remote_xlog and remote_basebackup attributes.
  • When deleting xlog or basebackup, remote_xlog or remote_basebackup attributes will be updated.

Additionally, this PR merge remote_basebackup attribute with state["backup_sites"][site]["basebackups"]:

self.state["backup_sites"][site]["basebackups"] = self.remote_basebackup[site]

Vampouille avatar Oct 02 '19 11:10 Vampouille

@Ormod Can you take a look at this merge request ? As discussed in #366, I want to keep track of remote files uploaded. So I added some attributes to store remote files and expose metrics:

  • total wal segments
  • continious wal segments
  • valid base backup count
  • missing wal segments
  • missing wal segments at end This PR might impact performances on database with high traffic so I would like to have your point of view.

Vampouille avatar Nov 19 '19 07:11 Vampouille