Unable to get metrics for ccp_backrest_* query

[KMikkey]

Overview

Accordingly to documentation on Helm installation I've enabled monitoring and provided image version accordingly to cluster version. All metrics are available expect ccp_backrest*.

Environment

Please provide the following details:

• Platform: OKE
• Platform Version: 1.25.4
• PGO Image Tag: 5.4.2-0
• Postgres Version: 13

Steps to Reproduce

Using helm installation of PGO with parameters:

• cluster image: 5.4.2-0
• operator image: 5.3.4-0
• postgres image: 13.8-1
• pgbackrest image: 2.41-2
• exporter image: 5.4.2-0
• monitoring: true

REPRO

Provide steps to get to the error condition:

1. In my case deployment the of PGO is done by Terraform
2. Try kubectl logs postgres-ha -n postgres -c exporter

EXPECTED

Exporter is exposing metrics for ccp_backrest*.

ACTUAL

All metrics are available expect those for ccp_backrest*.

Logs

ts=2024-01-29T12:05:40.037Z caller=namespace.go:236 level=info err="Error running query on database \"localhost:5432\": ccp_backrest_oldest_full_backup pq: cannot delete from table \"pgbackrest_info\" because it does not have a replica identity and publishes deletes" ts=2024-01-29T12:05:40.042Z caller=namespace.go:236 level=info err="Error running query on database \"localhost:5432\": ccp_backrest_last_full_backup pq: cannot delete from table \"pgbackrest_info\" because it does not have a replica identity and publishes deletes" ts=2024-01-29T12:05:40.085Z caller=namespace.go:236 level=info err="Error running query on database \"localhost:5432\": ccp_backrest_last_info pq: cannot delete from table \"pgbackrest_info\" because it does not have a replica identity and publishes deletes" ts=2024-01-29T12:05:40.089Z caller=namespace.go:236 level=info err="Error running query on database \"localhost:5432\": ccp_backrest_last_diff_backup pq: cannot delete from table \"pgbackrest_info\" because it does not have a replica identity and publishes deletes" ts=2024-01-29T12:05:40.096Z caller=namespace.go:236 level=info err="Error running query on database \"localhost:5432\": ccp_backrest_last_incr_backup pq: cannot delete from table \"pgbackrest_info\" because it does not have a replica identity and publishes deletes" ts=2024-01-29T12:05:40.115Z caller=postgres_exporter.go:716 level=error err="queryNamespaceMappings returned 5 errors"

Additional Information

After fixing the error with replica identity, it seems that exporter is not able to find the file usr/bin/pgbackrest-info.sh

##How to fix

On Postgres instance (master) execute ALTER TABLE monitor.pgbackrest_info REPLICA IDENTITY FULL; After few minutes cpp_backrest* metrics should be available.

[tjmoore4]

Hello @KMikkey. Sorry to hear you're hitting this issue. I did some local testing and was unable to duplicate the provided error. That said, there are a few things I can recommend trying.

To begin with, it is important to ensure your software versions match for a given release to minimize the chance of feature misalignment. That means your cluster, operator and exporter versions should always match for best results, so running operator version 5.3.4 is not recommended with exporter and cluster versions 5.4.2. Please see the release notes for more information.

Also, I will not that there have been some significant improvements to the monitoring stack in version 5.5.0 (see the release notes), so I would recommend trying out that version to see if it resolves the issue you are encountering. Hope this helps!

[tjmoore4]

Since we haven't heard back on this issue for some time, I am closing this issue. If you need further assistance, feel free to re-open this issue or ask a question in our Discord server.