pg_prometheus
pg_prometheus copied to clipboard
Strange behaviour of queries with 'irate' function
Hello, We are using prometheus-postgresql-adapter and pg-prometheus services for keeping prometheus metrics in database storage. I'm not sure which one is causing the problem, but after switching from prometheus local persistent storage to those two services, all queries containing 'irate' function seem to behave incorrectly. It is especially visible in Grafana dashboards - graphs with 'irate' are showing 'no data' error, or there are no charts visible, only values are shown, or charts are visible only for ~2 seconds, and they disappear again. After changing 'irate' to 'rate' function there are no problems and everything works just fine.
Hi @maciejprz,
In order to dig deeper into no data
error I would recommend to run prometheus-postgresql-adapter
with --log.level=debug
and in the log file you will see SQL queries that are running against the TimescaleDB to fetch Prometheus data. You can test those SQL queries explicitly to make sure that they do return the data.
The irate
behaviour seems strange. The remote storage integration is only responsible for moving data back and forth between Prometheus instance and remote storage so shouldn't have anything to do with the PromQL(eg. irate).
Let me know if looking into SQL queries has helped.