lxd icon indicating copy to clipboard operation
lxd copied to clipboard

Update metrics.md Changed /common/tls to /current/tls for metrics.crt, metrics.key & server.crt location

Open JohnHammell opened this issue 9 months ago • 6 comments

Changed: /var/snap/prometheus/common/tls/ to /var/snap/prometheus/current/tls/ as the location for the metrics.crt, metrics.key & server.crt files in Prometheus.

I was getting no data in Grafana after following this page.

Looked into my syslog and found the following message:

2024-05-11T03:08:48.449690+00:00 lxd-metrics prometheus.prometheus[2973]: ts=2024-05-11T03:08:48.448Z caller=manager.go:212 level=error component="scrape manager" msg="error creating new scrape pool" err="error creating HTTP client: unable to load specified CA cert /var/snap/prometheus/86/tls/server.crt: open /var/snap/prometheus/86/tls/server.crt: no such file or directory" scrape_pool=lxd

So I made a copy of server.crt in the /86/tls/ folder where the message states it was missing from and that error went away.

Then started seeing this next error regarding metrics.crt & metrics.key:

2024-05-11T03:10:31.336446+00:00 lxd-metrics prometheus.prometheus[3051]: ts=2024-05-11T03:10:31.335Z caller=manager.go:212 level=error component="scrape manager" msg="error creating new scrape pool" err="error creating HTTP client: unable to use specified client cert (/var/snap/prometheus/86/tls/metrics.crt) & key (/var/snap/prometheus/86/tls/metrics.key): open /var/snap/prometheus/86/tls/metrics.crt: no such file or directory" scrape_pool=lxd

Also moved metrics.crt & metrics.key into /86/tls/ and that second error went away.

As a test, I removed metrics.key, metrics.crt & server.crt from the original /common/tls/ location and that did not cause a problem.

It appears that Prometheus needs those .crt & .key files in the /var/snap/prometheus/86/tls folder rather than /var/snap/prometheus/common/tls , otherwise I get those errors about them being missing.

/current/ is a symlink to /86/ , so I figured there must be a reason for that and used /current/ in my edit instead of /86/

And, finally, in case this makes any difference, here is my setup:

Using Ubuntu 22.04 as my host LXD 5.21 server, I followed this page and setup Prometheus, Grafana, Loki & Promtail inside an Ubuntu 24.04 container. Proxied web access to Grafana through HAProxy.

JohnHammell avatar May 11 '24 16:05 JohnHammell