plugins prometheus: Revive the prometheus plugin

I also added myself as maintainer in the CODEOWNERS file.

Aug 13 '24 14:08 cdecker

Excited to see this PR! Will try it out soon :)

Aug 19 '24 18:08 chrisguida

Tests are flaky, i saw some "address already in use" in one of them.

Aug 27 '24 11:08 daywalker90

Interesting, looks like this is only broken on 23.11

Aug 31 '24 23:08 chrisguida

Gonna update the CI to add 24.08 and remove 23.11, then this should be good to go

Sep 12 '24 18:09 chrisguida

@cdecker any suggestions on how to use this? I'm not familiar with the prometheus protocol.

I think this is fine to merge since this plugin was already in the plugins repo, just archived due to failing tests, which this PR fixes.

ACK https://github.com/lightningd/plugins/pull/582/commits/78b17daa00444c9d72bc74cdfe68f85bf4f2fab6

Sep 27 '24 17:09 chrisguida

Sure, the plugin just exposes the /metrics URL on a given port (concurrent tests are causing the flakiness), and then you configure your prometheus server to scrape that endpoint, which puts the metrics in its timeseries DB. It's a very lightweight way of collecting metrics in a scalable way.

Oct 18 '24 08:10 cdecker

Taking a look at this, I have a raspiblitz running cln, going to clone the prometheus plugin branch in and run prometheus and point it at the endpoint.

Oct 23 '24 21:10 mikemoniker

I was able to install prometheus and get it to ingest the metrics and display them on port 9100. I had some difficulties with poetry and raspiblitz.

Since raspiblitz does everything with pip install --user, but I did not find an equivalent for poetry, I had to edit the shebang of the prometheus.py to use the venv python executable. @cdecker, I am curious how you run this on your system. Is there a more reproducible way? So it can be easier for future users.

Oct 23 '24 23:10 mikemoniker

We have been playing with a number of possible python installation options:

Using the system python install. This quickly leads to version conflicts for multiple python projects and often lags severely behind in it's version often going over EOL
Using the env helper to find the python virtualenv: #!/user/bin/env python3 picks up the python env from the $PATH envvar. However that envvar comes from lightningd so it is shared between plugins, potentially causing multiple plugins to cause a version clash. The advantage is that it is very flexible and doesn't require additional setup, but plugins share an env.
Using uv to manage both python interpreter as well as python dependencies. Setting a shebang of #!/ist/bin/env -S uv run will pick uv from the env, have it read the script (see how dependency declaration can be embedded in the script), create a virtualenv (cached), download a matching interpreter (cached), install any missing dependency (cached) and run the script with the env. Requires some setup so that uv is available, but provides the best env isolation and the least headaches I think.

Ideally we'd have something that works everywhere. Maybe a runner script that analyzes the env, adjusts it and then runs the interpreter could be useful, but at that point why not use uv directly?

Dec 07 '24 11:12 cdecker

Hmm uv sounds interesting! Will research...

Dec 09 '24 19:12 chrisguida

re-ACK https://github.com/lightningd/plugins/pull/582/commits/7c565ddc7560b05554ee5e96862a44400ebc1833

Dec 09 '24 19:12 chrisguida