A script that makes it easy to export c-lightning logs into Prometheus and Grafana

[clarketowson]

trafficstars

Hi Everyone,

I have a suggestion for making the logfiles from c-lightning easily presentable on the web.

Please take a close look at the node stats generated on this page: https://lqwdtech.com/node-stats/

Which essentially boil down to:

Total BTC Total Estimated $CURRENCY Total Transactions Routed by $NODENAME Running Total of Transactions for All Nodes Running Total of BTC Processed for All Nodes Total Transactions by Node Total Amount Routed Per Node (BTC) Number of Channels Routed Transactions Month by Month Routed Transactions Week by Week Routed Transactions Day by Day

• with associated pretty graphs.

It occurred to me that there is currently no simple method for node operators to massage data from their c-lightning nodes into a format that’s easily presentable on the web in the same manner that lqwd have using the metrics above. So I got to work looking closely at Grafana and Prometheus which I believe is how lqwd is presenting their nodes data. I had a go at writing a script myself but it's not easy to utilize the excellent JSON logfiles from lightning for ingestion into Prometheus for presentation to the web via Grafana and I think it would be best if such a script was available to others as well in an official release of c-lightning.

I think this kind of data would be extremely useful for node operators (particularly fintechs) to present via their websites. If there was a command line script provided with a future version of c-lightning to achieve this that would be really fantastic. The script could be run as a cron job on the node pushing the data on a regular basis into a Prometheus database to Grafana and then published to the node operators website.

I'm keen to hear others thoughts on this.

Keep up the great work on lightning!

Best Regards,

Clarke

Clarke Towson, BCMS (Bachelor of Computer & Mathematical Science) CEO INTJ Billing

m: +61 432 359 166 a: 7 Cullen Court Spotswood Victoria 3015 AUSTRALIA w: https://intjbilling.com/ e: [email protected]

Bitcoin Lightning Network Node Name: CHILLYCALENDAR Node Public Key: 025124c73ef7ecf527e0114ead02a0cc6e3ecbc0c99474ee3f5506c4503b089693 Status Page: https://intjbilling.com/INTJBillingStatusPage.php

amboss.space: https://amboss.space/node/025124c73ef7ecf527e0114ead02a0cc6e3ecbc0c99474ee3f5506c4503b089693 1ml.com: https://1ml.com/node/025124c73ef7ecf527e0114ead02a0cc6e3ecbc0c99474ee3f5506c4503b089693 Lightning network+: https://lightningnetwork.plus/nodes/025124c73ef7ecf527e0114ead02a0cc6e3ecbc0c99474ee3f5506c4503b089693

[vincenzopalazzo]

I think the script is not the optimal idea, I am working on something like that, and I hope that the PR https://github.com/ElementsProject/lightning/pull/6990 get merged for starting work on a plugin that export the logs

[cdecker]

For what it's worth, we have a prometheus plugin that could be extended to fulfill the metrics use-case, and in order to pipe logs from CLN into Loki I personally use systemd to run CLN, which puts logs into journald from where we can tail and inject into Loki using promtail.