connect promgen-prometheus

[josefoSe]

Hi, Paul, how are you? I annoy you because I am not able to connect promgen with prometheus and I am lost. I am trying a simple installation on my intranet and on the same subnet. I have promgen installed at 10.30.156.72. Prometheus, grafana, and alert_manager at 10.30.56.65. No firewall. The point is that what I modify in promgen is not reflected in prometheus (teams, alerts, groups, etc.). For example, the promgen.rule.yml was never created. Create it by hand, with permissions and nothing happened either. I installed promgen following the installation guide: https://github.com/line/promgen/blob/master/CONTRIBUTING.md and after installed and tested I continued with: https://promgen.readthedocs.io/en/latest/conf/index.html i opted for Cron Model (Pull) because i dont have celery installed, neither redis / broker promgen-prometheus.zip Is there something I am missing to install or configure? Thank you!

[kfdm]

Likely unrelated to this particular error, but I notice in your promgen.yml you have url listed twice under both prometheus and alertmanager. This is probably an error.

As to the targets/rules not showing up in Prometheus, can you verify your cron job ? https://promgen.readthedocs.io/en/latest/conf/prometheus.html#cron-model-pull

I would also verify the promgen api is working correctly and that you can get a list of targets/rules from the API. You should be able to test using something like. curl -v http://10.30.156.72/api/v1/targets and curl -v http://10.30.156.72/api/v1/rules to verify.

[josefoSe]

Okay. Thank you very much for your quick response! I had put both servers because I assumed that it could handle two prometheus with their alertManager in a single promgen. This is bad? I think I was missing the link between targets.json and prometheus.yml which was pointing to /etc/prometheus/promgen.json (file that is empty from the beginning). The same with the alerts. What I did was: in prometheus.yml:

rule_files:
# - "/etc/prometheus/promgen.rule.yml"
    - "/etc/prometheus/rules.json"

  - job_name: 'promgen'
    file_sd_configs:
      - files:
# - "/etc/prometheus/promgen.json"
        - "/etc/prometheus/targets.json"

and in the cron file that runs every 5 minutes:

#!/bin/sh
set -e
curl http://10.30.156.72/api/v1/targets --output /etc/prometheus/targets.tmp
curl http://10.30.156.72/api/v1/rules --output /etc/prometheus/rules.tmp
mv /etc/prometheus/targets.tmp /etc/prometheus/targets.json
mv /etc/prometheus/rules.tmp /etc/prometheus/rules.json
curl -XPOST http://localhost:9090/-/reload

This is good?

Edited formatting (You can use 3 ` characters to make a code block :) )

[kfdm]

/etc/prometheus/rules.json should likely be rules.yaml (Rules are yaml not json) but otherwise it looks ok to me.

The promgen configuration is yaml, so you can not have two of the same key (example two url) at the same level. It's not well explained, but the promgen.yml typically gets deployed to each server (when using celery) so I think I need to add more details to the documentation

[josefoSe]

Yes, I understand. The thing is that the cron takes the json from promgen (json), be it for rules, target or url. That is what I am putting inside prometheus.yaml. All json. Is there a way to take yaml or a way to translate them?

[kfdm]

Prometheus targets can be in either json or yaml (though Promgen exports only json for targets) Prometheus rules are always yaml