Localhost URLs in alert messages

[artm]

Description

Alert messages sent by PMM installed with the helm chart contain broken links with localhost in the URLs instead of the configured domain name. Related config file inside the pmm container:

[root@pmm-0 opt]# cat /etc/supervisord.d/alertmanager.ini 
; Managed by pmm-managed. DO NOT EDIT.

[program:alertmanager]
priority = 8
command =
        /usr/sbin/alertmanager
                --config.file=/etc/alertmanager.yml
                --storage.path=/srv/alertmanager/data
                --data.retention=336h
                --web.external-url=http://localhost:9093/alertmanager/
                --web.listen-address=127.0.0.1:9093
                --cluster.listen-address=""
...

(--web.external-url= seems to contain the wrong domain)

Expected Results

Links that lead to the alert page

Actual Results

Broken links

Version

PMM Helm Cart 1.2.0 PMM Server 2.35.0

Steps to reproduce

• install PMM on kubernetes / docker
• add an alert contact point
• send a test alert

Relevant logs

No response

Code of Conduct

• [X] I agree to follow Percona Community Code of Conduct

[artm]

1. Alertmanager should be using the Public Address set in the Advanced Settings.
2. The helm chart should initialize this setting to the value of ingress.hosts[0].host

[artm]

Now that I compare the --web.external-url=http://localhost:9093/alertmanager/ from /etc/supervisord.d/alertmanager.ini with the links in the alerts (e.g. https://localhost/graph/alerting/list) it seems unrelated after all (the port 9093 is missing as well as the path /alertmanager/)

[artemgavrilov]

Hi @artm! Since version 2.31 PMM uses grafana embedded Alertmanager, not this one that you see in supervisord configuration.

To fix your problem you can add domain option to /etc/grafana/grafana.ini file.

[server]
domain = example.com

We have setting PMM Public Address on the UI, but it doesn't update grafana.ini. I created bug report for that problem: https://jira.percona.com/browse/PMM-11770

[artm]

Thanks for the clarification Artem. Is there a way to fix the /etc/grafana/grafana.ini when using the helm chart or should we wait for the solution of PMM-11770?

Also, is jira a better place than github to report issues in the future?

[artemgavrilov]

Once this bug will be fixed you will be able to configure this parameter with PMM_PUBLIC_ADDRESS environment variable. Right now there is no option to configure it with helm. Also please take a note that for now changes in grafana.ini will not survive PMM Server updates.

Jira is our main issue tracker, so it's preferable. Thank you!