Support of sending custom header in Alertmanagers in operatorconfig

[kartik7153]

Hi, We are running our cortex and want to use alertmanager for cortex for routing the alerts. I couldn't see support for adding headers in alertmanagers

[lyanco]

Sorry, I don't understand... GMP has nothing to do with Cortex. What are you trying to accomplish?

[kartik7153]

I am trying to use cortex's alertmanager to push alert to respective channels.

[lyanco]

OK, I'm not familiar with that. Why do you need headers to be able to communicate with it?

[kartik7153]

So cortex's alert-manager needs X-Scope-OrgID to be able to authenticate the request and to be able to figure out the tenant. more on this here.

[kartik7153]

Hi any updates on this?

[pintohutch]

Hi @kartik7153 - IIUC you want the ability to add a custom header to the alert HTTP request going to the configured alertmanager endpoint?

[kartik7153]

Yes right.

[pintohutch]

At this point in time, Prometheus does not support custom HTTP headers in its config. The idea has been discussed from the scrape side, but was decided against.

However, this should be possible through a forwarding proxy in the proxy_url section of alertmanager_config. In fact, someone on that thread suggested a solution using an nginx container that works for scraping. I wonder if the same thing can work from the alerting side.

[pintohutch]

Note - this soon may be possible with https://github.com/prometheus/common/pull/416..

[gviczai]

At this point in time, Prometheus does not support custom HTTP headers in its config. The idea has been discussed from the scrape side, but was decided against.

However, this should be possible through a forwarding proxy in the proxy_url section of alertmanager_config. In fact, someone on that thread suggested a solution using an nginx container that works for scraping. I wonder if the same thing can work from the alerting side.

I have some issues about this "use a proxy" workaround as the "official solution". Also the "decided against" is actually a very short conversation, not a real discussion with arguments and counter-arguments, just an "I don't think this is the responsibility of the scraper." statement. I have a counter-argument however, I think it IS the responsibility of the scraper to set headers required by the exporter. No third-party man-in-the-middle service should be required just for doing that. Details in https://github.com/prometheus/prometheus/issues/13098.