Issue on Alerting WebHook since 0.24.0 alertmanager version

[sswieton]

What did you do?

I noticed a issue or change of behavior on the latest Alertmanager tag: 0.24.0 on the Receiver Webhook

We are using webhook receiver with a proxy configuration. This configuration was working fine with Alertmanager 0.23.0 We've upgraded to 0.24.0 and got this issue :

level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="webhook/webhook[0]: notify retry canceled after 2 attempts: Post "https://URL/": Post "https://TOKEN": dial tcp 10.X.X.X:443: connect: connection timed out

Do we need to update the webhook configuration ?

Thanks in advance regards

  - name: webhook
    webhook_configs:
    - send_resolved: true
      url: "https://URL"
      max_alerts: 4
      http_config:
        proxy_url: "http://PROXY"
        tls_config:
          insecure_skip_verify: true
        oauth2:
          client_id: CLIENT_ID
          client_secret: CLIENT_SECRET
          token_url: https://TOKEN_URL

What did you expect to see?

• Alertmanager version: 0.24.0

• Prometheus version: 2.35.0

[roidelapluie]

Could you try adding enable_http2: false ?

On 16 May 02:39, sswieton wrote:

What did you do?

I noticed a issue or change of behavior on the latest Alertmanager tag: 0.24.0 on the Receiver Webhook

We are using webhook receiver with a proxy configuration. This configuration was working fine with Alertmanager 0.23.0 We've upgraded to 0.24.0 and got this issue :

level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="webhook/webhook[0]: notify retry canceled after 2 attempts: Post "https://URL/": Post "https://TOKEN": dial tcp 10.X.X.X:443: connect: connection timed out

Do we need to update the webhook configuration ?

Thanks in advance regards

  - name: webhook
    webhook_configs:
    - send_resolved: true
      url: "https://URL"
      max_alerts: 4
      http_config:
        proxy_url: "http://PROXY"
        tls_config:
          insecure_skip_verify: true
        oauth2:
          client_id: CLIENT_ID
          client_secret: CLIENT_SECRET
          token_url: https://TOKEN_URL

What did you expect to see?

• Alertmanager version: 0.24.0

• Prometheus version: 2.35.0

-- Reply to this email directly or view it on GitHub: https://github.com/prometheus/alertmanager/issues/2924 You are receiving this because you are subscribed to this thread.

Message ID: @.***>

-- Julien Pivotto O11y - https://o11y.eu/

[sswieton]

Hi,

The param "enable_http2" is defined in the scraping jobs, so, updating it could have a large impact on the solution... My concern is just limited to the WebHook receiver.

is there an other way to set this param ? in the receiver for ex ? thanks

Whether to enable HTTP2.

[ enable_http2: | default: true ]

[roidelapluie]

This should be settable in the webhook receiver, that's what I am talking about

[sswieton]

ok. this is not documented in Alertmanager receiver spec. I guess this is the correct schema to use Thanks regards

 - name: webhook
    webhook_configs:
    - send_resolved: true
      enable_http2: false
      url: "https://URL"
      http_config:
        proxy_url: "http://PROXY"
        tls_config:
          insecure_skip_verify: true
        oauth2:
          client_id: CLIENT_ID
          client_secret: CLIENT_SECRET
          token_url: https://TOKEN_URL

[roidelapluie]

mmh indeed we would need to first update the alertmanager, it's not supported yet.

So you could try setting the env variable in the alertmanager:

PROMETHEUS_COMMON_DISABLE_HTTP2=x

[sswieton]

We deploy in Production, so, not possible to integrate this fix. Back to Alertmanager 0.23.0 waiting for the fix regarding this issue

Thanks