Support global settings and templates in the new Alertmanager's `spec.alertmanagerConfiguration` field

[simonpasquier]

What is missing?

https://github.com/prometheus-operator/prometheus-operator/pull/4220 added the ability to reference an AlertmanagerConfig object to provision the main Alertmanager configuration instead of using a raw secret. But users can't pass custom templates and global settings as defined by https://prometheus.io/docs/alerting/latest/configuration/#configuration-file.

The extended schema should look like:

apiVersion: monitoring.coreos.com/v1
kind: Alertmanager
metadata:
  name: main
spec:
   alertmanagerConfiguration:
     name: main
     global:
       smtpFrom: "" # string
       smtpSmarthost: "" # string
       smtpHello: "" # string
       smtpAuthUsername: "" # string
       smtpAuthIdentity: "" # string
       smtpAuthPassword: # *v1.SecretKeySelector
         key: "smtp_password"
         name: "am-credentials"
       smtpAuthSecret: # *v1.SecretKeySelector
         key: "smtp_secret"
         name: "am-credentials"      
       smtpRequireTls: true # *bool
       slackApiUrl: # *v1.SecretKeySelector
         key: "slack_api"
         name: "am-credentials"      
       victorOpsApiKey: # *v1.SecretKeySelector
         key: "vops_key"
         name: "am-credentials"
       victorOpsApiUrl: "" # string
       pagerDutyUrl: "" # string
       opsGenieApiKey: # *v1.SecretKeySelector
         key: "opsgenie_key"
         name: "am-credentials"
       opsGenieApiUrl: "" # string
       weChatApiSecret: # *v1.SecretKeySelector
         key: "wechat_key"
         name: "am-credentials"
       wechatApiUrl: "" # string
       wechatApiCorpId: "" # string
       resolveTimeout: "" # string (duration)
       httpConfig: {} # we need a new HTTPConfig type, it should be a 1:1 copy of v1alpha1.HTTPConfig
     templates: # list of ConfigMap references (e.g. []v1.ConfigMapKeySelector)
     - key: template1
       name: my-configmap

Why do we need it?

To have feature parity with the Alertmanager configuration format when using the AlertmanagerConfig CRD for the global config.

Environment

• Prometheus Operator version:

  Insert image tag or Git SHA here

Anything else we need to know?:

[simonpasquier]

cc @sunlintong

[sunlintong]

Ok I should be able to implement it in the next few days.

[simonpasquier]

Cool. And no pressure: if you don't have time, we totally understand.

[sunlintong]

   httpConfig: {} # we need a new HTTPConfig type, it should be a 1:1 copy of v1alpha1.HTTPConfig

For the httpConfig, can I move it's defination to monitoringv1 package, which means to update it's version from v1alpha1.HTTPConfig to v1.HTTPConfig.

So that we can refer it as *monitoringv1.HTTPConfig, and won't copy it.

https://github.com/prometheus-operator/prometheus-operator/blob/019631029ca02a0a025ac30a495af1d4ba0ae776/pkg/apis/monitoring/v1alpha1/alertmanager_config_types.go#L224-L227

I have tried, it seems no problem, because almost all of it's sub fields are defined in monitoringv1 package.

[jojotong]

   httpConfig: {} # we need a new HTTPConfig type, it should be a 1:1 copy of v1alpha1.HTTPConfig

For the httpConfig, can I move it's defination to monitoringv1 package, which means to update it's version from v1alpha1.HTTPConfig to v1.HTTPConfig.

So that we can refer it as *monitoringv1.HTTPConfig, and won't copy it.

https://github.com/prometheus-operator/prometheus-operator/blob/019631029ca02a0a025ac30a495af1d4ba0ae776/pkg/apis/monitoring/v1alpha1/alertmanager_config_types.go#L224-L227

I have tried, it seems no problem, because almost all of it's sub fields are defined in monitoringv1 package.

BTW, this is my annother account for @sunlintong, Maybe I can use this account to continue.

[simonpasquier]

For the httpConfig, can I move it's defination to monitoringv1 package, which means to update it's version from v1alpha1.HTTPConfig to v1.HTTPConfig.

I would keep a clear separation between the v1 and v1alpha1 packages for now. Duplicating the struct should be manageable and it should be easy to move everything to v1 later on. I'd also recommend going with small PRs to ease the review (one for templates + maybe 1 per integration settings).

[simonpasquier]

Something we can discuss too is whether we keep the flat structure of the Alertmanager configuration for globals or if we want something different more hierarchical:

apiVersion: monitoring.coreos.com/v1
kind: Alertmanager
metadata:
  name: main
spec:
   alertmanagerConfiguration:
     name: main
     global:
       smtp:
         from: "" # string
         smarthost: "" # string
         hello: "" # string
         authUsername: "" # string
         authIdentity: "" # string
         authPassword: # *v1.SecretKeySelector
           key: "smtp_password"
           name: "am-credentials"
         authSecret: # *v1.SecretKeySelector
           key: "smtp_secret"
           name: "am-credentials"      
         requireTls: true # *bool
       slack:
         apiUrl: # *v1.SecretKeySelector
           key: "slack_api"
           name: "am-credentials"      
...

I don't have a strong opinion. Drifting from the original Alertmanager config format introduces some confusion so I would probably lean towards the first form but happy to hear other opinions :)

[jojotong]

I would keep a clear separation between the v1 and v1alpha1 packages for now

That's also ok,

I don't have a strong opinion. Drifting from the original Alertmanager config format introduces some confusion so I would probably lean towards the first form but happy to hear other opinions :)

The second from is absolutely more beautiful. But IMO, operator data structure keep the same with origin alertmanager is easier to migrate.

I agree with the first ;)

[ap-vishal]

Is there any workaround with which we can use custom templates with the new version? We needed the global config but as soon as we upgraded all our alerts syntax broke.

[jojotong]

Is there any workaround with which we can use custom templates with the new version? We needed the global config but as soon as we upgraded all our alerts syntax broke.

In the latest release, maybe no solution. But I'll support the custom templates after https://github.com/prometheus-operator/prometheus-operator/pull/4622 be merged.

[jojotong]

@simonpasquier https://github.com/prometheus-operator/prometheus-operator/pull/4622 has been merged. I planed to open a new pr to support custom templates first, like below:

apiVersion: monitoring.coreos.com/v1
kind: Alertmanager
metadata:
  name: main
spec:
   alertmanagerConfiguration:
     name: main
     global:
       ...
     templates: # list of ConfigMap references (e.g. []v1.ConfigMapKeySelector)
     - key: template1
       name: my-configmap

What do you think?

[simonpasquier]

@jojotong I think it makes sense. v1.SecretOrConfigMap instead of v1.ConfigMapKeySelector might be a better choice since it provides more flexibility (e.g. you're not supposed to store secret data in templates but :man_shrugging:).

[jojotong]

That's a good idea 😄.

[uqix]

Looking forward to global.smtp/wechat... badly.👍

[rgarcia89]

Where can I find information about how the templates have to look? Seems like they are necessary in order to deploy these global parameters or am I wrong?