helm-charts icon indicating copy to clipboard operation
helm-charts copied to clipboard

Published 20 hours ago • verified publisher icon prometheus-community

[alertmanager] field region not found in type config.plain

Open pilchita opened this issue 2 years ago • 9 comments

Describe the bug a clear and concise description of what the bug is.

Hi, I have installed alertmanager through helm:

i see this error in the kubectl logs:

ts=2023-05-19T14:36:08.949Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml
ts=2023-05-19T14:36:08.950Z caller=coordinator.go:118 level=error component=configuration msg="Loading configuration file failed" file=/etc/alertmanager/alertmanager.yml err="yaml: unmarshal errors:\n  line 3: field groups not found in type config.plain\n  line 18: field region not found in type config.plain\n  line 27: cannot unmarshal !!map into []string"
ts=2023-05-19T14:36:08.950Z caller=cluster.go:690 level=info component=cluster msg="gossip not settled but continuing anyway" polls=0 elapsed=36.344814m

in the archio values is the following configuration:

global:
  route:
    group_wait: 30s
    group_interval: 5m
    repeat_interval: 12h
    receiver: 'amazon-sns'
    routes:
    - matchers:
        group: 'alert-test'
      receiver: 'amazon-sns'
  receivers:
  - name: 'amazon-sns'
    sns_configs:
     - region: 'us-east-1'
       topic_arn: 'arn:aws:sns:us-east-1:xxxxxx:sns'

  groups:
  - name: alert-test
    rules:
    - alert: HpaMaxedOut
      expr: |
        kube_hpa_status_current_replicas{hpa='apache-hpa'} == kube_hpa_status_desired_replicas{hpa='apache-hpa'}
      for: 3m
      labels:
        severity: 'low'
        hpa: apache-hpa
      annotations:
        summary: '{{ $labels.hpa }}'

Any ideas?

What's your helm version?

version.BuildInfo{Version:"v3.12.0", GitCommit:"c9f554d75773799f72ceef38c51210f1842a1dea", GitTreeState:"clean", GoVersion:"go1.20.4"}

What's your kubectl version?

Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.3", GitCommit:"9e644106593f3f4aa98f8a84b23db5fa378900bd", GitTreeState:"clean", BuildDate:"2023-03-15T13:33:11Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"darwin/arm64"} Kustomize Version: v4.5.7 Server Version: version.Info{Major:"1", Minor:"22+", GitVersion:"v1.22.17-eks-48e63af", GitCommit:"47b89ea2caa1f7958bc6539d6865820c86b4bf60", GitTreeState:"clean", BuildDate:"2023-01-24T09:34:06Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}

Which chart?

prometheus-community/alertmanager

What's the chart version?

v0.25.0

What happened?

No response

What you expected to happen?

No response

How to reproduce it?

No response

Enter the changed values of values.yaml?

No response

Enter the command that you execute and failing/misfunctioning.

helm install -f values.yaml prometheus-alertmanager -n prometheus prometheus-community/alertmanager

Anything else we need to know?

No response

pilchita avatar May 19 '23 22:05 pilchita

Some hints:

  1. groups belongs in a Prometheus' rules file, i.e. is a Prometheus' configuration parameter
  2. In routes[].matchers[], a list of label key/value pairs is expected (example), e.g.
matchers:
- alertname="HpaMaxedOut"
  1. sns_configs[] is incorrect as indicated in the error message. Please, refer to its configuration.

Considering the config above, the following would be an example of a syntactically valid alertmanager configuration:

route:
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 12h
  receiver: 'amazon-sns'
  routes:
  - receiver: 'amazon-sns'
    matchers:
    - alertname='HpaMaxedOut'
receivers:
- name: 'amazon-sns'
  sns_configs:
  - topic_arn: 'arn:aws:sns:us-east-1:xxxxxx:sns'
    api_url: https://sns.us-east-1.amazonaws.com
    sigv4:
      region: 'us-east-1'

Validating the configuration with amtool is very useful before rolling it out.

zeritti avatar May 20 '23 11:05 zeritti

Thanks for the clarification, I thought that the groups parametro was used on the side of alertmanager and not on prometheus, I apologize for asking something so silly.

pilchita avatar May 20 '23 13:05 pilchita

I’m back here again.

What is the correct way to update the alertmanager.yaml file?

I try this:

alertmanagerFiles:
  alertmanager.yml:
    alertmanager:
      config:
        global:
          route:
            group_wait: 30s
            group_interval: 5m
            repeat_interval: 12h
            receiver: 'amazon-sns'
          routes:
          - receiver: 'amazon-sns'
            matchers:
            - alertname='HpaMaxedOut'
          receivers:
          - name: 'amazon-sns'
            sns_configs:
            - topic_arn: 'arn:aws:sns:us-east-1:xxxxxxx:xxxxxx'
              api_url: https://global.sns-api.chatbot.amazonaws.com
            sigv4:
              region: 'us-east-1'

I see this error:

│ Error: cannot patch "prometheus-server" with kind StatefulSet: StatefulSet.apps "prometheus-server" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', 'updateStrategy' and 'minReadySeconds' are forbidden

What am I doing wrong?

pilchita avatar May 21 '23 16:05 pilchita

This is very difficult my God, I can’t find the problem.

I see this:

ts=2023-05-22T00:16:24.928Z caller=coordinator.go:118 level=error component=configuration msg="Loading configuration file failed" file=/etc/alertmanager/alertmanager.yml err="yaml: unmarshal errors:\n line 3: field sigv4 not found in type config.plain"

I was all day Sunday trying to solve the problem and I could not, it is very frustrating, see something wrong in the configuration?


replicaCount: 1

image:
  repository: quay.io/prometheus/alertmanager
  pullPolicy: IfNotPresent
  tag: ""

extraArgs: {}

extraSecretMounts: []

imagePullSecrets: []
nameOverride: ""
fullnameOverride: ""

serviceAccount:
  create: true
  annotations: {}
  name:

podSecurityContext:
  fsGroup: 65534
dnsConfig: {}
securityContext:
  runAsUser: 65534
  runAsNonRoot: true
  runAsGroup: 65534

additionalPeers: []

livenessProbe:
  httpGet:
    path: /
    port: http

readinessProbe:
  httpGet:
    path: /
    port: http

service:
  annotations: {}
  type: ClusterIP
  port: 9093
  clusterPort: 9094
  loadBalancerIP: ""  # Assign ext IP when Service type is LoadBalancer
  loadBalancerSourceRanges: []  # Only allow access to loadBalancerIP from these IPs

ingress:
  enabled: false
  className: ""
  annotations: {}
  hosts:
    - host: alertmanager.domain.com
      paths:
        - path: /
          pathType: ImplementationSpecific
  tls: []

resources: {}

nodeSelector: {}

tolerations: []

affinity: {}

podAntiAffinity: ""

podAntiAffinityTopologyKey: kubernetes.io/hostname

topologySpreadConstraints: []

statefulSet:
  annotations: {}

podAnnotations: {}
podLabels: {}

podDisruptionBudget: {}

command: []

persistence:
  enabled: true
  accessModes:
    - ReadWriteOnce
  size: 50Mi

config:
  global:

  route:
    group_wait: 30s
    group_interval: 5m
    repeat_interval: 12h
    receiver: 'amazon-sns'
    routes:
    - receiver: 'amazon-sns'
      matchers:
      - alertname='HpaMaxedOut'
  receivers:
  - name: 'amazon-sns'
    sns_configs:
    - topic_arn: 'arn:aws:sns:us-east-1:xxxxxx:sns'
      api_url: https://sns.us-east-1.amazonaws.com
      sigv4:
        region: 'us-east-1'

  templates:
    - '/etc/alertmanager/*.tmpl'

configmapReload:
  enabled: false

  name: configmap-reload

  image:
    repository: jimmidyson/configmap-reload
    tag: v0.8.0
    pullPolicy: IfNotPresent


  resources: {}

templates: {}

extraVolumeMounts: []

extraVolumes: []

extraEnv: []

pilchita avatar May 22 '23 00:05 pilchita

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

stale[bot] avatar Aug 10 '23 02:08 stale[bot]

I got it after a few hours of guessing, trial and error: Errors in the form field [field] not found in type config.plain mean that the config key [field] is not part of the validation scheme and unknown keys are not supported.

In your case sigv4 has probably been renamed and the docs and examples you're setting this up with are not up-to-date. I can't help you with a concrete fix, because I don't know the module, but I wanted to share the explanation of the error message.

krichter722 avatar Dec 12 '23 14:12 krichter722

my alert manager operator contains this errors:

Provision alertmanager configuration: failed to initialize from secret: yaml: unmarshal errors: line 3: field scrape_interval not found in type config.plain line 4: field time_intervals not found in type config.plain

Version: v0.26.0 Helm chart: helm install my-kube-prometheus-stack prometheus-community/kube-prometheus-stack --version 56.8.2

Vinaum8 avatar Feb 21 '24 17:02 Vinaum8

my alert manager operator contains this errors:

Provision alertmanager configuration: failed to initialize from secret: yaml: unmarshal errors: line 3: field scrape_interval not found in type config.plain line 4: field time_intervals not found in type config.plain

Version: v0.26.0 Helm chart: helm install my-kube-prometheus-stack prometheus-community/kube-prometheus-stack --version 56.8.2

I copied alert manager config gile and put then in chatgpt. This solved my problem, because it alerted me to the correct indentation and removed a directive that did not exist in the alert manager.

see here: https://prometheus.io/docs/alerting/latest/configuration/

Vinaum8 avatar Feb 21 '24 17:02 Vinaum8

Describe the bug a clear and concise description of what the bug is.

Hi, I have installed alertmanager through helm:

i see this error in the kubectl logs:

ts=2023-05-19T14:36:08.949Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml
ts=2023-05-19T14:36:08.950Z caller=coordinator.go:118 level=error component=configuration msg="Loading configuration file failed" file=/etc/alertmanager/alertmanager.yml err="yaml: unmarshal errors:\n  line 3: field groups not found in type config.plain\n  line 18: field region not found in type config.plain\n  line 27: cannot unmarshal !!map into []string"
ts=2023-05-19T14:36:08.950Z caller=cluster.go:690 level=info component=cluster msg="gossip not settled but continuing anyway" polls=0 elapsed=36.344814m

in the archio values is the following configuration:

global:
  route:
    group_wait: 30s
    group_interval: 5m
    repeat_interval: 12h
    receiver: 'amazon-sns'
    routes:
    - matchers:
        group: 'alert-test'
      receiver: 'amazon-sns'
  receivers:
  - name: 'amazon-sns'
    sns_configs:
     - region: 'us-east-1'
       topic_arn: 'arn:aws:sns:us-east-1:xxxxxx:sns'

  groups:
  - name: alert-test
    rules:
    - alert: HpaMaxedOut
      expr: |
        kube_hpa_status_current_replicas{hpa='apache-hpa'} == kube_hpa_status_desired_replicas{hpa='apache-hpa'}
      for: 3m
      labels:
        severity: 'low'
        hpa: apache-hpa
      annotations:
        summary: '{{ $labels.hpa }}'

Any ideas?

What's your helm version?

version.BuildInfo{Version:"v3.12.0", GitCommit:"c9f554d75773799f72ceef38c51210f1842a1dea", GitTreeState:"clean", GoVersion:"go1.20.4"}

What's your kubectl version?

Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.3", GitCommit:"9e644106593f3f4aa98f8a84b23db5fa378900bd", GitTreeState:"clean", BuildDate:"2023-03-15T13:33:11Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"darwin/arm64"} Kustomize Version: v4.5.7 Server Version: version.Info{Major:"1", Minor:"22+", GitVersion:"v1.22.17-eks-48e63af", GitCommit:"47b89ea2caa1f7958bc6539d6865820c86b4bf60", GitTreeState:"clean", BuildDate:"2023-01-24T09:34:06Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}

Which chart?

prometheus-community/alertmanager

What's the chart version?

v0.25.0

What happened?

No response

What you expected to happen?

No response

How to reproduce it?

No response

Enter the changed values of values.yaml?

No response

Enter the command that you execute and failing/misfunctioning.

helm install -f values.yaml prometheus-alertmanager -n prometheus prometheus-community/alertmanager

Anything else we need to know?

No response

https://prometheus.io/docs/alerting/latest/configuration/#sns_config see here:

receivers:
  - name: 'amazon-sns'
    sns_configs:
     - region: 'us-east-1'
       topic_arn: 'arn:aws:sns:us-east-1:xxxxxx:sns'

region not accept here, according to the documentation

Vinaum8 avatar Feb 21 '24 17:02 Vinaum8