not able to authenticate in ses

[dauntlessnomad]

What did you do? Configured ses to send alert emails.

What did you expect to see? emails receiving my inbox. What did you see instead? Under which circumstances?

Oct  8 09:51:02 ip-10-64-64-250 alertmanager[34700]: level=error ts=2020-10-08T09:51:02.908Z caller=dispatch.go:309 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="email/email[0]: notify retry canceled after 5 attempts: *smtp.plainAuth auth: 535 Authentication Credentials Invalid"
Oct  8 09:51:02 ip-10-64-64-250 alertmanager[34700]: level=error ts=2020-10-08T09:51:02.922Z caller=dispatch.go:309 component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="email/email[0]: notify retry canceled after 5 attempts: *smtp.plainAuth auth: 535 Authentication Credentials Invalid"
Oct  8 09:51:02 ip-10-64-64-250 alertmanager[34700]: level=error ts=2020-10-08T09:51:02.983Z caller=dispatch.go:309 component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="email/email[0]: notify retry canceled after 5 attempts: *smtp.plainAuth auth: 535 Authentication Credentials Invalid"
Oct  8 09:51:03 ip-10-64-64-250 alertmanager[34700]: level=error ts=2020-10-08T09:51:03.003Z caller=dispatch.go:309 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="email/email[0]: notify retry canceled after 5 attempts: *smtp.plainAuth auth: 535 Authentication Credentials Invalid"
Oct  8 09:51:03 ip-10-64-64-250 alertmanager[34700]: level=warn ts=2020-10-08T09:51:03.756Z caller=notify.go:674 component=dispatcher receiver=email integration=email[0] msg="Notify attempt failed, will retry later" attempts=1 err="*smtp.plainAuth auth: 535 Authentication Credentials Invalid"

Environment
production AWS

• System information: Centos 8. Linux 4.18.0-193.19.1.el8_2.x86_64 x86_64

• Alertmanager version: 0.21.0

• Prometheus version: v2.21.0.1

• Alertmanager configuration file:

global:
  resolve_timeout: 5m

route:
  group_by: ['alertname','instance', 'alert']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 1h
  receiver: 'email'
receivers:
  - name: email
    email_configs:
      - to: '[email protected]'
        from: '[email protected]'
        smarthost: 'email-smtp.us-east-1.amazonaws.com:587'
        auth_username: ''
        auth_password: ''
        require_tls: yes

inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'dev', 'instance']

• Logs:

Oct  8 09:51:02 ip-10-64-64-250 alertmanager[34700]: level=error ts=2020-10-08T09:51:02.908Z caller=dispatch.go:309 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="email/email[0]: notify retry canceled after 5 attempts: *smtp.plainAuth auth: 535 Authentication Credentials Invalid"
Oct  8 09:51:02 ip-10-64-64-250 alertmanager[34700]: level=error ts=2020-10-08T09:51:02.922Z caller=dispatch.go:309 component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="email/email[0]: notify retry canceled after 5 attempts: *smtp.plainAuth auth: 535 Authentication Credentials Invalid"
Oct  8 09:51:02 ip-10-64-64-250 alertmanager[34700]: level=error ts=2020-10-08T09:51:02.983Z caller=dispatch.go:309 component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="email/email[0]: notify retry canceled after 5 attempts: *smtp.plainAuth auth: 535 Authentication Credentials Invalid"
Oct  8 09:51:03 ip-10-64-64-250 alertmanager[34700]: level=error ts=2020-10-08T09:51:03.003Z caller=dispatch.go:309 component=dispatcher msg="Notify for alerts failed" num_alerts=12 err="email/email[0]: notify retry canceled after 5 attempts: *smtp.plainAuth auth: 535 Authentication Credentials Invalid"
Oct  8 09:51:03 ip-10-64-64-250 alertmanager[34700]: level=warn ts=2020-10-08T09:51:03.756Z caller=notify.go:674 component=dispatcher receiver=email integration=email[0] msg="Notify attempt failed, will retry later" attempts=1 err="*smtp.plainAuth auth: 535 Authentication Credentials Invalid" 

[simonpasquier]

535 Authentication Credentials Invalid indicates that the server rejects the authentication. You need to double check the credentials and maybe ask the SES support.

[dionisiemoscalu]

HI @tomahawk-pilot ,

Are you doing it via helm ? Thanks,

[dionisiemoscalu]

@simonpasquier is not related to the SES services, i have same problem

The workaround is to add password throw Alertmanger in Grafana UI (but is not right solution)