self-hosted icon indicating copy to clipboard operation
self-hosted copied to clipboard

SMTP TLS/SSL not working anymore

Open StefanIGit opened this issue 1 year ago • 27 comments

Environment

self-hosted (https://develop.sentry.dev/self-hosted/)

Steps to Reproduce

upgrade to 24.1.0 from 23.8.0 from sentry/config.yml

mail.backend: 'smtp'
mail.host: mail6.server.tld
mail.password: 'secret'
mail.username: '[email protected]'
mail.from: '[email protected]'
mail.use-ssl: true
mail.port: 587

it worked before the settings are correct since they work fine with Thunderbird

The server requires "STARTTLS" it is try mail.use-ssl: false or mail.use-tls: true or mail.use-ssl: false mail.use-tls: false I get error like SMTPServerDisconnected('Connection unexpectedly closed: timed out') and SMTPNotSupportedError('SMTP AUTH extension not supported by server.')

Expected Result

sending email (invite/pwrest) successfully

Actual Result

errors in log

sentry-self-hosted-worker-1                                        | 10:15:48 [ERROR] celery.app.trace: Task sentry.tasks.email.send_email[6b605357-fb39-4788-a43d-8e68b58d49cf] raised unexpected: SSLError(1, '[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1007)') (data={'hostname': 'celery@a75b7517a419', 'id': '6b605357-fb39-4788-a43d-8e68b58d49cf', 'name': 'sentry.tasks.email.send_email', 'exc': "SSLError(1, '[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1007)')", 'traceback': 'Traceback (most recent call last):\n  File "/usr/local/lib/python3.10/site-packages/celery/app/trace.py", line 477, in trace_task\n    R = retval = fun(*args, **kwargs)\n  File "/usr/local/lib/python3.10/site-packages/sentry_sdk/integrations/celery.py", line 306, in _inner\n    reraise(*exc_info)\n  File "/usr/local/lib/python3.10/site-packages/sentry_sdk/_compat.py", line 115, in reraise\n    raise value\n  File "/usr/local/lib/python3.10/site-packages/sentry_sdk/integrations/celery.py", line 301, in _inner\n    return f(*args, **kwargs)\n  File "/usr/local/lib/python3.10/site-packages/celery/app/trace.py", line 760, in __protected_call__\n    return self.run(*args, **kwargs)\n  File "/usr/local/lib/python3.10/site-packages/sentry/silo/base.py", line 145, in override\n    return original_method(*args, **kwargs)\n  File "/usr/local/lib/python3.10/site-packages/sentry/tasks/base.py", line 117, in _wrapped\n    result = func(*args, **kwargs)\n  File "/usr/local/lib/python3.10/site-packages/sentry/tasks/email.py", line 55, in send_email\n    send_messages([message])\n  File "/usr/local/lib/python3.10/site-packages/sentry/utils/email/send.py", line 17, in send_messages\n    sent = connection.send_messages(messages)\n  File "/usr/local/lib/python3.10/site-packages/django/core/mail/backends/smtp.py", line 102, in send_messages\n    new_conn_created = self.open()\n  File "/usr/local/lib/python3.10/site-packages/django/core/mail/backends/smtp.py", line 62, in open\n    self.connection = self.connection_class(self.host, self.port, **connection_params)\n  File "/usr/local/lib/python3.10/smtplib.py", line 1050, in __init__\n    SMTP.__init__(self, host, port, local_hostname, timeout,\n  File "/usr/local/lib/python3.10/smtplib.py", line 255, in __init__\n    (code, msg) = self.connect(host, port)\n  File "/usr/local/lib/python3.10/smtplib.py", line 341, in connect\n    self.sock = self._get_socket(host, port, self.timeout)\n  File "/usr/local/lib/python3.10/smtplib.py", line 1057, in _get_socket\n    new_socket = self.context.wrap_socket(new_socket,\n  File "/usr/local/lib/python3.10/ssl.py", line 513, in wrap_socket\n    return self.sslsocket_class._create(\n  File "/usr/local/lib/python3.10/ssl.py", line 1104, in _create\n    self.do_handshake()\n  File "/usr/local/lib/python3.10/ssl.py", line 1375, in do_handshake\n    self._sslobj.do_handshake()\nssl.SSLError: [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1007)\n', 'args': '()', 'kwargs': "{'message': <django.core.mail.message.EmailMultiAlternatives object at 0x7fe621961990>, '__start_time': 1706523347.505079}", 'description': 'raised unexpected', 'internal': False})

Product Area

Settings

Link

No response

DSN

No response

Version

24.1.0

StefanIGit avatar Jan 29 '24 10:01 StefanIGit

Assigning to @getsentry/support for routing ⏲️

getsantry[bot] avatar Jan 29 '24 10:01 getsantry[bot]

Is your server using TLS as well? We've also upgraded successfully without encountering this issue, so I don't think this is a general problem.

hubertdeng123 avatar Jan 30 '24 23:01 hubertdeng123

same here:

worker-1                                        | smtplib.SMTPServerDisconnected: Connection unexpectedly closed: timed out

Leask avatar Feb 16 '24 05:02 Leask

There was a breaking change at 23.11.0 that removed a server that was not on the default SMTP path. Is it possible that either of you were using it?

azaslavsky avatar Feb 20 '24 18:02 azaslavsky

We have the same problem since version sentry 24.*. However, it is due to a faulty server certificate. This was not noticed before the changeover to Django Mail Vers. 4.2, as the host entries were not checked. Since the new Django version this is now necessary.

Error: SSL: CERTIFICATE_VERIFY_FAILED This is due to the ssl_context.check_hostname field in the django.core.mail.backends.smtp.py file being set to True by default, starting from Django 4.2.

Unfortunately I have not found a way to work around this using skip ssl validation. In our case it only helps that our mail admin issues a valid server certificate.

chris-laack avatar Feb 28 '24 07:02 chris-laack

@chris-laack Thanks for your input. Does that help @StefanIGit ?

hubertdeng123 avatar Feb 29 '24 23:02 hubertdeng123

I have the same problem on 23.11.2 It worked before upgrade from 23.8.0

AwiOnline avatar Mar 14 '24 16:03 AwiOnline

What is the error message you are seeing @AwiOnline?

hubertdeng123 avatar Mar 15 '24 22:03 hubertdeng123

Here is the error message: "SMTPServerDisconnected('Connection unexpectedly closed: timed out')"

AwiOnline avatar Mar 16 '24 16:03 AwiOnline

Are you also using thunderbird? I am unable to reproduce this and we are using sendgrid

hubertdeng123 avatar Mar 19 '24 20:03 hubertdeng123

Are you also using thunderbird? I am unable to reproduce this and we are using sendgrid

I'm using Gmail as an SMTP relay

AwiOnline avatar Mar 20 '24 17:03 AwiOnline

I'm going to keep this open to see if there is further input from the community, as I'm afraid I'm not able to reproduce this issue.

hubertdeng123 avatar Mar 21 '24 20:03 hubertdeng123

same for me, struggling to set mailgun as SMTP,

mail.host: 'smtp.mailgun.org'
mail.port: 587
mail.username: '[email protected]'
mail.password: 'secret'
mail.use-tls: true
worker-1  |   File "/usr/local/lib/python3.10/site-packages/celery/app/trace.py", line 477, in trace_task
worker-1  |     R = retval = fun(*args, **kwargs)
worker-1  |   File "/usr/local/lib/python3.10/site-packages/sentry_sdk/integrations/celery.py", line 306, in _inner
worker-1  |     reraise(*exc_info)
worker-1  |   File "/usr/local/lib/python3.10/site-packages/sentry_sdk/_compat.py", line 115, in reraise
worker-1  |     raise value
worker-1  |   File "/usr/local/lib/python3.10/site-packages/sentry_sdk/integrations/celery.py", line 301, in _inner
worker-1  |     return f(*args, **kwargs)
worker-1  |   File "/usr/local/lib/python3.10/site-packages/celery/app/trace.py", line 760, in __protected_call__
worker-1  |     return self.run(*args, **kwargs)
worker-1  |   File "/usr/local/lib/python3.10/site-packages/sentry/silo/base.py", line 145, in override
worker-1  |     return original_method(*args, **kwargs)
worker-1  |   File "/usr/local/lib/python3.10/site-packages/sentry/tasks/base.py", line 117, in _wrapped
worker-1  |     result = func(*args, **kwargs)
worker-1  |   File "/usr/local/lib/python3.10/site-packages/sentry/tasks/email.py", line 55, in send_email
worker-1  |     send_messages([message])
worker-1  |   File "/usr/local/lib/python3.10/site-packages/sentry/utils/email/send.py", line 17, in send_messages
worker-1  |     sent = connection.send_messages(messages)
worker-1  |   File "/usr/local/lib/python3.10/site-packages/django/core/mail/backends/smtp.py", line 127, in send_messages
worker-1  |     new_conn_created = self.open()
worker-1  |   File "/usr/local/lib/python3.10/site-packages/django/core/mail/backends/smtp.py", line 85, in open
worker-1  |     self.connection = self.connection_class(
worker-1  |   File "/usr/local/lib/python3.10/smtplib.py", line 255, in __init__
worker-1  |     (code, msg) = self.connect(host, port)
worker-1  |   File "/usr/local/lib/python3.10/smtplib.py", line 343, in connect
worker-1  |     (code, msg) = self.getreply()
worker-1  |   File "/usr/local/lib/python3.10/smtplib.py", line 405, in getreply
worker-1  |     raise SMTPServerDisconnected("Connection unexpectedly closed")
worker-1  | smtplib.SMTPServerDisconnected: Connection unexpectedly closed
worker-1  | 14:43:10 [ERROR] celery.app.trace: Task sentry.tasks.email.send_email[997c31df-908e-4401-92df-b60cfcd3c453] raised unexpected: SMTPServerDisconnected('Connection unexpectedly closed') (data={'hostname': 'celery@58c1d45811e5', 'id': '997c31df-908e-4401-92df-b60cfcd3c453', 'name': 'sentry.tasks.email.send_email', 'exc': "SMTPServerDisconnected('Connection unexpectedly closed')", 'traceback': 'Traceback (most recent call last):\n  File "/usr/local/lib/python3.10/site-packages/celery/app/trace.py", line 477, in trace_task\n    R = retval = fun(*args, **kwargs)\n  File "/usr/local/lib/python3.10/site-packages/sentry_sdk/integrations/celery.py", line 306, in _inner\n    reraise(*exc_info)\n  File "/usr/local/lib/python3.10/site-packages/sentry_sdk/_compat.py", line 115, in reraise\n    raise value\n  File "/usr/local/lib/python3.10/site-packages/sentry_sdk/integrations/celery.py", line 301, in _inner\n    return f(*args, **kwargs)\n  File "/usr/local/lib/python3.10/site-packages/celery/app/trace.py", line 760, in __protected_call__\n    return self.run(*args, **kwargs)\n  File "/usr/local/lib/python3.10/site-packages/sentry/silo/base.py", line 145, in override\n    return original_method(*args, **kwargs)\n  File "/usr/local/lib/python3.10/site-packages/sentry/tasks/base.py", line 117, in _wrapped\n    result = func(*args, **kwargs)\n  File "/usr/local/lib/python3.10/site-packages/sentry/tasks/email.py", line 55, in send_email\n    send_messages([message])\n  File "/usr/local/lib/python3.10/site-packages/sentry/utils/email/send.py", line 17, in send_messages\n    sent = connection.send_messages(messages)\n  File "/usr/local/lib/python3.10/site-packages/django/core/mail/backends/smtp.py", line 127, in send_messages\n    new_conn_created = self.open()\n  File "/usr/local/lib/python3.10/site-packages/django/core/mail/backends/smtp.py", line 85, in open\n    self.connection = self.connection_class(\n  File "/usr/local/lib/python3.10/smtplib.py", line 255, in __init__\n    (code, msg) = self.connect(host, port)\n  File "/usr/local/lib/python3.10/smtplib.py", line 343, in connect\n    (code, msg) = self.getreply()\n  File "/usr/local/lib/python3.10/smtplib.py", line 405, in getreply\n    raise SMTPServerDisconnected("Connection unexpectedly closed")\nsmtplib.SMTPServerDisconnected: Connection unexpectedly closed\n', 'args': '()', 'kwargs': "{'message': <django.core.mail.message.EmailMultiAlternatives object at 0x7fe2f2523820>, '__start_time': 1713969790.239359}", 'description': 'raised unexpected', 'internal': False})

michaelkuty avatar Apr 24 '24 14:04 michaelkuty

@michaelkuty are you seeing the CERITIFICATE_VERIFY_FAILED error as well?

azaslavsky avatar Apr 25 '24 19:04 azaslavsky

@michaelkuty are you seeing the CERITIFICATE_VERIFY_FAILED error as well?

no, I dont see anything else then this error, this is also reason why this is kinda strange because Ive installed like 3 sentry-self hosted last year and without any issue and currently there is no additional debug info just this error and that is all

also I forgot to mention sentry version Sentry 24.1.1

michaelkuty avatar Apr 26 '24 06:04 michaelkuty

I am at sentry 24.4.2.

I am having issues with sendgrid SMTP:

Screenshot 2024-05-08 at 15 21 39 Screenshot 2024-05-08 at 15 18 53 d">
18:18:35 [INFO] sentry.superuser: superuser.request (url='http://localhost/api/0/internal/mail/' method='POST' ip_address='172.18.0.1' user_id=1)
18:18:41 [INFO] sentry.superuser: superuser.request (url='http://localhost/api/0/internal/mail/' method='POST' ip_address='172.18.0.1' user_id=1)
18:18:43 [INFO] sentry.access.api: api.access (method='POST' view='sentry.api.endpoints.relay.project_configs.RelayProjectConfigsEndpoint' response=200 user_id='None' is_app='None' token_type='None' is_frontend_request='False' organization_id='None' auth_id='None' path='/api/0/relays/projectconfigs/' caller_ip='172.18.0.53' user_agent='None' rate_limited='False' rate_limit_category='None' request_duration_seconds=0.015624046325683594 rate_limit_type='DNE' concurrent_limit='None' concurrent_requests='None' reset_time='None' group='None' limit='None' remaining='None')
18:18:53 [INFO] sentry.access.api: api.access (method='POST' view='sentry.api.endpoints.relay.project_configs.RelayProjectConfigsEndpoint' response=200 user_id='None' is_app='None' token_type='None' is_frontend_request='False' organization_id='None' auth_id='None' path='/api/0/relays/projectconfigs/' caller_ip='172.18.0.53' user_agent='None' rate_limited='False' rate_limit_category='None' request_duration_seconds=0.01541447639465332 rate_limit_type='DNE' concurrent_limit='None' concurrent_requests='None' reset_time='None' group='None' limit='None' remaining='None')
18:19:03 [INFO] sentry.access.api: api.access (method='POST' view='sentry.api.endpoints.relay.project_configs.RelayProjectConfigsEndpoint' response=200 user_id='None' is_app='None' token_type='None' is_frontend_request='False' organization_id='None' auth_id='None' path='/api/0/relays/projectconfigs/' caller_ip='172.18.0.53' user_agent='None' rate_limited='False' rate_limit_category='None' request_duration_seconds=0.016051292419433594 rate_limit_type='DNE' concurrent_limit='None' concurrent_requests='None' reset_time='None' group='None' limit='None' remaining='None')
18:19:10 [INFO] sentry.superuser: superuser.request (url='http://localhost/api/0/internal/mail/' method='POST' ip_address='172.18.0.1' user_id=1)
18:19:13 [INFO] sentry.access.api: api.access (method='POST' view='sentry.api.endpoints.relay.project_configs.RelayProjectConfigsEndpoint' response=200 user_id='None' is_app='None' token_type='None' is_frontend_request='False' organization_id='None' auth_id='None' path='/api/0/relays/projectconfigs/' caller_ip='172.18.0.53' user_agent='None' rate_limited='False' rate_limit_category='None' request_duration_seconds=0.015586614608764648 rate_limit_type='DNE' concurrent_limit='None' concurrent_requests='None' reset_time='None' group='None' limit='None' remaining='None')
18:19:23 [INFO] sentry.access.api: api.access (method='POST' view='sentry.api.endpoints.relay.project_configs.RelayProjectConfigsEndpoint' response=200 user_id='None' is_app='None' token_type='None' is_frontend_request='False' organization_id='None' auth_id='None' path='/api/0/relays/projectconfigs/' caller_ip='172.18.0.53' user_agent='None' rate_limited='False' rate_limit_category='None' request_duration_seconds=0.016732454299926758 rate_limit_type='DNE' concurrent_limit='None' concurrent_requests='None' reset_time='None' group='None' limit='None' remaining='None')
^[[B18:19:33 [INFO] sentry.access.api: api.access (method='POST' view='sentry.api.endpoints.relay.project_configs.RelayProjectConfigsEndpoint' response=200 user_id='None' is_app='None' token_type='None' is_frontend_request='False' organization_id='None' auth_id='None' path='/api/0/relays/projectconfigs/' caller_ip='172.18.0.53' user_agent='None' rate_limited='False' rate_limit_category='None' request_duration_seconds=0.016119956970214844 rate_limit_type='DNE' concurrent_limit='None' concurrent_requests='None' reset_time='None' group='None' limit='None' remaining='None')
18:19:43 [INFO] sentry.access.api: api.access (method='POST' view='sentry.api.endpoints.relay.project_configs.RelayProjectConfigsEndpoint' response=200 user_id='None' is_app='None' token_type='None' is_frontend_request='False' organization_id='None' auth_id='None' path='/api/0/relays/projectconfigs/' caller_ip='172.18.0.53' user_agent='None' rate_limited='False' rate_limit_category='None' request_duration_seconds=0.01533365249633789 rate_limit_type='DNE' concurrent_limit='None' concurrent_requests='None' reset_time='None' group='None' limit='None' remaining='None')
18:19:46 [INFO] sentry.access.api: api.access (method='POST' view='sentry.api.endpoints.internal.mail.InternalMailEndpoint' response=500 user_id='1' is_app='False' token_type='None' is_frontend_request='True' organization_id='None' auth_id='None' path='/api/0/internal/mail/' caller_ip='172.18.0.1' user_agent='Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36' rate_limited='False' rate_limit_category='None' request_duration_seconds=120.13616347312927 rate_limit_type='DNE' concurrent_limit='None' concurrent_requests='None' reset_time='None' group='None' limit='None' remaining='None')
18:19:46 [ERROR] django.request: Internal Server Error: /api/0/internal/mail/ (status_code=500 request=<WSGIRequest: POST '/api/0/internal/mail/'>)

lcsvcn avatar May 08 '24 18:05 lcsvcn

btw, the issue is exclusive with STARTTLS, if I use SSL works fine:

Screenshot 2024-05-08 at 15 29 26 Screenshot 2024-05-08 at 15 30 39

lcsvcn avatar May 08 '24 18:05 lcsvcn

same for me, struggling to set mailgun as SMTP,

mail.host: 'smtp.mailgun.org'
mail.port: 587
mail.username: '[email protected]'
mail.password: 'secret'
mail.use-tls: true
worker-1  |   File "/usr/local/lib/python3.10/site-packages/celery/app/trace.py", line 477, in trace_task
worker-1  |     R = retval = fun(*args, **kwargs)
worker-1  |   File "/usr/local/lib/python3.10/site-packages/sentry_sdk/integrations/celery.py", line 306, in _inner
worker-1  |     reraise(*exc_info)
worker-1  |   File "/usr/local/lib/python3.10/site-packages/sentry_sdk/_compat.py", line 115, in reraise
worker-1  |     raise value
worker-1  |   File "/usr/local/lib/python3.10/site-packages/sentry_sdk/integrations/celery.py", line 301, in _inner
worker-1  |     return f(*args, **kwargs)
worker-1  |   File "/usr/local/lib/python3.10/site-packages/celery/app/trace.py", line 760, in __protected_call__
worker-1  |     return self.run(*args, **kwargs)
worker-1  |   File "/usr/local/lib/python3.10/site-packages/sentry/silo/base.py", line 145, in override
worker-1  |     return original_method(*args, **kwargs)
worker-1  |   File "/usr/local/lib/python3.10/site-packages/sentry/tasks/base.py", line 117, in _wrapped
worker-1  |     result = func(*args, **kwargs)
worker-1  |   File "/usr/local/lib/python3.10/site-packages/sentry/tasks/email.py", line 55, in send_email
worker-1  |     send_messages([message])
worker-1  |   File "/usr/local/lib/python3.10/site-packages/sentry/utils/email/send.py", line 17, in send_messages
worker-1  |     sent = connection.send_messages(messages)
worker-1  |   File "/usr/local/lib/python3.10/site-packages/django/core/mail/backends/smtp.py", line 127, in send_messages
worker-1  |     new_conn_created = self.open()
worker-1  |   File "/usr/local/lib/python3.10/site-packages/django/core/mail/backends/smtp.py", line 85, in open
worker-1  |     self.connection = self.connection_class(
worker-1  |   File "/usr/local/lib/python3.10/smtplib.py", line 255, in __init__
worker-1  |     (code, msg) = self.connect(host, port)
worker-1  |   File "/usr/local/lib/python3.10/smtplib.py", line 343, in connect
worker-1  |     (code, msg) = self.getreply()
worker-1  |   File "/usr/local/lib/python3.10/smtplib.py", line 405, in getreply
worker-1  |     raise SMTPServerDisconnected("Connection unexpectedly closed")
worker-1  | smtplib.SMTPServerDisconnected: Connection unexpectedly closed
worker-1  | 14:43:10 [ERROR] celery.app.trace: Task sentry.tasks.email.send_email[997c31df-908e-4401-92df-b60cfcd3c453] raised unexpected: SMTPServerDisconnected('Connection unexpectedly closed') (data={'hostname': 'celery@58c1d45811e5', 'id': '997c31df-908e-4401-92df-b60cfcd3c453', 'name': 'sentry.tasks.email.send_email', 'exc': "SMTPServerDisconnected('Connection unexpectedly closed')", 'traceback': 'Traceback (most recent call last):\n  File "/usr/local/lib/python3.10/site-packages/celery/app/trace.py", line 477, in trace_task\n    R = retval = fun(*args, **kwargs)\n  File "/usr/local/lib/python3.10/site-packages/sentry_sdk/integrations/celery.py", line 306, in _inner\n    reraise(*exc_info)\n  File "/usr/local/lib/python3.10/site-packages/sentry_sdk/_compat.py", line 115, in reraise\n    raise value\n  File "/usr/local/lib/python3.10/site-packages/sentry_sdk/integrations/celery.py", line 301, in _inner\n    return f(*args, **kwargs)\n  File "/usr/local/lib/python3.10/site-packages/celery/app/trace.py", line 760, in __protected_call__\n    return self.run(*args, **kwargs)\n  File "/usr/local/lib/python3.10/site-packages/sentry/silo/base.py", line 145, in override\n    return original_method(*args, **kwargs)\n  File "/usr/local/lib/python3.10/site-packages/sentry/tasks/base.py", line 117, in _wrapped\n    result = func(*args, **kwargs)\n  File "/usr/local/lib/python3.10/site-packages/sentry/tasks/email.py", line 55, in send_email\n    send_messages([message])\n  File "/usr/local/lib/python3.10/site-packages/sentry/utils/email/send.py", line 17, in send_messages\n    sent = connection.send_messages(messages)\n  File "/usr/local/lib/python3.10/site-packages/django/core/mail/backends/smtp.py", line 127, in send_messages\n    new_conn_created = self.open()\n  File "/usr/local/lib/python3.10/site-packages/django/core/mail/backends/smtp.py", line 85, in open\n    self.connection = self.connection_class(\n  File "/usr/local/lib/python3.10/smtplib.py", line 255, in __init__\n    (code, msg) = self.connect(host, port)\n  File "/usr/local/lib/python3.10/smtplib.py", line 343, in connect\n    (code, msg) = self.getreply()\n  File "/usr/local/lib/python3.10/smtplib.py", line 405, in getreply\n    raise SMTPServerDisconnected("Connection unexpectedly closed")\nsmtplib.SMTPServerDisconnected: Connection unexpectedly closed\n', 'args': '()', 'kwargs': "{'message': <django.core.mail.message.EmailMultiAlternatives object at 0x7fe2f2523820>, '__start_time': 1713969790.239359}", 'description': 'raised unexpected', 'internal': False})

Seeing the date of your post, you might be hit by the requirement to configure the IP in the Allow list. Mailgun has introduced this in April.

MrKoopie avatar May 31 '24 19:05 MrKoopie

Hi We have the same issue on 24.5.0.

13:20:36 [ERROR] django.request: Internal Server Error: /api/0/internal/mail/ (status_code=500 request=<WSGIRequest: POST '/api/0/internal/mail/'>)

I can make an SSL handshake on container. Appliction could not complete SSL verification.

yildizozgur avatar Jun 06 '24 13:06 yildizozgur

Is there no other error information? A 500 error alone doesn't give us much to go on, and usually the logs contain more information about the failed SSL exchange.

azaslavsky avatar Jun 07 '24 23:06 azaslavsky

Hi, This is my settings:

  backend: smtp
  useTls: true
  useSsl: false
  host: my.mailserver.com:
  from: xxxx
  port: 25
  username: xxxx
  existingSecret: xxxx

I can make a success handshake on the pod.


I have no name!@sentry-qa-worker-xxxxx:/$ openssl s_client -starttls smtp -connect my.mailserver.com:25
CONNECTED(00000003)
depth=2 C = XX, O = XXXX, OU = xxxx.com, CN = XXXX Root CA 
verify error:num=19:self-signed certificate in certificate chain
verify return:1
.....
verify return:1
.....
verify return:1
......
verify return:1
---
Certificate chain
 0 s:O = xxxxxx...............
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 3756 bytes and written 433 bytes
Verification error: self-signed certificate in certificate chain
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 19 (self-signed certificate in certificate chain)
---
250 HELP

Here is the all worker certificate settings and pod errors:

/$ more /etc/sentry/sentry.conf.py
SENTRY_SDK_CONFIG = {
    'ca_certs': '/etc/pki/ca-trust/custom/ca.crt',
    "release": sentry.__build__,
    "environment": ENVIRONMENT,
    "in_app_include": ["sentry", "sentry_plugins"],
    "debug": True,
    "send_default_pii": True,
    "auto_enabling_integrations": False,
}

$ env | grep BUNDLE
REQUESTS_CA_BUNDLE=/etc/pki/ca-trust/custom/ca.crt
06:30:00 [INFO] sentry.tasks.auto_ongoing_issues: auto_transition_issues_new_to_ongoing started (first_seen_lte=1717396200 first_seen_lte_datetime=datetime.datetime(2024, 6, 3, 6, 30, tzinfo=datetime.timezone.utc))
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 477, in trace_task
    R = retval = fun(*args, **kwargs)
                 ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/integrations/celery/__init__.py", line 346, in _inner
    reraise(*exc_info)
  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/utils.py", line 1640, in reraise
    raise value
  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/integrations/celery/__init__.py", line 341, in _inner
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 760, in __protected_call__
    return self.run(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry/silo/base.py", line 146, in override
    return original_method(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry/tasks/base.py", line 117, in _wrapped
    result = func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry/tasks/email.py", line 54, in send_email
    send_messages([message])
  File "/usr/local/lib/python3.11/site-packages/sentry/utils/email/send.py", line 18, in send_messages
    sent = connection.send_messages(messages)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/mail/backends/smtp.py", line 128, in send_messages
    new_conn_created = self.open()
                       ^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/mail/backends/smtp.py", line 93, in open
    self.connection.starttls(context=self.ssl_context)
  File "/usr/local/lib/python3.11/smtplib.py", line 790, in starttls
    self.sock = context.wrap_socket(self.sock,
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/ssl.py", line 517, in wrap_socket
    return self.sslsocket_class._create(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/ssl.py", line 1104, in _create
    self.do_handshake()
  File "/usr/local/lib/python3.11/ssl.py", line 1382, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1006)
06:31:21 [ERROR] celery.app.trace: Task sentry.tasks.email.send_email[0c3eedcd-5946-4f52-a156-f4faa50c32f8] raised unexpected: SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1006)') (data={'hostname': 'celery@sentry-qa-worker-55ddc48d9d-ftlmp', 'id': '0c3eedcd-5946-4f52-a156-f4faa50c32f8', 'name': 'sentry.tasks.email.send_email', 'exc': "SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1006)')", 'traceback': 'Traceback (most recent call last):\n  File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 477, in trace_task\n    R = retval = fun(*args, **kwargs)\n                 ^^^^^^^^^^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/integrations/celery/__init__.py", line 346, in _inner\n    reraise(*exc_info)\n  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/utils.py", line 1640, in reraise\n    raise value\n  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/integrations/celery/__init__.py", line 341, in _inner\n    return f(*args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 760, in __protected_call__\n    return self.run(*args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/site-packages/sentry/silo/base.py", line 146, in override\n    return original_method(*args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/site-packages/sentry/tasks/base.py", line 117, in _wrapped\n    result = func(*args, **kwargs)\n             ^^^^^^^^^^^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/site-packages/sentry/tasks/email.py", line 54, in send_email\n    send_messages([message])\n  File "/usr/local/lib/python3.11/site-packages/sentry/utils/email/send.py", line 18, in send_messages\n    sent = connection.send_messages(messages)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/site-packages/django/core/mail/backends/smtp.py", line 128, in send_messages\n    new_conn_created = self.open()\n                       ^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/site-packages/django/core/mail/backends/smtp.py", line 93, in open\n    self.connection.starttls(context=self.ssl_context)\n  File "/usr/local/lib/python3.11/smtplib.py", line 790, in starttls\n    self.sock = context.wrap_socket(self.sock,\n                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/ssl.py", line 517, in wrap_socket\n    return self.sslsocket_class._create(\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/ssl.py", line 1104, in _create\n    self.do_handshake()\n  File "/usr/local/lib/python3.11/ssl.py", line 1382, in do_handshake\n    self._sslobj.do_handshake()\nssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1006)\n', 'args': '()', 'kwargs': "{'message': <django.core.mail.message.EmailMultiAlternatives object at 0x7f078855f850>, '__start_time': 1718001080.878142}", 'description': 'raised unexpected', 'internal': False})
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 477, in trace_task
    R = retval = fun(*args, **kwargs)
                 ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/integrations/celery/__init__.py", line 346, in _inner
    reraise(*exc_info)
  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/utils.py", line 1640, in reraise
    raise value
  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/integrations/celery/__init__.py", line 341, in _inner
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 760, in __protected_call__
    return self.run(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry/silo/base.py", line 146, in override
    return original_method(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry/tasks/base.py", line 117, in _wrapped
    result = func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry/tasks/email.py", line 54, in send_email
    send_messages([message])
  File "/usr/local/lib/python3.11/site-packages/sentry/utils/email/send.py", line 18, in send_messages
    sent = connection.send_messages(messages)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/mail/backends/smtp.py", line 128, in send_messages
    new_conn_created = self.open()
                       ^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/mail/backends/smtp.py", line 93, in open
    self.connection.starttls(context=self.ssl_context)
  File "/usr/local/lib/python3.11/smtplib.py", line 790, in starttls
    self.sock = context.wrap_socket(self.sock,
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/ssl.py", line 517, in wrap_socket
    return self.sslsocket_class._create(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/ssl.py", line 1104, in _create
    self.do_handshake()
  File "/usr/local/lib/python3.11/ssl.py", line 1382, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1006)
06:31:21 [ERROR] celery.app.trace: Task sentry.tasks.email.send_email[dd8c1177-6176-446a-820b-2f68dff3c94f] raised unexpected: SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1006)') (data={'hostname': 'celery@sentry-qa-worker-55ddc48d9d-ftlmp', 'id': 'dd8c1177-6176-446a-820b-2f68dff3c94f', 'name': 'sentry.tasks.email.send_email', 'exc': "SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1006)')", 'traceback': 'Traceback (most recent call last):\n  File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 477, in trace_task\n    R = retval = fun(*args, **kwargs)\n                 ^^^^^^^^^^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/integrations/celery/__init__.py", line 346, in _inner\n    reraise(*exc_info)\n  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/utils.py", line 1640, in reraise\n    raise value\n  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/integrations/celery/__init__.py", line 341, in _inner\n    return f(*args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 760, in __protected_call__\n    return self.run(*args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/site-packages/sentry/silo/base.py", line 146, in override\n    return original_method(*args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/site-packages/sentry/tasks/base.py", line 117, in _wrapped\n    result = func(*args, **kwargs)\n             ^^^^^^^^^^^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/site-packages/sentry/tasks/email.py", line 54, in send_email\n    send_messages([message])\n  File "/usr/local/lib/python3.11/site-packages/sentry/utils/email/send.py", line 18, in send_messages\n    sent = connection.send_messages(messages)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/site-packages/django/core/mail/backends/smtp.py", line 128, in send_messages\n    new_conn_created = self.open()\n                       ^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/site-packages/django/core/mail/backends/smtp.py", line 93, in open\n    self.connection.starttls(context=self.ssl_context)\n  File "/usr/local/lib/python3.11/smtplib.py", line 790, in starttls\n    self.sock = context.wrap_socket(self.sock,\n                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/ssl.py", line 517, in wrap_socket\n    return self.sslsocket_class._create(\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/ssl.py", line 1104, in _create\n    self.do_handshake()\n  File "/usr/local/lib/python3.11/ssl.py", line 1382, in do_handshake\n    self._sslobj.do_handshake()\nssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1006)\n', 'args': '()', 'kwargs': "{'message': <django.core.mail.message.EmailMultiAlternatives object at 0x7f07907cb5d0>, '__start_time': 1718001081.443496}", 'description': 'raised unexpected', 'internal': False})
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 477, in trace_task
    R = retval = fun(*args, **kwargs)
                 ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/integrations/celery/__init__.py", line 346, in _inner
    reraise(*exc_info)
  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/utils.py", line 1640, in reraise
    raise value
  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/integrations/celery/__init__.py", line 341, in _inner
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 760, in __protected_call__
    return self.run(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry/silo/base.py", line 146, in override
    return original_method(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry/tasks/base.py", line 117, in _wrapped
    result = func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry/tasks/email.py", line 54, in send_email
    send_messages([message])
  File "/usr/local/lib/python3.11/site-packages/sentry/utils/email/send.py", line 18, in send_messages
    sent = connection.send_messages(messages)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/mail/backends/smtp.py", line 128, in send_messages
    new_conn_created = self.open()
                       ^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/mail/backends/smtp.py", line 93, in open
    self.connection.starttls(context=self.ssl_context)
  File "/usr/local/lib/python3.11/smtplib.py", line 790, in starttls
    self.sock = context.wrap_socket(self.sock,
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/ssl.py", line 517, in wrap_socket
    return self.sslsocket_class._create(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/ssl.py", line 1104, in _create
    self.do_handshake()
  File "/usr/local/lib/python3.11/ssl.py", line 1382, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1006)
06:31:22 [ERROR] celery.app.trace: Task sentry.tasks.email.send_email[b2c63c9b-8536-49ea-be25-5e99672168fb] raised unexpected: SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1006)') (data={'hostname': 'celery@sentry-qa-worker-55ddc48d9d-ftlmp', 'id': 'b2c63c9b-8536-49ea-be25-5e99672168fb', 'name': 'sentry.tasks.email.send_email', 'exc': "SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1006)')", 'traceback': 'Traceback (most recent call last):\n  File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 477, in trace_task\n    R = retval = fun(*args, **kwargs)\n                 ^^^^^^^^^^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/integrations/celery/__init__.py", line 346, in _inner\n    reraise(*exc_info)\n  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/utils.py", line 1640, in reraise\n    raise value\n  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/integrations/celery/__init__.py", line 341, in _inner\n    return f(*args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 760, in __protected_call__\n    return self.run(*args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/site-packages/sentry/silo/base.py", line 146, in override\n    return original_method(*args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/site-packages/sentry/tasks/base.py", line 117, in _wrapped\n    result = func(*args, **kwargs)\n             ^^^^^^^^^^^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/site-packages/sentry/tasks/email.py", line 54, in send_email\n    send_messages([message])\n  File "/usr/local/lib/python3.11/site-packages/sentry/utils/email/send.py", line 18, in send_messages\n    sent = connection.send_messages(messages)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/site-packages/django/core/mail/backends/smtp.py", line 128, in send_messages\n    new_conn_created = self.open()\n                       ^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/site-packages/django/core/mail/backends/smtp.py", line 93, in open\n    self.connection.starttls(context=self.ssl_context)\n  File "/usr/local/lib/python3.11/smtplib.py", line 790, in starttls\n    self.sock = context.wrap_socket(self.sock,\n                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/ssl.py", line 517, in wrap_socket\n    return self.sslsocket_class._create(\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/ssl.py", line 1104, in _create\n    self.do_handshake()\n  File "/usr/local/lib/python3.11/ssl.py", line 1382, in do_handshake\n    self._sslobj.do_handshake()\nssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1006)\n', 'args': '()', 'kwargs': "{'message': <django.core.mail.message.EmailMultiAlternatives object at 0x7f0788594890>, '__start_time': 1718001081.931252}", 'description': 'raised unexpected', 'internal': False})
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 477, in trace_task
    R = retval = fun(*args, **kwargs)
                 ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/integrations/celery/__init__.py", line 346, in _inner
    reraise(*exc_info)
  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/utils.py", line 1640, in reraise
    raise value
  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/integrations/celery/__init__.py", line 341, in _inner
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 760, in __protected_call__
    return self.run(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry/silo/base.py", line 146, in override
    return original_method(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry/tasks/base.py", line 117, in _wrapped
    result = func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry/tasks/email.py", line 54, in send_email
    send_messages([message])
  File "/usr/local/lib/python3.11/site-packages/sentry/utils/email/send.py", line 18, in send_messages
    sent = connection.send_messages(messages)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/mail/backends/smtp.py", line 128, in send_messages
    new_conn_created = self.open()
                       ^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/mail/backends/smtp.py", line 93, in open
    self.connection.starttls(context=self.ssl_context)
  File "/usr/local/lib/python3.11/smtplib.py", line 790, in starttls
    self.sock = context.wrap_socket(self.sock,
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/ssl.py", line 517, in wrap_socket
    return self.sslsocket_class._create(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/ssl.py", line 1104, in _create
    self.do_handshake()
  File "/usr/local/lib/python3.11/ssl.py", line 1382, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1006)
06:31:22 [ERROR] celery.app.trace: Task sentry.tasks.email.send_email[d07059e3-fb47-4176-b158-db1587eadbd8] raised unexpected: SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1006)') (data={'hostname': 'celery@sentry-qa-worker-55ddc48d9d-ftlmp', 'id': 'd07059e3-fb47-4176-b158-db1587eadbd8', 'name': 'sentry.tasks.email.send_email', 'exc': "SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1006)')", 'traceback': 'Traceback (most recent call last):\n  File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 477, in trace_task\n    R = retval = fun(*args, **kwargs)\n                 ^^^^^^^^^^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/integrations/celery/__init__.py", line 346, in _inner\n    reraise(*exc_info)\n  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/utils.py", line 1640, in reraise\n    raise value\n  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/integrations/celery/__init__.py", line 341, in _inner\n    return f(*args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 760, in __protected_call__\n    return self.run(*args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/site-packages/sentry/silo/base.py", line 146, in override\n    return original_method(*args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/site-packages/sentry/tasks/base.py", line 117, in _wrapped\n    result = func(*args, **kwargs)\n             ^^^^^^^^^^^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/site-packages/sentry/tasks/email.py", line 54, in send_email\n    send_messages([message])\n  File "/usr/local/lib/python3.11/site-packages/sentry/utils/email/send.py", line 18, in send_messages\n    sent = connection.send_messages(messages)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/site-packages/django/core/mail/backends/smtp.py", line 128, in send_messages\n    new_conn_created = self.open()\n                       ^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/site-packages/django/core/mail/backends/smtp.py", line 93, in open\n    self.connection.starttls(context=self.ssl_context)\n  File "/usr/local/lib/python3.11/smtplib.py", line 790, in starttls\n    self.sock = context.wrap_socket(self.sock,\n                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/ssl.py", line 517, in wrap_socket\n    return self.sslsocket_class._create(\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File "/usr/local/lib/python3.11/ssl.py", line 1104, in _create\n    self.do_handshake()\n  File "/usr/local/lib/python3.11/ssl.py", line 1382, in do_handshake\n    self._sslobj.do_handshake()\nssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1006)\n', 'args': '()', 'kwargs': "{'message': <django.core.mail.message.EmailMultiAlternatives object at 0x7f078830e110>, '__start_time': 1718001082.394444}", 'description': 'raised unexpected', 'internal': False})

yildizozgur avatar Jun 10 '24 09:06 yildizozgur

@yildizozgur This may be a similar issue, but also not considering you're using sentry-kubernetes, which isn't supported here.

hubertdeng123 avatar Jun 11 '24 22:06 hubertdeng123

Hi @hubertdeng123 , yes We are running on K8s. But the problem is related on Application, it is not related to the environment. In the pod we can do SSL handshake successfully.

yildizozgur avatar Jun 12 '24 21:06 yildizozgur

I am having the same problem without using k8s.

marbon87 avatar Jun 14 '24 14:06 marbon87

Thanks for reporting, I'm going to backlog this item for now.

hubertdeng123 avatar Jun 14 '24 20:06 hubertdeng123

[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1006)

I had the same problem after following the guide in https://develop.sentry.dev/self-hosted/custom-ca-roots/. Please make sure that your certificate has the expected format and file extension. See https://github.com/getsentry/develop/pull/1334

otbutz avatar Jul 15 '24 09:07 otbutz