SSL error during "sentry cleanup"

[gbit-is]

Environment

self-hosted (https://develop.sentry.dev/self-hosted/)

Steps to Reproduce

On a self hosted sentry setup (docker), when running the cleanup there is an SSL error in the end, which doesn't really seem to affect anything in the cleanup itself? but it looks pretty bad in the shell and I guess it might leave orphaned tasks if tasks are registered server side (I don't have a deep enough understanding of sentry to know if this is the case, if tasks like this are centrally managed or not)

If I understand the code correctly (gave it a quick glance) this is the last step in the cleanup process, just closing up the sentry_sdk transaction class, I don't realise if this leaves a zombie transaction on the

Steps to reproduce:

1. Have a self hosted sentry setup in docker, with an external reverse proxy for port 80/443 with a custom CA (trusted by the host server and docker container) ( version: 24.1.2 )
2. run "docker exec -it sentry cleanup --days 99999" (days 9999 just to save time, skips most of the actual cleanup)
3. wait for failure in the end

I did some digging here and

Error is triggered after the try/finally section in : /usr/local/lib/python3.11/site-packages/sentry/runner/commands/cleanup.py

if transaction:
    transaction.__exit__(None, None, None)

if I, inside the container create "filename.py" and import requests, and do a simple unauthenticated post request to the URL, I do not get an SSL error, so the cert of the site is trusted by default by python

Any attempts to use enviromment variables to ignore SSL errors or link to CA bundles including this certificate before running the cleanup script do not change the results of this

I haven't gone down the rabbit hole of checking why sentry_sdk doesn't trust the cert

Expected Result

No error ?

Actual Result

Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 468, in _make_request self._validate_conn(conn) File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 1097, in _validate_conn conn.connect() File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 642, in connect sock_and_verified = _ssl_wrap_socket_and_match_hostname( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 783, in ssl_wrap_socket_and_match_hostname ssl_sock = ssl_wrap_socket( ^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/urllib3/util/ssl.py", line 471, in ssl_wrap_socket ssl_sock = ssl_wrap_socket_impl(sock, context, tls_in_tls, server_hostname) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/urllib3/util/ssl.py", line 515, in _ssl_wrap_socket_impl return ssl_context.wrap_socket(sock, server_hostname=server_hostname) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/ssl.py", line 517, in wrap_socket return self.sslsocket_class._create( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/ssl.py", line 1108, in _create self.do_handshake() File "/usr/local/lib/python3.11/ssl.py", line 1379, in do_handshake self._sslobj.do_handshake() ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 791, in urlopen response = self._make_request( ^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 492, in _make_request raise new_e urllib3.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/sentry_sdk/transport.py", line 535, in send_envelope_wrapper self._send_envelope(envelope) File "/usr/local/lib/python3.11/site-packages/sentry_sdk/transport.py", line 434, in _send_envelope self._send_request( File "/usr/local/lib/python3.11/site-packages/sentry/utils/sdk.py", line 241, in patched_send_request return _send_request(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/sentry_sdk/transport.py", line 245, in _send_request response = self._pool.request( ^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/urllib3/_request_methods.py", line 118, in request return self.request_encode_body( ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/urllib3/_request_methods.py", line 217, in request_encode_body return self.urlopen(method, url, **extra_kw) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/urllib3/poolmanager.py", line 443, in urlopen response = conn.urlopen(method, u.request_uri, **kw) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 875, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 875, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 875, in urlopen return self.urlopen( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 845, in urlopen retries = retries.increment( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/urllib3/util/retry.py", line 515, in increment raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='REDACTED', port=443): Max retries exceeded with url: /api/1/envelope/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)'))) 08:45:44 [ERROR] sentry_sdk.errors: Internal error in sentry_sdk

Product Area

Unknown

Link

No response

DSN

No response

Version

24.1.2

[getsantry[bot]]

Assigning to @getsentry/support for routing ⏲️

[souredoutlook]

Transferring to self-hosted o7

[azaslavsky]

It seems like the SSL certificates on the machine you are running self-hosted on may be out of date? Have you tried updating those?

[azaslavsky]

Just tried this on our local dogfood instance, and it seemed to work fine. What OS and docker setup are you using?

[csvan]

Getting the same error during cleanup here:

 [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate

I am reasonably sure our certs are correct though, ran this tool without finding any obvious problems:

https://github.com/drwetter/testssl.sh

We also have no problems making SSL connections from other tools on the same server.

[hubertdeng123]

Thanks for chiming in. I do not think this should affect how the cleanup job works, but it is a bug nonetheless.

[gbit-is]

I did some further digging and the issue seems to be that in

"/usr/local/lib/python3.11/site-packages/sentry_sdk/transport.py", line 203

` self._pool = self._make_pool( self.parsed_dsn, http_proxy=options["http_proxy"], https_proxy=options["https_proxy"], ca_certs=options["ca_certs"], proxy_headers=options["proxy_headers"],

    )`

options["ca_certs"] resolves to null

by adding a line above, this options["ca_certs"] = "/etc/ssl/certs/ca-certificates.crt"

This does not result in an error

[atocyo]

I have similar problem, where cleanup fails because it tries to connect to 443 port, but ssl is provided by load balancer, it should use internal prefix / port 80.

So this is bug where sentry_sdk should use 'system.internal-url-prefix' but is 'using system.url-prefix' instead.

` Cleaning up unused FileBlob references File Blobs: 100% |#########################################################################################################################| Time: 0:00:00 Received STOP_WORKER task

       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.11/site-packages/urllib3/util/connection.py", line 85, in create_connection raise err File "/usr/local/lib/python3.11/site-packages/urllib3/util/connection.py", line 73, in create_connection sock.connect(sa) ConnectionRefusedError: [Errno 111] Connection refused

The above exception was the direct cause of the following exception: ..........

........ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host=***********', port=443): Max retries exceeded with url: /api/1/envelope/ (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fb7eb58d010>: Failed to establish a new connection: [Errno 111] Connection refused')) 07:35:43 [ERROR] sentry_sdk.errors: Internal error in sentry_sdk

`

[hubertdeng123]

Thanks for chiming in. I’m going to backlog this for now

[lava83]

I have similar problem, where cleanup fails because it tries to connect to 443 port, but ssl is provided by load balancer, it should use internal prefix / port 80.

So this is bug where sentry_sdk should use 'system.internal-url-prefix' but is 'using system.url-prefix' instead.

` Cleaning up unused FileBlob references File Blobs: 100% |#########################################################################################################################| Time: 0:00:00 Received STOP_WORKER task

       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.11/site-packages/urllib3/util/connection.py", line 85, in create_connection raise err File "/usr/local/lib/python3.11/site-packages/urllib3/util/connection.py", line 73, in create_connection sock.connect(sa) ConnectionRefusedError: [Errno 111] Connection refused

The above exception was the direct cause of the following exception: ..........

........ urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host=***********', port=443): Max retries exceeded with url: /api/1/envelope/ (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fb7eb58d010>: Failed to establish a new connection: [Errno 111] Connection refused')) 07:35:43 [ERROR] sentry_sdk.errors: Internal error in sentry_sdk

`

Same error here.

[malsssh]

Same error here

Max retries exceeded with url: /api/1/envelope/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)'))) 14:13:14 [ERROR] sentry_sdk.errors: Internal error in sentry_sdk

tried with env variable REQUESTS_CA_BUNDLE=/certs.crt - without success