robusta icon indicating copy to clipboard operation
robusta copied to clipboard
verified publisher icon robusta-dev

SSL: CERTIFICATE_VERIFY_FAILED certificate verify failed: unable to get issuer certificate

Open styladj1 opened this issue 3 months ago • 6 comments

Describe the bug "SSL: CERTIFICATE_VERIFY_FAILED certificate verify failed: unable to get issuer certificate" when connecting to a Mattermost instance using a certificate signed by an internal company CA.

To Reproduce Steps to reproduce the behavior:

  1. Install Robusta Helm chart
  2. Create a bundle containing internal CAs - in my setup trust-manager does that for me
  3. Set additional_env_vars in the runner section in generated_values.yaml:
- name: REQUESTS_CA_BUNDLE
  value: /etc/ssl/certs/company-root-ca-bundle.crt
  1. Mount the ConfigMap with the custom ca bundle
  2. Re-deploy Robusta with updated values

Expected behavior SSL verification should succeed.

Logs Logs are attached. github_issue_robusta_ssl_verification_failed.txt

Desktop (please complete the following information):

  • OS: Debian 12
  • Robusta Helm chart version: 0.28.1
  • Kubernetes version: 1.30.14

Additional context Did some try and error and found out that SSL verification succeeds with tools like curl in the robusta-runner pods when the bundle with internal CAs is mounted. I went even one step further and created a bundle consisting of Python's certifi CAs and our internal CAs, but Robusta still fails to verify Mattermost's SSL certificate.

Please provide some guidance how to fix this issue. Thanks.

Br,

Matthias

styladj1 avatar Oct 08 '25 08:10 styladj1

Hi 👋, thanks for opening an issue! Please note, it may take some time for us to respond, but we'll get back to you as soon as we can!

  • 💬 Slack Community: Join Robusta team and other contributors on Slack here.
  • 📖 Docs: Find our documentation here.
  • 🎥 YouTube Channel: Watch our videos here.

github-actions[bot] avatar Oct 08 '25 08:10 github-actions[bot]

Thanks for reporting this @styladj1

Can you try adding the ca-bundle, b64 encoded using an env var named CERTIFICATE ?

Similar to what's described here: https://docs.robusta.dev/master/configuration/metric-providers-in-cluster.html#ssl-verification

Does that help?

arikalon1 avatar Oct 08 '25 08:10 arikalon1

Thanks for reporting this @styladj1

Can you try adding the ca-bundle, b64 encoded using an env var named CERTIFICATE ?

Similar to what's described here: https://docs.robusta.dev/master/configuration/metric-providers-in-cluster.html#ssl-verification

Does that help?

I tried it already before (sorry, I forgot to mention it above), but it did not work.

For quite some days I played around using a combined ca bundle (consisting of python certifi + out internal CAs). But this does not work at all.

However, I was able to fix the issue with the connection to Mattermost now.

  • regenerated the ca bundle with our internal CAs using trust-manager
  • recreated the configmap
  • redeployed Robusta

robusta-runner is able to use the bundle and connects to Mattermost properly.

I still see SSL: CERTIFICATE VERIFY FAILED errors in the runner logs, but these are related to the telemetry updates. I don't care about them, because I do not want to send telemetry data anyway. They are likely occurring due to the fact that I set REQUESTS_CA_BUNDLE as an environment variable.

2025-10-08 10:26:11.415 ERROR    Failed to run periodic telemetry update HTTPSConnectionPool(host='api.robusta.dev', port=443): Max retries exceeded with url: /telemetry (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1016)')))
Traceback (most recent call last):
  File "/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 716, in urlopen
    httplib_response = self._make_request(
                       ^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 404, in _make_request
    self._validate_conn(conn)
  File "/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 1061, in _validate_conn
    conn.connect()
  File "/venv/lib/python3.11/site-packages/urllib3/connection.py", line 419, in connect
    self.sock = ssl_wrap_socket(
                ^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.11/site-packages/urllib3/util/ssl_.py", line 458, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(
               ^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.11/site-packages/urllib3/util/ssl_.py", line 502, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/ssl.py", line 517, in wrap_socket
    return self.sslsocket_class._create(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/ssl.py", line 1104, in _create
    self.do_handshake()
  File "/usr/local/lib/python3.11/ssl.py", line 1382, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1016)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/venv/lib/python3.11/site-packages/requests/adapters.py", line 644, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 802, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.11/site-packages/urllib3/util/retry.py", line 594, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='api.robusta.dev', port=443): Max retries exceeded with url: /telemetry (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1016)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/src/robusta/runner/telemetry_service.py", line 63, in __log_periodic
    self.__log(tele)
  File "/app/src/robusta/runner/telemetry_service.py", line 72, in __log
    r = requests.post(self.endpoint, data=data.json(), headers={"Content-Type": "application/json"})
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.11/site-packages/requests/api.py", line 115, in post
    return request("post", url, data=data, json=json, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.11/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.11/site-packages/requests/adapters.py", line 675, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='api.robusta.dev', port=443): Max retries exceeded with url: /telemetry (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1016)')))

Is there a way to disable telemetry stuff in Robusta?

styladj1 avatar Oct 08 '25 10:10 styladj1

If you're not using the Robusta UI, you can disable telemetry by adding an env var: ENABLE_TELEMETRY: "false"

arikalon1 avatar Oct 08 '25 10:10 arikalon1

@styladj1 anything we can fix in the docs to make it more clear to the next person who needs to do this?

aantn avatar Oct 08 '25 10:10 aantn

@arikalon1 Thank you for the hint. Logs state now INFO Telemetry is disabled.

@aantn You could probably add a note in the installation section stating what needs to be done.

Something like:

If you plan to integrate Robusta with a self-hosted collaboration platform like Slack or Mattermost using certificates signed by an internal Certificate Authority (CA), additional steps are required to ensure proper SSL trust. You will need to distribute and install your internal CA certificates within your Kubernetes cluster and the Robusta pods. This typically involves creating a combined CA bundle that includes your internal CAs alongside system and trusted CAs, and configuring the environment so that Robusta trusts this bundle. Without it, SSL verification may fail when connecting to your collaboration platform endpoints.

styladj1 avatar Oct 08 '25 12:10 styladj1