SMTP certificate behaviour changed with 2023.8 forward

[joachimtingvold]

Describe the bug SMTP server configured with TLS fails server validation in 2023.8.*

To Reproduce Steps to reproduce the behavior:

1: Configure SMTP;

AUTHENTIK_EMAIL__HOST=smtp.foo.bar
AUTHENTIK_EMAIL__PORT=587
AUTHENTIK_EMAIL__USE_TLS=true
AUTHENTIK_EMAIL__USE_SSL=false
AUTHENTIK_EMAIL__TIMEOUT=10
AUTHENTIK_EMAIL__FROM='[email protected]'

2. Send mail from Authentik (f.ex. via dc exec authentik-worker ak test_email [email protected])
3. Watch it fail due to SSL validation

Expected behavior Email should be sent without issues.

Screenshots N/A

Logs

{"event": "Error sending email, retrying...", "exc": "SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)')", "level": "debug", "logger": "authentik.stages.email.tasks", "pid": 222, "timestamp": "2023-09-01T03:59:00.872432"}
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/manage.py", line 31, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.11/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.11/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.11/site-packages/django/core/management/base.py", line 412, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.11/site-packages/django/core/management/base.py", line 458, in execute
    output = self.handle(*args, **options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/core/management/base.py", line 106, in wrapper
    res = handle_func(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/authentik/stages/email/management/commands/test_email.py", line 36, in handle
    send_mail(message.__dict__, stage.pk)
  File "/usr/local/lib/python3.11/site-packages/celery/local.py", line 182, in __call__
    return self._get_current_object()(*a, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/celery/app/task.py", line 411, in __call__
    return self.run(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/celery/app/autoretry.py", line 60, in run
    ret = task.retry(exc=exc, **retry_kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/celery/app/task.py", line 720, in retry
    raise_with_context(exc or Retry('Task can be retried', None))
  File "/usr/local/lib/python3.11/site-packages/celery/app/autoretry.py", line 38, in run
    return task._orig_run(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/authentik/stages/email/tasks.py", line 103, in send_mail
    raise exc
  File "/authentik/stages/email/tasks.py", line 73, in send_mail
    backend.open()
  File "/usr/local/lib/python3.11/site-packages/django/core/mail/backends/smtp.py", line 92, in open
    self.connection.starttls(context=self.ssl_context)
  File "/usr/local/lib/python3.11/smtplib.py", line 790, in starttls
    self.sock = context.wrap_socket(self.sock,
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/ssl.py", line 517, in wrap_socket
    return self.sslsocket_class._create(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/ssl.py", line 1108, in _create
    self.do_handshake()
  File "/usr/local/lib/python3.11/ssl.py", line 1379, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)

Version and Deployment (please complete the following information):

• authentik version: 2023.8.1
• Deployment: docker-compose

Additional context

Same configuration works fine in 2023.6.2.

Root CA for the SMTP-server is mounted in /etc/ssl/certs (both when trying 2023.6.2 and 2023.8.1).

[drpetersen]

Tried with the exact same settings,

$ grep -i email authentik.env 
AUTHENTIK_EMAIL__HOST=smtp.******
AUTHENTIK_EMAIL__PORT=587
AUTHENTIK_EMAIL__USERNAME=admin@******
# AUTHENTIK_EMAIL__PASSWORD: Set in .env
AUTHENTIK_EMAIL__USE_TLS=true
AUTHENTIK_EMAIL__USE_SSL=false
AUTHENTIK_EMAIL__TIMEOUT=10
AUTHENTIK_EMAIL__FROM=admin@******

Works for me, with authentik 2023.8.2.

[joachimtingvold]

Is the certificate of the SMTP server you're using self-signed?

[drpetersen]

No, it's from letsencrypt.

[joachimtingvold]

Could be related to self-signed vs. public certificate, then. The SMTP servers I use have self-signed certificates (from internal PKI).

However, it works in 2023.6.2, and not in 2023.8.1, nor 2023.8.2. I only have to change the image parameter between the two to get working/non-working scenarios.

This works:

[…]
services:
  […]
  authentik-server:
    image: ghcr.io/goauthentik/server:2023.6.2
    environment:
      AUTHENTIK_EMAIL__HOST: smtp.foo.bar
      AUTHENTIK_EMAIL__PORT: 587
      AUTHENTIK_EMAIL__USE_TLS: true
      AUTHENTIK_EMAIL__USE_SSL: false
      AUTHENTIK_EMAIL__TIMEOUT: 10
      AUTHENTIK_EMAIL__FROM: 'login.foo.bar <[email protected]>'
    volumes:
      […]
      - /some/path/self_signed_ca.pem:/etc/ssl/certs/foobar_ca.pem:ro
      - /some/path/self_signed_issuing_ca.pem:/etc/ssl/certs/foobar_issuing_ca.pem:ro
    […]

This does not:

[…]
services:
  […]
  authentik-server:
    image: ghcr.io/goauthentik/server:2023.8.2
    environment:
      AUTHENTIK_EMAIL__HOST: smtp.foo.bar
      AUTHENTIK_EMAIL__PORT: 587
      AUTHENTIK_EMAIL__USE_TLS: true
      AUTHENTIK_EMAIL__USE_SSL: false
      AUTHENTIK_EMAIL__TIMEOUT: 10
      AUTHENTIK_EMAIL__FROM: 'login.foo.bar <[email protected]>'
    volumes:
      […]
      - /some/path/self_signed_ca.pem:/etc/ssl/certs/foobar_ca.pem:ro
      - /some/path/self_signed_issuing_ca.pem:/etc/ssl/certs/foobar_issuing_ca.pem:ro
    […]

[drpetersen]

Ok, I see. At first glance, I'd check for a permissions issue (although that would not explain why it used to work with 2023.6.2). But I assume you did that already?

docker exec authentik-worker md5sum /etc/ssl/certs/foobar*.pem
docker exec authentik-worker ls -l /etc/ssl/certs/foobar*.pem
docker exec authentik-worker id

Can the authentik user access your cert files?

I am also mounting custom certificates into the authentik-worker container, and in order to grant access, I assigned my authentik user to the ssl-cert group (put user 1000:114 in the authentik-worker service in my compose file, where 114 is the numeric id of my ssl-cert group) and grant that group access to the cert files

chgrp ssl-cert -R /etc/ssl/certs
chmod g+rX -R /etc/ssl/certs

Sorry if this is too obvious, it's the only idea I have right now.

[joachimtingvold]

Both of the certificates are world-readable, but I just double checked that the authentik user indeed can read them;

root@docker1:~# docker exec authentik-worker runuser -u authentik -- md5sum /etc/ssl/certs/foobar_ca.pem
ed1[…]3cb  /etc/ssl/certs/foobar_ca.pem

root@docker1:~# docker exec authentik-worker runuser -u authentik -- md5sum /etc/ssl/certs/foobar_issuing_ca.pem
7a2[…]bc0  /etc/ssl/certs/foobar_issuing_ca.pem

[joachimtingvold]

Authentik uses stages/email/tasks.py, which again uses django/core/mail/backends/smtp.py. Since Authentik doesn't pass the ssl_certfile parameter, ssl.create_default_context() is used (as per this code). According to the documentation, this triggers SSLContext.load_default_certs(), which again triggers SSLContext.set_default_verify_paths(). The latter method is allowed to softfail, which might or might not happen in my scenario. And it's also somewhat unclear to me what paths/folders its actually attempting to use (i.e. if it's limited to only checking /etc/ssl/certs/ca-certificates.crt, that is most likely the culprit, as /etc/ssl/certs/ca-certificates.crt is not automatically updated with new certificates).

I did not require to mount these CA files in 2023.6.2 in order for SMTP TLS to work (even if the servers have self-signed certificates), so not sure why that even worked in 2023.6.2... I'm pretty sure this problem is caused by updating the underlying Debian image (to 12/Bookworm), which I assume happened in 2023.8.*.

Simply adding CA files to /usr/share/ca-certificates/ and/or /etc/ssl/certs/ isn't enough. Even running update-ca-certificates after adding the CA files is not enough (you have to explicitly define the certificate relative path under /usr/share/ca-certificates/ in /etc/ca-certificates.conf, and then run update-ca-certificates).

Doing a docker compose build specification, I was able to update the system CA store, which I can confirm resolves the issue;

docker-compose.yml:

services:
  […]
  authentik-server:
    […]
    image: ghcr.io/goauthentik/server:latest
    build: ./build/authentik
    […]
  authentik-worker:
    […]
    image: ghcr.io/goauthentik/server:latest
    build: ./build/authentik
    […]

./build/authentik/Dockerfile:

FROM ghcr.io/goauthentik/server:latest
USER root
RUN mkdir -p /usr/share/ca-certificates/foobar
COPY foobar_ca.pem /usr/share/ca-certificates/foobar/
COPY foobar_issuing_ca.pem /usr/share/ca-certificates/foobar/
RUN echo 'foobar/foobar_ca.crt' >> /etc/ca-certificates.conf
RUN echo 'foobar/foobar_issuing_ca.crt' >> /etc/ca-certificates.conf
RUN update-ca-certificates

It's a bit cumbersome to do custom build to resolve this. Maybe Authentik should be able to set the ssl_keyfile and ssl_certfile parameters when invoking the Django mail module (django/core/mail/backends/smtp.py), so that we are able to pass custom CA-chains/certificates to the underlying SSL module... like AUTHENTIK_EMAIL__SSL_CERTFILE and AUTHENTIK_EMAIL__SSL_KEYFILE or similar.

[authentik-automation[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[joachimtingvold]

The issue is still present. Something changed in 2023.8.* that made this happen (probably Debian version change). I guess it's technically not a bug, but something that worked pre-2023.8.* is no longer working. I guess this issue can be rephrased to implement support for setting ssl_keyfile and ssl_certfile via the Authentik configuration file.

[simevo]

this might be relevant: https://stackoverflow.com/a/78474038

[simevo]

I have a draft here, it is untested, very rough and on top of version/2024.8.3.

If it has a chance to be accepted as MR, I am willing to polish it and make it ready for submission!

[lucpfi]

What is here the current status? Because I'm also having the same problem with Version 2024.10.2. Or does anyone has a good Workaround for a containerized environment.

@simevo ?

[joachimtingvold]

I'm still using the Docker compose build specification as a workaround, which works fine. But I agree that a fix should probably be implemented. The draft from @simevo seems to do the trick, not sure what's missing to get that incorporated, or if the devs want a different approach.

[rickyelopez]

I am also now hitting this issue. Are there any workarounds within authentik? (i.e. that don't involve another dockerfile and local build?) Any chance a solution like @simevo's could be implemented?

[alhazmy13]

Still the issue exist with latest version

[tziuhtli]

I am having the exact same problem. I also have a local root CA that creates certificates for all my other containers, including Authentik. I normally just install all my root CA on my docker host and mount the whole trust store to the docker container so they all trust each other.

    volumes:
      - /etc/ssl/certs/ca-certificates.crt:/etc/ssl/certs/ca-certificates.crt:ro

Unfortunately, even with this, Authentik does not trust the local email server.

@joachimtingvold Thanks for pointing out all the files where this error was coming from. I replicated the error message, although not sure if this is the exact same way in which Authentik gets it. As far as I see the ssl_context method has None for ssl_certfile and ssl_keyfile. I still tried both clauses of the if statement.

Some Python Tests

For all tests below, assume the SSL certificate of mailserver.local is trusted by the local trust store mounted at /etc/ssl/certs/ca-certificates.crt.

No error here [else clause]

import ssl
import socket

address = "mailserver.local"
context = ssl.create_default_context()
conn = context.wrap_socket(socket.socket(socket.AF_INET),server_hostname=address)
conn.connect((address, 443))
cert = conn.getpeercert()
print(cert)

works just as expected :)

Error here [if clause]

since we are not giving a specific ssl cert. I created dummy ones in case Authentik is somehow entering this clause. This failing implies the local trust store is not being loaded.

mkdir /ssl && cd /ssl
openssl genpkey -algorithm RSA -out private.key -aes256
openssl req -new -key private.key -out certificate.csr
openssl x509 -req -days 365 -in certificate.csr -signkey private.key -out certificate.crt

import ssl
import socket

address = "mailserver.local"
ssl_certfile = "/ssl/certificate.crt"
ssl_keyfile = "/ssl/private.key"
context = ssl.SSLContext(protocol=ssl.PROTOCOL_TLS_CLIENT)
context.load_cert_chain(ssl_certfile, ssl_keyfile)
conn = context.wrap_socket(socket.socket(socket.AF_INET),server_hostname=address)
conn.connect((address, 443))
# ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1010)

Still puzzled a bit, since following the code seems to indicate that ssl_certfile and ssl_keyfile are both None. If it is helpful for anyone, this is the commit where the method ssl_context was added, and this is where the else clause was added.

No error here (fixed?).

Manually load the local trust store in "if clause" too! using set_default_verify_paths()

import ssl
import socket

address = "mailserver.local"
ssl_certfile = "/ssl/certificate.crt"
ssl_keyfile = "/ssl/private.key"
context = ssl.SSLContext(protocol=ssl.PROTOCOL_TLS_CLIENT)
context.load_cert_chain(ssl_certfile, ssl_keyfile)
context.set_default_verify_paths() # trust local trust store
conn = context.wrap_socket(socket.socket(socket.AF_INET),server_hostname=address)
conn.connect((address, 443))
cert = conn.getpeercert()
print(cert)

[tziuhtli]

please ignore my previous post... I just found out how to fix the issue!

Here, deep in the docs for EMAIL_SSL_CERTFILE

... In such cases, the server's certificate (or the root certificate of the private CA) should be installed into the system's CA bundle. This can be done by following platform-specific instructions for installing a root CA certificate, or by using OpenSSL's SSL_CERT_FILE or SSL_CERT_DIR environment variables to specify a custom certificate bundle (if modifying the system bundle is not possible or desired).

Here is my working solution now! I have my local CA installed in my docker host. So appart from passing it to the container's default trust, we need to instruct OpenSSL to use it too using SSL_CERT_FILE according to the quote above. But other solutions should apply if using SSL_CERT_DIR.

    volumes:
      - /etc/ssl/certs/ca-certificates.crt:/etc/ssl/certs/ca-certificates.crt:ro
    environment:
      - SSL_CERT_FILE="/etc/ssl/certs/ca-certificates.crt"

I have now successfully sent a test email.

[dewi-tik]

Now documented in #16727. Thanks @tziuhtli