docker-mailserver-helm icon indicating copy to clipboard operation
docker-mailserver-helm copied to clipboard

Existing certificate from cert-manager is not being mounted

Open manyjuice opened this issue 5 months ago • 7 comments

I'm testing a minimal setup according to the guide on top of K3S. The mail server is available, but it uses the self-signed docker-mailserver tls.crt, tls.key instead of the one from cert-manager. It's possible that the first time I used the chart the cert manager's certificate wasn't issued yet, but the current deployment is made after issuing one successfully

kubectl get certificates -n mail -> LE cert from certmanager is available and valid in values.yaml I have

proxyProtocol:
  enabled: false

service:
  type: LoadBalancer
  externalTrafficPolicy: Local

certificate: mail-tls-certificate-rsa

deployment:
  env:
    OVERRIDE_HOSTNAME: mail.example.me
    ENABLE_UPDATE_CHECK: 1
    UPDATE_CHECK_INTERVAL: 1w
    ENABLE_CLAMAV: 1
    ENABLE_FAIL2BAN: 1
    POSTFIX_MESSAGE_SIZE_LIMIT: 102400000

helm upgrade doesn't help, I also tried redeploying it from scratch, to search for extra secrets, but I see nothing. Is there a way to mount it properly withouw workarounds like manual directory mounts? Now I'm gettings Verify return code: 18 (self-signed certificate) during the clients authorization even with the self-signed cert whitelisted both by STARTTLS and TLS.

I tried to add:

    SSL_TYPE: manual
    SSL_CERT_PATH: /tmp/dms/custom-certs/tls.crt
    SSL_KEY_PATH:  /tmp/dms/custom-certs/tls.key

It doesn't help. I can create mailboxes but not can't use them with TLS/STARTTLS. But the behavior is different


 openssl s_client -connect mail.example.me:465

CONNECTED(00000003)
40275D84057F0000:error:0A000126:SSL routines:ssl3_read_n:unexpected eof while reading:../ssl/record/rec_layer_s3.c:316:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 326 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
What I"m doing wrong?

manyjuice avatar Jun 26 '25 13:06 manyjuice

Its a bit hard to follow what you have setup. Sounds like you are using cert-manager. Are you using something like Let's Encrypt or a self-signed certificate with cert-manager?

And what do you mean "to search for extra secrets, but I see nothing". When you configure cert-manager you tell it what secret to store the certificate in. So verify that secret exists in the same namespace as docker-mailserver.

cfis avatar Jun 26 '25 19:06 cfis

Its a bit hard to follow what you have setup. Sounds like you are using cert-manager. Are you using something like Let's Encrypt or a self-signed certificate with cert-manager?

And what do you mean "to search for extra secrets, but I see nothing". When you configure cert-manager you tell it what secret to store the certificate in. So verify that secret exists in the same namespace as docker-mailserver.

Hello, thanks for the feedback. Yes, I have letsencrypt cert issued properly via cert-manager. It exists in mail namespace (the same docker-mailserver is) has a proper secret/cert name, inside it I see a valit certificate (tls.key, tls cert). It's specified in values of the chart, but docker-mailserver still uses it's default self-signed cert instead of the one from cert-manager.

values.yaml:
....
service:
  type: LoadBalancer
  externalTrafficPolicy: Local

certificate: mail-tls-certificate-rsa
....

Here is the cert details

mail-cert.yaml
...
apiVersion: cert-manager.io/v1
kind: Certificate

metadata:
  name: mail-tls-certificate-rsa

spec:
  secretName: mail-tls-certificate-rsa
  isCA: false
  privateKey:
    algorithm: RSA
    encoding: PKCS1
    size: 2048
  dnsNames:
    - mail.example.me
  issuerRef:
    name: letsencrypt-production
    kind: ClusterIssuer

By extra secrets I mean the default self-signed TLS cert of docker-mailserver, as it now uses this cert instead of the required cert-manager's letsencrypt. So it looks like docker-mailserver simply ignores this cert and uses the old one (stock), but I don't see this stock cert in secrets/certs in any namespace. Only the valid LE cert exists, but it's not being used.

So the main question: is it enough to specify the cert name certificate: mail-tls-certificate-rsa in the chart values and to ensure the valid LE cert-manager's cert exists in the same namespace? Of something else is needed to mount the cert to docker-mailserver?

Even after the full helm chart reinstallation:

 openssl s_client -connect mail.example.me:143 -starttls imap
CONNECTED(00000003)
depth=0 CN = docker-mailserver.invalid
verify error:num=18:self-signed certificate
verify return:1
depth=0 CN = docker-mailserver.invalid
verify return:1
---
Certificate chain
 0 s:CN = docker-mailserver.invalid
   i:CN = docker-mailserver.invalid

From within the container the default path shows old tls.* (seems to be older stock self-signed). But also I see a subdir with the timestamp in the name, it contains the corrent date (when LE cert was generated), but it's not in use and I don't understand, why the timestamp is in the subdir name at all

ls -lah /tmp/dms/custom-certs
total 4.0K
drwxrwxrwt 3 root root  120 Jun 26 19:36 .
drwxr-xr-x 3 root root 4.0K Jun 26 19:36 ..
drwxr-xr-x 2 root root   80 Jun 26 19:36 ..2025_06_26_19_36_39.2917257397
lrwxrwxrwx 1 root root   32 Jun 26 19:36 ..data -> ..2025_06_26_19_36_39.2917257397
lrwxrwxrwx 1 root root   14 Jun 26 19:36 tls.crt -> ..data/tls.crt
lrwxrwxrwx 1 root root   14 Jun 26 19:36 tls.key -> ..data/tls.key

If I remove "manual" from the chart, then redeploy, the error differs:

openssl s_client -connect mail.example.me:143 -starttls imap
Didn't find STARTTLS in server response, trying anyway...
40B7BC81B07F0000:error:0A000126:SSL routines:ssl3_read_n:unexpected eof while reading:../ssl/record/rec_layer_s3.c:316:
no peer certificate available
No client certificate CA names sent
SSL handshake has read 0 bytes and written 352 bytes
Verification: OK

openssl s_client -connect mail.example.me:465 -showcerts
CONNECTED(00000003)
40E7CBF6167F0000:error:0A000126:SSL routines:ssl3_read_n:unexpected eof while reading:../ssl/record/rec_layer_s3.c:316:
no peer certificate available
No client certificate CA names sent
SSL handshake has read 0 bytes and written 326 bytes
Verification: OK
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)

manyjuice avatar Jun 26 '25 19:06 manyjuice

From within the pod I see the correct Letsencrypt certificate by the correct path

kubectl -n mail exec -it deploy/docker-mailserver -- openssl x509 -in /tmp/dms/custom-certs/tls.crt -text -noout

But Posftfix, Dovecot still use the self-signed one. I don't see the cert specified there, the value is empty:

root@docker-mailserver-66d4dd6dc6-t6wn6:/# postconf smtpd_tls_cert_file
smtpd_tls_cert_file =

And Dovecot seems to be using the default self-signeb

grep -r 'ssl' /etc/dovecot
/etc/dovecot/conf.d/10-master.conf:    ssl = yes
/etc/dovecot/conf.d/10-ssl.conf:ssl = required
/etc/dovecot/conf.d/10-ssl.conf:ssl_cert = </etc/ssl/certs/ssl-cert-snakeoil.pem
/etc/dovecot/conf.d/10-ssl.conf:ssl_key = </etc/ssl/private/ssl-cert-snakeoil.key

Reinstallation the Helm chart, removing volumes etc doesn't help.

My current values:

proxyProtocol:
  enabled: false

service:
  type: LoadBalancer
  externalTrafficPolicy: Local

certificate: mail-tls-certificate-rsa

deployment:
  env:
    OVERRIDE_HOSTNAME: mail.example.me
    ENABLE_UPDATE_CHECK: 1
    UPDATE_CHECK_INTERVAL: 1w
    ENABLE_CLAMAV: 1
    ENABLE_FAIL2BAN: 1
    POSTFIX_MESSAGE_SIZE_LIMIT: 102400000

manyjuice avatar Jun 28 '25 23:06 manyjuice

I don't have experience with helm/k8s, but shouldn't you be setting the SSL_TYPE=letsencrypt ENV to enable the certs?

That ENV is written into our settings file within the container so if you shell into the container and run env command it should show SSL_TYPE if it's been configured correctly, and that would mean that this startup logic would have been applied to configure Postfix/Dovecot.

You will also see that our k8s docs page has SSL_TYPE configured. So I'm pretty sure you are meant to configure this. The docs at a glance cover both manual and letsencrypt types.

polarathene avatar Jun 29 '25 01:06 polarathene

@polarathene thanks for the information. I tried both SSL_TYPE: letsencrypt and SSL_TYPE: manual either with and without paths to files. Completely purged resources before reapplying The same result when specifying only the cert name or also SSL_TYPE/paths:

From within the container I see a correct certificate under a proper LE cert under by the expected location /tmp/dms/custom-certs/tls.crt. But I don't see SSL enabled properly in Dovecot or Postfix Config. So testing from an external client still gives me a default self-signed docker-mailserver cert.

As I see now, there are no problems with mouting the cert to the container via the helm chart

certificate: mail-tls-certificate-rsa

deployment:
    ...
    SSL_TYPE: manual
    SSL_CERT_PATH: /tmp/dms/custom-certs/tls.crt
    SSL_KEY_PATH: /tmp/dms/custom-certs/tls.key

it works the same way as the default minimal config manner:

certificate: mail-tls-certificate-rsa

Now, for example, the pod received env vars accordingly:

root@docker-mailserver-54cc4979b5-t49m2:/# printenv | grep SSL
SSL_CERT_PATH=/tmp/dms/custom-certs/tls.crt
SSL_KEY_PATH=/tmp/dms/custom-certs/tls.key
SSL_ALT_CERT_PATH=
SSL_TYPE=manual
SSL_ALT_KEY_PATH=

I'm thinking on adding extraDeploy or ConfigMap to explicitly specify the cert path for Dovecot, Postfix. But I'm confused it's not mentioned in the chart readme and looks as a dirty workaround.

manyjuice avatar Jun 29 '25 09:06 manyjuice

Can you confirm that the ENV was persisted into our container settings storage /etc/dms-settings? Try grep SSL_CERT_PATH /etc/dms-settings which should match your ENV.

If the ENV are written correctly to this file, but Postfix/Dovecot is not updated, please ensure you have set the ENV LOG_LEVEL=trace, it should have lines logged like:

Configuring SSL using 'letsencrypt'
Adding ${LETSENCRYPT_DOMAIN} SSL certificate to the postfix and dovecot configuration
SSL configured with 'letsencrypt' certificates

If it does, then it should be running this logic to update Postfix/Dovecot configs:

https://github.com/docker-mailserver/docker-mailserver/blob/6bc4d243e2f29f07a38ae3452e335fd8fc014ecf/target/scripts/helpers/ssl.sh#L47-L57

which relates to file paths for those config here:

https://github.com/docker-mailserver/docker-mailserver/blob/6bc4d243e2f29f07a38ae3452e335fd8fc014ecf/target/scripts/helpers/ssl.sh#L23-L25

Please note that the Postfix setting you checked with postconf does not match the one we're configuring which is smtpd_tls_chain_files.


If this is not happening, something must be off with your configuration, but as I don't know my way around Kubernetes and related tooling, you'll need to troubleshoot that yourself or hope someone else can assist you.

I did notice that your output shared for /tmp/dms/custom-certs has symlinks. Ensure that those resolve correctly within the container. There is advice on our TLS docs page for LetsEncrypt mounting specifically where users only mounted the symlinks into the container instead of actual certificates.

You can try cat /tmp/dms/custom-certs/tls.crt to verify that these can be read within the container. SSL_TYPE=manual differs from SSL_TYPE=letsencrypt, in that we make an internal copy so verify that was done correctly too cat /etc/dms/tls/cert. Both should have the same content of your PEM encoded X.509 TLS certificate.

polarathene avatar Jun 29 '25 23:06 polarathene

Thank you @manyjuice for the additional information. So this is where setting up the certificate is done:

https://github.com/docker-mailserver/docker-mailserver-helm/blob/90b9a2754ae39425a89852f92a777ab9c0ae785a/charts/docker-mailserver/templates/deployment.yaml#L112

 {{- if .Values.certificate }}
          - name: SSL_TYPE
            value: manual
          - name: SSL_CERT_PATH
            value: /tmp/dms/custom-certs/tls.crt
          - name: SSL_KEY_PATH
            value: /tmp/dms/custom-certs/tls.key
          {{- end }}

If you set .Values.certificate, which you have done, then the helm chart tells docker-mailserver that there is a manual certificate and sets the path to it. This setups up the ENV variables (@polarathene). Note the secret specified in .Values.certificate is mounted here:

https://github.com/docker-mailserver/docker-mailserver-helm/blob/90b9a2754ae39425a89852f92a777ab9c0ae785a/charts/docker-mailserver/templates/deployment.yaml#L141

I think this approach is correct because you are using Cert Manager to generate a Let's Encrypt certificate and not docker mailserer itself.

Note I have the exact same setup (using cert-manager to generate a Let's Encrypt certificate) and it works fine. So I wonder what's different in your setup?

cfis avatar Jun 30 '25 06:06 cfis

@manyjuice do you still have an issue or were you able to solve it?

cfis avatar Nov 07 '25 07:11 cfis

@manyjuice do you still have an issue or were you able to solve it?

Hi, I fixed it after reading your previous response. To be honest, I don't remember how exactly and I don't have access to the project anymore. But no overcomplicated workarounds were needed. This thread should be enough for clarification.

Thanks for help.

manyjuice avatar Nov 07 '25 08:11 manyjuice