traefik
Add support for subpath setting under volumes
Welcome!
- [X] Yes, I've searched similar issues on GitHub and didn't find any.
- [X] Yes, I've searched similar issues on the Traefik community forum and didn't find any.
What did you expect to see?
I am trying to mount a private root CA pem file under /etc/ssl/certs in order for traefik to trust certs issued by the CA.
I'm doing this by mounting via the following, in the helm chart:
volumes:
- mountPath: /etc/ssl/certs/root-ca.pem
name: root-ca
readOnly: false
subPath: root-ca.pem
type: configMap
However, the setting subPath: root-ca.pem is ignored by the helm chart, and therefore the cert is improperly mounted to:
/etc/ssl/certs/root-ca.pem/root-ca.pem when it should be mounted to /etc/ssl/certs/root-ca.pem
I'm able to modify the deployment manually and get the mount to be correct, but ideally this would be supported within the helm chart itself.
I doubt mounting your file in here would have any effect. Have you tried patching your existing deployment, leaving Helm aside?
I think you need to run some update-ca-certifcates. I would recommend an initContainer.
Trying to translate what I'ld usually do in this context, into values for traefik-helm-chart, ... First, I would declare my CA volume, and some emptyDir volume to share /etc/ssl/certs between Traefik and its init container. Something like this:
volumes:
- mountPath: /usr/local/share/ca-certificates
name: root-ca
type: configMap
- mountPath: /etc/ssl/certs
name: temp
type: emptyDir
Next, I would add an initContainer, that should update the system trusted certifcates database. Something like this:
deployment:
initContainers:
- name: init-pki
command:
- /bin/sh
args:
- -c
- |
update-ca-certificates || echo warning
volumeMounts:
- name: temp
mountPath: /etc/ssl/certs
- name: root-ca
mountPath: /usr/local/share/ca-certificates
Let us know how that goes.
Hi @faust64 I patched the deployment with the subPath setting, and this does indeed solve the issue:
volumeMounts:
- mountPath: /data
name: data
- mountPath: /tmp
name: tmp
- mountPath: /etc/ssl/certs/root-ca.pem
name: root-ca
readOnly: true
subPath: root-ca.pem #<-- patched here
https://paraspatidar.medium.com/add-self-signed-or-ca-root-certificate-in-kubernetes-pod-ca-root-certificate-store-cb7863cb3f87 has a fairly good write up under Method 3— Kubernets Deployment Yaml / Helm changes
The issue with overwriting /etc/ssl/certs completely with an emptyDir, is that it removes all "public"/well known certs from the pod, so if Traefik were to try to also reach out to an endpoint with a cert signed by a "public" CA, I'd then need to manage additional CAs which I don't want to do.
You may have noticed /etc/ssl/certs holds links. Running update-ca-certificates, you re-generate that database. No matter it's emptyDir, the initContainer is meant to build a fresh copy, that would include your custom CAs, stored in /usr/local/share/ca-certificates, alongside the defaults shipping with the ca-certificates package.
On debian, ubuntu, alpine and lookalikes, this is the recommended way to trust certificates https://manpages.ubuntu.com/manpages/xenial/man8/update-ca-certificates.8.html .
In practice, authorities are identified by their fingerprint. Links we find in /etc/ssl/certs should be named after that fingerprint for authority to be trusted. I'm not sure why Traefik would let this go.
Here's a concrete example, using openssl itself:
$ kubectl exec ... -it xx /bin/sh
$ cd /etc/ssl/certs
$ ls -l | grep kube
lrwxrwxrwx 1 1001 root 17 Sep 20 01:43 5d4eb271.0 -> kube-certs-ca.pem
lrwxrwxrwx 1 1001 root 50 Sep 20 01:43 kube-certs-ca.pem -> /usr/local/share/ca-certificates/kube-certs-ca.crt
This is my CA. Notice the 5d4eb271.0 file.
With this, I can connect to some service, that uses a certificate signed by that same CA:
$ echo y | openssl s_client -connect openldap-kube.ci.svc.cluster.local:1636 2>&1 | tail -10
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
DONE
Now, say I just remove that link, while keeping the others, which do include a copy of my CA.
$ rm -f 5d4eb271.0
$ echo y | openssl s_client -connect openldap-kube.ci.svc.cluster.local:1636 2>&1 | tail -10
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 19 (self signed certificate in certificate chain)
---
DONE
Verify failed. Self-signed certificate in chain.
Randomly writing files in /etc/ssl/certs doesn't make those trusted -- at the very least, you should the use c_rehash. Though this is not recommended either: update-ca-certificates is the way.
Which is option 2 in your article. Though I wouldn't recommend doing this in a Dockerfile. That's really a job for initContainers, contextualizing runtime.
Option 3 is just the author being lucky.
So... yeah. maybe it works, with traefik. I couldn't explain why. I can tell you it shouldn't.
@faust64 I've gone ahead and tried the solution you provided with the following additions/changes to the helm chart:
initContainers:
- args:
- '-c'
- |
update-ca-certificates || echo warning
command:
- /bin/sh
image: 'alpine:latest'
name: init-pki
volumeMounts:
- mountPath: /etc/ssl/certs
name: temp
- mountPath: /usr/local/share/ca-certificates
name: root-ca
volumes:
- mountPath: /usr/local/share/ca-certificates
name: root-ca
type: configMap
- mountPath: /etc/ssl/certs
name: temp
type: emptyDir
But I'm seeing the following error, and I'm not too familiar with this issue:
Error: failed to create containerd task: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: rootfs_linux.go:76: mounting "/var/lib/kubelet/pods/5a80e9df-a68c-48ad-bdae-547d6ec7bb23/volume-subpaths/root-ca/traefik/4" to rootfs at "/etc/ssl/certs/root-ca.pem" caused: open /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/traefik/rootfs/etc/ssl/certs/root-ca.pem: read-only file system: unknown
I'm not sure what's wrong. I've justed tested and confirmed, the following works:
volumes:
- mountPath: /usr/local/share/ca-certificates
name: root-ca
type: configMap
- mountPath: /etc/ssl/certs
name: temp
type: emptyDir
deployment:
enabled: true
kind: Deployment
podLabels:
foo: bar
replicas: 1
initContainers:
- name: init-pki
command:
- /bin/sh
args:
- -c
- |
update-ca-certificates || echo warning
image: docker.io/traefik:v2.4.13
securityContext:
capabilities:
drop:
- ALL
readOnlyRootFilesystem: true
runAsGroup: 65532
runAsNonRoot: true
runAsUser: 65532
volumeMounts:
- name: temp
mountPath: /etc/ssl/certs
- name: root-ca
mountPath: /usr/local/share/ca-certificates
The securityContext bit reproduces what's used later on by the main Traefik container
$ kubectl create cm root-ca -n ingress-traefik --from-file=custom-ca.crt=some-ca.crt
Which generates:
apiVersion: apps/v1
kind: Deployment
...
spec:
replicas: 1
...
template:
...
spec:
...
initContainers:
- args:
- -c
- |
update-ca-certificates || echo warning
command:
- /bin/sh
image: docker.io/traefik:v2.4.13
name: init-pki
securityContext:
capabilities:
drop:
- ALL
readOnlyRootFilesystem: true
runAsGroup: 65532
runAsNonRoot: true
runAsUser: 65532
volumeMounts:
- mountPath: /etc/ssl/certs
name: temp
- mountPath: /usr/local/share/ca-certificates
name: root-ca
containers:
- image: "traefik:2.4.13"
imagePullPolicy: IfNotPresent
name: traefik
...
securityContext:
capabilities:
drop:
- ALL
readOnlyRootFilesystem: true
runAsGroup: 65532
runAsNonRoot: true
runAsUser: 65532
volumeMounts:
- name: data
mountPath: /data
- name: tmp
mountPath: /tmp
- name: root-ca
mountPath: /usr/local/share/ca-certificates
readOnly: true
- name: temp
mountPath: /etc/ssl/certs
readOnly: true
args:
...
volumes:
- name: data
emptyDir: {}
- name: tmp
emptyDir: {}
- name: root-ca
configMap:
name: root-ca
- name: temp
securityContext:
fsGroup: 65532
Which does result in my custom CA being added to trust store:
/etc/ssl/certs $ ls -ltr|grep local/share
lrwxrwxrwx 1 65532 65532 46 Sep 26 21:31 ca-cert-custom-ca.pem -> /usr/local/share/ca-certificates/custom-ca.crt
While system CAs are still trusted:
/etc/ssl/certs $ ls -ltr|tail -5
lrwxrwxrwx 1 65532 65532 58 Sep 26 21:31 06dc52d5.0 -> ca-cert-SSL.com_EV_Root_Certification_Authority_RSA_R2.pem
lrwxrwxrwx 1 65532 65532 33 Sep 26 21:31 064e0aa9.0 -> ca-cert-QuoVadis_Root_CA_2_G3.pem
lrwxrwxrwx 1 65532 65532 35 Sep 26 21:31 062cdee6.0 -> ca-cert-GlobalSign_Root_CA_-_R3.pem
lrwxrwxrwx 1 65532 65532 44 Sep 26 21:31 03179a64.0 -> ca-cert-Staat_der_Nederlanden_EV_Root_CA.pem
lrwxrwxrwx 1 65532 65532 53 Sep 26 21:31 02265526.0 -> ca-cert-Entrust_Root_Certification_Authority_-_G2.pem
I'm not sure what's wrong in your case. Never seem that error message either. Why would it try to mount your "root-ca" into "/etc/ssl/certs"? Kubernetes should only bother about mounting your emptyDir, then link is written/reused by containers ... that's pretty weird.
Even using different images, privileges or securityContext in your initContainer, it should work. Could this be specific to k3s, maybe? I'll try to start a lab, soon ... might take a few days, I'm a bit busy lately ...
Now... if you've got it working your own way ... I didn't mean to break your stuff, feel free to submit a PR if that's easier for you. I'm not sure mounting a certificate directly in /etc/ssl/certs would always work. But if Traefik loads it today, that's good enough, just be careful there's no regression later on.
I wanted to provide an update on this as I was able to have some time to debug this. From what I tell @faust64 your example works. What I believe was the issue here, was that for my initContainer i was using the image alpine:latest, if I switched the container image to docker.io/traefik:v2.5.3 this works.
Since this solution works, and is technically the "correct" way to do this. This issue can either be closed, or it can be left open as an enhancement to the docs so that this type of solution can be officially documented for custom CA injection (I'm not sure how common knowledge this type of solution is)
Pretty weird, using the alpine image for your initContainer shouldn't be an issue -- my using the traefik image is mostly to avoid pulling 2 images during Pod startup. I'm not sure I understand what was wrong with your previous test. Maybe some k3s limitation/specific (?!) Glad you figured it out.