Unable to mount and read only file error

[rajeshbhoye]

Warning FailedMount 11m (x329 over 16h) kubelet Unable to attach or mount volumes: unmounted volumes=[tls], unattached volumes=[tls kube-api-access-2v6n8]: timed out waiting for the condition Warning FailedMount 88s (x482 over 16h) kubelet MountVolume.SetUp failed for volume "tls" : rpc error: code = Unknown desc = chmod /var/lib/kubelet/pods/0fd27403-622b-457c-b43f-606472572c59/volumes/kubernetes.io~csi/tls/mount: read-only file system

[ankurpshah]

+1

[glmdev]

I'm also experiencing this issue trying to mount to a container. Some more details, if it helps:

• Controller image: quay.io/jetstack/cert-manager-controller:v1.7.2
• CSI driver image: quay.io/jetstack/cert-manager-csi-driver:v0.2.0
• I'm running on Rancher K3s with kubelet 1.22.7. I have a few services using cert-manager certificates that are working fine.

Certificate definition:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: registry-tls
spec:
  secretName: registry-tls-secret
  dnsNames:
    - 'registry.redacted.net'
  issuerRef:
    name: letsencrypt-ca
    kind: ClusterIssuer

Pod definition:

apiVersion: v1
kind: Pod
metadata:
  name: docker-registry-pod
  labels:
    app: registry
spec:
  containers:
    - name: registry
      image: registry:2
      volumeMounts:
        - name: repo-vol
          mountPath: '/var/lib/registry'
        - name: tls
          mountPath: '/tls'
  volumes:
    - name: repo-vol
      persistentVolumeClaim:
        claimName: docker-registry-vol-claim
    - name: tls
      csi:
        readOnly: yes
        driver: csi.cert-manager.io
        volumeAttributes:
          csi.cert-manager.io/issuer-name: letsencrypt-ca
          csi.cert-manager.io/dns-names: registry.redacted.net

Logs from csi driver are a bunch of the following:

E0506 00:00:52.022851       1 server.go:109] driver "msg"="failed processing request" "error"="timed out waiting for the condition" "request"={} "rpc_method"="/csi.v1.Node/NodePublishVolume" 
I0506 00:01:24.122823       1 nodeserver.go:74] driver "msg"="Registered new volume with storage backend" "pod_name"="docker-registry-pod" 
I0506 00:01:24.122885       1 nodeserver.go:84] driver "msg"="Volume registered for management" "pod_name"="docker-registry-pod" 
I0506 00:01:25.124239       1 manager.go:504] manager "msg"="Triggering new issuance" "volume_id"="csi-d00aa30048a62abe239705e4427130b3f8b9bec2b54487c35153d9a5ca3db6c1" 
I0506 00:01:25.124309       1 manager.go:243] manager "msg"="Processing issuance" "volume_id"="csi-d00aa30048a62abe239705e4427130b3f8b9bec2b54487c35153d9a5ca3db6c1" 
I0506 00:01:25.137035       1 manager.go:369] manager "msg"="Deleted CertificateRequest resource" "volume_id"="csi-d00aa30048a62abe239705e4427130b3f8b9bec2b54487c35153d9a5ca3db6c1" "name"="eedc4e94-592d-4857-aef2-4905ecdf75d9" "namespace"="default"
E0506 00:01:25.482172       1 manager.go:506] manager "msg"="Failed to issue certificate, retrying after applying exponential backoff" "error"="waiting for request: certificaterequest.cert-manager.io \"eedc4e94-592d-4857-aef2-4905ecdf75d9\" not found" "volume_id"="csi-d00aa30048a62abe239705e4427130b3f8b9bec2b54487c35153d9a5ca3db6c1"

Logs from cert-manager:

I0506 00:00:23.433363       1 conditions.go:261] Setting lastTransitionTime for CertificateRequest "eedc4e94-592d-4857-aef2-4905ecdf75d9" condition "Approved" to 2022-05-06 00:00:23.433346151 +0000 UTC m=+1446937.641072660
I0506 00:00:23.456865       1 conditions.go:261] Setting lastTransitionTime for CertificateRequest "eedc4e94-592d-4857-aef2-4905ecdf75d9" condition "Ready" to 2022-05-06 00:00:23.456854023 +0000 UTC m=+1446937.664580514
I0506 00:00:23.456913       1 conditions.go:261] Setting lastTransitionTime for CertificateRequest "eedc4e94-592d-4857-aef2-4905ecdf75d9" condition "Ready" to 2022-05-06 00:00:23.456904982 +0000 UTC m=+1446937.664631475
I0506 00:00:23.457423       1 conditions.go:261] Setting lastTransitionTime for CertificateRequest "eedc4e94-592d-4857-aef2-4905ecdf75d9" condition "Ready" to 2022-05-06 00:00:23.457415688 +0000 UTC m=+1446937.665142182
I0506 00:00:23.457598       1 conditions.go:261] Setting lastTransitionTime for CertificateRequest "eedc4e94-592d-4857-aef2-4905ecdf75d9" condition "Ready" to 2022-05-06 00:00:23.457589161 +0000 UTC m=+1446937.665315652
I0506 00:00:23.457627       1 conditions.go:261] Setting lastTransitionTime for CertificateRequest "eedc4e94-592d-4857-aef2-4905ecdf75d9" condition "Ready" to 2022-05-06 00:00:23.457613946 +0000 UTC m=+1446937.665340437
I0506 00:00:23.492341       1 controller.go:161] cert-manager/certificaterequests-issuer-ca "msg"="re-queuing item due to optimistic locking on resource" "key"="default/eedc4e94-592d-4857-aef2-4905ecdf75d9" "error"="Operation cannot be fulfilled on certificaterequests.cert-manager.io \"eedc4e94-592d-4857-aef2-4905ecdf75d9\": the object has been modified; please apply your changes to the latest version and try again"
I0506 00:00:23.499850       1 controller.go:161] cert-manager/certificaterequests-issuer-acme "msg"="re-queuing item due to optimistic locking on resource" "key"="default/eedc4e94-592d-4857-aef2-4905ecdf75d9" "error"="Operation cannot be fulfilled on certificaterequests.cert-manager.io \"eedc4e94-592d-4857-aef2-4905ecdf75d9\": the object has been modified; please apply your changes to the latest version and try again"
I0506 00:00:23.500874       1 controller.go:161] cert-manager/certificaterequests-issuer-vault "msg"="re-queuing item due to optimistic locking on resource" "key"="default/eedc4e94-592d-4857-aef2-4905ecdf75d9" "error"="Operation cannot be fulfilled on certificaterequests.cert-manager.io \"eedc4e94-592d-4857-aef2-4905ecdf75d9\": the object has been modified; please apply your changes to the latest version and try again"
I0506 00:00:23.501078       1 controller.go:161] cert-manager/certificaterequests-issuer-selfsigned "msg"="re-queuing item due to optimistic locking on resource" "key"="default/eedc4e94-592d-4857-aef2-4905ecdf75d9" "error"="Operation cannot be fulfilled on certificaterequests.cert-manager.io \"eedc4e94-592d-4857-aef2-4905ecdf75d9\": the object has been modified; please apply your changes to the latest version and try again"

I can't really tell what's happening here, tbh. It looks like the certificate -is- being issued, but not in time? So the CSI driver deletes the certificate request and tries again which again times out?

I'm using the Cloudflare DNS challenge solver for cert-manager, so it's possible that the propagation delay is to blame here. I've also tried it with the disable-auto-renew attribute.

ETA: after messing around with this for a bit, I eventually gave up and mounted the TLS secret as a volume on the container.

[Yamakaky]

ETA: after messing around with this for a bit, I eventually gave up and mounted the TLS secret as a volume on the container.

@glmdev I have the same issue, what did you do to fix it?

[JoshVanL]

For anyone running into the above, can they please share what the status is of the CertificateRequest which is causing them problems? e.g.:

kubectl describe cr eedc4e94-592d-4857-aef2-4905ecdf75d9

Also setting a higher log level on the csi-driver should help give an idea on what is happening:

--log-level=5

[munnerz]

It sounds like everyone here is using Let's Encrypt with the CSI driver - is that correct?

I am not sure if the CSI driver is the best tool for this job if so, as the CSI driver will perform a whole new Order on every pod creation for a new certificate, which is likely going to eat up all your quota with their API.

Also to clarify, you don't need to use the CSI driver at all if you are creating a Certificate resource - you'll just want to mount the generated Secret into the Pod as a volume like you've resorted to above 😊

[Yamakaky]

For now I fell back to manual certificate management, I'll check that when I get the time.

No, I have a local CA generated with openssl.

[GarbageYard]

I am trying to use csi-driver for pod certs (mTLS). I am also getting similar error:

Events:
  Type     Reason       Age                   From               Message
  ----     ------       ----                  ----               -------
  Normal   Scheduled    46m                   default-scheduler  Successfully assigned mtls-demo/nginx-csi-app to worker-1
  Warning  FailedMount  10m (x16 over 44m)    kubelet            Unable to attach or mount volumes: unmounted volumes=[tls], unattached volumes=[tls kube-api-access-l5gxv]: timed out waiting for the condition
  Warning  FailedMount  4m10s (x23 over 45m)  kubelet            MountVolume.SetUp failed for volume "tls" : rpc error: code = Unknown desc = timed out waiting for the condition

cert-manager log:

I0819 12:46:49.655107       1 conditions.go:261] Setting lastTransitionTime for CertificateRequest "cc574045-998f-400a-87f6-c3641ada4bc3" condition "Approved" to 2022-08-19 12:46:49.655085489 +0000 UTC m=+20753.641048167
I0819 12:46:49.689897       1 conditions.go:261] Setting lastTransitionTime for CertificateRequest "cc574045-998f-400a-87f6-c3641ada4bc3" condition "Ready" to 2022-08-19 12:46:49.689824879 +0000 UTC m=+20753.675787511
I0819 12:46:49.690161       1 conditions.go:261] Setting lastTransitionTime for CertificateRequest "cc574045-998f-400a-87f6-c3641ada4bc3" condition "Ready" to 2022-08-19 12:46:49.690136449 +0000 UTC m=+20753.676099083
I0819 12:46:49.690568       1 conditions.go:261] Setting lastTransitionTime for CertificateRequest "cc574045-998f-400a-87f6-c3641ada4bc3" condition "Ready" to 2022-08-19 12:46:49.690537081 +0000 UTC m=+20753.676499702
I0819 12:46:49.690728       1 conditions.go:261] Setting lastTransitionTime for CertificateRequest "cc574045-998f-400a-87f6-c3641ada4bc3" condition "Ready" to 2022-08-19 12:46:49.690684788 +0000 UTC m=+20753.676647418
I0819 12:46:49.690923       1 conditions.go:261] Setting lastTransitionTime for CertificateRequest "cc574045-998f-400a-87f6-c3641ada4bc3" condition "Ready" to 2022-08-19 12:46:49.690901191 +0000 UTC m=+20753.676863811
I0819 12:46:49.775161       1 controller.go:161] cert-manager/certificaterequests-issuer-venafi "msg"="re-queuing item due to optimistic locking on resource" "error"="Operation cannot be fulfilled on certificaterequests.cert-manager.io \"cc574045-998f-400a-87f6-c3641ada4bc3\": the object has been modified; please apply your changes to the latest version and try again" "key"="mtls-demo/cc574045-998f-400a-87f6-c3641ada4bc3"
I0819 12:46:49.783453       1 controller.go:161] cert-manager/certificaterequests-issuer-acme "msg"="re-queuing item due to optimistic locking on resource" "error"="Operation cannot be fulfilled on certificaterequests.cert-manager.io \"cc574045-998f-400a-87f6-c3641ada4bc3\": the object has been modified; please apply your changes to the latest version and try again" "key"="mtls-demo/cc574045-998f-400a-87f6-c3641ada4bc3"
I0819 12:46:49.802477       1 controller.go:161] cert-manager/certificaterequests-issuer-vault "msg"="re-queuing item due to optimistic locking on resource" "error"="Operation cannot be fulfilled on certificaterequests.cert-manager.io \"cc574045-998f-400a-87f6-c3641ada4bc3\": the object has been modified; please apply your changes to the latest version and try again" "key"="mtls-demo/cc574045-998f-400a-87f6-c3641ada4bc3"
I0819 12:46:49.822315       1 controller.go:161] cert-manager/certificaterequests-issuer-selfsigned "msg"="re-queuing item due to optimistic locking on resource" "error"="Operation cannot be fulfilled on certificaterequests.cert-manager.io \"cc574045-998f-400a-87f6-c3641ada4bc3\": the object has been modified; please apply your changes to the latest version and try again" "key"="mtls-demo/cc574045-998f-400a-87f6-c3641ada4bc3"

$ kubectl describe cr cc574045-998f-400a-87f6-c3641ada4bc3 -n mtls-demo
Events:
  Type    Reason           Age   From                                                Message
  ----    ------           ----  ----                                                -------
  Normal  IssuerNotFound   85s   cert-manager-certificaterequests-issuer-ca          Referenced "ClusterIssuer" not found: clusterissuer.cert-manager.io "ca-issuer" not found
  Normal  cert-manager.io  85s   cert-manager-certificaterequests-approver           Certificate request has been approved by cert-manager.io
  Normal  IssuerNotFound   85s   cert-manager-certificaterequests-issuer-acme        Referenced "ClusterIssuer" not found: clusterissuer.cert-manager.io "ca-issuer" not found
  Normal  IssuerNotFound   85s   cert-manager-certificaterequests-issuer-selfsigned  Referenced "ClusterIssuer" not found: clusterissuer.cert-manager.io "ca-issuer" not found
  Normal  IssuerNotFound   85s   cert-manager-certificaterequests-issuer-vault       Referenced "ClusterIssuer" not found: clusterissuer.cert-manager.io "ca-issuer" not found
  Normal  IssuerNotFound   85s   cert-manager-certificaterequests-issuer-venafi      Referenced "ClusterIssuer" not found: clusterissuer.cert-manager.io "ca-issuer" not found

From what I understood from the doc, one doesn't need to create any Certificate resource (in which we define issuerRef kind as ClusterIssuer) explicitly. I still tried creating certificate (skipped specifying dnsNames) assuming that DNS name is already specified in the pod spec (csi.cert-manager.io/dns-names: ${POD_NAME}.${POD_NAMESPACE}.svc.cluster.local) but then my certificate failed with error which was actually expected:

admission webhook "webhook.cert-manager.io" denied the request: spec: Invalid value: "": at least one of commonName, dnsNames, uris ipAddresses, or emailAddresses must be set

Certificate:
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: csi-example-com
  namespace: mtls-demo
spec:
  secretName: csi-example-com-tls
  isCA: false
  issuerRef:
    name: ca-issuer
    name: letsencrypt-staging
    kind: ClusterIssuer

I will really appreciate any help.

[JoshVanL]

Hi @GarbageYard, looking at the events of the CertificateRequest, the ca-issuer doesn't exist yet. Either check that the issuer you are trying to use have the correct name and type, or head over to the docs for more information on how to setup an issuer.

The Certificate resource you are creating there has nothing to do with the csi-driver. The webhook error is still correct, and enforces that you must define one of those SANs on the Certificate resource for it to be accepted.

[GarbageYard]

Thanks, actually the name for the ClusterIssuer was incorrectly set. Now I am getting this error:

E0819 14:52:36.083386 1 sync.go:273] cert-manager/orders "msg"="failed to create Order resource due to bad request, marking Order as failed" "error"="400 urn:ietf:params:acme:error:rejectedIdentifier: Error creating new order :: Cannot issue for \"nginx-csi-app.mtls-demo.svc.cluster.local\": Domain name does not end with a valid public suffix (TLD)" "resource_kind"="Order" "resource_name"="9e3438d9-f5ff-4eb5-937a-a64f49f41d5a-1859061661" "resource_namespace"="mtls-demo" "resource_version"="v1"

Since .local is not reachable, I am wondering how the csi-driver is expected to work for pods then? I did check the docs for csi-driver and read that it doesn't support self-signed certs but for letsencrypt, it doesn't say such thing. I did read @munnerz comment above that csi-driver with letsencrypt is not really how it's meant to be used due to quota limit but I at least wanted to see this working with .local so I could then later use the same with real CA, venafi etc.