Docker Registry certificate does not autorenew

[petermikitsh]

The resource Certificate with name docker-reg-cert expires without auto-renewing. This breaks deployments, because once the certificate expires and deployments are issued, which leads to pods attempt to spin up, you'll get errors like:

  Warning  Failed     5s (x2 over 21s)   kubelet, epic-borg-ud9d  Failed to pull image "10.245.8.222:5000/digitalocean:1": rpc error: code = Unknown desc = Error response from daemon: Get https://10.245.8.222:5000/v2/: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "docker-registry.default.svc.cluster.local")

or, more commonly:

Warning  Failed     17s                kubelet, epic-borg-ud9v  Failed to pull image "10.245.8.222:5000/digitalocean:1": rpc error: code = Unknown desc = Error response from daemon: Get https://10.245.8.222:5000/v2/: x509: certificate has expired or is not yet valid

Configuring the certificate resource with the renewBefore option [0] should fix the problem, e.g.,

apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: example
spec:
  secretName: example-tls
  duration: 24h
  renewBefore: 12h <--- this is the important part we need to add
  dnsNames:
  - foo.example.com
  - bar.example.com
  issuerRef:
    name: my-internal-ca
    kind: Issuer

Additionally, we create a DaemonSet resource [1] that copies over the latest certificate daily to all nodes in all node pools. This permits the node pools to trust pulling images from the docker registry.

Here's a couple things to review to ensure high availability of the docker registry, and ensure the capability to deploy whenever within the cluster:

1. Ensure the renewBefore spec setting actually works as excepted
2. Ensure that the docker registry "hot reloads" the new certificate (if not, we need to force it to restart after the certificate is renewed)
3. Ensure that the DaemonSet is performing as we except it to (described above)