ERR_CERT_DATE_INVALID after renewal on gce

[ciokan]

I see in the logs of kube-lego that the certificate was renewed but my browser still shows ERR_CERT_DATE_INVALID. What do I need to do to use the new one, any ideas?

[ahmetb]

I don't know inner workings of kube-lego, but my guess is that it should have updated the secret on the Ingress (which I expect that it would have uploaded the new TLS secret to the Google Cloud and configure it on the Load Balancer).

1. Can you find the value of ssl-cert annotation in kubectl describe ingress <name> and check if this value matches the certificate column on the Load Balancer console? (example below)

2. Can you check the expiration date served to your browser, and see if that matches what's stored in the kubectl get secret <secretname> -o=yaml (I'll leave it up to you to figure out how to save and parse the certificate using openssl).

[ciokan]

The values don't match:

On the ingress describe I get: k8s-ssl-default-loadbalancer--cbee26f7d1895ddb and in the load balancer console I get: k8s-um-default-loadbalancer--41dc2c3938d90731

[ahmetb]

@ciokan Are you looking at the Certificate column in the console? It looks like you're looking at the title of the load balancer. See my screenshot, it has a Certificate column on the right.

What I will suggest is that:

1. check if values match
2. check if the cert thumbprint matches. First find the TLS certificate on the Google Cloud Load Balancer (you can click on the Certificate name and see a Serial Number, for example mine is: 03:82:05:24:0C:41:0F:D1:6C:A0:6E:A4:43:89:7D:5F:B1:88) Then go to browser's cert information window, compare these two strings.
3. Assuming (2) is matching, you should download the secret saved on Kubernetes to your machine and verify that the thumbprint is the same too. If the thumbprints are the same, kube-lego probably didn't save the renewed secret.

I am particularly interested in helping out as I've seen before that kube-lego log statements can be misleading and we all need this feature to work.

[ciokan]

yes the certificate or title point to the same "id":

k8s-ssl-default-loadbalancer--41dc2c3938d90731

Whatever is loaded in the loadbalancer expired on april 26:

Expires	
Apr 26, 2017, 3:03:00 AM
Serial Number	
03:46:43:E3:DB:BF:05:9B:B9:0E:9E:63:11:AD:B1:3F:23:D4
Certificate Issuer	
Let's Encrypt Authority X3

In the kube-lego pod logs I can see a different story:

2017-04-30T14:26:04.078240149Z time="2017-04-30T14:26:04Z" level=info msg="cert expires in 88.0 days, no renewal needed" context="ingress_tls" expire_time=2017-07-27 13:47:00 +0000 UTC name=loadbalancer namespace=default 
2017-04-30T14:26:04.078331927Z time="2017-04-30T14:26:04Z" level=info msg="no cert request needed" context="ingress_tls" name=loadbalancer namespace=default 

I'm curious as to why it's not present in the loadbalancer, it seems to have kept the old one.

[ahmetb]

Ok it looks like the certificate object on the ingress is not updated. I am not sure if Ingress is supposed to pick up values of an "updated" secret it is already configured with... Let's try inspecting the secret itself:

First, save the secret out:

kubectl get secret SECRET_NAME -o jsonpath='{.data.tls\.crt}' | base64 -d | tee tls.crt

then inspect the validity date of certificate:

 openssl x509 -in tls.crt -text

Can you verify if the serial number is different or the same on this output (and see the validity date).

Let's figure out if kube-lego has actually updated the secret just fine and that's why it thinks there's 88 more days to its expiration.

[ciokan]

This one seems ok.

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            03:86:3c:38:40:8a:1c:e3:be:41:7c:ab:8a:77:67:b8:4a:10
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=US, O=Let's Encrypt, CN=Let's Encrypt Authority X3
        Validity
            Not Before: Apr 28 13:47:00 2017 GMT
            Not After : Jul 27 13:47:00 2017 GMT

[ahmetb]

Thanks. This is a bit scary. I am nowadays evaluating the LetsEncrypt clients for Kubernetes out there and this is certainly something worth considering while using such projects.

I think now the owners of the project can be asked the following question:

Is kube-lego responsible for making sure that the Ingress is picking up the renewed secret? If this is the responsibility of the Ingress (and not kube-lego), has this guarantee been tested before with the GCE ingress? It looks like the Cloud Load Balancer is not reconfigured when the TLS secret associated with the Ingress is updated.

[ciokan]

Bummer. I loaded this thing into a lot of applications. I'm still hoping it's some sort of a "my fault". This project has quite some time behind it and I failed to find any similar ticket. Maybe someone from the dev team can shed a light.

[gianrubio]

Is kube-lego responsible for making sure that the Ingress is picking up the renewed secret? If this is the responsibility of the Ingress (and not kube-lego), has this guarantee been tested before with the GCE ingress? It looks like the Cloud Load Balancer is not reconfigured when the TLS secret associated with the Ingress is updated

No, kube-lego doesn't manage the certificates in the load balancer, it only create the certificate secrets. GCE ingress is watching the objects and it's responsible for change the ssl certificate at the load balancer.

I saw some improvements on gce when it's managing certificates #639. Please test the latest release and let us know if this solve this issue.

[ahmetb]

@gianrubio thanks for linking to this issue. I'm not sure if this patch solves updating TLS configuration on GCLB when the Secret(s) associated to an Ingress is updated :/

How does one update the ingress controller in GKE (I can't find it in my cluster). So I don’t know which version of ingress controller is even running. (Any ideas?) This will be hard to test anyway.

I wonder if the project authors have seen the cert refresh flow working on GCE/GKE before. A statement like “we never tested this” or “we’ve seen it working before” would kind of help.

[ciokan]

Apart from upgrading my kubernetes version to 1.6.2 (latest offering by GKE) and trying to re-apply my ingress in hope of it picking up the new changes I don't know what else to try.

My website is failing to respond due to this issue for 1 week already and I have no clue what else to try. It's a side project with nothing serious on it otherwise I would have bought a SSL immediately so I'll wait a bit more until we get to the bottom of this.

[ahmetb]

@ciokan It looks like your website is running on another certificate currently?

Here's what I would recommend to test if the upgrade fixes the problem:

1. Download the expires cert/key from Google Cloud Platform Console (I hope you still have these on your existing load balancer).
2. Delete your ingress.
3. Upgrade your GKE cluster
4. Remove kubernetes.io/tls-acme: "true" annotation from your ingress manifest
5. Delete your existing tls secret.
6. Create new secret (with the old name) and use the expired cert/key
7. Deploy your modified ingress manifest.
8. Make sure you get a cert expired error on browser.
9. Kube-lego won't do anything because there's no annotation.
10. Put the tls-acme annotation back and kubectl-apply.
11. Watch kube-lego logs to see if it refreshes the certificate
12. Go back to Google Cloud Platform Console and see if the Ingress Controller has updated the LB with the new certificate.

[gianrubio]

@ahmetb

@gianrubio thanks for linking to this issue. I'm not sure if this patch solves updating TLS configuration on GCLB when the Secret(s) associated to an Ingress is updated :/

How does one update the ingress controller in GKE (I can't find it in my cluster). So I don’t know which version of ingress controller is even running. (Any ideas?) This will be hard to test anyway.

I have no idea how to upgrade it, you could ask on http://github.com/kubernetes/ingress

I wonder if the project authors have seen the cert refresh flow working on GCE/GKE before. A statement like “we never tested this” or “we’ve seen it working before” would kind of help.

Last week I tested this feature , as I described on #180 on how to test the renew certificate. I bootstrapped a new cluster on gce (v.1.6.2), configured the env LEGO_MINIMUM_VALIDITY to a low value (1 hour) and all the certificates are correctly renewed.

Probably I'm addicted on how to run kube-lego. If you're able to do the same as I describe above can help us to figure it out.