Lego Pod disappearing from kube-lego-nginx Service, leading to 503 errors when requesting new certificate

[marceldegraaf]

I'm getting frequent HTTP 503 status errors from lego, when it's requesting new certificates. Using kube-lego and kubernetes-nginx-ingress on AWS.

Logs from the nginx pod:

W1116 14:34:20.128648       5 controller.go:869] service infra/kube-lego-nginx does not have any active endpoints
W1116 14:34:20.128719       5 controller.go:1100] ssl certificate "default/echoserver-ingress-tls" does not exist in local store
172.20.42.71 - [172.20.42.71] - - [16/Nov/2017:14:34:20 +0000] "GET /.well-known/acme-challenge/_selftest HTTP/1.1" 503 213 "-" "Go-http-client/1.1" 138 0.000 [] - - - -

Logs from the lego pod contain a whole lot of lines like these:

time="2017-11-16T14:34:28Z" level=debug msg="testing reachability of http://echo.syntaxis.systems/.well-known/acme-challenge/_selftest" context=acme domain=echo.syntaxis.systems 
time="2017-11-16T14:34:28Z" level=debug msg="error while authorizing: reachability test failed: wrong status code '503'" context=acme domain=echo.syntaxis.systems 

I'm able to inspect the NGINX config with kubectl -n infra exec nginx-1248418661-f20j4 cat /etc/nginx/nginx.conf, which shows this:

upstream default-echoserver-8080 {
  server 100.96.2.10:8080 max_fails=0 fail_timeout=0;
  server 100.96.1.9:8080 max_fails=0 fail_timeout=0;
  [...]
}

server {
  server_name echo.syntaxis.systems ;
  listen 80;
      
  location /.well-known/acme-challenge {
    # No endpoints available for the request
    return 503;
    }
}

Seems like kube-lego isn't properly configured as backend in the NGINX config? I would expect the location ./well-known/acme-challenge block to have a proxy_pass to a kube-lego backend, but neither the backend nor the proxy-pass are there.

There is an Ingress for kube-lego-nginx:

And there's a Service with no External endpoints:

I dug into this a bit further and found out that for some reason, after a certain amount of time, the lego Pod is no longer associated with the kube-lego-nginx Service. To fix this, I manually recreate the Service with kubectl apply -f service.yaml:

kind: Service
apiVersion: v1
metadata:
  name: kube-lego-nginx
  namespace: infra
spec:
  selector:
    app: lego
  ports:
    - protocol: TCP
      port: 8080
      targetPort: 8080

As long as the lego Pod is available in the kube-lego-nginx service, the NGINX config contains the correct upstream entry for the ACME endpoints, and SSL certificates are generated properly. As soon as the Pod disappears from the kube-lego-nginx service, the 503 errors start returning in the lego logs.

Versions

• nginx-ingress-controller: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.9.0-beta.17
• kube-lego: jetstack/kube-lego:0.1.5

[marceldegraaf]

To be clear; this is what I'm seeing in the kube-lego-nginx service while getting the 503 errors:

Note how the list of Pods is empty. After running kubectl apply -f service.yaml as described above, I see this:

However, for some reason the lego Pod disappears again after a while, with no obvious reason as to why this happens.

[marceldegraaf]

@simonswine @munnerz sorry if you're not the right people to ping but I'm wondering if this issue is on the radar? It's preventing me from using kube-lego at the moment.

Thanks!

[deltcloud]

The service will be created automaticly, and the service selector is app: kube-lego. So the deployment you create should with label app: kube-lego ,so you can check the deployment's label

[marceldegraaf]

@deltcloud thanks for your reply. I'm afraid I don't really understand what you mean though – I'm a bit new to Kubernetes and the related terms. Can you elaborate on your answer? Thanks!

[deltcloud]

when you create the lego, you need not create the service by yourself, the deployment will create the service with select label """app: kube-lego""", so you should create deployment with """app: kube-lego""", not """app: lego"""

[deltcloud]

By the way, you can have a look at chart kube-lego (https://github.com/kubernetes/charts/tree/master/stable/kube-lego)