kube-lego icon indicating copy to clipboard operation
kube-lego copied to clipboard

Published 20 hours ago • verified publisher icon jetstack

Secrets not created

Open jwaldrip opened this issue 7 years ago • 11 comments

Secrets are not getting created. My logs keep looping over the following messages:

21:16:05.000
time="2017-01-30T04:16:05Z" level=info msg="creating new secret" context=secret name="cert-4437b0d78785d405c05f834894c5e734abe44a5f" namespace=apps
21:16:08.000
time="2017-01-30T04:16:08Z" level=info msg="creating new secret" context=secret name="cert-4437b0d78785d405c05f834894c5e734abe44a5f" namespace=apps
21:16:11.000
time="2017-01-30T04:16:11Z" level=info msg="creating new secret" context=secret name="cert-4437b0d78785d405c05f834894c5e734abe44a5f" namespace=apps
21:16:12.000
time="2017-01-30T04:16:12Z" level=info msg="creating new secret" context=secret name="cert-4437b0d78785d405c05f834894c5e734abe44a5f" namespace=apps
21:16:15.000
time="2017-01-30T04:16:15Z" level=info msg="creating new secret" context=secret name="cert-4437b0d78785d405c05f834894c5e734abe44a5f" namespace=apps

Any idea why this would be happening?

jwaldrip avatar Jan 30 '17 04:01 jwaldrip

yup, seeing the same issue too

rimusz avatar Jan 31 '17 15:01 rimusz

Seeing the same issue as well :-(

FourSigma avatar Feb 01 '17 08:02 FourSigma

For a brief moment I've thought I had the same issue. Please notice that:

  1. you need to properly configure your domain in ingress.yaml
  2. you need to point dns to the assigned IP, otherwise it let's encrypt cannot properly verify the domain

orian avatar Feb 06 '17 13:02 orian

I think that might be a problem, with the log level not high enough. Can you try running the kube-lego pod with debug flags:

   env:
        - name: LEGO_LOG_LEVEL
          value: debug

simonswine avatar Feb 06 '17 18:02 simonswine

@FourSigma @rimusz @jwaldrip

simonswine avatar Feb 06 '17 18:02 simonswine

Not sure if mine is the same issue, but it seems that the secret is getting created incorrectly, since after creating the secret, I get the following error:

Error while process certificate requests: Secret \"app-dev-tls\" is invalid: [data[tls.crt]: Required value, data[tls.key]: Required value]" context=kubelego

huysamen avatar Feb 17 '17 07:02 huysamen

@huysamen please enable debug logging and provide a bit more info (K8S objects, ...)

simonswine avatar Feb 17 '17 08:02 simonswine

So this is related to #77 and #62 I hit the same issue.

I activated debug, but not much more help:

time="2017-02-24T16:20:59Z" level=debug msg="worker: begin processing true" context=kubelego 
time="2017-02-24T16:20:59Z" level=info msg="ignoring as has no annotiation 'kubernetes.io/tls-acme'" context=ingress name=kube-lego-nginx namespace=nginx 
time="2017-02-24T16:20:59Z" level=debug msg=reset context=provider provider=gce 
time="2017-02-24T16:20:59Z" level=debug msg=finialize context=provider provider=gce 
time="2017-02-24T16:20:59Z" level=debug msg=reset context=provider provider=nginx 
time="2017-02-24T16:20:59Z" level=debug msg=finialize context=provider provider=nginx 
time="2017-02-24T16:21:00Z" level=info msg="process certificates requests for ingresses" context=kubelego 
time="2017-02-24T16:21:00Z" level=info msg="creating new secret" context=secret name=tls namespace=production 
time="2017-02-24T16:21:00Z" level=info msg="no cert associated with ingress" context="ingress_tls" name=app namespace=production 
time="2017-02-24T16:21:00Z" level=info msg="requesting certificate for ***domainname***" context="ingress_tls" name=app namespace=production 
time="2017-02-24T16:21:00Z" level=info msg="creating new secret" context=secret name=kube-lego-account namespace=nginx 
time="2017-02-24T16:21:00Z" level=info msg="creating new secret" context=secret name=tls namespace=production 
time="2017-02-24T16:21:01Z" level=error msg="Error while process certificate requests: Secret \"tls\" is invalid: [data[tls.crt]: Required value, data[tls.key]: Required value]" context=kubelego 
time="2017-02-24T16:21:01Z" level=debug msg="worker: done processing true" context=kubelego

Here is my ingress:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: app
  annotations:
    kubernetes.io/tls-acme: 'true'
    kubernetes.io/ingress.class: nginx
spec:
  tls:
    - secretName: tls
      hosts:
        - ***domainname***
  rules:
    - host: ***domainname***
      http:
        paths:
          - path: /
            backend:
              serviceName: app
              servicePort: 3000

I did one suggestion, I added one container to the lego pod, exec'ed inside it and run the following:

ping 8.8.8.8
wget https://acme-v01.api.letsencrypt.org

And it worked as expected. I couldn't try inside the lego container itself. (It reminds me unikernel :) )

For information, I'm running on GKE, and I installed everything with helm:

helm install --namespace nginx --name nginx stable/nginx-ingress
helm install --namespace nginx --name lego -f k8s/values-lego.yml stable/kube-lego

(The values are just the lego API endpoint and my email)

Is there anything I can do to help debug this? Thanks a lot for your work!

pierreozoux avatar Feb 24 '17 16:02 pierreozoux

Edit:

I found my issue:

here was the value of :

LEGO_URL: Lhttps://acme-v01.api.letsencrypt.org/directory

You got it? Yes, me too... Lost 2 hours...

It would be a nice to have to have it slightly more verbose :)

Everything is working on my side! Have a wonderful week-end!

pierreozoux avatar Feb 24 '17 16:02 pierreozoux

Old comment text. Click to expand Same issue here. But in my case some secrets get created, but others don't. ``` time="2017-03-27T21:37:40Z" level=info msg="kube-lego 0.1.3-d425b293 starting" context=kubelego time="2017-03-27T21:37:40Z" level=info msg="connected to kubernetes api v1.5.3" context=kubelego time="2017-03-27T21:37:40Z" level=debug msg="start watching ingress objects" context=kubelego time="2017-03-27T21:37:40Z" level=info msg="server listening on http://:8080/" context=acme time="2017-03-27T21:37:40Z" level=debug msg="CREATE ingress/example-test2/testapp-1" context=kubelego time="2017-03-27T21:37:40Z" level=debug msg="CREATE ingress/example-test2/testapp-2" context=kubelego time="2017-03-27T21:37:40Z" level=debug msg="CREATE ingress/example-test1/testapp-3" context=kubelego time="2017-03-27T21:37:40Z" level=debug msg="CREATE ingress/monitoring/examplenet-monitoring" context=kubelego time="2017-03-27T21:37:40Z" level=debug msg="worker: begin processing true" context=kubelego time="2017-03-27T21:37:40Z" level=debug msg="CREATE ingress/monitoring/examplenet-testapp-4" context=kubelego time="2017-03-27T21:37:40Z" level=info msg="ignoring as has no annotiation 'kubernetes.io/tls-acme'" context=ingress name=kube-lego-nginx namespace=kube-system time="2017-03-27T21:37:40Z" level=debug msg=reset context=provider provider=gce time="2017-03-27T21:37:40Z" level=debug msg=finialize context=provider provider=gce time="2017-03-27T21:37:40Z" level=debug msg=reset context=provider provider=nginx time="2017-03-27T21:37:40Z" level=debug msg=finialize context=provider provider=nginx time="2017-03-27T21:37:40Z" level=info msg="process certificates requests for ingresses" context=kubelego time="2017-03-27T21:37:40Z" level=info msg="cert expires in 80.9 days, no renewal needed" context="ingress_tls" expire_time=2017-06-16 19:24:00 +0000 UTC name=examplenet-monitoring namespace=testspace time="2017-03-27T21:37:40Z" level=info msg="no cert request needed" context="ingress_tls" name=examplenet-monitoring namespace=testspace time="2017-03-27T21:37:40Z" level=info msg="creating new secret" context=secret name=tls-net-example-testapp-4 namespace=testspace time="2017-03-27T21:37:40Z" level=info msg="no cert associated with ingress" context="ingress_tls" name=examplenet-testapp-4 namespace=testspace time="2017-03-27T21:37:40Z" level=info msg="requesting certificate for testapp-1.example.net" context="ingress_tls" name=examplenet-testapp-4 namespace=testspace time="2017-03-27T21:37:41Z" level=debug msg="testing reachablity of http://testapp-1.example.net/.well-known/acme-challenge/_selftest" context=acme domain=testapp-1.example.net time="2017-03-27T21:37:56Z" level=debug msg="error while authorizing: reachabily test failed: wrong status code '504'" context=acme domain=testapp-1.example.net time="2017-03-27T21:38:06Z" level=debug msg="testing reachablity of http://testapp-1.example.net/.well-known/acme-challenge/_selftest" context=acme domain=testapp-1.example.net time="2017-03-27T21:38:07Z" level=debug msg="got authorization: &{URI:https://acme-v01.api.letsencrypt.org/acme/challenge/[REMOVED]/[REMOVED] Status:valid Identifier:{Type: Value:} Challenges:[] Combinations:[]}" context=acme domain=testapp-1.example.net time="2017-03-27T21:38:07Z" level=info msg="authorization successful" context=acme domain=testapp-1.example.net time="2017-03-27T21:38:08Z" level=info msg="creating new secret" context=secret name=tls-net-example-test2-testapp-1 namespace=example-test2 time="2017-03-27T21:38:08Z" level=info msg="no cert associated with ingress" context="ingress_tls" name=testapp-1 namespace=example-test2 time="2017-03-27T21:38:08Z" level=info msg="requesting certificate for testapp-1.example-test2.net" context="ingress_tls" name=testapp-1 namespace=example-test2 time="2017-03-27T21:38:08Z" level=debug msg="testing reachablity of http://testapp-1.example-test2.net/.well-known/acme-challenge/_selftest" context=acme domain=testapp-1.example-test2.net time="2017-03-27T21:38:10Z" level=debug msg="got authorization: &{URI:https://acme-v01.api.letsencrypt.org/acme/challenge/[REMOVED]/[REMOVED] Status:valid Identifier:{Type: Value:} Challenges:[] Combinations:[]}" context=acme domain=testapp-1.example-test2.net time="2017-03-27T21:38:10Z" level=info msg="authorization successful" context=acme domain=testapp-1.example-test2.net time="2017-03-27T21:38:10Z" level=info msg="creating new secret" context=secret name=tls-net-example-test2-testapp-2 namespace=example-test2 time="2017-03-27T21:38:10Z" level=info msg="no cert associated with ingress" context="ingress_tls" name=testapp-2 namespace=example-test2 time="2017-03-27T21:38:10Z" level=info msg="requesting certificate for testapp-2.example-test2.net" context="ingress_tls" name=testapp-2 namespace=example-test2 time="2017-03-27T21:38:11Z" level=debug msg="testing reachablity of http://testapp-2.example-test2.net/.well-known/acme-challenge/_selftest" context=acme domain=testapp-2.example-test2.net time="2017-03-27T21:38:12Z" level=debug msg="got authorization: &{URI:https://acme-v01.api.letsencrypt.org/acme/challenge/[REMOVED]/[REMOVED] Status:valid Identifier:{Type: Value:} Challenges:[] Combinations:[]}" context=acme domain=testapp-2.example-test2.net time="2017-03-27T21:38:12Z" level=info msg="authorization successful" context=acme domain=testapp-2.example-test2.net time="2017-03-27T21:38:12Z" level=info msg="creating new secret" context=secret name=tls-net-example-testapp-3 namespace=example-test1 time="2017-03-27T21:38:12Z" level=info msg="no cert associated with ingress" context="ingress_tls" name=testapp-3 namespace=example-test1 time="2017-03-27T21:38:12Z" level=info msg="requesting certificate for testapp-3.example.net" context="ingress_tls" name=testapp-3 namespace=example-test1 time="2017-03-27T21:38:13Z" level=debug msg="testing reachablity of http://testapp-3.example.net/.well-known/acme-challenge/_selftest" context=acme domain=testapp-3.example.net time="2017-03-27T21:38:15Z" level=debug msg="responding to challenge request" basePath="/.well-known/acme-challenge" context=acme host=testapp-3.example.net token="[REMOVED]" time="2017-03-27T21:38:17Z" level=debug msg="got authorization: &{URI:https://acme-v01.api.letsencrypt.org/acme/challenge/[REMOVED]/[REMOVED] Status:valid Identifier:{Type: Value:} Challenges:[] Combinations:[]}" context=acme domain=testapp-3.example.net time="2017-03-27T21:38:17Z" level=info msg="authorization successful" context=acme domain=testapp-3.example.net time="2017-03-27T21:38:17Z" level=info msg="creating new secret" context=secret name=tls-net-example-testapp-3 namespace=example-test1 time="2017-03-27T21:38:17Z" level=info msg="no cert associated with ingress" context="ingress_tls" name=testapp-3 namespace=example-test1 time="2017-03-27T21:38:17Z" level=info msg="requesting certificate for testapp-3.example.net" context="ingress_tls" name=testapp-3 namespace=example-test1 time="2017-03-27T21:38:18Z" level=debug msg="testing reachablity of http://testapp-3.example.net/.well-known/acme-challenge/_selftest" context=acme domain=testapp-3.example.net time="2017-03-27T21:38:19Z" level=debug msg="responding to challenge request" basePath="/.well-known/acme-challenge" context=acme host=testapp-3.example.net token="[REMOVED]" time="2017-03-27T21:38:21Z" level=debug msg="got authorization: &{URI:https://acme-v01.api.letsencrypt.org/acme/challenge/[REMOVED]/[REMOVED] Status:valid Identifier:{Type: Value:} Challenges:[] Combinations:[]}" context=acme domain=testapp-3.example.net time="2017-03-27T21:38:21Z" level=info msg="authorization successful" context=acme domain=testapp-3.example.net time="2017-03-27T21:38:22Z" level=info msg="successfully got certificate: domains=[testapp-3.example.net] url=https://acme-v01.api.letsencrypt.org/acme/cert/[REMOVED]" context=acme time="2017-03-27T21:38:22Z" level=debug msg="certificate pem data:\n-----BEGIN CERTIFICATE-----\n[REMOVED]\n-----END CERTIFICATE-----\n" context=acme time="2017-03-27T21:38:22Z" level=info msg="creating new secret" context=secret name=tls-net-example-testapp-3 namespace=example-test1 time="2017-03-27T21:38:22Z" level=error msg="Error while process certificate requests: error getting certificate: 429 urn:acme:error:rateLimited: Error creating new cert :: Too many certificates already issued for: example.net, error getting certificate: 429 urn:acme:error:rateLimited: Error creating new cert :: Too many certificates already issued for: example-test2.net, error getting certificate: 429 urn:acme:error:rateLimited: Error creating new cert :: Too many certificates already issued for: example-test2.net, error getting certificate: 429 urn:acme:error:rateLimited: Error creating new cert :: Too many certificates already issued for: example.net" context=kubelego time="2017-03-27T21:38:22Z" level=debug msg="worker: done processing true" context=kubelego time="2017-03-27T21:38:22Z" level=debug msg="worker: begin processing true" context=kubelego time="2017-03-27T21:38:22Z" level=info msg="ignoring as has no annotiation 'kubernetes.io/tls-acme'" context=ingress name=kube-lego-nginx namespace=kube-system time="2017-03-27T21:38:22Z" level=debug msg=reset context=provider provider=gce time="2017-03-27T21:38:22Z" level=debug msg=finialize context=provider provider=gce time="2017-03-27T21:38:22Z" level=debug msg=reset context=provider provider=nginx time="2017-03-27T21:38:22Z" level=debug msg=finialize context=provider provider=nginx time="2017-03-27T21:38:22Z" level=info msg="process certificates requests for ingresses" context=kubelego time="2017-03-27T21:38:22Z" level=info msg="cert expires in 90.0 days, no renewal needed" context="ingress_tls" expire_time=2017-06-25 20:38:00 +0000 UTC name=testapp-3 namespace=example-test1 time="2017-03-27T21:38:22Z" level=info msg="no cert request needed" context="ingress_tls" name=testapp-3 namespace=example-test1 time="2017-03-27T21:38:22Z" level=info msg="cert expires in 80.9 days, no renewal needed" context="ingress_tls" expire_time=2017-06-16 19:24:00 +0000 UTC name=examplenet-monitoring namespace=testspace time="2017-03-27T21:38:22Z" level=info msg="no cert request needed" context="ingress_tls" name=examplenet-monitoring namespace=testspace time="2017-03-27T21:38:22Z" level=info msg="creating new secret" context=secret name=tls-net-example-testapp-4 namespace=testspace time="2017-03-27T21:38:22Z" level=info msg="no cert associated with ingress" context="ingress_tls" name=examplenet-testapp-4 namespace=testspace time="2017-03-27T21:38:22Z" level=info msg="requesting certificate for testapp-1.example.net" context="ingress_tls" name=examplenet-testapp-4 namespace=testspace time="2017-03-27T21:38:23Z" level=debug msg="testing reachablity of http://testapp-1.example.net/.well-known/acme-challenge/_selftest" context=acme domain=testapp-1.example.net time="2017-03-27T21:38:24Z" level=debug msg="got authorization: &{URI:https://acme-v01.api.letsencrypt.org/acme/challenge/[REMOVED]/[REMOVED] Status:valid Identifier:{Type: Value:} Challenges:[] Combinations:[]}" context=acme domain=testapp-1.example.net time="2017-03-27T21:38:24Z" level=info msg="authorization successful" context=acme domain=testapp-1.example.net time="2017-03-27T21:38:24Z" level=info msg="creating new secret" context=secret name=tls-net-example-test2-testapp-1 namespace=example-test2 time="2017-03-27T21:38:24Z" level=info msg="no cert associated with ingress" context="ingress_tls" name=testapp-1 namespace=example-test2 time="2017-03-27T21:38:24Z" level=info msg="requesting certificate for testapp-1.example-test2.net" context="ingress_tls" name=testapp-1 namespace=example-test2 time="2017-03-27T21:38:25Z" level=debug msg="testing reachablity of http://testapp-1.example-test2.net/.well-known/acme-challenge/_selftest" context=acme domain=testapp-1.example-test2.net time="2017-03-27T21:38:26Z" level=debug msg="got authorization: &{URI:https://acme-v01.api.letsencrypt.org/acme/challenge/[REMOVED]/[REMOVED] Status:valid Identifier:{Type: Value:} Challenges:[] Combinations:[]}" context=acme domain=testapp-1.example-test2.net time="2017-03-27T21:38:26Z" level=info msg="authorization successful" context=acme domain=testapp-1.example-test2.net time="2017-03-27T21:38:27Z" level=info msg="creating new secret" context=secret name=tls-net-example-test2-testapp-2 namespace=example-test2 time="2017-03-27T21:38:27Z" level=info msg="no cert associated with ingress" context="ingress_tls" name=testapp-2 namespace=example-test2 time="2017-03-27T21:38:27Z" level=info msg="requesting certificate for testapp-2.example-test2.net" context="ingress_tls" name=testapp-2 namespace=example-test2 time="2017-03-27T21:38:27Z" level=debug msg="testing reachablity of http://testapp-2.example-test2.net/.well-known/acme-challenge/_selftest" context=acme domain=testapp-2.example-test2.net time="2017-03-27T21:38:28Z" level=debug msg="got authorization: &{URI:https://acme-v01.api.letsencrypt.org/acme/challenge/[REMOVED]/[REMOVED] Status:valid Identifier:{Type: Value:} Challenges:[] Combinations:[]}" context=acme domain=testapp-2.example-test2.net time="2017-03-27T21:38:28Z" level=info msg="authorization successful" context=acme domain=testapp-2.example-test2.net time="2017-03-27T21:38:29Z" level=info msg="creating new secret" context=secret name=tls-net-example-testapp-3 namespace=example-test1 time="2017-03-27T21:38:29Z" level=info msg="no cert associated with ingress" context="ingress_tls" name=testapp-3 namespace=example-test1 time="2017-03-27T21:38:29Z" level=info msg="requesting certificate for testapp-3.example.net" context="ingress_tls" name=testapp-3 namespace=example-test1 time="2017-03-27T21:38:29Z" level=debug msg="testing reachablity of http://testapp-3.example.net/.well-known/acme-challenge/_selftest" context=acme domain=testapp-3.example.net time="2017-03-27T21:38:30Z" level=debug msg="got authorization: &{URI:https://acme-v01.api.letsencrypt.org/acme/challenge/[REMOVED]/[REMOVED] Status:valid Identifier:{Type: Value:} Challenges:[] Combinations:[]}" context=acme domain=testapp-3.example.net time="2017-03-27T21:38:30Z" level=info msg="authorization successful" context=acme domain=testapp-3.example.net time="2017-03-27T21:38:31Z" level=error msg="Error while process certificate requests: error getting certificate: 429 urn:acme:error:rateLimited: Error creating new cert :: Too many certificates already issued for: example.net, error getting certificate: 429 urn:acme:error:rateLimited: Error creating new cert :: Too many certificates already issued for: example-test2.net, error getting certificate: 429 urn:acme:error:rateLimited: Error creating new cert :: Too many certificates already issued for: example-test2.net, error getting certificate: 429 urn:acme:error:rateLimited: Error creating new cert :: Too many certificates already issued for: example.net" context=kubelego time="2017-03-27T21:38:31Z" level=debug msg="worker: done processing true" context=kubelego time="2017-03-27T21:38:31Z" level=debug msg="worker: begin processing true" context=kubelego time="2017-03-27T21:38:31Z" level=info msg="ignoring as has no annotiation 'kubernetes.io/tls-acme'" context=ingress name=kube-lego-nginx namespace=kube-system time="2017-03-27T21:38:31Z" level=debug msg=reset context=provider provider=gce time="2017-03-27T21:38:31Z" level=debug msg=finialize context=provider provider=gce time="2017-03-27T21:38:31Z" level=debug msg=reset context=provider provider=nginx time="2017-03-27T21:38:31Z" level=debug msg=finialize context=provider provider=nginx ``` (The original domains have been replaced with mostly `example*.net`. The domains are valid and when using the `certbot` manually I got working certificates out)

It seems that because one certificate failed, kube-lego went into a loop and hit the rate limit in my case.

galexrt avatar Mar 27 '17 22:03 galexrt

I had the same problem.. forgot to update my email address in the template.. a better error would have saved me some time. :)

andrejpk avatar Dec 08 '17 18:12 andrejpk