JWS verification error

[mxrlkn]

After getting certificates for about 45 domains, caddy suddenly stopped and I got this error:

[INFO] [mydomain.com] acme: Obtaining bundled SAN certificate
http: TLS handshake error from 127.0.0.1:59836: EOF
[ERROR][mydomain.com] failed to obtain certificate: acme: error: 400 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:malformed :: JWS verification error, url: (attempt 1/3; challenge=tls-alpn-01)
[INFO] [mydomain.com] acme: Obtaining bundled SAN certificate
[ERROR][mydomain.com] failed to obtain certificate: acme: error: 400 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:malformed :: JWS verification error, url: (attempt 2/3; challenge=tls-alpn-01)
[INFO] [mydomain.com] acme: Obtaining bundled SAN certificate
[ERROR][mydomain.com] failed to obtain certificate: acme: error: 400 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:malformed :: JWS verification error, url: (attempt 3/3; challenge=tls-alpn-01)
[INFO] [mydomain.com] acme: Obtaining bundled SAN certificate
[ERROR][mydomain.com] failed to obtain certificate: acme: error: 400 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:malformed :: JWS verification error, url: (attempt 1/3; challenge=http-01)
[INFO] [mydomain.com] acme: Obtaining bundled SAN certificate
[ERROR][mydomain.com] failed to obtain certificate: acme: error: 400 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:malformed :: JWS verification error, url: (attempt 2/3; challenge=http-01)
[INFO] [mydomain.com] acme: Obtaining bundled SAN certificate
[ERROR][mydomain.com] failed to obtain certificate: acme: error: 400 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:malformed :: JWS verification error, url: (attempt 3/3; challenge=http-01)
http: TLS handshake error from 152.115.135.58:55802: failed to obtain certificate: acme: error: 400 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:malformed :: JWS verification error, url:

Happens on all new domains I add.

I'm running caddy 1.0.3.

[ldez]

Hello,

I think the error is related to caddy, maybe you are using a corrupted private key.

[mholt]

Interesting, I haven't considered that possibility.

@ldez Is there any way for lego to check if a key is corrupted before trying to use it? For example, parse or validate it?

If the key is replaced, is there way to update an account's key with lego yet?

[mholt]

@ldez The private key and its associated reg resource is confirmed to be valid: https://caddy.community/t/acme-auto-ssl-suddenly-stopped-working/6147/31?u=matt

So there is still something afoot... let me know how you want to go about pinpointing this.

[cpu]

Do you have any logs of the JWS that doesn't validate?

[mxrlkn]

I've pasted all the logs at the top. I don't have anything else regarding the error.

[cpu]

Unfortunately the logs at the top don't contain the JWS object. Perhaps @mholt knows if it's possible to have Caddy log the JWS with a config change or whether it would require changes in Caddy or Lego's code to achieve.

[mholt]

JWS's are abstracted away -- Caddy (and CertMagic) doesn't touch them at all. The logs would have to be emitted from lego.

[mholt]

@mxrlkn Can you keep your account key and metadata handy so that this can continue to be debugged while you use another one in the meantime?

This is interesting, since it's not exactly kosher to share your private key to have others debug it... 😅 thanks for your patience.

@ldez where do you recommend adding logs for this?

[mxrlkn]

Yes. It's on my test setup which isn't that important 🙂

[ldez]

You check the private key in NewJWS and the alg in SignContent

Maybe it's related to the algorithms used to create the private key.

[mholt]

Thanks. Do you think lego could also add more logs in relevant parts of the challenge process so that we can see what the actual errors are?

[ldez]

for now and related to the logger behavior, it will be far too verbose and precise to have a real interest for the majority of users.

[mholt]

It's too verbose to emit logs when there are errors?

[ldez]

Sorry misread, no problem to log the errors.

In this case, I think we already logs the error, and put the private key in a logs seems unsafe.

I don't know what is the safe way to get more information in this case.

[ldez]

I have an idea to improve errors, stay tune.

[cpu]

put the private key in a logs seems unsafe.

To start with I think the JWS and the account public key needed to verify the JWS would be sufficient.

[mbardelmeijer]

We ran into the same issue with using Caddy 1.0.4. When we requested a new LetsEncrypt account certification requests went through again.

[christianflintrup]

Experienced the same thing using Caddy 1.0.4, too. Switched to a new LetsEncrypt user and it worked again.

[Ma27]

I experienced the same problem with nginx and creating a new LE account fixed the problem for me as well. Just wondering, is there any reasonable way in lego to catch this kind of error?

[m1cr0man]

Hi! I have found a way to reproduce this error. I have detailed the instructions here, in the context of NixOS, however the same instructions still apply running lego on its own (just change the paths).

It seems to happen when the account ID and the key in the keys folder are mismatched. Let's Encrypt makes a 1:1 relation with accounts and keys, as their documentation hints, and this can return the error people are seeing.

Would any of the lego devs know why this would happen, seemingly at random? Would lego be able to deal with this situation and correct the account ID automatically?

[ldez]

Would any of the lego devs know why this would happen, seemingly at random?

I don't see any reason, the random doesn't exist :wink: so we have to find the real reason behind that.

Would lego be able to deal with this situation and correct the account ID automatically?

The first step will be to detect the problem.