nginx-proxy-manager icon indicating copy to clipboard operation
nginx-proxy-manager copied to clipboard

Published 20 hours ago • verified publisher icon NginxProxyManager

Auto SSL certificate renewal failing using AWS Route53 DNS

Open coreylane opened this issue 2 years ago • 17 comments

Checklist

  • Have you pulled and found the error with jc21/nginx-proxy-manager:latest docker image?
    • Yes
  • Are you sure you're not using someone else's docker image?
    • Yes
  • Have you searched for similar issues (both open and closed)?
    • Yes

Describe the bug Automatic SSL certificate renewal fails when using AWS Route53 DNS challenge, but renewing manually through NPM console works fine.

Based on initial review of the logs, it looks like the automated renewal command is not setting the required "AWS_CONFIG_FILE" environment variable:

[3/8/2022] [6:02:17 AM] [SSL      ] › ✖  error     Error: Command failed: certbot renew --non-interactive --quiet --config "/etc/letsencrypt.ini" --preferred-challenges "dns,http" --disable-hook-validation  
Failed to renew certificate npm-12 with error: Unable to locate credentials

Nginx Proxy Manager Version v2.9.16

To Reproduce Steps to reproduce the behavior:

  1. Creates Let's Encrypt certificate using AWS Route53 DNS challenge
  2. As certificate expiry date nears, the renewals fail

Expected behavior SSL certificate is renewed as the expiry date approaches

Operating System Unraid 6.10.0-rc2 Docker version 20.10.9, build c2ea9bc

Additional context Logs

[3/8/2022] [5:54:15 AM] [SSL      ] › ℹ  info      Renewing SSL certs close to expiry...
[3/8/2022] [6:02:17 AM] [SSL      ] › ✖  error     Error: Command failed: certbot renew --non-interactive --quiet --config "/etc/letsencrypt.ini" --preferred-challenges "dns,http" --disable-hook-validation  
Failed to renew certificate npm-12 with error: Unable to locate credentials
To use certbot-dns-route53, configure credentials as described at https://boto3.readthedocs.io/en/latest/guide/configuration.html#best-practices-for-configuring-credentials and add the necessary permissions for Route53 access.
AMAll renewals failed. The following certificates could not be renewed:
/etc/letsencrypt/live/npm-12/fullchain.pem (failure)
1 renew failure(s), 0 parse failure(s)

   at ChildProcess.exithandler (node:child_process:399:12)
   at ChildProcess.emit (node:events:520:28)
   at maybeClose (node:internal/child_process:1092:16)
   at Process.ChildProcess._handle.onexit (node:internal/child_process:302:5)

Workaround: Renewing through the console works as expected

[3/8/2022] [9:49:46 AM] [SSL      ] › ℹ  info      Renewing Let'sEncrypt certificates via Route 53 (Amazon) for Cert #12: *.darktower.one, darktower.one
[3/8/2022] [9:49:46 AM] [SSL      ] › ℹ  info      Command: AWS_CONFIG_FILE='/etc/letsencrypt/credentials/credentials-12' certbot renew --config "/etc/letsencrypt.ini" --cert-name "npm-12" --disable-hook-validation --no-random-sleep-on-renew 
[3/8/2022] [9:50:20 AM] [SSL      ] › ℹ  info      - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/npm-12.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Renewing an existing certificate for *.darktower.one and darktower.one

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Congratulations, all renewals succeeded: 
  /etc/letsencrypt/live/npm-12/fullchain.pem (success)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

coreylane avatar Mar 08 '22 16:03 coreylane

@coreylane I am having the same issue. Would you be kind enough to share the manual command, please? I could not find it. Thank you.

chaddm avatar Apr 22 '22 15:04 chaddm

@chaddm Try renewing through the NPM GUI, under SSL Certificates -> Renew Now

Screen Shot 2022-04-22 at 10 35 26 AM

coreylane avatar Apr 22 '22 15:04 coreylane

@coreylane Thank you. It worked correctly.

chaddm avatar Apr 22 '22 22:04 chaddm

Same issue. Renewing manually through GUI fixed the issue which was the renewal at 4/25/2022 1:01AM

`[4/24/2022] [10:41:18 PM] [SSL ] › ℹ info Renewing SSL certs close to expiry... [4/24/2022] [10:41:19 PM] [IP Ranges] › ℹ info Fetching IP Ranges from online services... [4/24/2022] [10:41:19 PM] [IP Ranges] › ℹ info Fetching https://ip-ranges.amazonaws.com/ip-ranges.json [4/24/2022] [10:41:19 PM] [IP Ranges] › ℹ info Fetching https://www.cloudflare.com/ips-v4 [4/24/2022] [10:41:19 PM] [IP Ranges] › ℹ info Fetching https://www.cloudflare.com/ips-v6 [4/24/2022] [10:41:19 PM] [Nginx ] › ℹ info Reloading Nginx [4/24/2022] [10:41:20 PM] [Setup ] › ℹ info Logrotate completed. [4/24/2022] [10:48:23 PM] [SSL ] › ✖ error Error: Command failed: certbot renew --non-interactive --quiet --config "/etc/letsencrypt.ini" --preferred-challenges "dns,http" --disable-hook-validation
Failed to renew certificate npm-1 with error: Unable to locate credentials To use certbot-dns-route53, configure credentials as described at https://boto3.readthedocs.io/en/latest/guide/configuration.html#best-practices-for-configuring-credentials and add the necessary permissions for Route53 access. All renewals failed. The following certificates could not be renewed: /etc/letsencrypt/live/npm-1/fullchain.pem (failure) 1 renew failure(s), 0 parse failure(s) at ChildProcess.exithandler (node:child_process:399:12) at ChildProcess.emit (node:events:526:28) at maybeClose (node:internal/child_process:1092:16) at Process.ChildProcess._handle.onexit (node:internal/child_process:302:5) [4/24/2022] [11:41:18 PM] [SSL ] › ℹ info Renewing SSL certs close to expiry... [4/24/2022] [11:43:16 PM] [SSL ] › ✖ error Error: Command failed: certbot renew --non-interactive --quiet --config "/etc/letsencrypt.ini" --preferred-challenges "dns,http" --disable-hook-validation
Failed to renew certificate npm-1 with error: Unable to locate credentials To use certbot-dns-route53, configure credentials as described at https://boto3.readthedocs.io/en/latest/guide/configuration.html#best-practices-for-configuring-credentials and add the necessary permissions for Route53 access. All renewals failed. The following certificates could not be renewed: /etc/letsencrypt/live/npm-1/fullchain.pem (failure) 1 renew failure(s), 0 parse failure(s) at ChildProcess.exithandler (node:child_process:399:12) at ChildProcess.emit (node:events:526:28) at maybeClose (node:internal/child_process:1092:16) at Process.ChildProcess._handle.onexit (node:internal/child_process:302:5) [4/25/2022] [12:41:18 AM] [SSL ] › ℹ info Renewing SSL certs close to expiry... [4/25/2022] [12:41:49 AM] [SSL ] › ✖ error Error: Command failed: certbot renew --non-interactive --quiet --config "/etc/letsencrypt.ini" --preferred-challenges "dns,http" --disable-hook-validation
Failed to renew certificate npm-1 with error: Unable to locate credentials To use certbot-dns-route53, configure credentials as described at https://boto3.readthedocs.io/en/latest/guide/configuration.html#best-practices-for-configuring-credentials and add the necessary permissions for Route53 access. All renewals failed. The following certificates could not be renewed: /etc/letsencrypt/live/npm-1/fullchain.pem (failure) 1 renew failure(s), 0 parse failure(s) at ChildProcess.exithandler (node:child_process:399:12) at ChildProcess.emit (node:events:526:28) at maybeClose (node:internal/child_process:1092:16) at Process.ChildProcess._handle.onexit (node:internal/child_process:302:5) Connection Error: Error: read ECONNRESET [4/25/2022] [1:01:25 AM] [Express ] › ⚠ warning connect ECONNREFUSED 172.16.0.3:3306 [4/25/2022] [1:01:31 AM] [SSL ] › ℹ info Renewing Let'sEncrypt certificates via Route 53 (Amazon) for Cert #1: *.NOTMYDOMAIN.com [4/25/2022] [1:01:31 AM] [SSL ] › ℹ info Command: AWS_CONFIG_FILE='/etc/letsencrypt/credentials/credentials-1' certbot renew --config "/etc/letsencrypt.ini" --cert-name "npm-1" --disable-hook-validation --no-random-sleep-on-renew [4/25/2022] [1:02:15 AM] [SSL ] › ℹ info - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Processing /etc/letsencrypt/renewal/npm-1.conf

Renewing an existing certificate for *.NOTMYDOMAIN.com

Congratulations, all renewals succeeded: /etc/letsencrypt/live/npm-1/fullchain.pem (success)

                                                                              • -`

nmbgeek avatar Apr 25 '22 01:04 nmbgeek

I can replicate this issue. Works manually via the GUI.

matthew-larner avatar May 17 '22 08:05 matthew-larner

I have the same issue. I'm unable to verify the logs (it's been a week or two since the last manual renewal) but symptoms are the same. Let's Encrypt via Route53 DNS, get e-mail from Let's Encrypt that the cert was not automatically renewed, manual renewal saves the day.

jonathanjuursema avatar Jun 08 '22 12:06 jonathanjuursema

I have the same Issue, Automatic renewal doesn't work, manual renewal does.

Below a snippet from the docker log:

Auto renewal

2022-06-17T09:47:57.071050992Z [6/17/2022] [9:47:57 AM] [SSL      ] › ℹ  info      Renewing SSL certs close to expiry...
2022-06-17T09:57:47.308169768Z [6/17/2022] [9:57:47 AM] [SSL      ] › ✖  error     Error: Command failed: certbot renew --non-interactive --quiet --config "/etc/letsencrypt.ini" --preferred-challenges "dns,http" --disable-hook-validation  
2022-06-17T09:57:47.309667756Z Failed to renew certificate npm-10 with error: Unable to locate credentials
2022-06-17T09:57:47.309676328Z To use certbot-dns-route53, configure credentials as described at https://boto3.readthedocs.io/en/latest/guide/configuration.html#best-practices-for-configuring-credentials and add the necessary permissions for Route53 access.
2022-06-17T09:57:47.309679394Z Failed to renew certificate npm-11 with error: Unable to locate credentials
2022-06-17T09:57:47.309687161Z To use certbot-dns-route53, configure credentials as described at https://boto3.readthedocs.io/en/latest/guide/configuration.html#best-practices-for-configuring-credentials and add the necessary permissions for Route53 access.
2022-06-17T09:57:47.309689779Z Failed to renew certificate npm-12 with error: Unable to locate credentials
2022-06-17T09:57:47.309692250Z To use certbot-dns-route53, configure credentials as described at https://boto3.readthedocs.io/en/latest/guide/configuration.html#best-practices-for-configuring-credentials and add the necessary permissions for Route53 access.
2022-06-17T09:57:47.309694859Z Failed to renew certificate npm-13 with error: Unable to locate credentials
2022-06-17T09:57:47.309697339Z To use certbot-dns-route53, configure credentials as described at https://boto3.readthedocs.io/en/latest/guide/configuration.html#best-practices-for-configuring-credentials and add the necessary permissions for Route53 access.
2022-06-17T09:57:47.309699886Z Failed to renew certificate npm-14 with error: Unable to locate credentials
2022-06-17T09:57:47.309702216Z To use certbot-dns-route53, configure credentials as described at https://boto3.readthedocs.io/en/latest/guide/configuration.html#best-practices-for-configuring-credentials and add the necessary permissions for Route53 access.
2022-06-17T09:57:47.309704851Z Failed to renew certificate npm-15 with error: Unable to locate credentials
2022-06-17T09:57:47.309707201Z To use certbot-dns-route53, configure credentials as described at https://boto3.readthedocs.io/en/latest/guide/configuration.html#best-practices-for-configuring-credentials and add the necessary permissions for Route53 access.
2022-06-17T09:57:47.309712085Z Failed to renew certificate npm-2 with error: Unable to locate credentials
2022-06-17T09:57:47.309714471Z To use certbot-dns-route53, configure credentials as described at https://boto3.readthedocs.io/en/latest/guide/configuration.html#best-practices-for-configuring-credentials and add the necessary permissions for Route53 access.
2022-06-17T09:57:47.309717083Z Failed to renew certificate npm-3 with error: Unable to locate credentials
2022-06-17T09:57:47.309719389Z To use certbot-dns-route53, configure credentials as described at https://boto3.readthedocs.io/en/latest/guide/configuration.html#best-practices-for-configuring-credentials and add the necessary permissions for Route53 access.
2022-06-17T09:57:47.309722585Z Failed to renew certificate npm-4 with error: Unable to locate credentials
2022-06-17T09:57:47.309726014Z To use certbot-dns-route53, configure credentials as described at https://boto3.readthedocs.io/en/latest/guide/configuration.html#best-practices-for-configuring-credentials and add the necessary permissions for Route53 access.
2022-06-17T09:57:47.309729790Z Failed to renew certificate npm-5 with error: Unable to locate credentials
2022-06-17T09:57:47.309733391Z To use certbot-dns-route53, configure credentials as described at https://boto3.readthedocs.io/en/latest/guide/configuration.html#best-practices-for-configuring-credentials and add the necessary permissions for Route53 access.
2022-06-17T09:57:47.309737128Z Failed to renew certificate npm-6 with error: Unable to locate credentials
2022-06-17T09:57:47.309740634Z To use certbot-dns-route53, configure credentials as described at https://boto3.readthedocs.io/en/latest/guide/configuration.html#best-practices-for-configuring-credentials and add the necessary permissions for Route53 access.
2022-06-17T09:57:47.309746021Z All renewals failed. The following certificates could not be renewed:
2022-06-17T09:57:47.309748402Z   /etc/letsencrypt/live/npm-10/fullchain.pem (failure)
2022-06-17T09:57:47.309751068Z   /etc/letsencrypt/live/npm-11/fullchain.pem (failure)
2022-06-17T09:57:47.309753448Z   /etc/letsencrypt/live/npm-12/fullchain.pem (failure)
2022-06-17T09:57:47.309755686Z   /etc/letsencrypt/live/npm-13/fullchain.pem (failure)
2022-06-17T09:57:47.309758011Z   /etc/letsencrypt/live/npm-14/fullchain.pem (failure)
2022-06-17T09:57:47.309760241Z   /etc/letsencrypt/live/npm-15/fullchain.pem (failure)
2022-06-17T09:57:47.309762491Z   /etc/letsencrypt/live/npm-2/fullchain.pem (failure)
2022-06-17T09:57:47.309764835Z   /etc/letsencrypt/live/npm-3/fullchain.pem (failure)
2022-06-17T09:57:47.309767558Z   /etc/letsencrypt/live/npm-4/fullchain.pem (failure)
2022-06-17T09:57:47.309769795Z   /etc/letsencrypt/live/npm-5/fullchain.pem (failure)
2022-06-17T09:57:47.309772019Z   /etc/letsencrypt/live/npm-6/fullchain.pem (failure)
2022-06-17T09:57:47.309774393Z 11 renew failure(s), 0 parse failure(s)
2022-06-17T09:57:47.309776639Z 
2022-06-17T09:57:47.309778811Z     at ChildProcess.exithandler (node:child_process:399:12)
2022-06-17T09:57:47.309781098Z     at ChildProcess.emit (node:events:520:28)
2022-06-17T09:57:47.309783608Z     at maybeClose (node:internal/child_process:1092:16)
2022-06-17T09:57:47.309785934Z     at Process.ChildProcess._handle.onexit (node:internal/child_process:302:5)

Manual renewal

2022-06-17T11:32:30.627108035Z [6/17/2022] [11:32:30 AM] [SSL      ] › ℹ  info      Renewing Let'sEncrypt certificates via Route 53 (Amazon) for Cert #10: *.domain.tld, domain.tld
2022-06-17T11:32:30.628592566Z [6/17/2022] [11:32:30 AM] [SSL      ] › ℹ  info      Command: AWS_CONFIG_FILE='/etc/letsencrypt/credentials/credentials-10' certbot renew --config "/etc/letsencrypt.ini" --cert-name "npm-10" --disable-hook-validation --no-random-sleep-on-renew 
2022-06-17T11:33:34.021023256Z [6/17/2022] [11:33:34 AM] [SSL      ] › ℹ  info      - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2022-06-17T11:33:34.021050555Z Processing /etc/letsencrypt/renewal/npm-10.conf
2022-06-17T11:33:34.021055186Z - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2022-06-17T11:33:34.021059019Z Renewing an existing certificate for *.domain.tld and domain.tld
2022-06-17T11:33:34.021062747Z 
2022-06-17T11:33:34.021066108Z - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2022-06-17T11:33:34.021069739Z Congratulations, all renewals succeeded: 
2022-06-17T11:33:34.021072948Z   /etc/letsencrypt/live/npm-10/fullchain.pem (success)
2022-06-17T11:33:34.021076289Z - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

NOTE: I've anonymized the domain names, but the structure is the same

emkookmer avatar Jun 17 '22 11:06 emkookmer

I'm still experiencing this issue on the latest version. Is anyone able to fix this issue?

matthew-larner avatar Aug 20 '22 11:08 matthew-larner

I'm still experiencing this issue on the latest version. Is anyone able to fix this issue?

@matthew-larner Renewing through the NPM GUI works fine for me, I posted a screenshot in an earlier comment. If this doesn't work you may have an issue with your AWS credentials. Perhaps the User doesn't have required permissions to route53. What do your logs say?

Required AWS permissions:

route53:ChangeResourceRecordSets
route53:ListHostedZones
route53:GetChange

coreylane avatar Aug 25 '22 21:08 coreylane

@coreylane renewing via the GUI works fine. Automatic renewals don't though. Since I have 50+ entries it's a real pain having to renew manually.

How much effort is required to fix the auto renewals?

Thanks for you help on this.

matthew-larner avatar Aug 29 '22 01:08 matthew-larner

It looks like the code use for auto renewal isnt using the same renewal functions as the UI, but a simple command to certbot

https://github.com/NginxProxyManager/nginx-proxy-manager/blob/14b889a85f2f8af9a13ed6122f5a0a91d64ecc36/backend/internal/certificate.js#L42-L106

It should use this function to renew the certificates https://github.com/NginxProxyManager/nginx-proxy-manager/blob/14b889a85f2f8af9a13ed6122f5a0a91d64ecc36/backend/internal/certificate.js#L999-L1028

I've "quick" fixed my installation by hardcoding the aws credentials:

const cmd = certbotCommand + ' renew --non-interactive --quiet ' +
        '--config "' + letsencryptConfig + '" ' +
        '--preferred-challenges "dns,http" ' +
        '--disable-hook-validation ' +
        (letsencryptStaging ? '--staging' : '');

let mainCmd = 'AWS_CONFIG_FILE=\'/etc/letsencrypt/credentials/credentials-3\' '+cmd

return utils.exec(/*cmd*/mainCmd)
        .then((result) => {

emkookmer avatar Aug 30 '22 09:08 emkookmer

@jc21 if I can push a fix for this would you support a PR?

matthew-larner avatar Sep 03 '22 04:09 matthew-larner

Still having this issue. I was able to resolve by passing the credentials as an environment variable in docker-compose:

environment: - AWS_CONFIG_FILE=/etc/letsencrypt/credentials/credentials-1

miztertea avatar Nov 04 '22 13:11 miztertea

Just to thank @miztertea for contributing a solution, and to confirm that the env variable solved the problem for me as well.

jangeador avatar Nov 08 '22 18:11 jangeador

How would I handle this for an install that has multiple credential files? I have 9 domains requiring certificates with all separate AWS keys. credentials-11, credentials-13...etc

mkzimms avatar Dec 02 '22 16:12 mkzimms

Unfortunately this workaround will only help with one credential file. I suppose you could create one IAM user that has access to all of your domains. As long as its the same IAM user, it won't matter which cred file you specify as they would all be the same access/secret pair

miztertea avatar Dec 02 '22 18:12 miztertea

Unfortunately this workaround will only help with one credential file. I suppose you could create one IAM user that has access to all of your domains. As long as its the same IAM user, it won't matter which cred file you specify as they would all be the same access/secret pair

This still would only work from one AWS tenant/account

emkookmer avatar Dec 09 '22 08:12 emkookmer

You can also pass the credentials directly to the certbot by setting two environment vars in your docker-compose file: AWS_ACCESS_KEY_ID: "AAAAAAAAAAAAAAAAAAAAA" AWS_SECRET_ACCESS_KEY: "BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB"

Ich-Eben avatar Sep 26 '23 15:09 Ich-Eben

That is excellent @Ich-Eben. Thanks!

jangeador avatar Sep 26 '23 16:09 jangeador

I implemented @emkookmer's suggestion in #3392

stevecrozz avatar Dec 15 '23 00:12 stevecrozz

It looks like the code use for auto renewal isnt using the same renewal functions as the UI, but a simple command to certbot

https://github.com/NginxProxyManager/nginx-proxy-manager/blob/14b889a85f2f8af9a13ed6122f5a0a91d64ecc36/backend/internal/certificate.js#L42-L106

It should use this function to renew the certificates

https://github.com/NginxProxyManager/nginx-proxy-manager/blob/14b889a85f2f8af9a13ed6122f5a0a91d64ecc36/backend/internal/certificate.js#L999-L1028

I've "quick" fixed my installation by hardcoding the aws credentials:

const cmd = certbotCommand + ' renew --non-interactive --quiet ' +
        '--config "' + letsencryptConfig + '" ' +
        '--preferred-challenges "dns,http" ' +
        '--disable-hook-validation ' +
        (letsencryptStaging ? '--staging' : '');

let mainCmd = 'AWS_CONFIG_FILE=\'/etc/letsencrypt/credentials/credentials-3\' '+cmd

return utils.exec(/*cmd*/mainCmd)
        .then((result) => {

Just came to thank you for your attention to detail here <3 Thank you!

sassy-xx avatar Apr 02 '24 09:04 sassy-xx