Can generate initial certificate, but silently fails to renew

[wisq]

trafficstars

This has happened a number of times now — as far as I'm aware, certbot-s3front has never successfully renewed my CloudFront certificate, even though it can deploy the initial cert just fine (e.g. if I delete the renewals/xxx.conf file and re-run the initial certbot command).

If I force a renewal:

% sudo /usr/bin/chpst -e /etc/letsencrypt/env.d -- certbot --force-renewal renew --cert-name i.wisq.net
Saving debug log to /var/log/letsencrypt/letsencrypt.log

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/i.wisq.net.conf
-------------------------------------------------------------------------------
Plugins selected: Authenticator certbot-s3front:auth, Installer certbot-s3front:installer
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for i.wisq.net
Found credentials in environment variables.
Starting new HTTPS connection (1): s3.amazonaws.com
Waiting for verification...
Cleaning up challenges
Starting new HTTPS connection (1): s3.amazonaws.com

-------------------------------------------------------------------------------
new certificate deployed with reload of certbot-s3front:installer server;
fullchain is /etc/letsencrypt/live/i.wisq.net/fullchain.pem
-------------------------------------------------------------------------------

-------------------------------------------------------------------------------

Congratulations, all renewals succeeded. The following certs have been renewed:
  /etc/letsencrypt/live/i.wisq.net/fullchain.pem (success)
-------------------------------------------------------------------------------```

The cert on disk is updated, but the actual cert remains unchanged in AWS:

% sudo /usr/bin/chpst -e /etc/letsencrypt/env.d -- aws iam list-server-certificates
{
    "ServerCertificateMetadataList": [
        {
            "Path": "/cloudfront/letsencrypt/",
            "Expiration": "2018-06-26T17:12:11Z",
            "ServerCertificateId": "ASCAIYHLWDV4CDQU5QFU2",
            "UploadDate": "2018-03-28T18:12:11Z",
            "Arn": "arn:aws:iam::067917669805:server-certificate/cloudfront/letsencrypt/le-i.wisq.net-1522260731",
            "ServerCertificateName": "le-i.wisq.net-1522260731"
        }
    ]
}

I'm using the sample AWS policy linked in the readme.

Anything I can do to diagnose this further?

[wisq]

Since my cert was about to expire, I went and deleted the cert and recreated it, and as usual, it worked fine on the initial create:

% sudo chpst -e /etc/letsencrypt/env.d -- certbot --agree-tos -a certbot-s3front:auth --certbot-s3front:auth-s3-bucket i.wisq.net -i certbot-s3front:installer --certbot-s3front:installer-cf-distribution-id E2L7B2VHN1HT68 -d i.wisq.net
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator certbot-s3front:auth, Installer certbot-s3front:installer
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for i.wisq.net
Found credentials in environment variables.
Starting new HTTPS connection (1): s3.amazonaws.com
Waiting for verification...
Cleaning up challenges
Starting new HTTPS connection (1): s3.amazonaws.com
Starting new HTTPS connection (1): iam.amazonaws.com
Starting new HTTPS connection (1): cloudfront.amazonaws.com

-------------------------------------------------------------------------------
Congratulations! You have successfully enabled https://i.wisq.net

You should test your configuration at:
https://www.ssllabs.com/ssltest/analyze.html?d=i.wisq.net
-------------------------------------------------------------------------------

Notably, there are extra HTTPS connections to iam.amazonaws.com and cloudfront.amazonaws.com, which I assume are the actual "let's update the cert" calls. I still have no idea why it doesn't make these calls on a renewal request.

[dlapiduz]

@wisq can you please try the new version? I just pushed a new version that supports the renew motions.

[wisq]

Hm, getting closer. Just tried this with 0.4.1, and it does seem to upload the new certificate (judging from the timestamps). However, it does not seem to update the CloudFront distribution to use the new cert (like it does when I first install a cert).

[selu]

I also tried version 0.4.1 and the renewal does not work because from renew_deploy only deploy_cert is called and save does not and only save updates the CloudFront distribution. However, I think, the code, what moved into save should be kept in deploy_cert, because it's simply part of the deploy in this case.

[shadow-light]

I also just discovered that my certs are being renewed but not deployed. So running renew reports everything as fine as the certs are not outdated, but my sites go down because the new version hasn't been uploaded/enabled in CF.

[ajorg]

I spent some time on this today as it was time to renew again. I'm sorry I haven't made time yet to actually work up a patch for the plugin, but here's a breakdown of what's still going on.

The certificate gets renewed and uploaded to IAM, but the CloudFront distribution config doesn't get updated. The new certificate ID is known:

$ grep -F ServerCertificateId ~/certbot/log/letsencrypt.log
b'<UploadServerCertificateResponse xmlns="https://iam.amazonaws.com/doc/2010-05-08/">\n  <UploadServerCertificateResult>\n    <ServerCertificateMetadata>\n      <Path>/cloudfront/letsencrypt/</Path>\n      <UploadDate>2019-11-09T17:57:26Z</UploadDate>\n      <Expiration>2020-02-07T16:57:24Z</Expiration>\n      <ServerCertificateName>le-andrew.jorgensenfamily.us-1573322246</ServerCertificateName>\n      <Arn>arn:aws:iam::246745595609:server-certificate/cloudfront/letsencrypt/le-andrew.jorgensenfamily.us-1573322246</Arn>\n      <ServerCertificateId>ASCATS4ZO73M6ISFBOSPD</ServerCertificateId>\n    </ServerCertificateMetadata>\n  </UploadServerCertificateResult>\n  <ResponseMetadata>\n    <RequestId>6cb46a72-8115-409d-8566-82226dec944e</RequestId>\n  </ResponseMetadata>\n</UploadServerCertificateResponse>\n'

But the distribution continues to use the old certificate ID:

$ aws cloudfront get-distribution-config --id EODMNMAEHV4AH | jq -r .DistributionConfig.ViewerCertificate.IAMCertificateId
ASCATS4ZO73MQGKXGCWT3

Here's what I've been doing to deploy the certificate (licensed to whomever under MIT No Attribution):

# Copyright (c) Andrew Jorgensen. All rights reserved.
# SPDX-License-Identifier: MIT-0

# Requires jq and assumes AWS CLI --output json

DISTRIBUTION_ID="<YOUR DISTRIBUTION ID>"
DISTRIBUTION_CONFIG="$(
  aws cloudfront get-distribution-config \
    --id "${DISTRIBUTION_ID}" \
    | jq -c .)"

# FRAGILE! Picks a certificate created today (UTC).
CERTIFICATE_ID="$(
  aws iam list-server-certificates \
    | jq -r ".ServerCertificateMetadataList[]
             | select(.UploadDate | startswith(\"$(date --utc +%F)T\"))
             | .ServerCertificateId")"

aws cloudfront update-distribution \
  --id "${DISTRIBUTION_ID}" \
  --if-match "$(jq -r '.ETag' <<<"${DISTRIBUTION_CONFIG}")" \
  --distribution-config "$(
    jq -c --arg certificate "${CERTIFICATE_ID}" \
      '.DistributionConfig
       | .ViewerCertificate.IAMCertificateId = $certificate
       | .ViewerCertificate.Certificate = $certificate' \
      <<<"${DISTRIBUTION_CONFIG}")"

I hope that's helpful to someone. I'll eventually get around to adding support to deploy the renewed and uploaded certificate, but I won't be sad if someone else gets to it before me.