acme.sh icon indicating copy to clipboard operation
acme.sh copied to clipboard
verified publisher icon acmesh-official

Can't renew ec-384 certificate after enabling FIPS mode.

Open easpeagle opened this issue 8 months ago • 7 comments

On my Rocky Linux 9.5, we can no longer renew our certificates from letsencrypt after we enabled FIPS mode:

[Fri Apr 11 20:57:15 UTC 2025] EC key
[Fri Apr 11 20:57:15 UTC 2025] Let's try ASN1 OID
[Fri Apr 11 20:57:15 UTC 2025] ECC oid: 
[Fri Apr 11 20:57:15 UTC 2025] Error creating new order.
[Fri Apr 11 20:57:15 UTC 2025] pid
[Fri Apr 11 20:57:15 UTC 2025] No need to restore nginx config, skipping.
[Fri Apr 11 20:57:15 UTC 2025] _clearupdns
[Fri Apr 11 20:57:15 UTC 2025] dns_entries
[Fri Apr 11 20:57:15 UTC 2025] Skipping dns.
[Fri Apr 11 20:57:15 UTC 2025] _on_issue_err
[Fri Apr 11 20:57:15 UTC 2025] Please add '--debug' or '--log' to see more information.
[Fri Apr 11 20:57:15 UTC 2025] See: https://github.com/acmesh-official/acme.sh/wiki/How-to-debug-acme.sh

This doesn't seem to be an issue with openssl ... was able to successfully generate a new key and csr locally without it barfing. Running this version:

# acme.sh version
https://github.com/acmesh-official/acme.sh
v3.1.1

I have certs that are going to expire in a couple of days... so this is urgent for me personally...

easpeagle avatar Apr 11 '25 20:04 easpeagle

Please upgrade to the latest code and try again first. Maybe it's already fixed. acme.sh --upgrade If it's still not working, please provide the log with --debug 2, otherwise, nobody can help you.

github-actions[bot] avatar Apr 11 '25 20:04 github-actions[bot]

Full debug 2 logging output:

# "/root/.acme.sh"/acme.sh --debug 2 --cron --home "/root/.acme.sh"
[Sun Apr 13 15:28:39 UTC 2025] Let's find the script directory.
[Sun Apr 13 15:28:39 UTC 2025] _SCRIPT_='/root/.acme.sh/acme.sh'
[Sun Apr 13 15:28:40 UTC 2025] _script='/root/.acme.sh/acme.sh'
[Sun Apr 13 15:28:40 UTC 2025] _script_home='/root/.acme.sh'
[Sun Apr 13 15:28:40 UTC 2025] Using config home: /root/.acme.sh
[Sun Apr 13 15:28:40 UTC 2025] LE_WORKING_DIR='/root/.acme.sh'
https://github.com/acmesh-official/acme.sh
v3.1.1
[Sun Apr 13 15:28:40 UTC 2025] Running cmd: cron
[Sun Apr 13 15:28:40 UTC 2025] Using config home: /root/.acme.sh
[Sun Apr 13 15:28:40 UTC 2025] default_acme_server
[Sun Apr 13 15:28:40 UTC 2025] ACME_DIRECTORY='https://acme.zerossl.com/v2/DV90'
[Sun Apr 13 15:28:40 UTC 2025] _ACME_SERVER_HOST='acme.zerossl.com'
[Sun Apr 13 15:28:40 UTC 2025] _ACME_SERVER_PATH='v2/DV90'
[Sun Apr 13 15:28:40 UTC 2025] ===Starting cron===
[Sun Apr 13 15:28:40 UTC 2025] Using config home: /root/.acme.sh
[Sun Apr 13 15:28:40 UTC 2025] ACME_DIRECTORY='https://acme.zerossl.com/v2/DV90'
[Sun Apr 13 15:28:40 UTC 2025] _ACME_SERVER_HOST='acme.zerossl.com'
[Sun Apr 13 15:28:40 UTC 2025] _ACME_SERVER_PATH='v2/DV90'
[Sun Apr 13 15:28:40 UTC 2025] _stopRenewOnError
[Sun Apr 13 15:28:40 UTC 2025] _server
[Sun Apr 13 15:28:40 UTC 2025] _set_level='2'
[Sun Apr 13 15:28:40 UTC 2025] di='/root/.acme.sh/mdm.toyon.com_ecc/'
[Sun Apr 13 15:28:40 UTC 2025] d='mdm.toyon.com_ecc'
[Sun Apr 13 15:28:40 UTC 2025] _renewServer
[Sun Apr 13 15:28:40 UTC 2025] Using config home: /root/.acme.sh
[Sun Apr 13 15:28:40 UTC 2025] ACME_DIRECTORY='https://acme.zerossl.com/v2/DV90'
[Sun Apr 13 15:28:40 UTC 2025] _ACME_SERVER_HOST='acme.zerossl.com'
[Sun Apr 13 15:28:40 UTC 2025] _ACME_SERVER_PATH='v2/DV90'
[Sun Apr 13 15:28:40 UTC 2025] DOMAIN_PATH='/root/.acme.sh/mdm.toyon.com_ecc'
[Sun Apr 13 15:28:40 UTC 2025] Renewing: 'mdm.toyon.com'
[Sun Apr 13 15:28:40 UTC 2025] Le_API='https://acme-v02.api.letsencrypt.org/directory'
[Sun Apr 13 15:28:40 UTC 2025] Renewing using Le_API=https://acme-v02.api.letsencrypt.org/directory
[Sun Apr 13 15:28:40 UTC 2025] initpath again.
[Sun Apr 13 15:28:40 UTC 2025] Using config home: /root/.acme.sh
[Sun Apr 13 15:28:40 UTC 2025] ACME_DIRECTORY='https://acme-v02.api.letsencrypt.org/directory'
[Sun Apr 13 15:28:40 UTC 2025] _ACME_SERVER_HOST='acme-v02.api.letsencrypt.org'
[Sun Apr 13 15:28:40 UTC 2025] _ACME_SERVER_PATH='directory'
[Sun Apr 13 15:28:40 UTC 2025] _main_domain='mdm.toyon.com'
[Sun Apr 13 15:28:40 UTC 2025] _alt_domains='autodiscover.toyon.com,mx.toyon.com,shareholders.toyon.com,dag.toyon.com'
[Sun Apr 13 15:28:40 UTC 2025] 'dns_aws' does not contain 'dns'
[Sun Apr 13 15:28:40 UTC 2025] 'dns_aws' does not contain 'dns'
[Sun Apr 13 15:28:40 UTC 2025] Le_NextRenewTime='1741987625'
[Sun Apr 13 15:28:40 UTC 2025] Using ACME_DIRECTORY: https://acme-v02.api.letsencrypt.org/directory
[Sun Apr 13 15:28:40 UTC 2025] _init API for server: https://acme-v02.api.letsencrypt.org/directory
[Sun Apr 13 15:28:40 UTC 2025] GET
[Sun Apr 13 15:28:40 UTC 2025] url='https://acme-v02.api.letsencrypt.org/directory'
[Sun Apr 13 15:28:40 UTC 2025] timeout=
[Sun Apr 13 15:28:40 UTC 2025] _CURL='curl --silent --dump-header /root/.acme.sh/http.header  -L  --trace-ascii /tmp/tmp.vDXkByEoHn  -g '
[Sun Apr 13 15:28:40 UTC 2025] ret='0'
[Sun Apr 13 15:28:40 UTC 2025] response='{
  "NkH-VW5hC4c": "https://community.letsencrypt.org/t/adding-random-entries-to-the-directory/33417",
  "keyChange": "https://acme-v02.api.letsencrypt.org/acme/key-change",
  "meta": {
    "caaIdentities": [
      "letsencrypt.org"
    ],
    "profiles": {
      "classic": "https://letsencrypt.org/docs/profiles#classic",
      "shortlived": "https://letsencrypt.org/docs/profiles#shortlived (not yet generally available)",
      "tlsserver": "https://letsencrypt.org/docs/profiles#tlsserver (not yet generally available)"
    },
    "termsOfService": "https://letsencrypt.org/documents/LE-SA-v1.5-February-24-2025.pdf",
    "website": "https://letsencrypt.org"
  },
  "newAccount": "https://acme-v02.api.letsencrypt.org/acme/new-acct",
  "newNonce": "https://acme-v02.api.letsencrypt.org/acme/new-nonce",
  "newOrder": "https://acme-v02.api.letsencrypt.org/acme/new-order",
  "renewalInfo": "https://acme-v02.api.letsencrypt.org/draft-ietf-acme-ari-03/renewalInfo",
  "revokeCert": "https://acme-v02.api.letsencrypt.org/acme/revoke-cert"
}'
[Sun Apr 13 15:28:40 UTC 2025] ACME_KEY_CHANGE='https://acme-v02.api.letsencrypt.org/acme/key-change'
[Sun Apr 13 15:28:40 UTC 2025] ACME_NEW_AUTHZ
[Sun Apr 13 15:28:41 UTC 2025] ACME_NEW_ORDER='https://acme-v02.api.letsencrypt.org/acme/new-order'
[Sun Apr 13 15:28:41 UTC 2025] ACME_NEW_ACCOUNT='https://acme-v02.api.letsencrypt.org/acme/new-acct'
[Sun Apr 13 15:28:41 UTC 2025] ACME_REVOKE_CERT='https://acme-v02.api.letsencrypt.org/acme/revoke-cert'
[Sun Apr 13 15:28:41 UTC 2025] ACME_AGREEMENT='https://letsencrypt.org/documents/LE-SA-v1.5-February-24-2025.pdf'
[Sun Apr 13 15:28:41 UTC 2025] ACME_NEW_NONCE='https://acme-v02.api.letsencrypt.org/acme/new-nonce'
[Sun Apr 13 15:28:41 UTC 2025] Using CA: https://acme-v02.api.letsencrypt.org/directory
[Sun Apr 13 15:28:41 UTC 2025] _on_before_issue
[Sun Apr 13 15:28:41 UTC 2025] _chk_main_domain='mdm.toyon.com'
[Sun Apr 13 15:28:41 UTC 2025] _chk_alt_domains='autodiscover.toyon.com,mx.toyon.com,shareholders.toyon.com,dag.toyon.com'
[Sun Apr 13 15:28:41 UTC 2025] 'dns_aws' does not contain 'no'
[Sun Apr 13 15:28:41 UTC 2025] Le_LocalAddress
[Sun Apr 13 15:28:41 UTC 2025] d='mdm.toyon.com'
[Sun Apr 13 15:28:41 UTC 2025] Checking for domain='mdm.toyon.com'
[Sun Apr 13 15:28:41 UTC 2025] _currentRoot='dns_aws'
[Sun Apr 13 15:28:41 UTC 2025] d='autodiscover.toyon.com'
[Sun Apr 13 15:28:41 UTC 2025] Checking for domain='autodiscover.toyon.com'
[Sun Apr 13 15:28:41 UTC 2025] _currentRoot='dns_aws'
[Sun Apr 13 15:28:41 UTC 2025] d='mx.toyon.com'
[Sun Apr 13 15:28:41 UTC 2025] Checking for domain='mx.toyon.com'
[Sun Apr 13 15:28:41 UTC 2025] _currentRoot='dns_aws'
[Sun Apr 13 15:28:41 UTC 2025] d='shareholders.toyon.com'
[Sun Apr 13 15:28:41 UTC 2025] Checking for domain='shareholders.toyon.com'
[Sun Apr 13 15:28:41 UTC 2025] _currentRoot='dns_aws'
[Sun Apr 13 15:28:41 UTC 2025] d='dag.toyon.com'
[Sun Apr 13 15:28:41 UTC 2025] Checking for domain='dag.toyon.com'
[Sun Apr 13 15:28:41 UTC 2025] _currentRoot='dns_aws'
[Sun Apr 13 15:28:41 UTC 2025] d
[Sun Apr 13 15:28:41 UTC 2025] 'dns_aws' does not contain 'apache'
[Sun Apr 13 15:28:41 UTC 2025] _saved_account_key_hash='xrR26DFNAwZQWohs2/8/mnoDbvGet8qBsIOq9zDk58E='
[Sun Apr 13 15:28:41 UTC 2025] _saved_account_key_hash was not changed, skipping account registration.
[Sun Apr 13 15:28:41 UTC 2025] Read key length: ec-384
[Sun Apr 13 15:28:41 UTC 2025] _createcsr
[Sun Apr 13 15:28:41 UTC 2025] domain='mdm.toyon.com'
[Sun Apr 13 15:28:41 UTC 2025] domainlist='autodiscover.toyon.com,mx.toyon.com,shareholders.toyon.com,dag.toyon.com'
[Sun Apr 13 15:28:41 UTC 2025] csrkey='/root/.acme.sh/mdm.toyon.com_ecc/mdm.toyon.com.key'
[Sun Apr 13 15:28:41 UTC 2025] csr='/root/.acme.sh/mdm.toyon.com_ecc/mdm.toyon.com.csr'
[Sun Apr 13 15:28:41 UTC 2025] csrconf='/root/.acme.sh/mdm.toyon.com_ecc/mdm.toyon.com.csr.conf'
[Sun Apr 13 15:28:41 UTC 2025] _is_idn_d='autodiscover.toyon.com,mx.toyon.com,shareholders.toyon.com,dag.toyon.com'
[Sun Apr 13 15:28:41 UTC 2025] _idn_temp
[Sun Apr 13 15:28:41 UTC 2025] domainlist='autodiscover.toyon.com,mx.toyon.com,shareholders.toyon.com,dag.toyon.com'
[Sun Apr 13 15:28:41 UTC 2025] seg='mdm'
[Sun Apr 13 15:28:41 UTC 2025] _is_idn_d='mdm.toyon.com'
[Sun Apr 13 15:28:41 UTC 2025] _idn_temp
[Sun Apr 13 15:28:41 UTC 2025] seg='autodiscover'
[Sun Apr 13 15:28:41 UTC 2025] seg='mx'
[Sun Apr 13 15:28:41 UTC 2025] seg='shareholders'
[Sun Apr 13 15:28:41 UTC 2025] seg='dag'
[Sun Apr 13 15:28:41 UTC 2025] Multi domain='DNS:mdm.toyon.com,DNS:autodiscover.toyon.com,DNS:mx.toyon.com,DNS:shareholders.toyon.com,DNS:dag.toyon.com'
[Sun Apr 13 15:28:41 UTC 2025] _is_idn_d='mdm.toyon.com'
[Sun Apr 13 15:28:41 UTC 2025] _idn_temp
[Sun Apr 13 15:28:41 UTC 2025] _csr_cn='mdm.toyon.com'
[Sun Apr 13 15:28:41 UTC 2025] seg='mdm'
[Sun Apr 13 15:28:41 UTC 2025] Getting domain auth token for each domain
[Sun Apr 13 15:28:41 UTC 2025] seg='mdm'
[Sun Apr 13 15:28:41 UTC 2025] _is_idn_d='mdm.toyon.com'
[Sun Apr 13 15:28:42 UTC 2025] _idn_temp
[Sun Apr 13 15:28:42 UTC 2025] d='autodiscover.toyon.com'
[Sun Apr 13 15:28:42 UTC 2025] seg='autodiscover'
[Sun Apr 13 15:28:42 UTC 2025] _is_idn_d='autodiscover.toyon.com'
[Sun Apr 13 15:28:42 UTC 2025] _idn_temp
[Sun Apr 13 15:28:42 UTC 2025] d='mx.toyon.com'
[Sun Apr 13 15:28:42 UTC 2025] seg='mx'
[Sun Apr 13 15:28:42 UTC 2025] _is_idn_d='mx.toyon.com'
[Sun Apr 13 15:28:42 UTC 2025] _idn_temp
[Sun Apr 13 15:28:42 UTC 2025] d='shareholders.toyon.com'
[Sun Apr 13 15:28:42 UTC 2025] seg='shareholders'
[Sun Apr 13 15:28:42 UTC 2025] _is_idn_d='shareholders.toyon.com'
[Sun Apr 13 15:28:42 UTC 2025] _idn_temp
[Sun Apr 13 15:28:42 UTC 2025] d='dag.toyon.com'
[Sun Apr 13 15:28:42 UTC 2025] seg='dag'
[Sun Apr 13 15:28:42 UTC 2025] _is_idn_d='dag.toyon.com'
[Sun Apr 13 15:28:42 UTC 2025] _idn_temp
[Sun Apr 13 15:28:42 UTC 2025] d
[Sun Apr 13 15:28:42 UTC 2025] _identifiers='{"type":"dns","value":"mdm.toyon.com"},{"type":"dns","value":"autodiscover.toyon.com"},{"type":"dns","value":"mx.toyon.com"},{"type":"dns","value":"shareholders.toyon.com"},{"type":"dns","value":"dag.toyon.com"}'
[Sun Apr 13 15:28:42 UTC 2025] _notBefore
[Sun Apr 13 15:28:42 UTC 2025] _notAfter
[Sun Apr 13 15:28:42 UTC 2025] STEP 1, Ordering a Certificate
[Sun Apr 13 15:28:42 UTC 2025] =======Sending Signed Request=======
[Sun Apr 13 15:28:42 UTC 2025] url='https://acme-v02.api.letsencrypt.org/acme/new-order'
[Sun Apr 13 15:28:42 UTC 2025] payload='{"identifiers": [{"type":"dns","value":"mdm.toyon.com"},{"type":"dns","value":"autodiscover.toyon.com"},{"type":"dns","value":"mx.toyon.com"},{"type":"dns","value":"shareholders.toyon.com"},{"type":"dns","value":"dag.toyon.com"}]}'
[Sun Apr 13 15:28:42 UTC 2025] EC key
[Sun Apr 13 15:28:42 UTC 2025] Let's try ASN1 OID
[Sun Apr 13 15:28:42 UTC 2025] ECC oid: 
[Sun Apr 13 15:28:42 UTC 2025] Error creating new order.
[Sun Apr 13 15:28:42 UTC 2025] pid
[Sun Apr 13 15:28:42 UTC 2025] No need to restore nginx config, skipping.
[Sun Apr 13 15:28:42 UTC 2025] _clearupdns
[Sun Apr 13 15:28:42 UTC 2025] dns_entries
[Sun Apr 13 15:28:42 UTC 2025] Skipping dns.
[Sun Apr 13 15:28:42 UTC 2025] _on_issue_err
[Sun Apr 13 15:28:42 UTC 2025] Please add '--debug' or '--log' to see more information.
[Sun Apr 13 15:28:42 UTC 2025] See: https://github.com/acmesh-official/acme.sh/wiki/How-to-debug-acme.sh
[Sun Apr 13 15:28:42 UTC 2025] _chk_vlist
[Sun Apr 13 15:28:42 UTC 2025] 'dns_aws' does not contain 'dns'
[Sun Apr 13 15:28:42 UTC 2025] Diagnosis versions: 
openssl:openssl
OpenSSL 3.2.2 4 Jun 2024 (Library: OpenSSL 3.2.2 4 Jun 2024)
Apache:
Apache doesn't exist.
nginx:
nginx doesn't exist.
socat:
socat by Gerhard Rieger and contributors - see www.dest-unreach.org
socat version 1.7.4.1 on Apr 15 2024 00:00:00
   running on Linux version #1 SMP PREEMPT_DYNAMIC Thu Apr 3 12:12:16 UTC 2025, release 5.14.0-503.35.1.el9_5.x86_64, machine x86_64
features:
  #define WITH_STDIO 1
  #define WITH_FDNUM 1
  #define WITH_FILE 1
  #define WITH_CREAT 1
  #define WITH_GOPEN 1
  #define WITH_TERMIOS 1
  #define WITH_PIPE 1
  #define WITH_UNIX 1
  #define WITH_ABSTRACT_UNIXSOCKET 1
  #define WITH_IP4 1
  #define WITH_IP6 1
  #define WITH_RAWIP 1
  #define WITH_GENERICSOCKET 1
  #define WITH_INTERFACE 1
  #define WITH_TCP 1
  #define WITH_UDP 1
  #define WITH_SCTP 1
  #define WITH_LISTEN 1
  #define WITH_SOCKS4 1
  #define WITH_SOCKS4A 1
  #define WITH_VSOCK 1
  #define WITH_PROXY 1
  #define WITH_SYSTEM 1
  #define WITH_EXEC 1
  #define WITH_READLINE 1
  #define WITH_TUN 1
  #define WITH_PTY 1
  #define WITH_OPENSSL 1
  #undef WITH_FIPS
  #undef WITH_LIBWRAP
  #define WITH_SYCLS 1
  #define WITH_FILAN 1
  #define WITH_RETRY 1
  #define WITH_MSGLEVEL 0 /*debug*/
[Sun Apr 13 15:28:42 UTC 2025] Return code: 1
[Sun Apr 13 15:28:42 UTC 2025] Error renewing mdm.toyon.com_ecc.
[Sun Apr 13 15:28:42 UTC 2025] _error_level='1'
[Sun Apr 13 15:28:42 UTC 2025] _set_level='2'
[Sun Apr 13 15:28:42 UTC 2025] The NOTIFY_HOOK is empty, will just return.
[Sun Apr 13 15:28:42 UTC 2025] ===End cron===

easpeagle avatar Apr 11 '25 21:04 easpeagle

Looks like this is the problem... in FIPS mode... openssl doesn't want to print out the key details that acme.sh is looking for to determine key type:

Tested with creating a dummy key and immediately trying to read it:

# openssl ecparam -name secp384r1 -noout -genkey >foo.key
# openssl ec -noout -text -in foo.key 
read EC key
unable to print EC key

but... if I add -provider=fips ...

openssl ec -provider=fips -noout -text -in foo.key 
read EC key
Private-Key: (384 bit)
...

Looks like some checking needs to be added around FIPS mode...

easpeagle avatar Apr 13 '25 16:04 easpeagle

This is an issue with RHEL/Rocky and other Enterprise Linuxes it's supposed to automatically use the correct provider but there appears to be a bug in openssl or the RHEL patches on top of openssl.

You need to disable the default module in the openssl configuration either at /etc/ssl/openssl.cnf or at /etc/pki/tls/openssl.cnf

[default_sect]
activate = 1

And change it to

[default_sect]
activate = 0

Or remove all the references to default_sect in the config

Afterwards you can test that it's working correctly with openssl list -providers If FIPS is setup correctly and the default openssl module is disabled it should only list the FIPS provider. If you don't have the FIPS module or you still have the default module enabled make sure you have already enabled FIPS properly with fips-mode-setup --enable and have rebooted to enable FIPS mode.

Further explanation

RedHat decided to load an extra file without telling anyone. The closest they have to documenting it is in some release notes about changing an option for connecting to non-FIPS systems. You can see the openssl patch here

This is supposed to work per the openssl documentation but per this part of the documentation their appears to be a bug in openssl or at least RedHat's build of openssl where it fails to read the fips property.

I'd check if ubuntu does the same thing but they don't have any sources available that I can see

gwelch-contegix avatar Aug 18 '25 21:08 gwelch-contegix

Hmm... that worked to disable the default_sect ... but I still see these providers:

# openssl list -providers
Providers:
  base
    name: OpenSSL Base Provider
    version: 3.2.2
    status: active
  fips
    name: Red Hat Enterprise Linux 9 - OpenSSL FIPS Provider
    version: 3.0.7-395c1a240fbfffd8
    status: active

And acme.sh still complains with the same error...

easpeagle avatar Sep 12 '25 16:09 easpeagle

The base provider is always required, if it's not loaded then it will complain. There might be another bug as well, I'd have to check on a non-RHEL based os to verify

gwelch-contegix avatar Sep 12 '25 19:09 gwelch-contegix

I just hacked the acme.sh binary to add -provider fips where needed on the openssl ec command lines. Seems like better detection of FIPS mode or a simple flag to assert this might be needed.

easpeagle avatar Sep 13 '25 19:09 easpeagle