"dns_challenge" block behavior has changed with cloudflare dns provider (error: one or more domains had a problem)

[yashjain10038]

Issue

The dns_challenge { } inside "acme_certificate" resource block is throwing below error while initializing with vancluever/acme v2.18.0 & above.

resource "acme_certificate" "subdomain_cert" {
  account_key_pem    = var.acme_account_key_pem
  common_name        = "*.${local.zone_name}"
  min_days_remaining = var.acme_min_days_remaining
  recursive_nameservers = ["1.1.1.1:53"]

  dns_challenge {
    provider = "cloudflare"
    config = {
      CF_DNS_API_TOKEN = var.cloudflare_api_token
    }
  }
}

Error: error: one or more domains had a problem:
[*.qa.devtest.companydomain.com] [*.qa.devtest.companydomain.com] acme: error presenting token: 2 errors occurred:
	rpc error: code = Unknown desc = cloudflare: failed to find zone internal.qa.devtest.companydomain.com.: zone could not be found
	error encountered while presenting token for DNS challenge: rpc error: code = Unknown desc = cloudflare: failed to find zone internal.qa.devtest.companydomain.com.: zone could not be found

Observations

In terraform debug logs, we found the dns lookup behavior difference in old (2.8.0) & latest versions (2.18.0 & above) as shown below :-

2.18.0 / 2.19.0 (Latest) - Both versions trying to do _acme CNAME record lookup which results to fail.

[DEBUG] plugin.terraform-provider-acme_v2.18.0.exe [INFO] Found CNAME entry for "_acme-challenge.qa.devtest.companydomain.com.": "_acme-challenge.internal.qa.devtest.companydomain.com." [INFO] provider.terraform-provider-acme_v2.18.0.exe: [DEBUG] lego: [.qa.devtest.companydomain.com] acme: Cleaning DNS-01 challenge: [DEBUG] provider.terraform-provider-acme_v2.18.0.exe: [DEBUG] plugin.terraform-provider-acme_v2.18.0.exe: [INFO] Found CNAME entry for "_acme-challenge.qa.devtest.companydomain.com.": "_acme-challenge.internal.qa.devtest.companydomain.com." [INFO] provider.terraform-provider-acme_v2.18.0.exe: [DEBUG] lego: [.qa.devtest.companydomain.com] acme: cleaning up failed: 2 errors occurred: rpc error: code = Unknown desc = cloudflare: failed to find zone internal.qa.devtest.companydomain.com (Opens in new window or tab).: zone could not be found error encountered while cleaning token for DNS challenge: rpc error: code = Unknown desc = cloudflare: failed to find zone internal.qa.devtest.companydomain.com (Opens in new window or tab).: zone could not be found:

2.7.0 / 2.8.0 - Here, dns lookup behavior is different, (DNS record propagation using [1.1.1.1:53]). No CNAME DNS request generated for acme. It used the recursive_nameservers attribute value to do dns lookup.

[INFO] provider.terraform-provider-acme_v2.8.0.exe: [DEBUG] lego: cloudflare: new record for qa.devtest.companydomain.com (Opens in new window or tab), ID b25fc558d7d70e37331a916a74b9970a: [INFO] provider.terraform-provider-acme_v2.8.0.exe: [DEBUG] lego: [.qa.devtest.companydomain.com] acme: Trying to solve DNS-01: [INFO] provider.terraform-provider-acme_v2.8.0.exe: [DEBUG] lego: [.qa.devtest.companydomain.com] acme: Checking DNS record propagation using [1.1.1.1:53]

Preventive action taken (temporarily)

We had to downgrade the provider version to v2.8.0 to prevent the production deployment issues. Please have a look at this issue and share the fix.