Error when issuing a wildcard certificate on Windows

[qgustavor]

From the error logs seems wildcard certificate related data is placed in a folder with a invalid name:

[Fri Apr 10 17:14:14.598191 2020] [md:error] [pid 19856:tid 644] (OS 123)The filename, directory name, or volume label syntax is incorrect.  : rename from C:/path/to/apache/md/staging/redacted.example.com to C:/path/to/apache/md/staging/*.redacted.example.com
[Fri Apr 10 17:14:14.598191 2020] [md:error] [pid 19856:tid 644] (OS 123)The filename, directory name, or volume label syntax is incorrect.  : AH10073: syncing 1 mds to registry

Checking this StackOverflow answer * can be used in filenames in Linux/Unix but not Windows.

[tlhackque]

While it is legal to use '*' in filenames on Unix (and VMS many others), it's never a good idea. Someone always goes to delete the file without the proper quoting. As far back as the 1960s, I remember "pranksters" creating such files in someone else's directory, and watching as their target deleted their work... Back then, the prankster usually was nice enough to have saved a backup copy for the target. But those were gentler times. You also wouldn't want a filename of example.txt>/etcpasswd...legal, but unwise.

mod_md needs to replace the '*' with some other token. Unfortunately, there aren't good choices.

_wild would seem like a good choice - '_' isn't valid in a DNS hostname. But it is valid in a DNS label (e.g. see RFC 2782, which uses _service.__proto - e.g. _ldap._tcp.example.com). ACME supports certificate subjects other than hostnames, so we need to be careful not to run into this in a future case. (e.g. a certificate issued for an e-mail address, or a person's name.)

Probably the safest thing to do is to create the filename by hashing the subject - which means either an index or reading each file to look for an unknown subject name. The hash cand be constrained to e.g. [A-Z0-9-] (some filesystems are case insensitive).

Alternatively, an escaping scheme could be used, e.g. _hexcode_ for any characer not in a safe list. (With _ being _3f_. Bracketing the code allows for wide characters.) That's easier for humans to read and doesn't require opening the file to learn the domain name, though it might take a couple of iterations to establish a "safe" list that works for all OSs on which httpd runs. And because it expands the name it can run into length limits on filenames.

Unfortuately, either choice also means a database conversion when that version of mod_md is released. It's very doable, but does require more code. Sigh.

@icing should probably handle this one.

Meantime, you might try using one of the covered names as the domain name, and make the wildcard an MDMember. It's harmless to include a hostname that's also covered by the wildcard. (E.g. <MDDomain www.example.net><MDMember *.example.net></MDDomain> or <MDDomain www.example.net *.example.net>) The files store entire domains - I don't think there's a case where a domain member is used as a filename.

That should work for any client that checks the SANs for a wildcard, but if the CN must be a wildcard, you'll have to wait for a complete fix.

[qgustavor]

I prefer the wildcard being a SAN, but if I configure like that DNS verification fails. Seems that it sets the TXT for both hostnames first then tries to verify instead of setting the TXT then verify for each hostname in order. The DNS service I'm using allows changing TXT records, but only "globally", in other words, if you set TXT records for both hostnames first then just the last record will be saved as it will overwrite the first.

[tlhackque]

The validator is doing the right thing. Otherwise you'd run into a number of timing issues with caches...

That sounds like a bug with either the DNS service or the client. You can have any number of TXT records in a domain. But DNS operators are notorious for unique (and broken) update mechanisms. RFC 2136 is largely ignored...

I ran into something like this when updating GoDaddy DNS - they don't have an API for adding to a RRSET; you have to replace the whole set. That means update requires reading all the existing TXT records, removing only those that you don't need any more, adding any new ones, then replacing the whole RRSET. (Obviously, there are race conditions with multiplel updaters...) It's possible to make it work for this use case.

I don't see how what you described can work in practice, though. Are you sure that a given domain can have only one TXT record? Most have many - a TXT record for SPF, one for Google statistics, and various others. So it seems most likely that the issue is in whatever tool you're using to make the updates. For example:

isc.org.                7156    IN      TXT     "MS=2583E0E0DCCBF33EC6DAFF0B4BFBAF78FFCF7AD0"
isc.org.                7156    IN      TXT     "google-site-verification=wg4m8dWi3XVPRYmtBoy6xSKp3juhxIWa6UdxjZkNQCg"
isc.org.                7156    IN      TXT     "v=spf1 a mx ip4:204.152.184.0/21 ip4:149.20.0.0/16 ip6:2001:04F8::0/32 include:servers.mcsv.net ip6:2001:500:60::65/128 ~all"
isc.org.                7156    IN      TXT     "google-site-verification=6v652rgkk_kI6Ky32iGdxqXjQ4_BAd5DYKsrnRXKUiE"
isc.org.                7156    IN      TXT     "_globalsign-domain-verification=ckxXdoIq27XGYE4ATbBYQOBeV7PTJWRxYe-PXDyzMX"

Or, two hosts in one domain:

hagrid.sb.litts.net.    86400   IN      TXT     "v=spf1" " ip4:96.233.62.58" " ip4:96.233.62.59" " ip4:96.233.62.60" " ip4:96.233.62.61" " ip4:96.233.62.62" " ip6:2001:470:8f95:940::/60" " a:micro.litts.net" " a:nano.litts.net" " a:pico.sb.litts.net" " a:overkill.sb.litts.net" " a:hagrid.sb.litts.net" " a:smtp.litts.net" " -all"
overkill.sb.litts.net.  86400   IN      TXT     "v=spf1" " ip4:96.233.62.58" " ip4:96.233.62.59" " ip4:96.233.62.60" " ip4:96.233.62.61" " ip4:96.233.62.62" " ip6:2001:470:8f95:940::/60" " a:micro.litts.net" " a:nano.litts.net" " a:pico.sb.litts.net" " a:overkill.sb.litts.net" " a:hagrid.sb.litts.net" " a:smtp.litts.net" " -all"

I added DNS update scripts for various providers (from another project) to the mod_md repo recently as part of some work-in-progress. You might find some answers there.

I haven't needed LE wildcards though - so no promises that those scripts will work as-is for you.

[qgustavor]

I know that's possible because I also work with other services, but it's a limitation of this service: as their documentation says "your TXT record will apply to all sub-subdomains under your domain". I can use other service of course which don't have this limitation, but that's not in my plans at the moment.

I used other LE clients before and some had this issue, others don't. If you say the implementation is correct, well, I'm fine with that, I had other issues with other clients anyway.

[tlhackque]

I'm saying that LE's validator is doing the right thing.

I agree with you that mod_md should not be trying to create (or read) a file name containing `*``.

Who is the DNS service that you use?

I could read the sentence that you quote as "software that reads the TXT record will (may) apply it to subdomains" - which is reasonable, if technically incorrrect. Some software will, some won't. They may be describing a particular use case. That shouldn't preclude others.

Or as you have, that "you can't create TXT records for more than one subdomain/host in a domain". That may be true - but if so, it's clearly wrong.

It's very reasonable to want several TXT records for several hosts for other use cases. For example, you have two hosts, and the TXT records are SPF policies, which are different. Or they're the contact information for security events - and you have two different IT groups. Or...

In any case, you have a valid use case (LE wildcard validation), and your provider needs to support it.

Your service provider deserves a bug report from you. Either their documentation is confusing, or their API needs work. Or maybe both!

As for mod_md, I'm sure @icing will put this issue on his TODO list.

[tlhackque]

Also - whether the wildcard is primary or only in the SAN list, LE will still need to validate two TXT records for it.

So I suspect that while you're encountering the filename issue in mod_md when the request is recorded, your DNS provider issue will be encountered at the next step even when it is fixed. Why do you think that the validation sequence will be different if the wildcard is primary?

I'm confused. You indicate that validation succeeds in that case. For that to be true, I think you must be encountering a timing/caching issue. Something like because of propagation time, the validator sees one TXT record after you've deleted/replaced it with the second. If this is true, it wouldn't be reliable. Relying on timing would be a bad idea - in addition to the validator's workload being unpredictable, note that LE is starting to validate each record from multiple servers, which are in different places on the network. I must be missing something.

I also noticed a LE post on wildcards that says they will reject redundant requests such as *.example.net, www.example.net. So a SAN work-around for this issue might have to be a bit more creatitve, e.g. dummy-host.x.example.net, *.example.net. (Wildcards in DNS only apply to a single, terminal level.) dummy-host.x.example.net would not exist, but LE would validate it as ususal.

I agree that mod_md has a bug that should be fixed. But I don't see how LE wildcards can work with a DNS service that won't provision 2 TXT records in a domain.

[qgustavor]

Check documentation, it's just how it works. I thought it was possible to verify one hostname, store the verification, verify the other and then use both verifications to get the certificate. If that's not possible then I can use two certificates, one just for the wildcard. It's just for a development server: I plan in future to use wildcards to I can test multiple projects more easily. I will wait for the filename issue to be fixed.

[tlhackque]

That's a very strange service - it doesn't conform to the DNS RFCs. Note that they do say

This can be used for example to prove your ownership with letsencrypt.org

So why not send DuckDNS a bug report and point out that they don't support wildcard DNS from LE? Seems like something that they can fix. They have a forum at https://groups.google.com/forum/#!forum/duckdns

Again, even when the filename issue is fixed, I don't think this service will work until they meet LE's requirement to provision two distinct TXT records.

Verification is not driven by mod_md. LE tells mod_md (or any client) what it wants, and mod_md responds. It's LE that requests/tracks verifications. mod_md is passive in that process. The sequence is roughly:

• C: Please issue a certificate (mod_md writes a file to track the progress) - this is where we stop now
• S: To prove you own the domain, install these records
• C: (installs records)
• C: Ready for you to check my work
• C: Are you done yet (repeats)
• S: Checks DNS for both records - this is where I think you'll fail once the filename is fixed
• S: Yes, you can delete the records. Here's where to get your certificate.

I believe that LE does cache verification results - but for wildcards, the verification doesn't have a result until both records are successfully seen.

In any case, for webserver projects, a wildcard doesn't save much work. mod_md will detect that you've added a new VHOST and request a new certificate automatically. Since there is no charge, there's really no reason to use a wildcard certificate for that. If you are using the certificate for a lot of other applications, (mail, new, notes,...) it may be easier to have a wildcard than to update all of them. Or if you have a LOT of vhosts (LE supports ~100/per certificate), a wildcard certificate is smaller, which conserves network bandwidth.

[qgustavor]

I prefer wildcards over automatic certificate creation because in this way subdomains don't show up in certificate transparency logs, but I will consider that for future projects. I also could run my own CA, as that's just for local development, but from what I read I find it harder to set up than using LE.

[qgustavor]

@tlhackque About the DuckDNS issue I've said "I used other LE clients before and some had this issue, others don't.". After searching a bit seems there is a small issue with your explanation: as shown here it will work if the client don't all challenges in a batch, I just got it working now.

Probably it's hard to implement it here and, of course, it's slower, so I don't expect it being fixed here, I just wanted to share this info.