autohttps: Implement `auto_https prefer_wildcard` option

[francislavoie]

Closes https://github.com/caddyserver/caddy/issues/5447

This implements a new auto_https prefer_wildcard option, which drops automation policies for non-wildcard domains when there's already a wildcard in another policy.

This allows users to flatten their config, and instead of using a pattern like https://caddyserver.com/docs/caddyfile/patterns#wildcard-certificates they can instead do something like:

{
	auto_https prefer_wildcard
}

*.example.com {
	tls {
		dns <provider>
	}
	respond "fallback"
}

foo.example.com {
	respond "foo"
}

This would only produce a single wildcard certificate, and no individual certificate for foo.example.com since it's already covered by the wildcard.

This also allows specifying multiple arguments to auto_https, so you can do the following to set multiple Automatic HTTPS options. Previously only one could be set at a time, which was generally fine because there wasn't actually any usecase where it would be useful:

{
	auto_https prefer_wildcard disable_redirects
}

I've only manually (visually) tested with a few simple usecases. Unfortunately we have a big lack of tests for the Automatic HTTPS logic because it manipulates config at runtime. I probably need help with testing this to make sure it doesn't have weird side effects. Thankfully, this should be safe/backwards compatible as long as users don't enable this option. We could call it experimental for now.

[mholt]

Thanks Francis, this looks appealing. Will wait for one or two people to field-test it.

[abjugard]

This looks amazing and a super useful feature to reduce subdomain discovery through certificate transparency!

I would suggest a small addition in order to minimise this even further.

E.g. to make Caddy always use wildcards.

This could maybe be implemented as another option for this directive, called force_wildcard, which figures out the minimal number of wildcard certs that need to be generated to cover all sites, and then uses your logic from this code to apply those certs to the correct sites.

Say I have two sites setup, for serviceA.domain.tld and serviceB.domain.tld, and use this setting, Caddy would then generate *.domain.tld and use it.

If I have hostX.serviceA.domain.tld, hostY.serviceA.domain.tld, hostX.serviceB.domain.tld, hostY.serviceB.domain.tld, then all of those would be covered by the *.*.domain.tld certificate (does ACME even allow such certificates? 🤔).

Alternatively, add another directive that allows users to specify a list of wildcard certs that will always be managed regardless of if they're used by sites or not, and then your feature here will be able to use those to avoid generating specific certs. A little less "magic" than the force_wildcard idea, but probably much easier to implement.

What do you think about this @francislavoie?

I guess it might be necessary to give Caddy a list of domains we control in order for this to work when using multiple domains so it doesn't see domain1.tld and domain2.tld and tries to generate *.tld... 😂

[jeda]

If I have hostX.serviceA.domain.tld, hostY.serviceA.domain.tld, hostX.serviceB.domain.tld, hostY.serviceB.domain.tld, then all of those would be covered by the *.*.domain.tld certificate (does ACME even allow such certificates? 🤔).

@abjugard *.domain.tld won't cover sub-sub-domains. Caddy would need to generate certs for *.serviceA.domain.tld and *.serviceB.domain.tld. At least Let's Encrypt doesn't allow for *.*.domain.tld, AFAIK.

[abjugard]

If I have hostX.serviceA.domain.tld, hostY.serviceA.domain.tld, hostX.serviceB.domain.tld, hostY.serviceB.domain.tld, then all of those would be covered by the *.*.domain.tld certificate (does ACME even allow such certificates? 🤔).

@abjugard *.domain.tld won't cover sub-sub-domains. Caddy would need to generate certs for *.serviceA.domain.tld and *.serviceB.domain.tld. At least Let's Encrypt doesn't allow for *.*.domain.tld, AFAIK.

Right, unfortunate limitation but understandable.

Then I think we can simplify the behaviour and not need to specify which domains are owned, but just let Caddy automatically figure out to make wildcards for the leftmost label.

Perhaps we should just wait for this PR to get merged then I can take a crack at a force-wildcard PR?

[mholt]

I'm actually implementing part of this in CertMagic -- likely will become a "subject transformer" that allows changes to the subject name when managing certs, so, e.g. one can lob off the leftmost label and replace it with * to make it a wildcard. Of course, the Caddy interface will still need to be developed, which this PR could be. But I kind of want to come back to this after CertMagic has its change made, since it coincides with https://github.com/caddyserver/certmagic/issues/280.

[mholt]

@francislavoie How would you feel if we/(I?) updated this PR to use the new SubjectTransformer introduced in the linked issue/commit? It should be relatively simple I think, if the global option is set, then set the SubjectTransformer for CertMagic configs to be a 3-line function.

[francislavoie]

I'm not sure how that would look :thinking: maybe make a branch off this one if you think it's simple?

[mholt]

I will try soon! :)

[mholt]

Actually the SubjectTransformer might be slightly tangential to this, rather than directly related -- the transformer is for, like, "I have a.b.c and b.b.c, but I want you to manage a single wildcard instead of individual certs for each specific domain." Whereas this change is, "I have *.b.c and a.b.c, and I want a.b.c to use the wildcard cert."

There's another aspect I want to consider as well, that is some users want just specific domains to be served under a wildcard, while the others shouldn't be. For example, in the config above, if there was also a site, bar.example.com, but they wanted that served with its own cert (maybe to try to obscure the fact that other subdomains are being served?) how would they do that?

I think I want to give this more thought, even before I implement anything here.

This is a needed change though and I think it's a good start. I just don't want to commit to its API/syntax/implementation quite yet until we have a better picture of the bigger picture.

[francislavoie]

There's another aspect I want to consider as well, that is some users want just specific domains to be served under a wildcard, while the others shouldn't be. For example, in the config above, if there was also a site, bar.example.com, but they wanted that served with its own cert (maybe to try to obscure the fact that other subdomains are being served?) how would they do that?

I think that's already handled by my approach, because prefer_wildcard only applies to a subdomain if there's already a wildcard cert in the config being managed (in which case it uses that policy). For other domains if you just don't have a wildcard that covers it, it'll still make a cert for that single domain. If you want to opt-out for just one domain that's covered by a wildcard, then don't use this feature and do it the handle way :man_shrugging:

[mholt]

Ah, okay. Hmm.

I might still wait on this until after 2.8 so I can give this a little more thought. We now have a way, in CertMagic, of mapping/transforming one subject name (domain name) to another, for the purposes of cert management. Even if this PR ends up being good as-is, I just want a little more time on it.

[omltcat]

Glad there is a PR for this and I really look forward to it. Thank you for your great work. One big advantage of having separate site blocks is when using caddy-docker-proxy labels.

Since these labels can be distributed across multiple docker-compose files, there can be some possibilities of misconfiguration somewhere (especially done by multiple people). With the current handle approach, if one of the subdomain is misconfigured, causes the ENTIRE wildcard site block to be removed, bringing down everything. With this PR, the failure would at least be localized (hopefully😊)

[polarathene]

If you want to opt-out for just one domain that's covered by a wildcard, then don't use this feature and do it the handle way 🤷‍♂️

Could there not just be a tls or similar directive for a more explicit opt-out? (I don't need such functionality myself though)

It already seems to be an issue according to this report where an internal wildcard cert is being used instead of the LetsEncrypt one for an explicit site address.

Alternatively, you could go the other way around like with local_certs / tls internal, and instead have something like tls internal_wildcard or prefer_wildcard in the actual tls directive options? (assuming that could also be used to prefer FQDN as an override too).

---

Since the certmagic subject transformer feature is available and Caddy 2.8 is released, is there anything that can be done to assist moving this feature forward?

I've only manually (visually) tested with a few simple usecases. Unfortunately we have a big lack of tests for the Automatic HTTPS logic because it manipulates config at runtime. I probably need help with testing this to make sure it doesn't have weird side effects.

If you can provide a rough outline of what to test, I could put together configs to verify?

---

~~One potential bug (without this PR) that already appears to exist is assigning a domain a wildcard cert from external files, and another site block for that domain with a different subdomain implicitly using the wildcard.~~ (EDIT: Nope, that was due to incorrect auto_https mode set)