Another use case: http-01 with remote server

[tlhackque]

I've encountered another use case that mod_md needs help handling.

Consider this situation (handled by the client I'm trying to retire...):

Remote server, a VPS with a vendor who doesn't support LE, needs a certificate; updating DNS is painful as the DNS provider has no supported API. Currently, I get/renew the certificates for the remote server from one of my local machines.

The remote server has a disk area that maps to /.well-known/acme-challenge.

The current client, running locally (not on the remote server), will call-out to a script, providing the token & key authorization. The script constructs a file (named by the token) containing the key authorization. It then uses scp to place the token on disk. Validation happens. The script is called to remove the token. Certificate issued.

For mod_md to support this, it needs a call-out like MDMessageCmd that provides an action (http-01-add|http-01-remove), domain, the token and the key authorization (could be the filename in $MD_STORE/challenges) to be stored. It needs to be configured on a per MD basis. Your current handler is great for local certificates!

Ideally, the MDHttpProvider directive is compatible with MDMessageCmd - in fact, I want the same script to handle both types of events. (My message events handler already has the necessary configuration and deployment infrastructure.)

When MDHttpProvider is not configured in a domain, your handler would continue to work. When it is configured, you will never see the GET (it goes to the remote server). But you do get the challenge (invoke and wait for the provider script), respond with the empty object. And when the status changes from "pending" to "valid" or "invalid", invoke the provider to de-provision.

Of course, I expect the MDMessageCmd handler to be called with 'installed' as usual - it knows how to get the certificate installed.

It doesn't seem like a big change - the hardest parts are finding the right places to call out, and adding to your test system...

[icing]

That sounds a bit like MDChallengeDns01 but for other challenge types as well. Except it would be optional.

I think we are drifting towards a more general event interface, like MDMessageCmd but for all sorts of things - including challenge setup/teardowns. This could help solving use cases of other people as well, that have distributed installations where not all domain names end up at the same host.

In Apache internals, there are hooks for callbacks and these MDEventsshould also trigger one, I guess, so one can also write modules that act on these. One of the callbacks is the md internal one that calls a configurable command with event name, domain, an environment, etc. (I think the whole httpd should have something like this, but one thing at a time...)

An MDEvent is something that is either pre or post a thing that happened. It has a variable list of named parameters and a common list of variables (for all events), the environment.

An MDEventHandler is a callback (external command or internal hook) that gets invoked for all such events. It does not have a return value. (MDChallengeDns01 has a return value. It is not an event handler but a necessary part of the ACME process.)

We send events synchronous. They are invoked just before/right after things happen. The should not block for long, else they block the mod_md watchdog jobs. But it is no harm if they take a second or two.

Which events do you want for your use case?

[tlhackque]

Yes, I realized that the external http-01 events are almost the same as the Dns-01 case when I woke up!

I'll give this some more thought when I'm even more awake :-)

To start with, I think MDEvents are the current list used by MDMessageCmd and MDChallengeDns01'. Adding optional args and environ args to MDChallengeDns01 might work - 'setup and 'teardown meaning 'DNS' as now. Perhaps 'setup-http01 and 'teardown-http01` coud indicate the external http-01 case. Pass the domain name and content as for DNS - but also the token name & MD_STORE. It does need a return value - e.g. if the remote is unreachable and the token can't be placed.

Optional args would help for DNS too - for example, my DNS server uses TSIG keys, so being able to pass the filename for the key and server name to use would help...

You'll see in my current handler that one of the challenges is config data - reserving a place in the MD_STORE tree for event config would be simpler for the handler.

There might be a case for an API to store/access config - but it needs to be simple enough for a script...

One issue with passing data via args and environ variables is visiblity. E.g. the ps and --password problem.

As I mentioned elsewhere, being able to deploy on renewal would be helpful - perhaps a root thread could be kept around for notifying trusted event handlers. This would also eliminate the need for periodic reloads of the server to install...

Hope this helps - I'll give this some more thought later.

[tlhackque]

I pushed a commit with a VERY preliminary and rough version of md_events and some supporting files (for dns-01). Read the commit comment for details.

This is not production ready - but hopefully is a good base for defining MDEvents.

FWIW, I used the term "events" for the current MDMessageCmd interface from day 0...

Out of time for now...

[tlhackque]

I noticed one related issue in dns-01 handling.

Often, it can take some time for a challenge to become visible to the validation servers. This may be due to negative caching in the DNS servers they use, or other network effects.

In any case, with other clients I've seen it take up to several minutes between when the record is placed in the DNS and when the validation servers admit to seeing it. Some clients try to poll an external DNS server to see if the record is visible, and don't send the 'proceed' to LE until it is. I've seen 50-100 such probes at 10 sec intervals before success... Even then, the LE servers can take a while to see the record.

Obviously, a MDChallengeDns01 script should not block long enough for this to settle out. Instead, I think the mod_md process should requeue/suspend its thread after a successful invocation of the MDChallengeDns01 program and before sending the "proceed". After the delay, it can send the "proceed".

Waiting minimizes the chance of negative caching.

A good value for the delay time is hard to estimate. It should be configurable. There's really no urgency. Perhaps MDDnsPropagationTime with a default of 5 or 10 minutes?

[icing]

• I agree that the domain setup/status polling needs to suspend/resume jobs, so that other renewal may proceed in the mean time.
• MDCAChallenges should better be named MDChallenges, maybe I make an alias.
• An MDChallengeStartDelay duration [ chtype=duration ... ] could do the job.

[tlhackque]

MDChallengeResponseDelay might be better. The validation server starts the challenge, mod_md responds.... The challenge is started - it's the response that's delayed.

I don't have a problem with MDCAChallenges - while MDChallenges is better, not sure it's worth adding another alias that has to be documented/explained/tested...

[tlhackque]

Also, the required delay depends on the challenge type. DNS can take time to propagate. tls-apln and mod_md internal http are instantly available. external http usually is visible when the script returns from the copy commend - but there might be a cache/CDN to update. ..

So scope needs to be per-MDomain. Since mod_md can announce ability to respond to more than one challenge type, server can choose.

So perhaps syntax is `MDChallengeResponseDelay http-01=[duration] tls-alpn=[duration] ...

OTOH, it doesn't have to be TOO precise. Doesn't make sense for internal responses to wait 10s of minutes for getting the first certificate. But the slow ones have to work. Cert renewal happens every 60 days, so a coarse solution should suffice.

[tlhackque]

I pushed a document (event_interface_notes.txt) that tried to collect the threads of this conversation.