Add another hook for extended waitready operations
In case a downstream of Microcluster (e.g. Micro*) does start some additional services besides the Microcluster daemon itself, there should be a way to be able to wait on those services to become ready. This effectively allows a user to wait until everything is ready so he can issue further commands/requests against the started services.
As part of its app struct Microcluster already has a Ready function which blocks until the daemon is ready. It returns right after the OnStart hook has finished.
So someone could already start additional services as part of the OnStart hook and simply don't return until the services are ready. The hook already receives the daemon's shutdown context which won't be cancelled until the daemon is requested to shutdown through signals.
While this might already be a valid option, we should detach the "wait" operation from the general startup procedure to allow the daemon coming up quickly.
Instead another hook PostStart(ctx) should be introduced which allows any downstreams to supply their own waitready logic.
This new hooks will be called in its own routine right after the daemon closed its ready channel. It's then the duty of the downstream to exit this hook if everything is started or the retries are exceeded. But we should not limit the maximal amount of retries or set a fixed deadline on the provided context. Instead the hook receives the daemon's shutdown context so it can react properly.
cc @mkalcok
As discussed for MicroCeph it would be useful to be able to differentiate which service / condition exactly should be waited on, e.g.
microcephd readyonce the microcephd is upmon quorum formedonce the mons have quorumstorage readyonce the cluster can accept write i/o