allow integration clock to be altered during runtime

[rolandshoemaker]

Currently Clock() in cmd/clock_integration.go returns a clock that always returns the time specified in the envvar FAKECLOCK. This means that if we want to change the time we need to restart all the relevant components so that they instantiate new clocks.

Ideally we could change this during runtime. The easy solution would be to change the returned clock to always check FAKECLOCK before returning time, this way we can change the time simply by changing the envvar. One thing that is slightly complicated here is that clock.FakeClock will advance when things like Clock.Sleep are called, which we rely on in a number of cases, so we need to not tread on the advancing. The naive solution is to simply record the value of FAKECLOCK and check if it has changed each time we need to return time. If it has changed we update the clock to the new time, if not we simply return the current clock state.

[rolandshoemaker]

@jsha rightly points out that you can't really change the env of a running proc. Better to do this as a HTTP endpoint under /debug or something.

[beautifulentropy]

It's doable gdb attach and calling putenv():

https://www.systutorials.com/how-to-change-another-process-environment-variable-in-linux/

[aarongable]

Here's my concrete proposal:

• newStatsRegistry already registers a bunch of http handlers for various debug things. Let's pass the clock (which may or may not be fake) into this function, and then if it is a fake clock and we're building with the -integration build flag register a new handler which receives a textual timestamp, parses it as a time.Time, and calls fc.Set() to set the process' fake time to the requested timestamp.
• Let's move startservers from python to go. We've wanted to do this for a while, and it's another important step towards getting rid of our python test harness entirely. But in this case it gives us one distinct advantage: now there will be a piece of Go code which knows the address and debug port of every single boulder service running in the integration test environment. That same piece of code can then expose a updateFakeClock(time.Time) function which posts the requested time to the new endpoint on every service at the same time. Maybe also a resetFakeClock() which posts time.Now() everywhere. Then any non-t.Parallel() integration test can use that to easily manipulate the fake time.

[jsha]

If possible I'd like to avoid exposing a network accessible call that sets the clock. Even if it's protected behind the integration build flag, it just feels too close to making a mistake. Right now the fake clock is protected by three things:

• It needs the integration build tag.
• It can only be set at startup.
• You have to control the local environment at startup (which is a high degree of control over the running binary).

I forget why we originally wanted this. Certainly it would be faster to avoid startup and shutdown of the whole boulder stack, but I think there's already some low hanging fruit we can hit to speed that up, like starting all components at once rather than waiting on health check to pass for each one.