Make `max_time` in `SynchronousComputeService.start` check occur in separate thread
Currently, max_time is checked alongside max_tasks at the end of each task execution. For long-running tasks (e.g. hours), this could make max_time a bit useless. Instead, we'd like for max_time to be checked periodically via a separate thread in the same way we are doing heartbeats.
Actually, it's very useful to have two separate max times that can be set, with different meanings:
-
max_time_soft: amount of time to run, beyond which the service will shut off after it has finished a cycle -
max_time_hard: amount of time to run, beyond which the service will shut off no matter what it is doing
These can of course be used together, with max_time_soft used to set the time beyond which no new tasks should be taken on (perhaps because the service is running in an HPC job, and there wouldn't be enough time to finish anything), and max_time_hard could be set to just before the job absolutely must finish, ensuring clean shutdown even if a task isn't completed.