alchemiscale icon indicating copy to clipboard operation
alchemiscale copied to clipboard

Make `max_time` in `SynchronousComputeService.start` check occur in separate thread

Open dotsdl opened this issue 2 years ago • 1 comments

Currently, max_time is checked alongside max_tasks at the end of each task execution. For long-running tasks (e.g. hours), this could make max_time a bit useless. Instead, we'd like for max_time to be checked periodically via a separate thread in the same way we are doing heartbeats.

dotsdl avatar Apr 17 '23 19:04 dotsdl

Actually, it's very useful to have two separate max times that can be set, with different meanings:

  • max_time_soft: amount of time to run, beyond which the service will shut off after it has finished a cycle
  • max_time_hard: amount of time to run, beyond which the service will shut off no matter what it is doing

These can of course be used together, with max_time_soft used to set the time beyond which no new tasks should be taken on (perhaps because the service is running in an HPC job, and there wouldn't be enough time to finish anything), and max_time_hard could be set to just before the job absolutely must finish, ensuring clean shutdown even if a task isn't completed.

dotsdl avatar Apr 29 '23 19:04 dotsdl