openff-evaluator
openff-evaluator copied to clipboard
Investigate feasibility of turning a DaskKubernetesBackend loose
Is your feature request related to a problem? Please describe.
Currently DaskKubernetesBackend is set-up to auto-shutdown the k8s cluster when the server or backend is closed or if an error is raised (this works with the error... sometimes.) I think that means for a long-running job, one needs to maintain a process that pings the EvaluatorServer at regular intervals to check if everything's done yet.
Describe the solution you'd like
It would be quite nice to just spin up an independent EvaluatorServer on k8s and submit a job to it, and just let it run remotely. I think this can already be technically be done, I'm just not sure if it's a great idea.
Describe alternatives you've considered
Additional context