Make it less disruptive to queue new experiments

Open arteymix opened this issue 6 years ago • 1 comments

Jan 10 '20 18:01 arteymix

Luigi has an undocumented feature called "assistant". It allows background workers to be launched to execute tasks that are pending. New tasks can be queued by launching them with zero workers.

The only caveat is that the assistants have to be restarted when the pipeline code is modified.

There are systemd unit templates ready to use for that purpose.

https://luigi.readthedocs.io/en/stable/api/luigi.interface.html#luigi.interface.core.assistant

We could have a systemd timer to pull new work from the spreadsheet.

Oct 10 '25 23:10 arteymix