fyrd icon indicating copy to clipboard operation
fyrd copied to clipboard

Merge python-pipeline into this project

Open MikeDacre opened this issue 8 years ago • 1 comments

The python-pipeline project is an effort to make it easy to create complex pipelines with python. It isn't that useful outside of a multithreading environment, so it makes sense to merge it in here and implement native multithreading in that project through the cluster module.

Rather than keep it as a separate project, the pipeline package should be added as a separate package alongside cluster to be used if the user wishes. However, it is important that its usage is not required in order to use the cluster package. ie. pipeline should depend on cluster, but cluster should not depend on pipeline.

MikeDacre avatar Jun 28 '16 05:06 MikeDacre

Most of the pipeline code is not needed, but add a Pipeline class that jobs can be added to.

This class should write itself to disk and be easily restorable from the name and directory alone. All jobs will have the track option in #80 set, which will allow pipeline restoration.

MikeDacre avatar Oct 22 '17 20:10 MikeDacre