fyrd
fyrd copied to clipboard
Merge python-pipeline into this project
The python-pipeline project is an effort to make it easy to create complex pipelines with python. It isn't that useful outside of a multithreading environment, so it makes sense to merge it in here and implement native multithreading in that project through the cluster module.
Rather than keep it as a separate project, the pipeline package should be added as a separate package alongside cluster to be used if the user wishes. However, it is important that its usage is not required in order to use the cluster package. ie. pipeline should depend on cluster, but cluster should not depend on pipeline.
Most of the pipeline code is not needed, but add a Pipeline
class that jobs can be added to.
This class should write itself to disk and be easily restorable from the name and directory alone. All jobs will have the track option in #80 set, which will allow pipeline restoration.