nprintml
nprintml copied to clipboard
background tasks
The pipeline might be sped up by performing in the background sub-tasks which are necessary for completion but not for the pipeline itself.
For example, the user may have requested that nPrint data and/or features are saved to disk; however, the pipeline needn't necessarily wait for these to be serialized and written to disk before continuing to the next step in the pipeline.
Instead, such work might be performed by a background thread/process (off of a standard queue), while the pipeline proceeds.
This should be considered and tested to ensure we're able to measurably improve performance this way and to ensure that whatever we do doesn't interfere with the performance of the pipeline itself.
Agreed on the testing -- this seems significantly less important given the results in #55, though as the files grow in size, it could become more important to consider this.