Neuraxle
Neuraxle copied to clipboard
Bug: Reuse workers on multiple transforms inside QueuedPipeline (SequentialQueuedPipeline Or ParallelFeatureUnionQueuedPipeline)
Instead of stopping the workers on _did_transform, it would be better to stop them on teardown or upon saving.
This is how it looks like right now in the code:
def _did_transform(self, data_container: DataContainer, context: ExecutionContext) -> DataContainer:
"""
Stop all of the workers after transform. Also, join the data using self.data_joiner.
:param data_container: data container
:type data_container: DataContainer
:param context: execution context
:type context: ExecutionContext
:return: data container
:rtype: DataContainer
"""
for name, step in self[:-1]:
step.stop()
return self.data_joiner.handle_transform(data_container, context)