Neuraxle icon indicating copy to clipboard operation
Neuraxle copied to clipboard

Bug: Reuse workers on multiple transforms inside QueuedPipeline (SequentialQueuedPipeline Or ParallelFeatureUnionQueuedPipeline)

Open alexbrillant opened this issue 5 years ago • 0 comments

Instead of stopping the workers on _did_transform, it would be better to stop them on teardown or upon saving.

This is how it looks like right now in the code:

  def _did_transform(self, data_container: DataContainer, context: ExecutionContext) -> DataContainer:
        """
        Stop all of the workers after transform. Also, join the data using self.data_joiner.
        :param data_container: data container
        :type data_container: DataContainer
        :param context: execution context
        :type context: ExecutionContext
        :return: data container
        :rtype: DataContainer
        """
        for name, step in self[:-1]:
            step.stop()

        return self.data_joiner.handle_transform(data_container, context)

alexbrillant avatar Oct 08 '20 21:10 alexbrillant