distilabel icon indicating copy to clipboard operation
distilabel copied to clipboard

[FEATURE] Define flag allowing to execute light steps in main process

Open gabrielmbmb opened this issue 10 months ago • 0 comments

Description

The current Pipeline implementation creates a subprocess to execute each Step. There are some Steps that only execute simpler transformations and that are very light to execute, therefore, the overhead added of creating a process and then sending the batches via queues is not worth it.

Add a flag for the Step and GlobalStep that allows executing the step in the main process without the need of creating an additional process.

gabrielmbmb avatar Apr 23 '24 13:04 gabrielmbmb