distilabel
distilabel copied to clipboard
[FEATURE] Define flag allowing to execute light steps in main process
Description
The current Pipeline
implementation creates a subprocess to execute each Step
. There are some Step
s that only execute simpler transformations and that are very light to execute, therefore, the overhead added of creating a process and then sending the batches via queues is not worth it.
Add a flag for the Step
and GlobalStep
that allows executing the step in the main process without the need of creating an additional process.