distilabel
distilabel copied to clipboard
[FEATURE] Assign a load stage to steps of a pipeline manually
Is your feature request related to a problem? Please describe.
In https://github.com/argilla-io/distilabel/pull/760 the logic to load the steps in stages was added. Currently, load stages are defined by the position of the GlobalStep
s in the pipeline.
Describe the solution you'd like I would like to be able to specify in which stage a step should be loaded. This could be helpful to allow executing bigger pipelines on machines with lower resources at the cost of a lower parallelisation. For example, a pipeline that uses several LLMs could be executed in a machine with 1 GPU if there is only one LLM being executed at a time i.e. there is only 1 step using a GPU per stage.