distilabel icon indicating copy to clipboard operation
distilabel copied to clipboard

[FEATURE] Assign a load stage to steps of a pipeline manually

Open gabrielmbmb opened this issue 7 months ago • 0 comments

Is your feature request related to a problem? Please describe. In https://github.com/argilla-io/distilabel/pull/760 the logic to load the steps in stages was added. Currently, load stages are defined by the position of the GlobalSteps in the pipeline.

Describe the solution you'd like I would like to be able to specify in which stage a step should be loaded. This could be helpful to allow executing bigger pipelines on machines with lower resources at the cost of a lower parallelisation. For example, a pipeline that uses several LLMs could be executed in a machine with 1 GPU if there is only one LLM being executed at a time i.e. there is only 1 step using a GPU per stage.

gabrielmbmb avatar Jul 23 '24 14:07 gabrielmbmb