distilabel icon indicating copy to clipboard operation
distilabel copied to clipboard

[FEATURE] Improve caching mechanism to include reading from other cached pipelines

Open plaguss opened this issue 7 months ago • 0 comments

Is your feature request related to a problem? Please describe. If we have a pipeline a >> b >> c and we create another a >> b >> c >> d, we should be able to grab the first pipeline and continue just with step d. This is currently not possible, a given Pipeline only reads the folder corresponding to its own signature.

Describe the solution you'd like We should be able to read from all the pipelines created in the cache folder and check if there are pipelines that could be continued.

Describe alternatives you've considered Nothing

Additional context To be done after https://github.com/argilla-io/distilabel/pull/766

plaguss avatar Jul 18 '24 14:07 plaguss