distilabel icon indicating copy to clipboard operation
distilabel copied to clipboard

Published 20 hours ago • verified publisher icon argilla-io

[FEATURE] Sequential execution for local pipeline

Open gabrielmbmb opened this issue 10 months ago • 1 comments

Description

As mentioned by @alvarobartt and Ellamind team, it would be nice to have a sequential model for executing the pipeline, in which no multiprocessing & batching is used.

The idea would be to load each step, process all the data, unload the step, load the next step, ...

gabrielmbmb avatar Apr 25 '24 09:04 gabrielmbmb

Just adding here that a goal of this would be to enable proper debugging within steps/tasks in the pipeline. Thanks for picking this up! :)

bjoernpl avatar Apr 25 '24 21:04 bjoernpl