distilabel icon indicating copy to clipboard operation
distilabel copied to clipboard

[BUG] Batches are not prepended correctly when CTRL+C

Open gabrielmbmb opened this issue 9 months ago • 0 comments

Describe the bug When stopping a pipeline with CTRL+C, the batches from the step input queues are prepended back to the _BatchManager so no information is lost.

The _BatchManagerStep contains a data dictionary with batches from the predecessors steps, but when we're reingesting the batches from the input queue of a certain step, the batch.step_name is now equal to the step (not the predecessor step). This causes a KeyError exception.

Expected behaviour The batch is added correctly back to the _BatchManagerStep and it can be used in the next pipeline execution if use_cache=True.

gabrielmbmb avatar May 18 '24 11:05 gabrielmbmb