distilabel
distilabel copied to clipboard
[BUG] Batches are not prepended correctly when CTRL+C
Describe the bug
When stopping a pipeline with CTRL+C, the batches from the step input queues are prepended back to the _BatchManager
so no information is lost.
The _BatchManagerStep
contains a data
dictionary with batches from the predecessors steps, but when we're reingesting the batches from the input queue of a certain step, the batch.step_name
is now equal to the step (not the predecessor step). This causes a KeyError
exception.
Expected behaviour
The batch is added correctly back to the _BatchManagerStep
and it can be used in the next pipeline execution if use_cache=True
.