distilabel icon indicating copy to clipboard operation
distilabel copied to clipboard

Create `CTRL + C` signal handler for gracefully pipeline stopping

Open gabrielmbmb opened this issue 11 months ago • 1 comments

Create a signal handler that captures SIGINT (ctrl + c) and stops the pipeline gracefully.

gabrielmbmb avatar Mar 11 '24 09:03 gabrielmbmb

This consistently stops me from being able to stop the execution of a pipeline if or when it got stuck somewhere. Would probably be best to have some override like exiting if crtl+c was pressed more than once or twice.

bjoernpl avatar Apr 25 '24 21:04 bjoernpl

To reproduce this @plaguss you can just create an infinite running step as:

import time

from distilabel.steps import StepInput, step
from distilabel.steps.typing import StepOutput

@step()
def InfiniteStep(*inputs: StepInput) -> StepOutput:
    time.sleep(100)

And the CTRL+C will be captured, but the SIGTERM signal won't be sent.

alvarobartt avatar May 10 '24 09:05 alvarobartt

Hi @bjoernpl in the next release this should be tackled: https://github.com/argilla-io/distilabel/pull/630

plaguss avatar May 14 '24 14:05 plaguss