conduit icon indicating copy to clipboard operation
conduit copied to clipboard

Feature: Force stop. Pipeline status should be stopped, not degraded

Open raulb opened this issue 1 year ago • 1 comments

Feature description

When a user tries to force stop a pipeline, the result is that the status of the pipeline ends up being degraded and not stopped.

I think it would make sense that it's stopped instead.

Steps to reproduce

  1. Run a pipeline
  2. Force stop it: curl -X POST localhost:8080/v1/pipelines/my-pipeline/stop -d '{"force": true}'
  3. Fetch status pipeline: curl 'http://localhost:8080/v1/pipelines/my-pipeline' | jq .state.status

Expected

STATUS_STOPPED

Actual

STATUS_DEGRADED

raulb avatar Oct 04 '24 10:10 raulb

This is what the statuses mean right now:

  • STATUS_STOPPED - the pipeline gracefully stopped, any in-flight records were flushed, acks were delivered back to the source, no error was encountered while stopping the pipeline.
  • STATUS_DEGRADED - an error caused the pipeline to stop. Any in-flight records were dropped, not all acks might have been delivered back to the source. Starting a degraded pipeline again could potentially result in duplicated data (depending on how the destination connector handles records), since some data might get re-delivered.

So given that force stopping a pipeline results in the second behavior (i.e. nodes stop immediately without flushing in-flight records) I'm not sure it would be correct to set the pipeline to "stopped".

lovromazgon avatar Oct 04 '24 17:10 lovromazgon

@lovromazgon Good point. I think it's important to make this distinction between gracefully stopped vs not. This is being documented on https://github.com/ConduitIO/conduit-site/pull/168.

raulb avatar Oct 21 '24 10:10 raulb