data-prepper icon indicating copy to clipboard operation
data-prepper copied to clipboard

Add option to shut down and fail data prepper when any sub-pipelines are invalid

Open graytaylor0 opened this issue 9 months ago • 1 comments

Is your feature request related to a problem? Please describe. The default behavior of Data Prepper core is to only shut down when no sub-pipelines are valid. This can lead to a confusing user experience, as someone who wants all sub-pipelines to be running correctly for data prepper to start.

Describe the solution you'd like An optional parameter in the data-prepper-config.yaml to change this behavior to fail to start Data Prepper when any of the sub-pipelines are not configured properly. This would then throw an exception that stops Data Prepper when any sub-pipelines are misconfigured.

fail_on_any_sub_pipeline_failure: true // defaults to false

Describe alternatives you've considered (Optional) A clear and concise description of any alternative solutions or features you've considered.

Additional context Add any other context or screenshots about the feature request here.

graytaylor0 avatar May 13 '24 15:05 graytaylor0