Stop pipeline when ExampleValidator finds anomalies

[rcrowe-google]

trafficstars

In cases where any anomalies that are found by ExampleValidator should stop the pipeline execution, the standard components don't have a way of stopping it. That's because none of the standard components depend on the results from ExampleValidator.

We can write a custom component to take the ExampleAnomalies artifact from ExampleValidator, along with the Examples artifact from ExampleGen, and parse the ExampleAnomalies artifact.

1. If no anomalies are found, the component passes through the Examples artifact from ExampleGen
2. If any anomalies are found, the component will fail

Downstream components which depend on the Examples artifact from ExampleGen can instead depend on the Examples artifact from this new custom component. When it fails, the pipeline will stop.

An enhancement of this would be to either include a user code module to check the type of anomaly and decide whether to fail or not, or some sort of configuration parameter.

[deep-diver]

this sounds very reasonable. what is the current status of this idea?

[deep-diver]

and is there a way to terminate pipeline from a component? maybe just raise an error?

[rcrowe-google]

I would think system that sys.exit() would work.

[deep-diver]

sounds good and sounds easy to implement I guess. What is your plan on this?

[rcrowe-google]

I don't have a plan to work on this, it's a project idea for someone to pick up. I agree, it sounds fairly simple, except for the part about user code to check the type of anomaly and decide whether to fail or not, or some sort of configuration parameter. Even that shouldn't be too hard.