[Feature] Optionally Distinguish Between Different Failure Severity Levels

[jdimatteo]

Is your feature request related to a problem? Please describe.

Sometimes some expectation failures are more severe than others, and analysts would like to be alerted differently for more severe errors so that they can follow up on more severe errors sooner (and maybe follow up on less severe errors later at a more convenient time). Currently expectations are either passing or failed, with no way to distinguish different severity levels.

Describe the solution you'd like

Define at least two failure severity levels:

1. Warning
2. Error

Warnings and errors should be reported separately in the CheckpointResult and the Data Docs.

For example, the CheckpointResult should include top level counts of each failure severity, e.g. with an additional member failure_counts that is a dictionary with items for each count like:

failure_counts = {
  'warning': 42,
  'error': 137,
}

This could be used to drive different behavior for programs that process the CheckpointResult.

Additionally it would be helpful to label each expectation with the severity level in the Data Docs and be able to filter by severity level, e.g. to only see failures that are an error severity level (and not warning severity level).

The severity level could default to error but could be overridden, e.g. using the same example detailed in #4170:

(venv) jdimatteo@erel:~/dev/great_expectations_fork_2/feature_example$ git diff -U6
diff --git a/feature_example/great_expectations_example.py b/feature_example/great_expectations_example.py
index f8caff28c..e6e3bd31b 100644
--- a/feature_example/great_expectations_example.py
+++ b/feature_example/great_expectations_example.py
@@ -56,12 +56,13 @@ suite.add_expectation(
     expectation_configuration=ExpectationConfiguration(
         expectation_type="expect_column_values_to_be_between",
         kwargs={
             "column": "age",
             "min_value": 0,
             "max_value": 150,
+            "severity_level": "WARNING",
         },
     )
 )
 context.save_expectation_suite(
     expectation_suite=suite, expectation_suite_name=SUITE_AND_ASSET_NAME
 )
(venv) jdimatteo@erel:~/dev/great_expectations_fork_2/feature_example$ 

Describe alternatives you've considered

We've considered just generating different checkpoints for each severity level (one checkpoint for warning severity expectations and another for error severity expectations), but this results harder to read Data Docs because expectations for the same table are split between two different suites / Data Docs pages (when the same table has expectations with different severity levels).

Additional context

We intend to use this feature in combination with #4170 and #4181 and #4185, and the example here builds off the example detailed in #4170.

[austiezr]

Hey @jdimatteo ! Thanks for reaching out with these; a lot of super interesting functionality here. We'll review internally over the next week and continue the conversation.

[kyleaton]

Hey @jdimatteo, We have reviewed this item and added it to our feature roadmap. We do not have an estimated time to start work on this, but we will notify you when we do.

[rdodev]

Added to roadmap.