great_expectations icon indicating copy to clipboard operation
great_expectations copied to clipboard

Issue with ignore_row_if argument in certain expectations

Open NehaNRane opened this issue 10 months ago • 0 comments

I've encountered a problem with the ignore_row_if argument in three expectations: expect_column_pair_values_a_to_be_greater_than_b, expect_multicolumn_sun_to_equal, and expect_compound_columns_to_be_unique. While the argument works as expected when either_value_is_missing or any_value_is_missing is passed, it behaves unexpectedly when both_values_are_missing / all_values_are_missing or neither / never is used.

Specifically, when using both_values_are_missing / all_values_are_missing or neither / never in the ignore_row_if argument, the following metrics are incorrect:

  • unexpected_count
  • unexpected_percent
  • partial_unexpected_list
  • unexpected_percent_total
  • unexpected_percent_nonmissing

This issue impacts the accuracy of the expectations and could lead to incorrect interpretations of data quality.

Steps to Reproduce:

  1. Use the affected expectations (expect_column_pair_values_a_to_be_greater_than_b, expect_multicolumn_sun_to_equal, expect_compound_columns_to_be_unique).
  2. Pass both_values_are_missing / all_values_are_missing or neither / never in the ignore_row_if argument.
  3. Observe the incorrect values for the mentioned metrics.

Expected Behavior: The ignore_row_if argument should accurately handle the specified conditions ( both_values_are_missing / all_values_are_missing or neither / never) and produce correct metrics for the affected expectations.

Additional Context:

  • Great Expectations version: 0.18.12
  • Python version: 3.9.6
  • snowflake-connector-python: 3.8.1
  • snowflake-sqlalchemy: 1.5.1

NehaNRane avatar Apr 12 '24 14:04 NehaNRane