spark-expectations icon indicating copy to clipboard operation
spark-expectations copied to clipboard

Published 20 hours ago • verified publisher icon Nike-Inc

[FEATURE] Update documentation

Open IMC07 opened this issue 9 months ago • 0 comments

Is your feature request related to a problem? Please describe. Add explanation of the meaning of the configuration settings: enable_for_source_dq_validation and enable_for_target_dq_validation

The explanation: SE has three phases:

  1. Take the source dataframe and run query_dq and agg_dq on it, when you enable_for_source_dq_validation is true
  2. If the first step succeeds, run row_dq
  3. Take the dataframe from row_dq and run query_dq and agg_dq on it, when you enable_for_target_dq_validation is true

Would also be nice to have the explanation of all the other configuration settings in place.

IMC07 avatar May 23 '24 13:05 IMC07