spark-expectations icon indicating copy to clipboard operation
spark-expectations copied to clipboard

[FEATURE] Enhancements to stats table

Open pinnapuvijay opened this issue 1 year ago • 3 comments

Is your feature request related to a problem? Please describe. During our testing/adoption,

  1. we feel like a framework level flag to determine how the stats should be stored(relational vs nested constructs)
  2. Storing the stats also for the rules that pass would be helpful
  3. Capturing the actual value of the rule output( especially aggregations) so they can be referenced to determine if a rule is passed/failed in the subsequent executions of the rule. Basically, looking at historical metric(last 7 days avg, etc) to determine the status of the rule.
  4. Not all rules get executed even if one of the rules marked as FAIL fails. A framework level flag to exit or continue with the other rules when a rule fails would.
  5. stats for the rule that is marked as FAIL need to be logged as well when the rule fails.

Describe the solution you'd like A clear and concise description of what you want to happen.

Describe alternatives you've considered May be a stats_detail that supplements the stats table will also be a good alternative. This detail table will be in relational format and would also contain granular data that is requested above

Additional context Add any other context or screenshots about the feature request here.

pinnapuvijay avatar Oct 16 '23 14:10 pinnapuvijay