spark-expectations
spark-expectations copied to clipboard
[FEATURE] Provide an interface for users to implement custom rule types that can be integrated with spark-expectations
Is your feature request related to a problem? Please describe.
As of now spark-expectations
support row_dq
, agg_dq
and query_dq
rule types. If users want to add their own rule types for example running data quality rules on subset of data using windowing functions
SE should provide a way for users to integrate such rule types
Describe the solution you'd like TBD