spark-expectations
spark-expectations copied to clipboard
Nike-Inc
A Python Library to support running data quality rules while the spark job is running⚡
**Is your feature request related to a problem? Please describe.** It would be nice to be able to express the desire for the new data to "look like" the old...
**Is your feature request related to a problem? Please describe.** In order to continue to work with the Spark ecosystem, we need to ensure we can continue to support the...
**Is your feature request related to a problem? Please describe.** Data Quality rules are being managed differently by different teams. Some manage it by directly entering it into the table...
**Is your feature request related to a problem? Please describe.** Test all the features of SE with 3.5 and above, make respective changes for it to work. Update documentation accordingly...
**Is your feature request related to a problem? Please describe.** When a rule fails, irrespective of the action - if user chooses to create an incident for a rule, then...
**Is your feature request related to a problem? Please describe.** There should be Alert enabled for all types of DQ when there is "ignore" scenario. **Describe the solution you'd like**...
**Describe the bug** Spark expectations fails on databricks runtime 14 and above with following error SparkExpectationsMiscException: Input rules_df is not of dataframe type **To Reproduce** Steps to reproduce the behavior:...
