spark-expectations
spark-expectations copied to clipboard
Nike-Inc
A Python Library to support running data quality rules while the spark job is running⚡
**Is your feature request related to a problem? Please describe.** Add explanation of the meaning of the configuration settings: enable_for_source_dq_validation and enable_for_target_dq_validation The explanation: SE has three phases: 1) Take...
## Description Pluggy dependencies as follows `pluggy = "^1"` ## Related Issue #93 ## Motivation and Context Currently pluggy version is hardcoded to `pluggy = "1.0.0"` which is not allowing...
**Is your feature request related to a problem? Please describe.** Currently pluggy version is hardcoded to `pluggy = "1.0.0"` which is not allowing to use latest versions of `pytest`. **Describe...
**Describe the bug** When data is written to error table, there is no information on which rule(s) caused this row to be written to the error table **To Reproduce** Steps...
## Description I've expanded the notification capability within the spark_expectations library to include Zoom alongside the existing Slack and Team notifications. This improvement allows users to receive updates on data...
**Is your feature request related to a problem? Please describe.** Email alerts are enabled for each Expectation that is triggered. These emails are not verbose barring a few status indicators...
**Is your feature request related to a problem? Please describe.** Currently, the absence of notifications on Zoom chat within the spark_expectation library limits its accessibility to users reliant on Zoom...
**Describe the bug** A clear and concise description of what the bug is. **To Reproduce** Steps to reproduce the behavior: 1. Go to '...' 2. Click on '....' 3. Scroll...
**Is your feature request related to a problem? Please describe.** Data Profiling feature helps to analyse the trends on the table overtime. **Describe the solution you'd like** Explore profiling features...
**Is your feature request related to a problem? Please describe.** The query DQ feature does provide output in terms of boolean vales. The boolean value of FALSE does inform us...