modern-data-warehouse-dataops
modern-data-warehouse-dataops copied to clipboard
Add data quality part for parking_sensor_synapse using great expectations library
Type of PR
- Code changes
Purpose
- Add the data validataion part for parking_sensor_synapse using great expectations library, following the similar logic in the databricks.
- For both 02_standardize and 03_transform notebook, add the following steps for data validation. 0. Create mount point path for spark job 1. Configure DataContext 2. Create a BatchRequest based on dataframe 3. Define Expecation Suite and corresponding Data Expectations 4. Configure a checkpoint and run Expectation suite using checkpoint
Does this introduce a breaking change? If yes, details on what can break
No
Author pre-publish checklist
- [x] Executed test to prove my fix is effective or new feature works
- [ ] No PII in logs
- [ ] Made corresponding changes to the documentation
Validation steps
- Run the notebooks: 02_standardize, 03_transform
Issues Closed or Referenced
- Closes #issue_number
- References #issue_number