modern-data-warehouse-dataops icon indicating copy to clipboard operation
modern-data-warehouse-dataops copied to clipboard

Add data quality part for parking_sensor_synapse using great expectations library

Open cchenshu opened this issue 1 year ago • 1 comments

Type of PR

  • Code changes

Purpose

  • Add the data validataion part for parking_sensor_synapse using great expectations library, following the similar logic in the databricks.
  • For both 02_standardize and 03_transform notebook, add the following steps for data validation. 0. Create mount point path for spark job 1. Configure DataContext 2. Create a BatchRequest based on dataframe 3. Define Expecation Suite and corresponding Data Expectations 4. Configure a checkpoint and run Expectation suite using checkpoint

Does this introduce a breaking change? If yes, details on what can break

No

Author pre-publish checklist

  • [x] Executed test to prove my fix is effective or new feature works
  • [ ] No PII in logs
  • [ ] Made corresponding changes to the documentation

Validation steps

  • Run the notebooks: 02_standardize, 03_transform

Issues Closed or Referenced

  • Closes #issue_number
  • References #issue_number

cchenshu avatar Jun 20 '23 02:06 cchenshu