zenml icon indicating copy to clipboard operation
zenml copied to clipboard

Upgrade Great Expectations integration to 1.0.4

Open wjayesh opened this issue 5 months ago • 4 comments

Describe changes

I upgraded great expectations to 1.0.4.

  • Data Profiling is no longer supported since profiles have been moved out of Great Expectations open-source.
  • Data Validation is amended to now take in a list of Expectations directly from the user, or an Expectation Suite that the user has already created.
  • There's a new class to help users define their Expectations and expectation arguments.
  • The materializer is updated to better support the updated CheckpointResult object.
  • Checkpoints now take in a validation definition which is built with a batch definition and expectation suite. More here.

Testing

This has only been tested with a subset of Expectations on an ephemeral storage so far. Some more tests need to happen and will be done incrementally.

Pre-requisites

Please ensure you have done the following:

  • [ ] I have read the CONTRIBUTING.md document.
  • [ ] If my change requires a change to docs, I have updated the documentation accordingly.
  • [ ] I have added tests to cover my changes.
  • [ ] I have based my new branch on develop and the open PR is targeting develop. If your branch wasn't based on develop read Contribution guide on rebasing branch to develop.
  • [ ] If my changes require changes to the dashboard, these changes are communicated/requested.

Types of changes

  • [ ] Bug fix (non-breaking change which fixes an issue)
  • [ ] New feature (non-breaking change which adds functionality)
  • [ ] Breaking change (fix or feature that would cause existing functionality to change)
  • [ ] Other (add details above)

wjayesh avatar Sep 18 '24 15:09 wjayesh