zenml
zenml copied to clipboard
Upgrade Great Expectations integration to 1.0.4
Describe changes
I upgraded great expectations to 1.0.4.
- Data Profiling is no longer supported since profiles have been moved out of Great Expectations open-source.
- Data Validation is amended to now take in a list of Expectations directly from the user, or an Expectation Suite that the user has already created.
- There's a new class to help users define their Expectations and expectation arguments.
- The materializer is updated to better support the updated CheckpointResult object.
- Checkpoints now take in a validation definition which is built with a batch definition and expectation suite. More here.
Testing
This has only been tested with a subset of Expectations on an ephemeral storage so far. Some more tests need to happen and will be done incrementally.
Pre-requisites
Please ensure you have done the following:
- [ ] I have read the CONTRIBUTING.md document.
- [ ] If my change requires a change to docs, I have updated the documentation accordingly.
- [ ] I have added tests to cover my changes.
- [ ] I have based my new branch on
develop
and the open PR is targetingdevelop
. If your branch wasn't based on develop read Contribution guide on rebasing branch to develop. - [ ] If my changes require changes to the dashboard, these changes are communicated/requested.
Types of changes
- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
- [ ] Other (add details above)