astro-sdk
astro-sdk copied to clipboard
Try to change Astro Python SDK check example DAGs to use GreatExpecatations
Context We've been having discussions about the Astro Python SDK check strategy. It feels there is an overlap with Great Expectations.
As part of this, we created:
- https://github.com/astro-projects/astro/issues/298
- https://github.com/astro-projects/astro/issues/301
The idea with this ticket is, once the ticket https://github.com/astro-projects/astro/issues/301 is completed, to change these (check) example DAGs to use the Great Expectations operator.
Acceptance criteria
- [ ] Pair with someone who has been involved with the development of the GreatExpectations operator (Benji Lampel?)
- [ ] Have a version of the (check) example DAGs using Great Expectations
- [ ] Based on this experience, propose a way to move forward, documenting the findings
@dimberman To take a look after completing https://github.com/astro-projects/astro/pull/322
Since this ticket was initially logged, @utkarsharma2 has led a new work stream on data validation. We can create new ones in future as needed. @utkarsharma2 it would be great if you could add links following up on the current validation story state.