astro-sdk icon indicating copy to clipboard operation
astro-sdk copied to clipboard

Try to change Astro Python SDK check example DAGs to use GreatExpecatations

Open tatiana opened this issue 2 years ago • 1 comments

Context We've been having discussions about the Astro Python SDK check strategy. It feels there is an overlap with Great Expectations.

As part of this, we created:

  1. https://github.com/astro-projects/astro/issues/298
  2. https://github.com/astro-projects/astro/issues/301

The idea with this ticket is, once the ticket https://github.com/astro-projects/astro/issues/301 is completed, to change these (check) example DAGs to use the Great Expectations operator.

Acceptance criteria

  • [ ] Pair with someone who has been involved with the development of the GreatExpectations operator (Benji Lampel?)
  • [ ] Have a version of the (check) example DAGs using Great Expectations
  • [ ] Based on this experience, propose a way to move forward, documenting the findings

tatiana avatar Apr 08 '22 09:04 tatiana

@dimberman To take a look after completing https://github.com/astro-projects/astro/pull/322

kaxil avatar May 03 '22 18:05 kaxil

Since this ticket was initially logged, @utkarsharma2 has led a new work stream on data validation. We can create new ones in future as needed. @utkarsharma2 it would be great if you could add links following up on the current validation story state.

tatiana avatar Jan 17 '23 09:01 tatiana