airflow-provider-great-expectations
airflow-provider-great-expectations copied to clipboard
astronomer
Operator Overhaul
This commit consolidates work from a PR opened in the astronomer- providers repository (https://github.com/astronomer/astronomer-providers/pull/503). The changes include making the operator more Airflow-centric, notably by handling connections, and allowing the checkpoint to be built automatically with provided default values. The goal is to make using Great Expectations more streamlined with Airflow.
@talagluck I moved your commits over here as well, so we can just use this one branch to do the operator updates. I'm going to add that extra link in for this and try to test anything at all tomorrow (Friday).
One big issue remaining with data docs is the actual link being given in my testing has been a local file. This seems to not work from inside the container (clicking the button in the Airflow UI doesn't work), but I can copy and paste the url and it works in my browser (running locally, this does not work in astro).
@alexsherstinsky really appreciate your thoughtful feedback here. Tried to implement everything that I knew made sense to implement--especially the GE util you pointed out, that was a great improvement. I think it should be all set now, would appreciate an approval if you think it's ready!
@virajmparekh @kaxil , I would definitely appreciate some input on how we can get the data docs working from a local file.
@talagluck I'm noticing that the SQLite pytest is creating a file called host when it's running, clearly making the actual SQLite "database", I think this should get cleaned up or just moved to a tests/sqlite folder.