astro-sdk icon indicating copy to clipboard operation
astro-sdk copied to clipboard

Improvements on OpenLineage implementation

Open utkarsharma2 opened this issue 2 years ago • 0 comments

Please describe the feature you'd like to see

  1. ~~We can add better unit-test cases where we test the output of get_openlineage_facets_on_complete and get_openlineage_facets_on_start right now we rely on Marquez UI to test working~~ We should rename the tests to be more searchable
  2. We can refactor since this function is too big and as a result of this we can also make them unit-testable https://github.com/astronomer/astro-sdk/blob/e09a88ce86578c8a52cc450d48ab92a9e8ace748/python-sdk/src/astro/sql/operators/load_file.py#L211
  3. We should also avoid running computations as part of get_openlineage_facets_on_complete or get_openlineage_facets_on_start since it's a reporting mechanism - https://github.com/astronomer/astro-sdk/blob/e09a88ce86578c8a52cc450d48ab92a9e8ace748/python-sdk/src/astro/sql/operators/load_file.py#L227
  4. For postgres - If a user creates a table object with a different schema, we are not currently not taking it into account - https://github.com/astronomer/astro-sdk/blob/main/python-sdk/src/astro/databases/postgres.py#L219-L226

utkarsharma2 avatar Feb 10 '23 06:02 utkarsharma2