astro-sdk icon indicating copy to clipboard operation
astro-sdk copied to clipboard

Openlineage support - Add Extractor for `MergeOperator`

Open kaxil opened this issue 1 year ago • 0 comments

Please describe the feature you'd like to see We should be able to extract open lineage info from the MergeOperator.

Describe the solution you'd like

  • Add a method get_openlineage_facets on the Operator (based on this doc)
  • The "PythonSDKExtractor" built-in https://github.com/astronomer/astro-sdk/issues/898 should be able to work with MergeOperator.get_openlineage_facets. Test it to make sure it works

Acceptance Criteria

  • [ ] Post the screenshot of how it looks in the Openlineage/Marquez UI
  • [ ] All checks and tests in the CI should pass
  • [ ] Unit tests (90% code coverage or more, once available)
  • [ ] Integration tests (if the feature relates to a new database or external service)
  • [ ] Docstrings in reStructuredText for each of methods, classes, functions and module-level attributes (including Example DAG on how it should be used)
  • [ ] Exception handling in case of errors
  • [ ] Logging (are we exposing useful information to the user? e.g. source and destination)
  • [ ] Improve the documentation (README, Sphinx, and any other relevant)

kaxil avatar Sep 23 '22 18:09 kaxil