Ability to add custom facet in OpenLineage events

[anandhimurali]

Motivation Currently, users need to write custom extractor inorder to add custom facets in the lineage events. Instead, providing a way to inject custom facets by accepting a config.

Modifications The get_custom_facets function in the OpenLineage provider is enhanced to fetch the Airflow config AIRFLOW__OPENLINEAGE__CUSTOM_FACET_FUNCTIONS, execute those functions and append their return values to the run facet of the lineage event for all the operators.

Result Enables users to add custom facets to the lineage events by defining functions instead of custom extractor.

Configs documentation for AIRFLOW__OPENLINEAGE__CUSTOM_FACET_FUNCTIONS

---

^ Add meaningful description above Read the Pull Request Guidelines for more information. In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed. In case of a new dependency, check compliance with the ASF 3rd Party License Policy. In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[boring-cyborg[bot]]

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst) Here are some useful points:

• Pay attention to the quality of your code (ruff, mypy and type annotations). Our pre-commits will help you with that.
• In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
• Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
• Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
• Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
• Be sure to read the Airflow Coding style.
• Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits. Apache Airflow is a community-driven project and together we are making it better 🚀. In case of doubts contact the developers at: Mailing List: [email protected] Slack: https://s.apache.org/airflow-slack

[XD-DENG]

Hi @eladkal , @anandhimurali is my colleague from the same team and I'm really excited about her 1st PR to Airflow!

Do you have an idea who among the committers may be the best to help review this openlineage-related change?

Any suggestion/input would be greatly appreciated. Thanks a lot!

[eladkal]

Ping @mobuchowski for review :)

[mobuchowski]

Hey @anandhimurali @XD-DENG I think ability to add custom facets to arbitrary operators is a great idea.

I have two points I think it's worth addressing:

1. Whether we pass all the necessary info. I think given task_instance we can call get_template_context and get all the data, so this might not be an issue.
2. Whether this acts "per DAG/Task" or it's a mechanism that works for all the tasks equally - this might be defined inside the function so it still might not be an issue other than complicating the function a bit more.

Overall I like the solution, but might be good to specify the goals regarding 1) and 2) - because for sure, later, someone will come and ask "how to make this custom facet work only on some particular tasks" and "how do I pass some custom info to my function".

[kacpermuda]

Hey @anandhimurali @XD-DENG, is there anything i could help you with here? I really like the idea and would like to see it happen, let me know 😄

[anandhimurali]

Hey @anandhimurali @XD-DENG, is there anything i could help you with here? I really like the idea and would like to see it happen, let me know 😄

Hi @kacpermuda, Thanks for the interest and I'm excited to hear that. Sorry I've been busy for the last few weeks. I'll look into the comments this week.

[anandhimurali]

I have two points I think it's worth addressing:

1. Whether we pass all the necessary info. I think given task_instance we can call get_template_context and get all the data, so this might not be an issue.
2. Whether this acts "per DAG/Task" or it's a mechanism that works for all the tasks equally - this might be defined inside the function so it still might not be an issue other than complicating the function a bit more.

Overall I like the solution, but might be good to specify the goals regarding 1) and 2) - because for sure, later, someone will come and ask "how to make this custom facet work only on some particular tasks" and "how do I pass some custom info to my function".

Hi @mobuchowski,

1. Yes, I think task_instance is sufficient to fetch necessary info about the run context.
2. The custom_facet_functions defined works equally for all tasks. Yes, user can choose to return None for scenarios where they won't to add the custom facet.

Let me know if this needs to be documented anywhere. Thanks.

[mobuchowski]

@anandhimurali any interest in getting this PR into shape and approved? I think this is good, really useful feature.

[anandhimurali]

@anandhimurali any interest in getting this PR into shape and approved? I think this is good, really useful feature.

@mobuchowski Yes, definitely. I just have to add some documentation and will get this done this week. Thanks and sorry for the delay.

[ahidalgob]

I think it would be good to rename this functionality to custom_run_facets, we might need to inject other facets too, specifically Job.facets.

This is a great feature:)

[anandhimurali]

I think it would be good to rename this functionality to custom_run_facets, we might need to inject other facets too, specifically Job.facets.

Addressed in 3a11b7625c896252c681bf1f217f0cce1f5eca57.

[anandhimurali]

Getting some documentation error on breeze build-docs openlineage. Trying to debug and fix this.

[potiuk]

Getting some documentation error on breeze build-docs openlineage. Trying to debug and fix this.

FYI. Sometimes --clean-build flag helps.

[anandhimurali]

FYI. Sometimes --clean-build flag helps.

Thanks @potiuk, --clean-build was helpful to overcome those documentation build errors.

[kacpermuda]

@anandhimurali If you need any help to make the CI pass let me know, If you run the pre-commits locally then all of the static checks will pass. When it comes to 2.7 compatibility tests, you simply need to add start_date to the test DAGs you are creating and it should also work fine.

[anandhimurali]

@anandhimurali If you need any help to make the CI pass let me know, If you run the pre-commits locally then all of the static checks will pass. When it comes to 2.7 compatibility tests, you simply need to add start_date to the test DAGs you are creating and it should also work fine.

@kacpermuda, Thanks for letting me know about this. The necessary fixes are addressed in 81a8282b60c5bb28e46b39672ca99707dd7c7454 after the pre-commit checks.

Let me know if any other changes are needed. Thanks.

[boring-cyborg[bot]]

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.

[mobuchowski]

Thanks @anandhimurali for implementing this feature 🚀