airflow icon indicating copy to clipboard operation
airflow copied to clipboard

fix BigQueryInsertJobOperator's return value and openlineage extraction in deferrable mode

Open kacpermuda opened this issue 1 year ago • 0 comments

This PR fixes two things:

  1. Return value of BigQueryInsertJobOperator's execute_complete() (so execution in deferrable mode) will now be an actual BQ job id instead of None. As described in this docs, when deferring: No state will persist, such as local variables or attributes set on self., so assigning value to self.job_id within execute() had no effect and was not actually used in execute_complete().

  2. We are re-assigning the job_id in execute_complete() as an instance attribute, so that we can later use it within OpenLineage method even when in deferrable mode. We are also re-creating any attributes (hook) that may be missing as a result of the state keeping logic mentioned above.


^ Add meaningful description above Read the Pull Request Guidelines for more information. In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed. In case of a new dependency, check compliance with the ASF 3rd Party License Policy. In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

kacpermuda avatar Jun 27 '24 14:06 kacpermuda