fix BigQueryInsertJobOperator's return value and openlineage extraction in deferrable mode
This PR fixes two things:
-
Return value of
BigQueryInsertJobOperator'sexecute_complete()(so execution in deferrable mode) will now be an actual BQ job id instead of None. As described in this docs, when deferring:No state will persist, such as local variables or attributes set on self., so assigning value to self.job_id withinexecute()had no effect and was not actually used inexecute_complete(). -
We are re-assigning the job_id in
execute_complete()as an instance attribute, so that we can later use it within OpenLineage method even when in deferrable mode. We are also re-creating any attributes (hook) that may be missing as a result of the state keeping logic mentioned above.
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.