airflow icon indicating copy to clipboard operation
airflow copied to clipboard

fix(asset-parition): add a mutex lock to AssetModel when updating APDR

Open Lee-W opened this issue 3 weeks ago • 3 comments

Why

closes: https://github.com/apache/airflow/issues/58919

What

Add a row lock to AssetModel when updating APDR for MySQL and PostgreSQL. Since SQLIte does not support row lock, add retry mechanism if OperationError with Database is locked is encountered.


^ Add meaningful description above Read the Pull Request Guidelines for more information. In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed. In case of a new dependency, check compliance with the ASF 3rd Party License Policy. In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

Lee-W avatar Dec 08 '25 03:12 Lee-W

I don't think it's a good solution

SELECT FOR UPDATE will not lock anything when ADPR lock is missing.

If the task switch happens after ROW lock in first event handling, both select for updates will run and return no row, and this will create both ADPRs:

Time Event 1 Event 2
0 SELECT FOR UPDATE ADR (NO ROW FOUND)
1 SELECT FOR UPDATE ADR (NO ROW FOUND)
2 CREATE ADPR
3 CREATE ADPR

potiuk avatar Dec 08 '25 23:12 potiuk

Just updated it to lock AssetModel inatead of APDR

Lee-W avatar Dec 09 '25 10:12 Lee-W

I think you need to rebase to get rid of the issues that were fixed in main already

potiuk avatar Dec 10 '25 19:12 potiuk