airflow icon indicating copy to clipboard operation
airflow copied to clipboard

Fix TIPydantic serialization of MappedOperator

Open dstandish opened this issue 1 year ago • 1 comments

Previously we relied on SerializedBaseOperator.serialize_operator for serialization of all Operator objects but this is no bueno because it does not work for mapped operator. Instead we use BaseSerialization.serialize, which calls the right method for each obj type. To make this actually work for db isolation though, we had to do a few more things. 1. operator_class obj was not deserialized properly so fixed that. It's a weird case cus it roundtrips to a dict -- not a class. Also expand_input did not work quite right because it roundtrips to _ExpandInputRef. Then we had to make a small change to get_map_type_key since we might have _ExpandInputRef here.

dstandish avatar Apr 27 '24 06:04 dstandish

@uranusjr do you have any concerns with this PR?

dstandish avatar May 16 '24 19:05 dstandish

@uranusjr just going to merge this for now; if you look and find issues, we can make changes.

dstandish avatar May 24 '24 21:05 dstandish

Hey @dstandish -> this one has been merged without rebasing after it has not been run for a month or so and it broke main quite heavily. I will revert it now and it should be re-created and fixed

potiuk avatar May 26 '24 10:05 potiuk

Update - it was not a month, but I think it has clashed with other changes in the meantime.

potiuk avatar May 26 '24 10:05 potiuk

I will also see if I can fix those issues in paralell, while checking if this is indeed the problem - so I might be able to fix it.

potiuk avatar May 26 '24 10:05 potiuk