apache
Determine needs_expansion on client side when DB isolated
This is necessary because Operator does not serialize quite right. It doesn't (either completely or at all) include the dag object or its groups. So we can't properly determine 'needs expansion' on server side when it's in a mapped task group.
It doesn't (either completely or at all) include the dag object or its groups.
And… we somehow don’t want it to do that? I feel we need to do a better job recreating the heirarchy. The code base is getting more and more difficult to maintain with all the special cases.
It doesn't (either completely or at all) include the dag object or its groups.
And… we somehow don’t want it to do that? I feel we need to do a better job recreating the heirarchy. The code base is getting more and more difficult to maintain with all the special cases.
Yeah I agree. Yeah I mean these PRs are in part like .... illustrating problems by showing what we have to do to make things work. i'm not sure what we want to / need to / ought to do with regard to serialization of task objects.
but like ... also... @uranusjr, using this example... there actually are some reasonable arguments for not serializing the complete task object. for one it's more efficient. why serialize the full task (taken to include the full dag and all other tasks) and send over the network just so that "needs expansion" can be evaluated on the server, when you can just determine locally whether it needs expansion and tell the server what to do and save the net traffic and processing time? further do we not run into recursion problem serializing at task that has the full dag when the dag also has all tasks each of which has the full dag and on and on.....