apache
Send important executor logs to task logs
If the executor fails to start a task, the user will not see any logs in the UI because the task has not started. This PR leverages TaskContextLogger implemented in #32646. It forwards the important error messages when an executor fail to execute a task to the task logs.
cc @o-nikolas
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.
Left a few comments. Also what about the Batch executor? Do you plan on fast following with that after this PR?
Yep, my plan is to first do it with one executor and receive feedbacks, address feedbacks etc ... Then when the direction is set, I'll create another PR for BatchExecutor
Had a look and I don't have anything extra to add; I like the direction this is going. I'd need to see some examples to have any real opinion on the question of one or one-per-executor but otherwise, I think it looks good after that one change to the session creation that Niko mentioned.
@RNHTTR you might be good person to review from the perspective of "are these circumstances where user would otherwise not have insight into what happened to the task" slash "are these types errors too frequent" re impact to scheduler.
From the different conversations in this PR, are there still some concerns or feedbacks?
From the different conversations in this PR, are there still some concerns or feedbacks?
All good from my end :+1:
