Support manual task annotation.

[sharingan-no-kakashi]

trafficstars

Description

I have been an airflow user for since a long time and I have often asked myself "what did happen to this task 4 months ago?". Usually, this question starts a long and tedious process of going through slack/github/jira/emails to understand what was the problem.

Wouldn't be nice having a small airflow component where users can insert notes about tasks?

What do you want to happen? The idea is to have a "task note" which is related to a task instance. We would have a new table: task_notes which has 5 columns dag_id, task_id, execution_date, task_notes, timestamp, (potentially others such a user etc) We would have a new view similar to the xcom (second screenshot) and we modify the task instance view and model to refer to this task note view (first screenshot). Users would be able to insert notes from this view.

In my current implementation, it looks something similar to this.

-->

Are you willing to submit a PR?

Yes. I have been working on this for fun as a POC.

Related Issues

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template!

[potiuk]

Nice. I think It's a nice addition. Go ahead with the PR!

[Jorricks]

+1 seems like a valuable contribution.

[sharingan-no-kakashi]

Ok! I am going to work on this! Thank you

[ashb]

See also https://github.com/apache/airflow/issues/12199 which asks for a new state -- So that you can "acknowledge" the failure without marking it as success.

[sharingan-no-kakashi]

Hi, @potiuk I think I am almost done with my base implementation, but before opening a PR (I have read the CONTRIBUTION and PULL_REQUEST instructions) I would like to ask a few questions 😄.

1. I cannot find in the codebase similar unit tests to be added when a view (and a new template) is added here airflow/www/views.py. (In my case the new view would be Task Note with a get and a post operation) Would you be able to point me to something similar in the codebase/github?

2. In case I would like to collect feedback for my work that is not 100% completed (missing tests), can I open a PR nevertheless, maybe in DRAFT or is there any specific guideline for this?

Thank you.

[sharingan-no-kakashi]

See also #12199 which asks for a new state -- So that you can "acknowledge" the failure without marking it as success.

They are indeed very similar. Has it already been brainstormed/implemented?

[uranusjr]

1. Not sure if I understand the question correctly. For a new (web) view, you should add a file in tests/www/views to test it. Other files in the same directory should provide some hints how the tests can be configured/implemented.
2. Yes, a draft PR would be best.

[Acehaidrey]

Hi all can you provide a status for this PR. What do we think, can we work to allow this to record notes during actions? Is that still the goal here

[Acehaidrey]

@sharingan-no-kakashi can you let me know are you planning to get this PR in? that would be great for us.

Sorry I see it was not implemented in. It was closed. Any way to get you to consider merging it again?

[sharingan-no-kakashi]

Hi @Acehaidrey I have a relatively free weekend! I can rebase the work of this PR and the other one into 1 with the minimal POC and I or someone else(and feel free to continue this work if you fancy) can improve it/finish it.

[hkc-8010]

Hello Committers,

Are we planning to implement this feature in Airflow? This would really help to identify which tasks were manually marked as success or failure and would add a great feature in terms of platform observability.

[potiuk]

The features are implemented when someone implements it.. There has to be someone who would like to take on the feature and work on it. It does not depend on 'committers' as a whole but on the willingness of someone to do it. And it absolutely does not have to be commiter - if you for example want this feature badly you could take on the task and implement it. If your company has a need they could let it be done by someone who they employ or who they pay for.

That's how it works for open source project. If you really need it and have no skills to implement it, convince someone to do it :).

[Jorricks]

The features are implemented when someone implements it.. There has to be someone who would like to take on the feature and work on it. It does not depend on 'committers' as a whole but on the willingness of someone to do it. And it absolutely does not have to be commiter - if you for example want this feature badly you could take on the task and implement it. If your company has a need they could let it be done by someone who they employ or who they pay for.

That's how it works for open source project. If you really need it and have no skills to implement it, convince someone to do it :).

I was wondering, what would be the best DB schema change? Should it be a separate table or would it make more sense to add an extra field to the DagRun and TaskInstance tables?

[uranusjr]

I'd say it's better to add a column.

[potiuk]

I'd say it's better to add a column.

Agree.