airflow icon indicating copy to clipboard operation
airflow copied to clipboard

adding callback hook for on/off state of dag

Open Bowrna opened this issue 1 year ago • 5 comments

closes: #36516


^ Add meaningful description above Read the Pull Request Guidelines for more information. In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed. In case of a new dependency, check compliance with the ASF 3rd Party License Policy. In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

Bowrna avatar Apr 02 '24 16:04 Bowrna

I have added the changes and included the toggle_dag_callback inside models. But I am not sure how I can test the changes that adds a new table. Can anyone guide me with any sample PR to have a look? ( I tried figuring out but I couldn't find one that would be useful for me)

Bowrna avatar May 01 '24 17:05 Bowrna

@eladkal I checked the other issue. It is about adding another mode drain to the paused/unpaused state of DAG. While this PR is to send a callback on this state change. in this PR it monitors the difference of state and invokes the callback if it varies from the previous state of the DAG. Do you think this has to be blocked until the other issue work is done?

Bowrna avatar May 02 '24 06:05 Bowrna

@eladkal I checked the other issue. It is about adding another mode drain to the paused/unpaused state of DAG. While this PR is to send a callback on this state change. in this PR it monitors the difference of state and invokes the callback if it varies from the previous state of the DAG. Do you think this has to be blocked until the other issue work is done?

Yes. Removed the temporary blocking. We can continue with reviewing and merge when ready

eladkal avatar May 03 '24 17:05 eladkal

I have added the changes and included the toggle_dag_callback inside models. But I am not sure how I can test the changes that adds a new table. Can anyone guide me with any sample PR to have a look? ( I tried figuring out but I couldn't find one that would be useful for me)

@uranusjr @potiuk Can you give me directions in this part?

Bowrna avatar May 08 '24 15:05 Bowrna

@uranusjr @potiuk Can you give me directions in this part?

Look at history of airflow/migrations -> this is where migration scripts are maintained and we use alembic to generate / maintain those. Simple description of what needs to be done is in https://github.com/apache/airflow/blob/main/contributing-docs/13_metadata_database_updates.rst - but I personally have never done any of those (somewhat surprisingly) from scratch :)

potiuk avatar May 09 '24 12:05 potiuk

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

github-actions[bot] avatar Jul 14 '24 00:07 github-actions[bot]

@potiuk I have added the migration changes. But I am not sure how I can apply those changes and run the code to test the changes. I tried checking around the repo to find help, but I didn't have any clue that I could see. Could you point me to place where I can get help on this part/ tag person who can help? May be I can add those changes and update the documentation part too.

Bowrna avatar Jul 26 '24 04:07 Bowrna

@potiuk I have added the migration changes. But I am not sure how I can apply those changes and run the code to test the changes. I tried checking around the repo to find help, but I didn't have any clue that I could see. Could you point me to place where I can get help on this part/ tag person who can help? May be I can add those changes and update the documentation part too.

@potiuk A gentle reminder here as I am struck here too. Thanks!

Bowrna avatar Aug 05 '24 06:08 Bowrna

Generally speaking - airflow db migrate should migrate the db - also all our tests are running migration up / down -including the latest changed - there is a separate step doing it before tests are run https://github.com/apache/airflow/actions/runs/10106338999/job/27948352857?pr=38683 - and the latest version of it failed with missing airlfow import.

potiuk avatar Aug 05 '24 09:08 potiuk