packit-service Cancel running jobs when new ones are started for a PR

It can easily happen that a PR can receive a push or a /packit copr-build command and then another one in a short time-span.

In order to not to waste resources and avoid publishing results from a parallel set of jobs into the same PR, we should cancel old jobs (builds, tests) and submit new ones.

Breakdown

[x] https://github.com/packit/packit/issues/2534
[x] https://github.com/packit/packit-service/issues/2725
[x] https://github.com/packit/packit-service/issues/2727
[ ] https://github.com/packit/packit-service/issues/2800
[ ] Improvements for Fedora CI
- [ ] https://github.com/packit/packit/issues/2535
[ ] Improvements in general
- [ ] https://github.com/packit/packit-service/issues/2728
- [ ] https://github.com/packit/packit-service/issues/2729
- [ ] cancelling via GitHub Check Run action
- [ ] allow configurable behavior
- [ ] allow cancelling jobs for commit trigger

Jan 09 '19 12:01 TomasTomecek

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

Nov 05 '19 00:11 stale[bot]

I wonder if copr optimizes this on its own - cancels old builds if new are requested in a project.

Nov 05 '19 10:11 TomasTomecek

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. We are doing this to be sure that the issue is still relevant. Anyone can comment to remove the stale state. (The issues marked with pinned, security, bug or EPIC label are not considered stale.)

Jan 04 '20 10:01 stale[bot]

with postgres this can be now done so easily honestly

Mar 02 '20 08:03 TomasTomecek

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. We are doing this to be sure that the issue is still relevant. Anyone can comment to remove the stale state. (The issues marked with pinned, security, bug or EPIC label are not considered stale.)

May 22 '20 14:05 stale[bot]

we should still investigate if copr can do this on it own

Jun 05 '20 15:06 TomasTomecek

This issue has been marked as stale because it hasn't seen any activity for the last 60 days.

Stale issues are closed after 14 days, unless the label is removed by a maintainer or someone comments on it.

This is done in order to ensure that open issues are still relevant.

Thank you for your contribution! :unicorn: :rocket: :robot:

(Note: issues labeled with pinned, security, bug or EPIC are never marked as stale.)

Aug 04 '20 16:08 stale[bot]

This issue has been marked as stale because it hasn't seen any activity for the last 60 days.

Stale issues are closed after 14 days, unless the label is removed by a maintainer or someone comments on it.

This is done in order to ensure that open issues are still relevant.

Thank you for your contribution! :unicorn: :rocket: :robot:

(Note: issues labeled with pinned, security, bug or EPIC are never marked as stale.)

Oct 04 '20 10:10 stale[bot]

Related to #785

Oct 05 '20 07:10 lachmanfrantisek

This issue has been marked as stale because it hasn't seen any activity for the last 60 days.

Stale issues are closed after 14 days, unless the label is removed by a maintainer or someone comments on it.

This is done in order to ensure that open issues are still relevant.

Thank you for your contribution! :unicorn: :rocket: :robot:

(Note: issues labeled with pinned or EPIC are never marked as stale.)

Dec 04 '20 20:12 stale[bot]

Still nice to have.

Dec 07 '20 08:12 lachmanfrantisek

This issue has been marked as stale because it hasn't seen any activity for the last 60 days.

Stale issues are closed after 14 days, unless the label is removed by a maintainer or someone comments on it.

This is done in order to ensure that open issues are still relevant.

Thank you for your contribution! :unicorn: :rocket: :robot:

(Note: issues labeled with pinned or EPIC are never marked as stale.)

Feb 06 '21 15:02 stale[bot]

Can we bump priority here? @mrc0mmand mentioned this again when talking about expectations ("Error Budget").

Apr 06 '21 16:04 jpopelka

Yes, sure. I am adding the issue to the project board so we can groom and plan for the next sprint(s).

Apr 07 '21 08:04 lachmanfrantisek

We discussed the following approach today:

Upgrade the pipeline-abstraction so that it's possible to tell if a pipeline is for a PR or branch is still in progress.
Add a new celery task at the beginning of the pipeline which would check if there is an another pipeline already running for the PR or branch, and cancel the that pipeline.
Cancelling the pipeline means:
- terminating SRPM builds if they are running
- terminating Copr builds if they are running
- terminating TF tests if they are running
- internally marking the pipeline as cancelled
- (GitHub flags should be already updated by this point, so we should just make sure they are not overwritten by the termination process).

May 04 '21 12:05 csomh

This issue has been marked as stale because it hasn't seen any activity for the last 60 days.

Stale issues are closed after 14 days, unless the label is removed by a maintainer or someone comments on it.

This is done in order to ensure that open issues are still relevant.

Thank you for your contribution! :unicorn: :rocket: :robot:

(Note: issues labeled with pinned or EPIC are never marked as stale.)

Jul 04 '21 04:07 stale[bot]

Let's start with something more easier and doable in short-term:

[ ] Terminate an old Copr build when submitting a new one.

Jul 08 '21 07:07 lachmanfrantisek

This issue has been marked as stale because it hasn't seen any activity for the last 60 days.

Stale issues are closed after 14 days, unless the label is removed by a maintainer or someone comments on it.

This is done in order to ensure that open issues are still relevant.

Thank you for your contribution! :unicorn: :rocket: :robot:

(Note: issues labeled with pinned or EPIC are never marked as stale.)

Sep 14 '21 07:09 stale[bot]

This issue has been marked as stale because it hasn't seen any activity for the last 60 days.

Stale issues are closed after 14 days, unless the label is removed by a maintainer or someone comments on it.

This is done in order to ensure that open issues are still relevant.

Thank you for your contribution! :unicorn: :rocket: :robot:

(Note: issues labeled with pinned or EPIC are never marked as stale.)

Nov 25 '21 20:11 stale[bot]

This issue has been marked as stale because it hasn't seen any activity for the last 60 days.

Stale issues are closed after 14 days, unless the label is removed by a maintainer or someone comments on it.

This is done in order to ensure that open issues are still relevant.

Thank you for your contribution! :unicorn: :rocket: :robot:

(Note: issues labeled with pinned or EPIC are never marked as stale.)

Mar 02 '22 09:03 stale[bot]

This issue has been marked as stale because it hasn't seen any activity for the last 60 days.

Stale issues are closed after 14 days, unless the label is removed by a maintainer or someone comments on it.

This is done in order to ensure that open issues are still relevant.

Thank you for your contribution! :unicorn: :rocket: :robot:

(Note: issues labeled with pinned or EPIC are never marked as stale.)

May 01 '22 21:05 stale[bot]

This issue has been marked as stale because it hasn't seen any activity for the last 60 days.

Stale issues are closed after 14 days, unless the label is removed by a maintainer or someone comments on it.

This is done in order to ensure that open issues are still relevant.

Thank you for your contribution! :unicorn: :rocket: :robot:

(Note: issues labeled with pinned or EPIC are never marked as stale.)

Jul 10 '22 13:07 stale[bot]

This issue has been marked as stale because it hasn't seen any activity for the last 60 days.

Stale issues are closed after 14 days, unless the label is removed by a maintainer or someone comments on it.

This is done in order to ensure that open issues are still relevant.

Thank you for your contribution! :unicorn: :rocket: :robot:

(Note: issues labeled with pinned or EPIC are never marked as stale.)

Sep 21 '22 02:09 stale[bot]

We discussed this and decided we will drop this issue since we are not able to plan this in the near future and the implementation could be more complex (since the codebase changed a lot since opening this issue) and lead to race conditions.

Dec 06 '22 12:12 lbarcziova

Reopening given the increased user base and since it would help with resource management and address challenges such as handling outages where queues become overloaded. This would prevent unnecessary triggers of builds/tests when a pull request is updated during such outages.

This was also asked by @evgeni in the chat recently.

May 15 '24 06:05 lbarcziova

+1 to this RFE!

When I trigger copr building and integration test execution on https://github.com/oamg/convert2rhel PRs, that's a lot of jobs. When I trigger that all again, for example by pushing to the PR branch, before even copr builds are created then the Testing Farm jobs that follow up after copr builds don't need to be even run. That might even save some GitHub App requests which are limited and our project has been hiting the limit https://docs.github.com/en/rest/using-the-rest-api/rate-limits-for-the-rest-api?primary-rate-limit-for-github-app-installations=&apiVersion=2022-11-28#primary-rate-limit-for-github-app-installations.

May 23 '24 14:05 bocekm