Remove jobs no longer referenced in the codebase

[fredden]

Description

When upgrading a Magento website over time, some cron jobs are left in the cron_schedule table for jobs which no longer exist. This can happen when a module is removed, or when a module removes a cronjob, or when a module changes the name of one of its cronjobs.

This pull request adds a daily clean-up task to remove such jobs from the schedule. This means that left-over jobs are pruned safely and the database table does have useless data.

Manual testing scenarios

1. Run the Magento cron so that sitemap_generate gets scheduled. This belongs to the Magento_Sitemap module.
2. Disable the Magento_Sitemap module.
3. Run the Magento cron.
4. Observe that there is (or not) a left-over entry for sitemap_generate in the cron_schedule database table.

Related issues

Fixes https://github.com/magento/magento2/issues/38217

Contribution checklist

• [x] Pull request has a meaningful description of its purpose
• [x] All commits are accompanied by meaningful commit messages
• [x] All new or changed code is covered with unit/integration tests (if applicable)
• [x] README.md files for modified modules are updated and included in the pull request if any README.md predefined sections require an update
• [x] All automated tests passed successfully (all builds are green)

[m2-assistant[bot]]

Hi @fredden. Thank you for your contribution! Here are some useful tips on how you can test your changes using Magento test environment.

Add the comment under your pull request to deploy test or vanilla Magento instance:

• @magento give me test instance - deploy test instance based on PR changes
• @magento give me 2.4-develop instance - deploy vanilla Magento instance

:exclamation: Automated tests can be triggered manually with an appropriate comment:

• @magento run all tests - run or re-run all required tests against the PR changes
• @magento run <test-build(s)> - run or re-run specific test build(s) For example: @magento run Unit Tests

<test-build(s)> is a comma-separated list of build names.

Allowed build names are:

1. Database Compare
2. Functional Tests CE
3. Functional Tests EE
4. Functional Tests B2B
5. Integration Tests
6. Magento Health Index
7. Sample Data Tests CE
8. Sample Data Tests EE
9. Sample Data Tests B2B
10. Static Tests
11. Unit Tests
12. WebAPI Tests
13. Semantic Version Checker

You can find more information about the builds here :information_source: Run only required test builds during development. Run all test builds before sending your pull request for review.

---

For more details, review the Code Contributions documentation. Join Magento Community Engineering Slack and ask your questions in #github channel.

[fredden]

@magento run all tests

[hostep]

Nice! Good job!

@fredden: can you add https://github.com/magento/magento2/issues/38217 to the description of this PR? As this fixes that issue, so it automatically gets closed when this PR is merged.

I made the argument in there that maybe we should add the cleanup of unknown jobs to the Recurring setup script that exists in the Cron module, so it only does the cleanup when bin/magento setup:upgrade is being executed. Because disabling modules, or removing cronjobs, also requires bin/magento setup:upgrade to be ran. That way, we don't need to check this daily. But I'm also fine with the daily cronjob like you implemented it now. It was just a thought.

[fredden]

add https://github.com/magento/magento2/issues/38217 to the description of this PR? As this fixes that issue

Yes, I'll update this later today. Thanks for highlighting; I hadn't seen that report before now.

maybe we should add the cleanup of unknown jobs to the Recurring setup script that exists in the Cron module

I considered this approach too. This would catch most of the scenarios where groups or jobs are removed. There are two (main) reasons why I went with a periodic clean-up instead.

If the clean-up is in the setup:... process, then it should clean up all orphaned entries, not just "old" orphaned entries. This means that we immediately lose history for these jobs. When removing a module forever, this is typically fine; sufficient backing up of data is part of good maintenance. When a module simply renames a cronjob however (eg, to fix a typo), then losing history prematurely isn't ideal. (I'm thinking about the "timeline" view of a certain cronjobmanager module.)

The other reason for the periodic clean-up, is that I have seen modules dynamically create jobs, which then never get cleaned up by Magento when that module's configuration changes.

[fredden]

@magento run all tests

[engcom-Hotel]

@magento run all tests

[engcom-Hotel]

@magento run Database Compare, Functional Tests B2B, Functional Tests CE, Functional Tests EE, Integration Tests, Unit Tests, WebAPI Tests

[fredden]

@engcom-Hotel thanks for your note. I don't know why I added a circular reference here. I've removed it now.

[fredden]

@magento run all tests

[engcom-Hotel]

@magento run Functional Tests B2B, Functional Tests EE, Functional Tests CE, WebAPI Tests

[fredden]

I don't know why I added a circular reference here. I've removed it now.

I've worked out that this was a typing error. I had intended to add a dependency on Magento_Config. I've adjusted the commits here to reflect this.

[fredden]

@magento run all tests

[engcom-Bravo]

Hi @fredden,

Thanks for the collaboration & contribution!

:heavy_check_mark: QA Passed

Preconditions:

• Install fresh Magento 2.4-develop

Steps to reproduce

As per the Description.

Before: :heavy_multiplication_x: 

After: :heavy_check_mark:  

The Jobs for Disabled Modules are removed from the cron_schedule table from the database.

Builds are failed. Hence, moving this PR to Extended Testing.

Thanks.

[engcom-Echo]

@magento run Functional Tests EE,Functional Tests CE,Functional Tests B2B

[engcom-Echo]

Functional Tests B2B,Functional Tests CE,Functional Tests EE are different on last two run on same commit. Also it contains some known failure.

ACQE-6523 - AdminReorderWithCatalogPriceRuleDiscountTest ACQE-6331 - StorefrontCreateOrderAllQuantityGroupedProductOptionDefaultStockTest

Functional Tests B2B Run1

Run2

Functional Tests CE Run1

Run2

Functional Tests EE Run1

Run2

Hence moving it to merge in progress

[MaximGns]

Questioning why you should execute this by using another cron job? I guess cronjobs that are removed/changed name is only discoverd after a setup upgrade, and would be better do run this job after a setup upgrade (during deployment to production for example). Some project will not build anything to production for months, therefore no cronjobs will be removed/changed by name in the meantime...

[fredden]

@MaximGns see https://github.com/magento/magento2/pull/38693#issuecomment-2092442675