Scheduling issue on Production LIVE environment where CMS page update delayed with 1 hour

[MilenV]

Preconditions and environment

• Magento version - Adobe Commerce ver. 2.4.5-p2
• Anything else that would help a developer reproduce the bug - This issue is not reproducible on Staging environment. We have done some more testing around scheduling templates on the staging UK homepage. I have found that the correct template deployed at the correct time. Here is a screenshot of the scheduling time and scheduling name. Here is a screenshot at 15:34 ( as seen in the bottom right corner) the correct template went live at the correct time. and The issue is only experienced on Production LIVE environment.

Friday morning the UK "better than black friday" the Homepage was supposed to go live however it did not go live at the time it was scheduled to go live (10:00am) GMT. This was then manually updated. An hour later the homepage updated to to the template that was originally supposed to go live at 10:00 GMT because of this we tested on the "UK dummy page" for the page to go live at 12:30 GMT and found that the homepage updated an hour later at 1:30 GMT meaning that the scheduling times were out by an hour. This issue is also NOT happening on staging. Other scheduling live banners and promo codes worked correctly. It seams to only be on happening on the Production live pages.

This is how the Homepage scheduling was set up, this is how all homepages have been set up - we think that there could possibly be an issue regarding the timing of Magento in the back end. This is the only thing that i could possibly think could have caused this issue.

Steps to reproduce

Example:

1. Once logged onto backend(admin panel) go to Content -> Pages and select to edit page "UK Dummy Home Page"

2. Add an schedule for updating the CMS page "UK Dummy Home Page" template to start date and time 17th Nov and time to 10 am GMT. or Assign to Existing Update:

3. The template being applied at 10 am GMT but it takes an affect on 11 am GMT with delayed time of one hour.

Expected result

CMS page "UK Dummy Home Page" template should be applied for start time at 10 am GMT.

Actual result

CMS page "UK Dummy Home Page" template is applied for start time at 11 am GMT which one hour delay.

Additional information

If you Go to Stores -> Configuration. Make sure the Store View is set to: Default Config. Go to General -> General. Scroll down to Locale Options you will notice the Timezone is GMT(Greenwich Mean Time Europe/London) which is identical on Staging and Production when compared manually.

Release note

No response

Triage and priority

• [ ] Severity: S0 - Affects critical data or functionality and leaves users without workaround.
• [ ] Severity: S1 - Affects critical data or functionality and forces users to employ a workaround.
• [X] Severity: S2 - Affects non-critical data or functionality and forces users to employ a workaround.
• [ ] Severity: S3 - Affects non-critical data or functionality and does not force users to employ a workaround.
• [ ] Severity: S4 - Affects aesthetics, professional look and feel, “quality” or “usability”.

[m2-assistant[bot]]

Hi @MilenV. Thank you for your report. To speed up processing of this issue, make sure that the issue is reproducible on the vanilla Magento instance following Steps to reproduce. To deploy vanilla Magento instance on our environment, Add a comment to the issue:

• @magento give me 2.4-develop instance - upcoming 2.4.x release
• For more details, review the Magento Contributor Assistant documentation.
• Add a comment to assign the issue: @magento I am working on this
• To learn more about issue processing workflow, refer to the Code Contributions.

---

Join Magento Community Engineering Slack and ask your questions in #github channel. :warning: According to the Magento Contribution requirements, all issues must go through the Community Contributions Triage process. Community Contributions Triage is a public meeting. :clock10: You can find the schedule on the Magento Community Calendar page. :telephone_receiver: The triage of issues happens in the queue order. If you want to speed up the delivery of your contribution, join the Community Contributions Triage session to discuss the appropriate ticket.

[m2-assistant[bot]]

Hi @engcom-Dash. Thank you for working on this issue. In order to make sure that issue has enough information and ready for development, please read and check the following instruction: :point_down:

• 1. Verify that issue has all the required information. (Preconditions, Steps to reproduce, Expected result, Actual result).
• 2. Verify that issue has a meaningful description and provides enough information to reproduce the issue.
• 3. Add Area: XXXXX label to the ticket, indicating the functional areas it may be related to.
• 4. Verify that the issue is reproducible on 2.4-develop branch
     Details

     - Add the comment @magento give me 2.4-develop instance to deploy test instance on Magento infrastructure.
     - If the issue is reproducible on 2.4-develop branch, please, add the label Reproduced on 2.4.x.
     - If the issue is not reproducible, add your comment that issue is not reproducible and close the issue and stop verification process here!

---

• Join Magento Community Engineering Slack and ask your questions in #github channel.

[engcom-Dash]

@magento give me 2.4-develop instance

[magento-deployment-service[bot]]

Hi @engcom-Dash. Thank you for your request. I'm working on Magento instance for you.

[magento-deployment-service[bot]]

Hi @engcom-Dash, here is your Magento Instance: https://c1d3046c7d3af807a032370b3a23014b.instances-prod.magento-community.engineering Admin access: https://c1d3046c7d3af807a032370b3a23014b.instances-prod.magento-community.engineering/admin_e121 Login: 7d210bc1 Password: fd915ac6c394

[engcom-Dash]

@magento give me 2.4-develop instance

[magento-deployment-service[bot]]

Hi @engcom-Dash. Thank you for your request. I'm working on Magento instance for you.

[magento-deployment-service[bot]]

Hi @engcom-Dash, here is your Magento Instance: https://c1d3046c7d3af807a032370b3a23014b.instances-prod.magento-community.engineering Admin access: https://c1d3046c7d3af807a032370b3a23014b.instances-prod.magento-community.engineering/admin_633c Login: b17199f4 Password: ee98fffe2bff

[engcom-Dash]

@magento give me 2.4-develop instance

[engcom-Dash]

@magento give me 2.4-develop instance

[magento-deployment-service[bot]]

Hi @engcom-Dash. Thank you for your request. I'm working on Magento instance for you.

[magento-deployment-service[bot]]

Hi @engcom-Dash, here is your Magento Instance: https://c1d3046c7d3af807a032370b3a23014b.instances-prod.magento-community.engineering Admin access: https://c1d3046c7d3af807a032370b3a23014b.instances-prod.magento-community.engineering/admin_02ad Login: c09dac57 Password: 868d811b2200

[m2-assistant[bot]]

Hi @engcom-November. Thank you for working on this issue. In order to make sure that issue has enough information and ready for development, please read and check the following instruction: :point_down:

• [ ] 1. Verify that issue has all the required information. (Preconditions, Steps to reproduce, Expected result, Actual result).
• [ ] 2. Verify that issue has a meaningful description and provides enough information to reproduce the issue.
• [ ] 3. Add Area: XXXXX label to the ticket, indicating the functional areas it may be related to.
• [ ] 4. Verify that the issue is reproducible on 2.4-develop branch
  Details

  - Add the comment @magento give me 2.4-develop instance to deploy test instance on Magento infrastructure.
  - If the issue is reproducible on 2.4-develop branch, please, add the label Reproduced on 2.4.x.
  - If the issue is not reproducible, add your comment that issue is not reproducible and close the issue and stop verification process here!
• [ ] 5. Add label Issue: Confirmed once verification is complete.
• [ ] 6. Make sure that automatic system confirms that report has been added to the backlog.

[engcom-November]

Hello @MilenV,

Thank you for the report and collaboration!

Verififed this issue on 2.4-develop, but the issue was not reproducible. In our case the scheduled changes are getting reflected at the start time, we did not see any delay, and this was done in production mode. Please verify this issue on latest magento version, and let us know if the issue persist.

Thank you.

[MilenV]

Hello Team,

After discussion with Adobe Dev Team it was encountered that this issue was reproducible on their end for Production environment.

Here is the solution provided by them: `I thank you for your patience while we investigated the issue in question, it is great you're investigating from your side too, and also thanks for the update from your side it is very valuable. At the same time I would like to apologise for the delay, please see my findings. I tried to investigate it deeply and during the investigation, I noticed that your Cron schedule is quite heavily busy which holds queue processing quite busy and is getting stuck sometimes which may cause delays for Cron jobs, namely and for example these jobs below are one of many that takes time to be processed, example of 2 types of Cron jobs you have, there are also API-related (adyen-php-api-library), etc.: xtento_trackingimport_profile_40_cron ---> where number 40 is sequence number from 1 to 40 or more xtento_orderexport_profile_98_cron_1 ---> where number 98 is sequence number from 1 to 98 or more

So, it may be necessary for you to adjust various configurations to find the optimal combination that reduces the queue processing time and provides the best results. Since there is no fixed or recommended approach for configuring these parameters due to variations in environments and usage patterns, experimentation is required. One of the areas of focus is the "max_messages" configuration may need to be increased. Considering these factors, in my investigation, I suggested increasing the "max_messages" value from 100 (you have now) to 250 and setting the "CONSUMERS_WAIT_FOR_MAX_MESSAGES" configuration to false, see below how it is expected to be updated in the .magento.env.yaml file: stage: ... deploy: ... CRON_CONSUMERS_RUNNER: ... max_messages: 250 ... CONSUMERS_WAIT_FOR_MAX_MESSAGES: false ...

During my investigation I was able to reproduce and fix an issue on my local environment that you're experiencing by investigating and setting test configurations from the above to the app/etc/env.php file, in your case you will need to update the .magento.env.yaml file as was mentioned above.

However, CMS Cron jobs are successfully finished after applying the above configuration, that configuration was for test purposes only and should be defined by your developer team for your particular case as well as identify the right value for "max_messages" and similar configuration parameters such as "multiple_processes" configuration, etc., as well as custom cron jobs and groups, please see more details about on the dev docs: https://experienceleague.adobe.com/docs/commerce-operations/configuration-guide/message-queues/manage-message-queues.html?lang=en https://experienceleague.adobe.com/docs/commerce-cloud-service/user-guide/configure/env/stage/variables-deploy.html?lang=en#cron_consumers_runner https://experienceleague.adobe.com/docs/commerce-operations/configuration-guide/crons/custom-cron.html?lang=en

I also hope that the finding you found so far is also helpful in the resolution.

It is strongly recommended to apply and test changes in an integration/staging environment, as well as create dumps before applying to production.

Hope it helps!

Please confirm that the solution provided to you has resolved the reported issue. Support requests will be set to solved if no response is received within 3 days following this notification.`

Further more we have encountered an error in Production cronjob schedule "indexer_reindex_all_invalid" SQLSTATE[23000]: Integrity constraint violation: 1452 Cannot add or update a child row: a foreign key constraint fails (vka64lyl5tm4m.amasty_amrules_purchase_history_index, CONSTRAINT AMASTY_AMRULES_PURCHASE_HISTORY_IDX_CSTR_ID_CSTR_ENTT_ENTT_ID FOREIGN KEY (customer_id) REFERENCES customer_entity (e), query was: INSERT INTO amasty_amrules_purchase_history_index (customer_id,sum_amount,orders_count) VALUES (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?), (?, ?, ?),`

Regards, Milen Velinov

[engcom-November]

Hello @MilenV,

As you mentioned here https://github.com/magento/magento2/issues/38212#issuecomment-1893359368, by changing the configuration of max_messages, were you able to resolve the issue? Please let us know on the same.

Thank you.

[MilenV]

Hello @engcom-November,

We have raised ticket with our client for handling the configuration of max_messages. We awaits Dev Team to confirm it.

Regards, Milen Velinov

[engcom-November]

Hello @MilenV,

Please let us know if this resolved the issue when you hear back from your dev team.

Thank you.

[engcom-November]

Hello @MilenV,

As there is no activity on this issue for a long time, we believe the issue has been resolved, hence closing this issue. Feel free to raise a new issue or reopen this if you need more assistance.

Thank you.