magento2 Added support for process fork to mview indexation

Description (*)

Added support for process fork to MView indexation

Related Pull Requests

https://github.com/magento/magento2/pull/36155

Fixed Issues (if relevant)

magento/magento2#38246

Manual testing scenarios (*)

Prepare MView action and use Magento\Indexer\Model\ProcessManager inside
Set environment variable MAGE_INDEXER_THREADS_COUNT=2
Run indexer_update_all_views cron job

Contribution checklist (*)

[ ] Pull request has a meaningful description of its purpose
[ ] All commits are accompanied by meaningful commit messages
[ ] All new or changed code is covered with unit/integration tests (if applicable)
[ ] README.md files for modified modules are updated and included in the pull request if any README.md predefined sections require an update
[ ] All automated tests passed successfully (all builds are green)

Dec 06 '23 16:12 MateuszMesek

Hi @MateuszMesek. Thank you for your contribution! Here are some useful tips on how you can test your changes using Magento test environment.

Add the comment under your pull request to deploy test or vanilla Magento instance:

@magento give me test instance - deploy test instance based on PR changes
@magento give me 2.4-develop instance - deploy vanilla Magento instance

:exclamation: Automated tests can be triggered manually with an appropriate comment:

@magento run all tests - run or re-run all required tests against the PR changes
@magento run <test-build(s)> - run or re-run specific test build(s) For example: @magento run Unit Tests

<test-build(s)> is a comma-separated list of build names.

Allowed build names are:

Database Compare
Functional Tests CE
Functional Tests EE
Functional Tests B2B
Integration Tests
Magento Health Index
Sample Data Tests CE
Sample Data Tests EE
Sample Data Tests B2B
Static Tests
Unit Tests
WebAPI Tests
Semantic Version Checker

You can find more information about the builds here :information_source: Run only required test builds during development. Run all test builds before sending your pull request for review.

For more details, review the Code Contributions documentation. Join Magento Community Engineering Slack and ask your questions in #github channel.

Dec 06 '23 16:12 m2-assistant[bot]

@magento run all tests

Dec 06 '23 17:12 MateuszMesek

The requested builds are added to the queue. You should be able to see them here within a few minutes. Please message the #magento-devops slack channel if they don't show in a reasonable amount of time and a representative will look into any issues.

Dec 06 '23 17:12 magento-automated-testing[bot]

@magento run all tests

Dec 06 '23 18:12 MateuszMesek

The requested builds are added to the queue. You should be able to see them here within a few minutes. Please message the #magento-devops slack channel if they don't show in a reasonable amount of time and a representative will look into any issues.

Dec 06 '23 18:12 magento-automated-testing[bot]

@magento run all tests

Jan 31 '24 06:01 engcom-Charlie

The requested builds are added to the queue. You should be able to see them here within a few minutes. Please message the #magento-devops slack channel if they don't show in a reasonable amount of time and a representative will look into any issues.

Jan 31 '24 06:01 magento-automated-testing[bot]

@magento run all tests

Feb 09 '24 07:02 engcom-Hotel

Hello @MateuszMesek,

Thanks for the collaboration!

:x: QA not Passed

We are unable to reproduce the issue in the latest development branch 2.4-develop. Please refer to the below steps we have followed to reproduce the original issue:

Set up a vanilla instance with the latest development branch i.e. 2.4-develop.
Added MAGE_INDEXER_THREADS_COUNT and set its value as 4
Run bin/magento setup:upgrade
Add products in the instance:
Start MySQL show full processlist with the interval of every 1 second with the command below to monitor the queries: mysqladmin --verbose -u root -p -i 1 processlist
Run cron via the below command: bin/magento cron:run --group=index

But the process went well and we do not see any Delete query in the process list.

Please let us know if we missed anything to reproduce the issue.

Thanks

Feb 09 '24 11:02 engcom-Hotel

Hello @MateuszMesek,

Have you had a chance to check this comment, https://github.com/magento/magento2/pull/38248#issuecomment-1935720379?

Thanks

Feb 19 '24 05:02 engcom-Hotel

Hello @engcom-Hotel

This issue occurs only when Mview action use Magento\Indexer\Model\ProcessManager. Please check Manual testing scenarios in my first comment, https://github.com/magento/magento2/pull/38248#issue-2028972194

This issue can be reproduce only by prepare custom code, you can't reproduce in vanilla instance.

Feb 28 '24 23:02 MateuszMesek

Hello @MateuszMesek,

Thanks for the reply!

We have tried with custom mview action using Magento\Indexer\Model\ProcessManager, but the issue is still not reproducible for us.

Please find attached the module for reference:

Magz.zip

Let us know if we have missed anything.

Thanks

Mar 05 '24 15:03 engcom-Hotel

Hello @engcom-Hotel

I updated code of your module to allow test our issue.

Magz.zip

My steps to replicate issue are:

Install module
Enable module ./bin/magento mod:ena Magz_Merchandizing
Enable schedule mode of new indexer ./bin/magento indexer:set-mode schedule merchandizing_popular
Add changes to changelog table INSERT INTO merchandizing_popular_order_cl (entity_id) VALUES (1), (2), (3), (4), (5), (6), (7), (8);
Run indexer in multi threads mode MAGE_INDEXER_THREADS_COUNT=4 ./bin/magento cron:run --group=index --bootstrap=standaloneProcessStarted=1

Mar 05 '24 16:03 MateuszMesek

Hello @MateuszMesek,

Thanks for the updated code, but the issue is still not reproducible for us with the latest development branch i.e. 2.4-develop. I request you to please try to reproduce the issue in the latest development branch and let us know if this is still reproducible for you.

Thanks

Mar 11 '24 09:03 engcom-Hotel

Hello @MateuszMesek,

I trust you've reviewed this comment. We are still unable to reproduce the issue with the latest development branch i.e. 2.4-develop. Kindly reconsider this and inform us if anything has been overlooked.

Thanks

Mar 19 '24 05:03 engcom-Hotel

Hello @engcom-Hotel

Yes, I can still reproduce this issue on latest commit (c971859e7f82412dbc505086bc79469efa0908c2) in development branch 2.4-develop.

We have modified code of Magz_Merchandizing module to display processed ids in output in case when we call cron job.

On my local environment in case when I add list of ids to changelog table (INSERT INTO merchandizing_popular_order_cl (entity_id) VALUES (1), (2), (3), (4), (5), (6), (7), (8);) And next when I call cron job then on output I see:

MAGE_INDEXER_THREADS_COUNT=4 ./bin/magento cron:run --group=index --bootstrap=standaloneProcessStarted=1

1
Ran jobs by schedule.

And inside var/log/cron.log file:

[2024-03-19T08:05:01.374918+00:00] main.CRITICAL: PDOException: SQLSTATE[42S02]: Base table or view not found: 1146 Table 'magento2.merchandizing_popular_order_cl__tmp65f9472d57b710_49733668' doesn't exist in /var/www/html/magento2/lib/internal/Magento/Framework/DB/Statement/Pdo/Mysql.php:90

Please remember about have installed pcntl module in PHP. You can check if correct method was used by Magento\Indexer\Model\ProcessManager by add var_dump(__METHOD__) inside simpleThreadExecute and multiThreadsExecute methods of this class. Output example in my local environment with this var_dump is:

MAGE_INDEXER_THREADS_COUNT=4 ./bin/magento cron:run --group=index --bootstrap=standaloneProcessStarted=1
string(57) "Magento\Indexer\Model\ProcessManager::multiThreadsExecute"

1
Ran jobs by schedule.

Mar 19 '24 08:03 MateuszMesek

Hello @MateuszMesek,

Appreciate your response!

Before I proceed, could you please resolve the conflicts? Once that's done, we can move forward with the next steps.

Thanks

Mar 20 '24 05:03 engcom-Hotel

@magento run all tests

Mar 21 '24 18:03 MateuszMesek

@engcom-Hotel done

Mar 21 '24 19:03 MateuszMesek

Thanks @MateuszMesek for the reply!

We are able to reproduce the issue on the development branch.

:heavy_check_mark: QA Passed

Added support for process fork to MView indexation

Manual testing scenario:

Install module
Enable module ./bin/magento mod:ena Magz_Merchandizing
Enable schedule mode of new indexer ./bin/magento indexer:set-mode schedule merchandizing_popular
Add changes to changelog table INSERT INTO merchandizing_popular_order_cl (entity_id) VALUES (1), (2), (3), (4), (5), (6), (7), (8);
Run indexer in multi threads mode MAGE_INDEXER_THREADS_COUNT=4 ./bin/magento cron:run --group=index --bootstrap=standaloneProcessStarted=1

Actual Result: :heavy_check_mark: Running on several threads, the indexer works. Or to be more precise: Running on several threads, the ChangelogBatchWalker doesn't delete the table used by other threads.

After: :heavy_check_mark: after_pr