[Issue] Fix concurrency issue in static content deploy

[m2-assistant[bot]]

This issue is automatically created based on existing pull request: magento/magento2#39954: Fix concurrency issue in static content deploy

---

Summary

This fixes a bug in which multiple concurrent processes spin up to handle the same theme package, depending on how the themes are defined with their parents.

We saw a lot of issues like

DEBUG: STDERR:Compilation from source /path/to/some.js failed
Magento\Framework\Exception\FileSystemException: The file or directory "/app/var/view_preprocessed/pub/static/frontend/path/to/some.js" cannot be copied to "/app/pub/static/frontend/path/to/some.js"

We also saw less compilation intermittently failing to generate complete styles-m.css and styles-l.css files, as multiple processes were fighting and doing work on the same file at the same time. This cause the frontend to be deployed in a broken manner which needed a redeploy to solve.

For affected setups this may also improve the performance of this process, by the idea that we have a process no longer doing duplicated work.

To reproduce

Reproduce

Clone https://github.com/convenient/magento-bug-reproduction-static-content-concurrency then run composer install, you can now generate static content.

Without the bugfix we can see multiple executions of the same package, see that frontend/Custom/ztheme/default is executed multiple times

$ rm -rf pub/static var/view_preprocessed/ var/cache ; php bin/magento > /dev/null ;  php ./bin/magento setup:static-content:deploy --no-ansi --no-interaction -f -s compact -vvv --jobs 4 --no-html-minify en_US en_GB en_IE -vvv | grep -E 'Execute: frontend/Custom/ztheme|Prevent'
Execute: frontend/Custom/ztheme/default
Execute: frontend/Custom/ztheme/default
Execute: frontend/Custom/ztheme/en_US
Execute: frontend/Custom/ztheme/en_GB
Execute: frontend/Custom/ztheme/en_IE

With the bugfix we can see each is only executed once

$ rm -rf pub/static var/view_preprocessed/ var/cache ; php bin/magento > /dev/null ;  php ./bin/magento setup:static-content:deploy --no-ansi --no-interaction -f -s compact -vvv --jobs 4 --no-html-minify en_US en_GB en_IE -vvv | grep -E 'Execute: frontend/Custom/ztheme|Prevent'
Execute: frontend/Custom/ztheme/default
Preventing duplicate execution of package as it is in progress: frontend/Custom/ztheme/default (pid: 67830)
Execute: frontend/Custom/ztheme/en_US
Execute: frontend/Custom/ztheme/en_GB
Execute: frontend/Custom/ztheme/en_IE

Verification

I have used this fix on production for over a year without any issues. But we can also double check on the example reproduction repository provided above.

Get a baseline

rm -rf pub/static/
php ./bin/magento setup:static-content:deploy --no-ansi --no-interaction -f -s compact -vvv --jobs 4 --no-html-minify en_US en_GB
find ./pub/static -type f -exec md5sum {} + | awk '{print $1, $2}' > before.txt

Apply the bugfix then do the following

rm -rf pub/static/
php ./bin/magento setup:static-content:deploy --no-ansi --no-interaction -f -s compact -vvv --jobs 4 --no-html-minify en_US en_GB
find ./pub/static -type f -exec md5sum {} + | awk '{print $1, $2}' > after.txt

See that all files (that matter) were generated the same

$ diff  before.txt after.txt
1c1
< 04d5dfe8b06b35834eeb526128ef11ba ./pub/static/deployed_version.txt
---
> b5e5f1122730b93197d32286a8dc65ab ./pub/static/deployed_version.txt

Contribution checklist (*)

• [x] Pull request has a meaningful description of its purpose
• [x] All commits are accompanied by meaningful commit messages
• [ ] All new or changed code is covered with unit/integration tests (if applicable)
• [ ] README.md files for modified modules are updated and included in the pull request if any README.md predefined sections require an update
• [ ] All automated tests passed successfully (all builds are green)

[engcom-Dash]

Hello @convenient Thank you for report & collaboration!

We have tried to reproduce the issue in the latest development branch i.e. 2.4-develop and the issue has been reproducible for us.

Several concurrent processes get initiated to manage the same theme package, depending on how the themes are structured and their relationships with parent theme is defined.

Please refer to the below screenshots for reference:

rm -rf pub/static var/view_preprocessed/ var/cache ; php bin/magento > /dev/null ; php ./bin/magento setup:static-content:deploy --no-ansi --no-interaction -f -s compact -vvv --jobs 4 --no-html-minify en_US en_GB en_IE -vvv | grep -E 'Execute: frontend/Magento/blank|Prevent'

Hence, confirming the issue.

[m2-assistant[bot]]

Hi @engcom-November. Thank you for working on this issue. In order to make sure that issue has enough information and ready for development, please read and check the following instruction: :point_down:

• [ ] 1. Verify that issue has all the required information. (Preconditions, Steps to reproduce, Expected result, Actual result).
• [ ] 2. Verify that issue has a meaningful description and provides enough information to reproduce the issue.
• [ ] 3. Add Area: XXXXX label to the ticket, indicating the functional areas it may be related to.
• [ ] 4. Verify that the issue is reproducible on 2.4-develop branch
  Details

  - If the issue is reproducible on 2.4-develop branch, please, add the label Reproduced on 2.4.x.
  - If the issue is not reproducible, add your comment that issue is not reproducible and close the issue and stop verification process here!
• [ ] 5. Add label Issue: Confirmed once verification is complete.
• [ ] 6. Make sure that automatic system confirms that report has been added to the backlog.

[github-jira-sync-bot]

:white_check_mark: Jira issue https://jira.corp.adobe.com/browse/AC-14944 is successfully created for this GitHub issue.

[m2-assistant[bot]]

:white_check_mark: Confirmed by @engcom-Dash. Thank you for verifying the issue.
Issue Available: @engcom-Dash, You will be automatically unassigned. Contributors/Maintainers can claim this issue to continue. To reclaim and continue work, reassign the ticket to yourself.