[Issue] Fix concurrency issue in static content deploy

[m2-assistant[bot]]

This issue is automatically created based on existing pull request: magento/magento2#39954: Fix concurrency issue in static content deploy

---

Summary

This fixes a bug in which multiple concurrent processes spin up to handle the same theme package, depending on how the themes are defined with their parents.

We saw a lot of issues like

DEBUG: STDERR:Compilation from source /path/to/some.js failed
Magento\Framework\Exception\FileSystemException: The file or directory "/app/var/view_preprocessed/pub/static/frontend/path/to/some.js" cannot be copied to "/app/pub/static/frontend/path/to/some.js"

We also saw less compilation intermittently failing to generate complete styles-m.css and styles-l.css files, as multiple processes were fighting and doing work on the same file at the same time. This cause the frontend to be deployed in a broken manner which needed a redeploy to solve.

For affected setups this may also improve the performance of this process, by the idea that we have a process no longer doing duplicated work.

To reproduce

Reproduce

Clone https://github.com/convenient/magento-bug-reproduction-static-content-concurrency then run composer install, you can now generate static content.

Without the bugfix we can see multiple executions of the same package, see that frontend/Custom/ztheme/default is executed multiple times

$ rm -rf pub/static var/view_preprocessed/ var/cache ; php bin/magento > /dev/null ;  php ./bin/magento setup:static-content:deploy --no-ansi --no-interaction -f -s compact -vvv --jobs 4 --no-html-minify en_US en_GB en_IE -vvv | grep -E 'Execute: frontend/Custom/ztheme|Prevent'
Execute: frontend/Custom/ztheme/default
Execute: frontend/Custom/ztheme/default
Execute: frontend/Custom/ztheme/en_US
Execute: frontend/Custom/ztheme/en_GB
Execute: frontend/Custom/ztheme/en_IE

With the bugfix we can see each is only executed once

$ rm -rf pub/static var/view_preprocessed/ var/cache ; php bin/magento > /dev/null ;  php ./bin/magento setup:static-content:deploy --no-ansi --no-interaction -f -s compact -vvv --jobs 4 --no-html-minify en_US en_GB en_IE -vvv | grep -E 'Execute: frontend/Custom/ztheme|Prevent'
Execute: frontend/Custom/ztheme/default
Preventing duplicate execution of package as it is in progress: frontend/Custom/ztheme/default (pid: 67830)
Execute: frontend/Custom/ztheme/en_US
Execute: frontend/Custom/ztheme/en_GB
Execute: frontend/Custom/ztheme/en_IE

Verification

I have used this fix on production for over a year without any issues. But we can also double check on the example reproduction repository provided above.

Get a baseline

rm -rf pub/static/
php ./bin/magento setup:static-content:deploy --no-ansi --no-interaction -f -s compact -vvv --jobs 4 --no-html-minify en_US en_GB
find ./pub/static -type f -exec md5sum {} + | awk '{print $1, $2}' > before.txt

Apply the bugfix then do the following

rm -rf pub/static/
php ./bin/magento setup:static-content:deploy --no-ansi --no-interaction -f -s compact -vvv --jobs 4 --no-html-minify en_US en_GB
find ./pub/static -type f -exec md5sum {} + | awk '{print $1, $2}' > after.txt

See that all files (that matter) were generated the same

$ diff  before.txt after.txt
1c1
< 04d5dfe8b06b35834eeb526128ef11ba ./pub/static/deployed_version.txt
---
> b5e5f1122730b93197d32286a8dc65ab ./pub/static/deployed_version.txt

Contribution checklist (*)

• [x] Pull request has a meaningful description of its purpose
• [x] All commits are accompanied by meaningful commit messages
• [ ] All new or changed code is covered with unit/integration tests (if applicable)
• [ ] README.md files for modified modules are updated and included in the pull request if any README.md predefined sections require an update
• [ ] All automated tests passed successfully (all builds are green)