Magento 2.4.1 Media Gallery....not ready for prime time. Fails when you have large amounts of data (images)

[perfpcs]

Preconditions (*)

The new media gallery introduced in Magento 2.4.1 certainly was not vetted using a large robust shop. Maybe it was designed with 100-200 products like the using Magento sample data but I can tell you from my experience trying to use this, it does NOT work when you have a large amount images. Since we use compression software to have images ready for desktop, table, phone and WEBP use. We have generated over a half a million images for our shop. What happens is that the images will load but the directory tree fails to generate due to sessions being destroyed and memory exhaustion. Even setting php.ini to basically unlimited use of ram and setting our instance to 192 GB physical ram, the media gallery still fails to load the directory tree. This is very disappointing! What I don't understand either is why would the media galley be set with /media as the storage root? This is the main problem as 95% of the images in the media gallery are product images. Most shops don't use product images to create banners or make CMS blocks or pages. The correct root for media gallery in my opinion should have been the /wysiwyg directory. That is what is used when dealing with media and the CMS block and pages creation. If this was set as root then these issues with session destruction and memory failures would never happen. Or, the media gallery should exempt the catalog/product images like it is in the old media gallery. If you go to those catalog/product folders you will see there are no images. That is another correct way this gallery should have been setup, in my humble opinion.

How to produce error (*)

1. Install M 2.4.1
2. Move your media content from your current or old website that has 500,000 or more images
3. Go to content/media galley
4. Media gallery will open with images but directory tree is missing. You can't find folders or see any folders or even see new ones if you make any. Note the folders are created but because the tree is not there, you cannot see the folders. Also it is random whether the search, sort or other fields show up. Sometimes they do and sometimes they dont.

Steps to reproduce (*)

1. Install M 2.4.1
2. Move your media content from your current or old website that has 500,000 or more images
3. Go to content/media galley
4. Open Media gallery

Expected result (*)

1. I expect that the directory tree, search, sort and full media gallery opens, not a crippled version of the media gallery that does not even work.

see various images. It should look like this:

Actual result (*)

1. Many errors in php log:

[19-Oct-2020 04:26:51 UTC] PHP Fatal error: Allowed memory size of 792723456 bytes exhausted (tried to allocate 8192 bytes) in /home/cloudpanel/htdocs/www.performance-pcs.com/vendor/magento/framework/Filesystem/Driver/File.php on line 962 [19-Oct-2020 04:26:51 UTC] PHP Fatal error: Allowed memory size of 792723456 bytes exhausted (tried to allocate 135168 bytes) in /home/cloudpanel/htdocs/www.performance-pcs.com/vendor/magento/framework/Session/SessionManager.php on line 151 [19-Oct-2020 04:26:51 UTC] PHP Fatal error: Allowed memory size of 792723456 bytes exhausted (tried to allocate 65536 bytes) in /home/cloudpanel/htdocs/www.performance-pcs.com/vendor/colinmollenhour/php-redis-session-abstract/src/Cm/RedisSession/Handler.php on line 793 [19-Oct-2020 04:26:51 UTC] PHP Fatal error: Uncaught RuntimeException: Unknown: Cannot call session save handler in a recursive manner in [no active file]:0[19-Oct-2020 08:29:18 UTC] PHP Warning: DOMXPath::query(): Invalid expression in /home/cloudpanel/htdocs/www.performance-pcs.com/vendor/pelago/emogrifier/src/Emogrifier.php on line 494 [19-Oct-2020 08:29:18 UTC] PHP Warning: DOMXPath::query(): Invalid expression in /home/cloudpanel/htdocs/www.performance-pcs.com/vendor/pelago/emogrifier/src/Emogrifier.php on line 494 [19-Oct-2020 08:29:18 UTC] PHP Warning: Invalid argument supplied for foreach() in /home/cloudpanel/htdocs/www.performance-pcs.com/vendor/pelago/emogrifier/src/Emogrifier.php on line 503 [19-Oct-2020 09:06:41 UTC] PHP Fatal error: Allowed memory size of 792723456 bytes exhausted (tried to allocate 8192 bytes) in /home/cloudpanel/htdocs/www.performance-pcs.com/vendor/magento/framework/Filesystem/Driver/File.php on line 962 [19-Oct-2020 09:06:41 UTC] PHP Fatal error: Allowed memory size of 792723456 bytes exhausted (tried to allocate 40960 bytes) in /home/cloudpanel/htdocs/www.performance-pcs.com/vendor/magento/framework/Session/SessionManager.php on line 151 [19-Oct-2020 09:06:41 UTC] PHP Fatal error: Allowed memory size of 792723456 bytes exhausted (tried to allocate 65536 bytes) in /home/cloudpanel/htdocs/www.performance-pcs.com/vendor/colinmollenhour/php-redis-session-abstract/src/Cm/RedisSession/Handler.php on line 793 [19-Oct-2020 09:06:41 UTC] PHP Fatal error: Uncaught RuntimeException: Unknown: Cannot call session save handler in a recursive manner in [no active file]:0[19-Oct-2020 09:54:47 UTC] PHP Fatal error: Uncaught RuntimeException: ObjectManager isn't initialized in /home/cloudpanel/htdocs/www.performance-pcs.com/vendor/magento/framework/App/ObjectManager.php:36 Stack trace: #0 /home/cloudpanel/htdocs/www.performance-pcs.com/vendor/magento/framework/Filesystem/Directory/Read.php(78): Magento\Framework\App\ObjectManager::getInstance() #1 /home/cloudpanel/htdocs/www.performance-pcs.com/vendor/magento/framework/Filesystem/Directory/Read.php(212): Magento\Framework\Filesystem\Directory\Read->validatePath() #2 /home/cloudpanel/htdocs/www.performance-pcs.com/vendor/magento/framework/Code/GeneratedFiles.php(123): Magento\Framework\Filesystem\Directory\Read->isExist() #3 /home/cloudpanel/htdocs/www.performance-pcs.com/vendor/magento/framework/Code/GeneratedFiles.php(98): Magento\Framework\Code\GeneratedFiles->isCleanGeneratedFilesAllowed() #4 /home/cloudpanel/htdocs/www.performance-pcs.com/vendor/magento/framework/App/ObjectManagerFactory.php(114): Magento\Framework\Code\GeneratedFiles->cleanGeneratedFiles() #5 /home/ in /home/cloudpanel/htdocs/www.performance-pcs.com/vendor/magento/framework/App/ObjectManager.php on line 36

2. Screenshots

Things we tried and results:

1. Our Host of AWS Cloud server adjusted these settings in this order: a. From the error shown you have reached the limit of 768M that was set on the php config file. We have increased it and set to memory_limit = 1024M b. The php-fpm of cli had memory_limit = 4048M. We have set this to unilimited c. So we have increased the memory limit to :8092M on file ".user.ini" on Magento root & inside pub/ location. mgt@performance-pcs:/home/cloudpanel/htdocs/www.performance-pcs.com/pub$ cat .user.ini memory_limit = 8092M
2. How we fixed: Well we didn't fix actually but we proved to ourselves that it was the number of images that caused the problem. So we moved all our images out of media except for about 200 or so. And vola! The media gallery opened fine with the proper tree structure, search and sort fields and all functions worked.

So now what?? We need to access all our NON-PRODUCT IMAGES!. I can turn back on old media gallery and it does work but I would like to use the new one with the Adobe image features, that is handy!

Please provide Severity assessment for the Issue as Reporter. This information will help during Confirmation and Issue triage processes.

• [ ] Severity: S0 - Affects critical data or functionality and leaves users without workaround.
• [X ] Severity: S1 - Affects critical data or functionality and forces users to employ a workaround.
• [ ] Severity: S2 - Affects non-critical data or functionality and forces users to employ a workaround.
• [ ] Severity: S3 - Affects non-critical data or functionality and does not force users to employ a workaround.
• [ ] Severity: S4 - Affects aesthetics, professional look and feel, “quality” or “usability”.

Additional info from Engcom

To reproduce the issue

1. Execute this file to generate empty txt files in media folder https://gist.github.com/Nazar65/61c756ce76afa8c6718bf670794d548c execute in CMD bash glob.sh
2. Set memory_limit in php to 100MB
3. Open Media Galley

Actual Result: Media gallery directory tree component renders directory tree component

[m2-assistant[bot]]

Hi @perfpcs. Thank you for your report. To help us process this issue please make sure that you provided the following information:

• Summary of the issue
• Information on your environment
• Steps to reproduce
• Expected and actual results

Please make sure that the issue is reproducible on the vanilla Magento instance following Steps to reproduce. To deploy vanilla Magento instance on our environment, please, add a comment to the issue:

@magento give me 2.4-develop instance - upcoming 2.4.x release

For more details, please, review the Magento Contributor Assistant documentation.

Please, add a comment to assign the issue: @magento I am working on this

---

• Join Magento Community Engineering Slack and ask your questions in #github channel.

:warning: According to the Magento Contribution requirements, all issues must go through the Community Contributions Triage process. Community Contributions Triage is a public meeting.

:clock10: You can find the schedule on the Magento Community Calendar page.

:telephone_receiver: The triage of issues happens in the queue order. If you want to speed up the delivery of your contribution, please join the Community Contributions Triage session to discuss the appropriate ticket.

:movie_camera: You can find the recording of the previous Community Contributions Triage on the Magento Youtube Channel

:pencil2: Feel free to post questions/proposals/feedback related to the Community Contributions Triage process to the corresponding Slack Channel

[m2-assistant[bot]]

Hi @engcom-Bravo. Thank you for working on this issue. In order to make sure that issue has enough information and ready for development, please read and check the following instruction: :point_down:

• [ ] 1. Verify that issue has all the required information. (Preconditions, Steps to reproduce, Expected result, Actual result).

  Details

  If the issue has a valid description, the label Issue: Format is valid will be added to the issue automatically. Please, edit issue description if needed, until label Issue: Format is valid appears.

• [ ] 2. Verify that issue has a meaningful description and provides enough information to reproduce the issue. If the report is valid, add Issue: Clear Description label to the issue by yourself.

• [ ] 3. Add Component: XXXXX label(s) to the ticket, indicating the components it may be related to.

• [ ] 4. Verify that the issue is reproducible on 2.4-develop branch

  Details

  - Add the comment @magento give me 2.4-develop instance to deploy test instance on Magento infrastructure.
  - If the issue is reproducible on 2.4-develop branch, please, add the label Reproduced on 2.4.x.
  - If the issue is not reproducible, add your comment that issue is not reproducible and close the issue and stop verification process here!

• [ ] 5. Add label Issue: Confirmed once verification is complete.

• [ ] 6. Make sure that automatic system confirms that report has been added to the backlog.

[Nazar65]

@magento I am working on this

[magento-engcom-team]

@engcom-Bravo Thank you for verifying the issue.

Unfortunately, not enough information was provided to acknowledge ticket. Please consider adding the following:

• [ ] Add "Reproduced on " label(s) to this ticket based on verification result

Once all required information is added, please add label "Issue: Confirmed" again. Thanks!

[magento-engcom-team]

:white_check_mark: Confirmed by @engcom-Bravo Thank you for verifying the issue. Based on the provided information internal tickets MC-38701 were created

Issue Available: @engcom-Bravo, You will be automatically unassigned. Contributors/Maintainers can claim this issue to continue. To reclaim and continue work, reassign the ticket to yourself.

[engcom-Bravo]

Additional info from Engcom

To reproduce the issue

1. Execute this file to generate empty txt files in media folder https://gist.github.com/Nazar65/61c756ce76afa8c6718bf670794d548c execute in CMD bash glob.sh
2. Set memory_limit in php to 100MB
3. Open Media Galley

Actual Result: Media gallery directory tree component renders directory tree component

[perfpcs]

Just an update here, the new file does render the directory tree but no images show up when you go to each folder. Also the whole media gallery locks up with a spinning ajax wheel with most times no search box, no filter or sort shows up. So, yes the tree now shows but the gallery is useless to use as it is now.

[perfpcs]

So I spent many many hours testing things this weekend and trying every possible combination of fixes. What I finally determined was the problem for me was having both my static files and media files on 2 different AWS Cloudfront Distributions. Here is what I determined this weekend:

1. #30629 this pull does actually work properly with large amounts of images, In my case over 2 million images. This is true only when using NO CDN and Magento Base URL's are set as URL's local to the server and not pointed to AWS Cloudfront CDN distributions. See settings:

2. #31749 this pull does actually work properly with large amounts of images, In my case over 2 million. Again this works with settings pointing Static and Media to the local server via URL. Same as above.

3. Both #30629 and #31749 DO NOT work when Magento Static Files and Media files are served from an AWS CDN. While the directory tree loads, the images do not always load. Clicking on folders changes nothing in views. Also the search bar does not appear and clicking on any folder shows no active filters to view or click and cancel. Also Adobe Stock Images does not work at all, you can not click and save images or previews. Clicking on anything in Adobe Stock does nothing.

4.#30629 DOES work fine with large amounts of images (in my case over 2 million images) when the Static URL is change back to local server. I WAS able to use the media Cloudfront distribution though. So, something about Magento files does not like the static files being served from a Cloudfront Distribution. Bottom line, with #30629 pull, Static URL pointed to local and media URL pointed to Cloudfront server, everything worked perfect. The search bar shows, clicking on folders adds them as the active filters that you can X out and close. Search terms work fine. Adobe Stock Images works perfectly and I can add previews and images with no problems. Basically, I can say everything works perfectly as expected with #30629 and just changing Static URL to the local server.

5. #31749 for me does not work using any CDNs. I tried both Static and Media both, individual each, and nothing works with AWS Cloudfront. So #31749 ONLY works with static and media URLs pointed to local URLs.

So, I hope this is helpful to those working on these files. I would love to have a fix to be able to use my Static file CDN on AWS. I am not sure what the issue is, I can only report what is happening. Thanks

[thomas-kl1]

Right now I don't see the correlation between the base_url of statics and media and the filesystem lookup. Do you have many environments where sometimes you have NAS or EFS?

[perfpcs]

@thomas-kl1 I am not sure I really understand your question. But many shops use Cloundfront, Cloudflare, Fastly, etc for CDN of static and media. All I can report is that when changing the URLs in the General/Web Magento settings pointing to the CDN server vs the local server url's, for both my Magento 2.41 projects, the gallery does not work at all using your 2 files. I don't know why though, sorry.

[thomas-kl1]

what is actually interesting is that you say the directory tree works (which the two PR tend to fix). So your issue is about filters and images when CDN is used, right?

[perfpcs]

@thomas-kl1 yes Sir, the directory tree is now forming with large data amount with both PR. So the remaining issues is with CDN and active filters and using the folders.

[perfpcs]

Just an update, after Amasty fixed some issues with their Extended Product Grid Extension that was conflicting with the new media gallery js files, I am finally able to have a fully working media gallery in the latest Magento 2.4.2 with the one file changed from #30629. vendor/magento/module-media-gallery-ui/Model/Directories/GetDirectoryTree.php

With this one file change all now works correctly for us including using static files on CDN. The directory tree forms, the active filters work fine now and all aspects of Adobe Stock and adding images works properly.

So hopefully #30629 is added to the next Magento release and I think all will be fine. thanks for peoples help.

[Nazar65]

@perfpcs it would be great to have it on next release, but @shiftedreality https://github.com/magento/magento2/pull/30629#discussion_r532666413 decided to deliver this only in 2.5 :(

[perfpcs]

@Nazar65 well I think a lot of people with big catalogs will struggle with the new media gallery as it probably wont work for them. But, I guess It's the Magento teams decision, not mine.

[iphigenie]

I don't have a large catalog or large amounts of CMS images but I wanted to use a subdomain for my images to have different cloudflare settings, and as a result I have a different path for static files (unchanged) and media file (subdomain) and I am getting

• gallery related errors in admin (full on error 500 when opening cms pages or blocks grid or products grid that vanishes when doing a regenerate di.xml, flush, reindex)
• errors in logs for frontend access, occasional, leaving in the log the following

ObjectManager isn't initialized
#0 /vendor/magento/framework/File/Mime.php(84): Magento\Framework\App\ObjectManager::getInstance()
#1 /pub/get.php(61): Magento\Framework\File\Mime->__construct()

This started when I upgraded to 2.4.2 and enabled the new gallery

I think the pattern here could explain my issues - very grateful at @perfpcs ! will look at #30629

[Nazar65]

@iphigenie Hello, how it's related to this issue ? and why it's gallery related error ?

[oscarsanch]

@Nazar65 Hello, I think this issue has already been fixed, MC-41854: Page builder gallery load issue (https://github.com/magento/magento2/commit/84e590c66b0fd3c3585bc50d784325f8dfcd81eb) The issue was recursively reading directories(magento2.4.2) \Magento\Cms\Block\Adminhtml\Wysiwyg\Images\Tree::getTreeJson line 79 $nestedDirectories = $this->getMediaDirectory()->readRecursively($item->getFilename());

Could you check it out?

[Nazar65]

@oscarsanch looks like, no bcs. this commit is for the default gallery, but also does not looks good at all :) it still go over all directories and files , which is very bad if you have a lot of images. My changes which magneto does not want accept is a performance fix which allows you to open new media gallery with large catalog, but still does not resolve issue when you have really a lot of folders like 20k or 30k which will be displayed in media gallery. To Fix it at all we need to change implementation of Folder tree to load one by one only when you click for that directory. But issue that you mention not related at all to the new media gallery.

[jorgb90]

Damn I am glad this issue is fixed! Performance is terrible out of the box! #30629 did the trick for me.

Is there a way to make the wysiwyg directory as root for the gallery? No one needs access to the complete media dir..

[thomas-kl1]

@jorgb90 you can exclude directories with the following file, in the etc folder of your module:

directory.xml

<?xml version="1.0"?>
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="urn:magento:module:Magento_MediaGalleryApi:etc/directory.xsd">
    <exclude>
        <patterns>
            <pattern name="myIdentifier">/^myFolder/</pattern>
        </patterns>
    </exclude>
</config>

[pmonosolo]

I have the same issue on 2.4.2-p1 with 300,000 products. The #30629 fixes the issue. Just using the XML folder exclude does not work.

This issue also bricks the old media gallery, so it definitely needs to be fixed.

[thomas-kl1]

@pmonosolo the exclude config does not works because it's applied after that all files have been resolved by Magento instead of filtering directories while discovering the files. (this is true, no joke...)

[pmonosolo]

@pmonosolo the exclude config does not works because it's applied once that all files are resolved by Magento instead of filtering directories while discovering the files. (this is true, no joke...)

I already did a override for the GetDirectoryTree.php file, is there a way to resync the directories?

I have seen this:

php bin/magento media-gallery:sync
php bin/magento media-content:sync

but have not tried it.

[pmonosolo]

I have seen this:

php bin/magento media-gallery:sync
php bin/magento media-content:sync

but have not tried it.

Nope, this didn't help.

[pmonosolo]

How'd ya'll override this class? I have tried preference and plugin methods but both did not work.

UPDATE: If anybody interested I posted an answer here(sorry for dirty implementation, I'll rewrite it later on): https://magento.stackexchange.com/questions/341510/overriding-getdirectorytree-php-preference-not-working/341549#341549

[luka4]

I fixed this with following patch (logic taken from magento version 2.3.3-p1)

Index: vendor/magento/module-cms/Block/Adminhtml/Wysiwyg/Images/Tree_original.php
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
--- a/vendor/magento/module-cms/Block/Adminhtml/Wysiwyg/Images/Tree.php	(revision a5387d040efc9efcbaebd6a6c28ba591f87d07b5)
+++ b/vendor/magento/module-cms/Block/Adminhtml/Wysiwyg/Images/Tree.php	(date 1628075064738)
@@ -76,8 +76,7 @@
                 'path' => substr($item->getFilename(), strlen($storageRoot)),
                 'cls' => 'folder',
             ];
-            $nestedDirectories = $this->getMediaDirectory()->readRecursively($item->getFilename());
-            $hasNestedDirectories = count($nestedDirectories) > 0;
+            $hasNestedDirectories = count(glob($item->getFilename() . '/*', GLOB_ONLYDIR)) > 0;
 
             // if no nested directories inside dir, add 'leaf' state so that jstree hides dropdown arrow next to dir
             if (!$hasNestedDirectories) {

[barryvdh]

This also broke our Image browser. It caused the browser to logout the admin session everytime. Replacing it with the global like in previous versions fixed it.

[cjnewbs]

@luka4 Thank you for this! I was using a custom patch that just commented out lines 79-85 but your patch kept the functionality.

For context we had about 260K files in 40K directories and were getting Cloudflare 524 timeout errors. Bypassing Cloudflare resulted in a 7 min TTFB for when loading the media root.

Applying this patch reduced the TTFB to 1.7 sec. Thats on my local so on staging/prod is likely to be less.

[perfpcs]

@cjnewbs Did you load the 3 files in the official fix and adjust these 3 lines? Or use the default Magento files and only adjust these 3 lines? I've loaded the patch but, I'm still very disappointed on how long it takes to load the media gallery compared to the old version of media gallery. This version of media gallery, although flashy looking and access to Adobe, is so incredibly slow that it takes at least 30-60 seconds on first load in admin. And even then it is 50% chance the directory tree is missing. Maybe with 2-5 page reloads I can get it to appear. So I do not consider this a good patch or really fixed. Right now its like putting a small Bandaid on a giant bloody wound. Is anyone working on anything better to fix this glaring problem??