core Extrem long save time at save items at BE

look at https://github.com/contao-community-alliance/dc-general/issues/201

we found the first bottle necks...

1.) ToolboxFile.php https://github.com/MetaModels/core/blob/08b502234efe150003450cf01b5611b00f9ebd57/src/MetaModels/Helper/ToolboxFile.php#L700

    public static function convertValueToPath($varValue)
    {
+        if (empty($varValue)) {
+            return '';
+        }
        $objFiles = FilesModel::findByPk($varValue);

2.) with "MM-1" and attribute select to "MM-2" which also a have relation per select to "MM-3" we found a strong increase of database queries... may be, we can "cache" all existing queries https://github.com/MetaModels/core/blob/08b502234efe150003450cf01b5611b00f9ebd57/src/MetaModels/MetaModel.php#L704

...
   /**
     * The service container.
     *
     * @var IMetaModelsServiceContainer
     */
    protected $serviceContainer;
+
+    private $existingIds = [];

...

    public function getIdsFromFilter($objFilter, $strSortBy = '', $intOffset = 0, $intLimit = 0, $strSortOrder = 'ASC')
    {
        $arrFilteredIds = $this->getMatchingIds($objFilter);

        // If desired, sort the entries.
        if ($arrFilteredIds && $strSortBy != '') {
            if ($objSortAttribute = $this->getAttribute($strSortBy)) {
                $arrFilteredIds = $objSortAttribute->sortIds($arrFilteredIds, $strSortOrder);
            } elseif (in_array($strSortBy, array('id', 'pid', 'tstamp', 'sorting'))) {
+                if (array_intersect($arrFilteredIds, $this->existingIds) == $arrFilteredIds) {
+                    return $arrFilteredIds;
+                }
                // Sort by database values.
                $arrFilteredIds = $this
                    ->getDatabase()
                    ->prepare(
                        sprintf(
                            'SELECT id FROM %s WHERE id IN(%s) ORDER BY %s %s',
                            $this->getTableName(),
                            $this->buildDatabaseParameterList($arrFilteredIds),
                            $strSortBy,
                            $strSortOrder
                        )
                    )
                    ->execute($arrFilteredIds)
                    ->fetchEach('id');
+                $this->existingIds = array_merge($this->existingIds, $arrFilteredIds);
            } elseif ($strSortBy == 'random') {
                shuffle($arrFilteredIds);
            }
        }
...

May 25 '17 06:05 zonky2

I have this message : Fatal error: Cannot redeclare MetaModels\MetaModel::$serviceContainer in /homepages/4/d406509772/htdocs/.../system/modules/metamodels/classes/src/MetaModels/MetaModel.php on line 692

I can't clear this line ? protected $serviceContainer;

I noted that the save time increases when i have a lot of items in MM (5000 or 10 000 items) before or after the changes.

May 29 '17 09:05 Damsgit

add only private $existingIds = [];

May 29 '17 09:05 zonky2

I have one page that accesses different MetaModels. It uses about 1200 database queries to load the site (Contao 4 / Symfony debug console). I changed the following in the MetaModel.php and was able to reduce the number of database queries to about 400(!!!).

+   private $cache;

    public function findById($intId, $arrAttrOnly = array())
    {
         if (!$intId) {
             return null;
         }
+
+        if (isset($this->cache[$intId])) {
+            return $this->cache[$intId];
+        }
+
         $objItems = $this->getItemsWithId(array($intId), $arrAttrOnly);
         if ($objItems && $objItems->first()) {
-            return $objItems->getItem();
+            $this->cache[$intId] = $objItems->getItem();
+            return $this->cache[$intId];
         }
         return null;
     }

Maybe there is a way to reach the same result by using doctrine cache / APCu??

Aug 03 '18 07:08 richardhj

may be it´s a good idea to equalize the terms private $cache; to private $existingIds = [];

Aug 03 '18 08:08 zonky2

@richardhj Your change introduces a problem when $attrOnly is not empty, therefore we can not walk down that road. :crying_cat_face:

Aug 03 '18 08:08 discordier

@discordier Did you ever enabled APCu doctrine cache? Is this available for Contao?

Aug 03 '18 08:08 richardhj

PRs: #1240 and #1241

Aug 05 '18 09:08 zonky2

Hotfix bottle neck attribute tags: https://github.com/MetaModels/attribute_tags/pull/68

Aug 11 '18 19:08 zonky2

I'm afraid we'll find more... I reopen the issue as a "collective issue".

Aug 12 '18 16:08 zonky2

see https://github.com/MetaModels/core/pull/1339

Oct 02 '19 12:10 zonky2

see https://github.com/MetaModels/core/issues/1149

Dec 05 '19 08:12 zonky2

see #1379

Mar 16 '20 13:03 zonky2

see #1409

May 29 '20 05:05 zonky2

As mentioned in #1352:

we should add cache for non-existent files at Template.php https://github.com/MetaModels/core/blob/33244d82cce0469c7b4040355ba22f9fdf7001b4/src/Render/Template.php#L250

Current workaround is to add a .text template next to the custom named .html5 template. However, this became significantly harder since Contao does not allow to specify custom template extensions anymore.

Jun 24 '20 06:06 discordier

As mentioned in #1352:

Fixed in MM 2.2

Jan 23 '22 16:01 zonky2

Read any tip to speed up MM: https://metamodels.readthedocs.io/de/latest/cookbook/tips/speed-up-backend-list.html

Jan 23 '22 16:01 zonky2

Currently there is still the problem in the BE, if many attributes are selected to the filter and there are many items...

@discordier has solution proposals for this

Aug 23 '22 19:08 zonky2

core core copied to clipboard

Extrem long save time at save items at BE

core
core copied to clipboard