[4.x]: Column not found when applying project config (matrix field)

[wallacio]

What happened?

Description

I have added a Link Field to an existing Matrix block, working correctly (editing/rendering) in the dev environment on which it was created. Applying project config on another environment fails, with an SQL Column not found error. The migration that's auto-generated appears to be looking for columns on the content table which do not exist (or at least which exist, but are called something slightly different).

Similar to https://github.com/craftcms/cms/issues/9749 and https://github.com/sebastian-lenz/craft-linkfield/issues/163 except this bug is present when applying project config. Is this a Craft issue, or a Plugin issue @sebastian-lenz ?

The pertinent error is Column not found: 1054 Unknown column 'content.field_fullWidthCta_videoUrl' in 'field list' but looking at the SELECT that's generated, the same is happening for other Matrix columns. The Matrix field name is being added to the column name, but that doesn't reflect what's in the database schema (i.e. it contains `content.field_videoUrl, not content.field_fullWidthCta_videoUrl etc.).

Part of the content table structure:

The changes in the Matrix block include a new Link Field (sebastian-lenz/craft-linkfield, the plugin installed at the same time. The Craft installation was updated from 3.7 to 4.x a few weeks ago.

Full output of the project-config/apply below.

Steps to reproduce

1. Add a Link Field to a Matrix block, apply the project-config changes to production server.

Expected behavior

The project-config is applied correctly and the Link Field is present in the Matrix block editor.

Actual behavior

Application of project config fails:

Applying changes from your project config files ...
- removing matrixBlockTypes.38e31717-13d0-4507-9208-97b907246bc6.fields.626ea6a6-4dda-46bf-ac28-33ce80a7ef7b.settings ...
  - updating matrixBlockTypes.38e31717-13d0-4507-9208-97b907246bc6 ...
error: SQLSTATE[42S22]: Column not found: 1054 Unknown column 'content.field_fullWidthCta_videoUrl' in 'field list'
The SQL being executed was: SELECT `elements`.`id`, `elements`.`canonicalId`, `elements`.`fieldLayoutId`, `elements`.`uid`, `elements`.`enabled`, `elements`.`archived`, `elements`.`dateLastMerged`, `elements`.`dateCreated`, `elements`.`dateUpdated`, `elements_sites`.`id` AS `siteSettingsId`, `elements_sites`.`slug`, `elements_sites`.`siteId`, `elements_sites`.`uri`, `elements_sites`.`enabled` AS `enabledForSite`, `entries`.`sectionId`, `entries`.`typeId`, `entries`.`authorId`, `entries`.`postDate`, `entries`.`expiryDate`, `content`.`id` AS `contentId`, `content`.`title`, `content`.`field_fullWidthCta_videoUrl`, `content`.`field_fullWidthCta_footertext`, `content`.`field_fullWidthCta_headerSelection`, `content`.`field_fullWidthCta_homepageAnimation`, `content`.`field_fullWidthCta_homepageLezButtonsIntro`, `content`.`field_fullWidthCta_linkToAnExternalUrl`, `content`.`field_fullWidthCta_makeSiteGrayscale_imwglwgy`, `content`.`field_fullWidthCta_metaDescription`, `content`.`field_fullWidthCta_metaTags`, `content`.`field_fullWidthCta_seo`, `content`.`field_fullWidthCta_openLinkInNewWindow`, `content`.`field_fullWidthCta_pagecontent`, `content`.`field_fullWidthCta_pagesubtitle`, `content`.`field_fullWidthCta_metaPageTitle`, `content`.`field_fullWidthCta_pagetitle`, `content`.`field_fullWidthCta_showInPrimaryNavigation`, `content`.`field_fullWidthCta_newsSummary`, `content`.`field_fullWidthCta_vrcUnavailable_tbehhjcb`, `structureelements`.`root`, `structureelements`.`lft`, `structureelements`.`rgt`, `structureelements`.`level`, `structureelements`.`structureId`
FROM (SELECT `elements`.`id` AS `elementsId`, `elements_sites`.`id` AS `elementsSitesId`, `content`.`id` AS `contentId`, `structureelements`.`structureId`
FROM `elements` `elements`
INNER JOIN `entries` `entries` ON `entries`.`id` = `elements`.`id`
INNER JOIN `elements_sites` `elements_sites` ON `elements_sites`.`elementId` = `elements`.`id`
INNER JOIN `content` `content` ON `content`.`elementId` = `elements`.`id`
LEFT JOIN `structureelements` `structureelements` ON (`structureelements`.`elementId` = `elements`.`id`) AND (EXISTS (SELECT *
FROM `structures` use index(primary)
WHERE (`id` = `structureelements`.`structureId`) AND (`dateDeleted` IS NULL)))
WHERE (`entries`.`postDate` > '2023-02-24 10:20:02') AND (`elements`.`archived`=FALSE) AND (`elements`.`dateDeleted` IS NULL) AND (`elements`.`draftId` IS NULL) AND (`elements`.`revisionId` IS NULL)
ORDER BY `postDate`
LIMIT 1) `subquery`
INNER JOIN `elements` `elements` ON `elements`.`id` = `subquery`.`elementsId`
INNER JOIN `elements_sites` `elements_sites` ON `elements_sites`.`id` = `subquery`.`elementsSitesId`
INNER JOIN `entries` `entries` ON `entries`.`id` = `subquery`.`elementsId`
INNER JOIN `content` `content` ON `content`.`id` = `subquery`.`contentId`
LEFT JOIN `structureelements` `structureelements` ON (`structureelements`.`elementId` = `subquery`.`elementsId`) AND (`structureelements`.`structureId` = `subquery`.`structureId`)
ORDER BY `postDate`

Partial stack trace from console.log:

{
    "trace": [
        "#0 /var/www/vtdocs/tslez/vendor/yiisoft/yii2/db/Command.php(1307): yii\\db\\Schema->convertException()",
        "#1 /var/www/vtdocs/tslez/vendor/yiisoft/yii2/db/Command.php(1168): yii\\db\\Command->internalExecute()",
        "#2 /var/www/vtdocs/tslez/vendor/yiisoft/yii2/db/Command.php(424): yii\\db\\Command->queryInternal()",
        "#3 /var/www/vtdocs/tslez/vendor/yiisoft/yii2/db/Query.php(287): yii\\db\\Command->queryOne()",
        "#4 /var/www/vtdocs/tslez/vendor/craftcms/cms/src/db/Query.php(275): yii\\db\\Query->one()",
        "#5 /var/www/vtdocs/tslez/vendor/craftcms/cms/src/elements/db/ElementQuery.php(1506): craft\\db\\Query->one()",
        "#6 /var/www/vtdocs/tslez/vendor/sebastianlenz/linkfield/src/listeners/ElementListenerState.php(233): craft\\elements\\db\\ElementQuery->one()",
        "#7 /var/www/vtdocs/tslez/vendor/sebastianlenz/linkfield/src/listeners/ElementListenerState.php(153): lenz\\linkfield\\listeners\\ElementListenerState->loadNextEntryChangeDate()",
        "#8 /var/www/vtdocs/tslez/vendor/sebastianlenz/linkfield/src/listeners/ElementListenerState.php(47): lenz\\linkfield\\listeners\\ElementListenerState->reset()",
        "#9 /var/www/vtdocs/tslez/vendor/sebastianlenz/linkfield/src/listeners/ElementListenerState.php(283): lenz\\linkfield\\listeners\\ElementListenerState->__construct()",
        "#10 /var/www/vtdocs/tslez/vendor/sebastianlenz/linkfield/src/fields/LinkField.php(102): lenz\\linkfield\\listeners\\ElementListenerState::getInstance()",
        "#11 /var/www/vtdocs/tslez/vendor/craftcms/cms/src/services/Fields.php(1712): lenz\\linkfield\\fields\\LinkField->afterSave()",
        "#12 /var/www/vtdocs/tslez/vendor/craftcms/cms/src/services/Matrix.php(312): craft\\services\\Fields->applyFieldSave()",
        "#13 /var/www/vtdocs/tslez/vendor/craftcms/cms/src/base/ApplicationTrait.php(1698): craft\\services\\Matrix->handleChangedBlockType()",
        "#14 /var/www/vtdocs/tslez/vendor/craftcms/cms/src/services/ProjectConfig.php(1125): craft\\console\\Application->craft\\base\\{closure}()",
        "#15 [internal function]: craft\\services\\ProjectConfig->handleChangeEvent()",
        "#16 /var/www/vtdocs/tslez/vendor/yiisoft/yii2/base/Component.php(633): call_user_func()",
        "#17 /var/www/vtdocs/tslez/vendor/craftcms/cms/src/models/ProjectConfigData.php(82): yii\\base\\Component->trigger()",
        "#18 /var/www/vtdocs/tslez/vendor/craftcms/cms/src/services/ProjectConfig.php(1115): craft\\models\\ProjectConfigData->commitChanges()",
        "#19 [internal function]: craft\\services\\ProjectConfig->handleChangeEvent()",
        "#20 /var/www/vtdocs/tslez/vendor/yiisoft/yii2/base/Component.php(633): call_user_func()",
        "#21 /var/www/vtdocs/tslez/vendor/craftcms/cms/src/models/ProjectConfigData.php(76): yii\\base\\Component->trigger()",
        "#22 /var/www/vtdocs/tslez/vendor/craftcms/cms/src/services/ProjectConfig.php(1231): craft\\models\\ProjectConfigData->commitChanges()",
        "#23 /var/www/vtdocs/tslez/vendor/craftcms/cms/src/services/ProjectConfig.php(1237): craft\\services\\ProjectConfig->craft\\services\\{closure}()",
        "#24 /var/www/vtdocs/tslez/vendor/craftcms/cms/src/services/ProjectConfig.php(577): craft\\services\\ProjectConfig->_applyChanges()",
        "#25 /var/www/vtdocs/tslez/vendor/craftcms/cms/src/console/controllers/ProjectConfigController.php(303): craft\\services\\ProjectConfig->applyExternalChanges()",
        "#26 [internal function]: craft\\console\\controllers\\ProjectConfigController->actionApply()",
        "#27 /var/www/vtdocs/tslez/vendor/yiisoft/yii2/base/InlineAction.php(57): call_user_func_array()",
        "#28 /var/www/vtdocs/tslez/vendor/yiisoft/yii2/base/Controller.php(178): yii\\base\\InlineAction->runWithParams()",
        "#29 /var/www/vtdocs/tslez/vendor/yiisoft/yii2/console/Controller.php(180): yii\\base\\Controller->runAction()",
        "#30 /var/www/vtdocs/tslez/vendor/craftcms/cms/src/console/Controller.php(227): yii\\console\\Controller->runAction()",
        "#31 /var/www/vtdocs/tslez/vendor/yiisoft/yii2/base/Module.php(552): craft\\console\\Controller->runAction()",
        "#32 /var/www/vtdocs/tslez/vendor/yiisoft/yii2/console/Application.php(180): yii\\base\\Module->runAction()",
        "#33 /var/www/vtdocs/tslez/vendor/craftcms/cms/src/console/Application.php(90): yii\\console\\Application->runAction()",
        "#34 /var/www/vtdocs/tslez/vendor/yiisoft/yii2/console/Application.php(147): craft\\console\\Application->runAction()",
        "#35 /var/www/vtdocs/tslez/vendor/craftcms/cms/src/console/Application.php(121): yii\\console\\Application->handleRequest()",
        "#36 /var/www/vtdocs/tslez/vendor/yiisoft/yii2/base/Application.php(384): craft\\console\\Application->handleRequest()",
        "#37 /var/www/vtdocs/tslez/craft(22): yii\\base\\Application->run()",
        "#38 {main}"
    ]
}

Craft CMS version

4.3.10

PHP version

8.0.28

Operating system and version

Debian 5.10.158-2

Database type and version

MariaDB 10.5.18

Image driver and version

No response

Installed plugins and versions

• Blitz 4.3.3
• Redactor 3.0.3
• Retcon 2.6.1
• Retour 4.1.10
• SEO v4.0.3
• Super Table 3.0.7
• Typed link field 2.1.5 (Being added in this update)

[wallacio]

Well here's fuel for thought, something which makes no sense.

I copied loaded project-config (i.e. not the modified version) and the database from the production server, recreated the environment locally. Git pull the changes that have caused this grief, then run project-config/apply, and it works... WHY.

[i-just]

Hi, thanks for getting in touch. I’m having trouble replicating this on a clean Craft 4 installation. Do you still have the config/project directory from before you pulled in the production config and production database and the composer.lock file from before the changes were applied? If so, could you please send those over to [email protected] so we can dig deeper?

[jurriaanr]

I am running into the same problem now. I just joined an existing project, needed to make a copy with a clean database and now when installing craft I get the same error when the code is applying the project config (I did the install through the console). I am not sure if the project code started out as craft 3, but I assume it did. Still want the config files and the composer.lock file?

More food for thought.. I can run project-config/apply right after and that will actually work (maybe also the reason why @wallacio successfully applied it after). Only problem now is that all the installation steps after applying the config have not been executed. I am not sure which steps those are, but in f.e. the admin user was not created.

Also, after switching to the source database and doing a composer update (which should not have installed new versions, since i very recently already did that) and then retrying to install this time it did work, so very similar to the original problem