Volume Settings / Sync between Environments

[davidhellmann]

I'm not sure if I do something wrong or what going wrong here but I try to explain.

general.php My aliases

'aliases' => [
    '@basePath' => getenv('CRAFTENV_BASE_PATH'),
    '@baseUrl' => getenv('CRAFTENV_BASE_URL'),
],

volumes.php

return [

    // All environments
    '*' => [
        'siteImages' => [
            'path' => '@basePath/uploads/images/',
            'url' => '@baseUrl/uploads/images/',
        ],
        'siteProducts' => [
            'path' => '@basePath/uploads/products/',
            'url' => '@baseUrl/uploads/products/',
        ],
        'siteDownloads' => [
            'path' => '@basePath/uploads/downloads/',
            'url' =>'@baseUrl/uploads/downloads/',
        ],
        'siteGraphics' => [
            'path' => '@basePath/uploads/graphics/',
            'url' => '@baseUrl/uploads/graphics/',
        ],
        'siteUsers' => [
            'path' => '@basePath/uploads/users/',
            'url' =>'@baseUrl/uploads/users/',
        ],
        'test' => [
            'path' => '@basePath/uploads/test/',
            'url' =>'@baseUrl/uploads/test/',
        ],
    ],

    // Live (production) environment
    'live'  => [
    ],

    // Staging (pre-production) environment
    'staging'  => [
    ],

    // Staging (pre-production) environment
    'stagingNeubau'  => [
    ],

    // Local (development) environment
    'local'  => [
    ],
];

So now, when I'm look into the backend it looks fine:

But when I look into the DB I've absolute URLs

{
  "optimizedImageUrls": {
    "200": "https://stage.foobarbaz.com/uploads/images/_200x133_crop_center-center_82_line/btr-ranger-5.jpg",
    "400": "https://stage.foobarbaz.com/uploads/images/_400x266_crop_center-center_82_line/btr-ranger-5.jpg",
    "800": "https://stage.foobarbaz.com/uploads/images/_800x533_crop_center-center_82_line/btr-ranger-5.jpg",
    "1200": "https://stage.foobarbaz.com/uploads/images/_1200x800_crop_center-center_82_line/btr-ranger-5.jpg"
  },
  "optimizedWebPImageUrls": {
    "200": "https://stage.foobarbaz.com/uploads/images/_200x133_crop_center-center_82_line/btr-ranger-5.jpg.webp",
    "400": "https://stage.foobarbaz.com/uploads/images/_400x266_crop_center-center_82_line/btr-ranger-5.jpg.webp",
    "800": "https://stage.foobarbaz.com/uploads/images/_800x533_crop_center-center_82_line/btr-ranger-5.jpg.webp",
    "1200": "https://stage.foobarbaz.com/uploads/images/_1200x800_crop_center-center_82_line/btr-ranger-5.jpg.webp"
  },
  "variantSourceWidths": [
    "1200",
    "800",
    "400",
    "200"
  ],
  "focalPoint": {
    "x": 0.5,
    "y": 0.5
  },
  "originalImageWidth": "4000",
  "originalImageHeight": "2667",
  "placeholder": "/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkzODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2P/2wBDARESEhgVGC8aGi9jQjhCY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2P/wAARCAAKABADASIAAhEBAxEB/8QAFgABAQEAAAAAAAAAAAAAAAAAAwIE/8QAIBAAAQMEAwEBAAAAAAAAAAAAAQIREgADBCEFMUETYf/EABQBAQAAAAAAAAAAAAAAAAAAAAL/xAAVEQEBAAAAAAAAAAAAAAAAAAAAEf/aAAwDAQACEQMRAD8A04q8izAi9MrIaSz1tnH7U5Gdn40xcTcBG3BcDXVLjEnkLSSSUhSmHgal5RR+8XMVRkPDv2iUf//Z",
  "placeholderSvg": "",
  "colorPalette": [
    "#cabfb9",
    "#363b3a",
    "#797c79",
    "#61584e",
    "#716b55"
  ],
  "placeholderWidth": 1200,
  "placeholderHeight": 800
}

When I'm sync the DB via craft-scripts I've always the wrong URLs :( What's wrong?

[davidhellmann]

No Idea? :)

[khalwat]

So I want to say that this is normal (though maybe not desirable?)l. The URLs in the OptimizedImages fields get resolved at asset save time. So the full absolute URL to each image will resolve to whatever the alias is.

Potentially this resolution could be postponed until runtime, but in the interim, if you re-index your assets on the environment.

[davidhellmann]

Sure, reindexing works but I takes several hours to do that. And when I sync DB / Assets from Live to Stage or Stage to Local I've to reindex all that? Why not /uploads/*.jpg or something else?

hm

[stevehurst]

I've reindexed and cleared cache, but I'm still getting the cached URLs from the different environment in my templates.

[khalwat]

@stevehurst Go to the Asset Volume in question, and click on Save -- it will cause it to recreate all of the image variants

[stevehurst]

@khalwat I've tried saving the Asset Volume settings, as well as saving the ImageOptimize fields associated. I see the "optimizing" animation in the sidebar, but the database tables don't get updated with the different environment URL.

[stevehurst]

@khalwat to reiterate, I have migrated a local database to a dev server.

I have:

1. Re-saved Asset Volume settings
2. Re-indexed Asset Volumes
3. Re-saved ImageOptimize fields
4. Cleared all caches

And my local environment URLs are still saved in my database and used in my templates for ImageOptimize tags. If I use a simple asset tag, the dev environment URL is used.

[khalwat]

How is your multi-environment setup configured? Are you using Aliases in your volumes.php @stevehurst ?

[stevehurst]

@khalwat basePath and baseUrl

    // All environments
    '*' => [
        'ASSET_HANDLE' => [
            'path' => '@basePath/ASSET_PATH',
            'url' => '@baseUrl/ASSET_PATH',
        ],
    ],

[stevehurst]

And I'm using this multi-environment config: https://github.com/nystudio107/craft3-multi-environment

[stevehurst]

@khalwat I see you're the developer of that as well. I used version 1.0.5 for my install and haven't updated. Is there an easy way to update to 1.0.7?

[khalwat]

Huh, I'm really curious why these URLs are not being updated. If you're available to do a screen share, ping me on Craft Slack?

[davidhellmann]

would be nice when you share the outcome :)

[khalwat]

This ended up just being that he had an SVG that wasn't able to render to other formats on his staging environment (probably due to lack of Imagick), and that happened to be the one image he was testing.

[joshuabaker]

Just ran into this issue myself. A way of addressing this issue to avoid having to rebuild an entire asset volume is definitely desirable.

[luke-nehemedia]

Same problem here. It would be very desirable to be able to change the paths and not have to reindex.

I use this plugin on one clients website. Right now I am considering using it on a couple of other projects, but because of this issue I struggle to suggest this plugin.

[joshuabaker]

@khalwat Please can we look at a solution for this. For some of the bigger sites the reindex can be quite painful.

[qrazi]

Running into something similar, I guess. We changed some paths on volume storage, resulting in the URLs also changing slightly. Forcing a recreate of all variants would take quite a bit of time, as we have thousands of assets, a lot of which aren't very small. So we manually moved the existing variants in the new structure. And we're now using a standard Craft / Yii2 migration to update the paths. Something along the lines of:

// Query all fields of type OptimizedImages and get any Element using them and iterate over them

// Per element / field combo get the model
$optimized_image = $element->{$field->handle};

// Using Laravel Collections here, but of course that is not required
// We're using preg_replace to change the subdir that we changed
$optimized_image->optimizedImageUrls = collect(
    $optimized_image->optimizedImageUrls
)
    ->map(fn ($url) => preg_replace("/^(.*oldsubdir\\/)((?!.*newsubdir)[^\\/]*)(\\/.*)\$/", "\$1newsubdir\$3", $url))
    ->toArray();

// Now save the updated model
// Save code copied from `nystudio107\imageoptimize\services\OptimizedImages::updateOptimizedImageFieldData`
$element->setFieldValue($field->handle, $field->serializeValue($optimized_image));
$table = $element->getContentTable();
$column = $element->getFieldColumnPrefix() . $field->handle;
$data = Json::encode($field->serializeValue($element->getFieldValue($field->handle), $element));
Craft::$app->db->createCommand()
    ->update($table, [
        $column => $data,
    ], [
        'elementId' => $element->getId(),
    ], [], false)
    ->execute();

Disclaimer; I haven't actually run it yet against our production environment, currently testing. But it illustrates a strategy for solving this I think.

[khalwat]

@qrazi I've been thinking about adding a flag to the CLI that regenerates the image variants that would skip the actual variant creation (but would update everything else) -- would this be useful to you?

[joshuabaker]

@khalwat That would certainly be a helpful task to have. 😊

Is there a reason it can’t be done within normalizeValue (or similar) when getting the value during the request?

[qrazi]

@khalwat Last night I ran the migration and our production environment is now running from the updated paths.

I think using migrations makes sense in that they are meant to update content and in that they are meant for singular changes. But, adding this as a CLI command would have saved me time writing my migration. So for the user base I can see how it's a valuable addition.

[khalwat]

Actually, I've been over the code, and I'm thinking just running the existing CLI command should do what you want.

The reason is that ImageOptimize uses regular old Craft CMS transforms... and they will not redo a transform if the file already exists on disk.

So if you make changes, when it goes through and resaves all of the OptimizedImages yes it'll chunk through each image, but as long as the transformed image exists on disk, it shouldn't be a particularly length process.

Have you tried this out on a staging or local dev environment @qrazi ?

[qrazi]

@khalwat I'll try and reproduce locally again to see why it didn't update for me (or find out it does work, but I didn't see it correctly before... 😅 )

[terryupton]

Was there a resolution for this that doesn't involve running the command line in each environment? It is giving me a few issues at the moment and would be great if the urls were environmentally aware like they use {siteUrl} or similar...

[khalwat]

Currently, no.

[turnstylerj]

I'm running into this issue getting a local in-development site deployed to a staging server. Reading this thread, sounds like I should be re-saving the assets directory, but running php craft resave/assets on the staging server (as well as my local server) is returning:

error: Calling unknown method: craft\console\Request::getPathInfo()

for each of the assets it's trying to resave. If I load one of the assets up in the control panel on the server and click the "save" button, I get the following output in queue.log:

2022-02-25 13:45:08 [-][1][-][info][craft\queue\QueueLogBehavior::beforeExec]  [477] Updating search indexes (attempt: 1, pid: 348) - Started
2022-02-25 13:45:08 [-][1][-][info][craft\queue\QueueLogBehavior::afterExec]  [477] Updating search indexes (attempt: 1, pid: 348) - Done (time: 0.028s)

but the image paths still don't seem to update (just looking at the variants in the CP to take the front end out of the equation). New uploads are working correctly and showing the expected paths in both environments.

Any idea what might be going on here, or anything I can do to get you some more useful debugging information?

Craft v3.7.34 ImageOptimize v1.6.42

[khalwat]

@turnstylerj please use the ImageOptimize CLI for regenerating the images:

https://nystudio107.com/docs/image-optimize/using.html#forcing-image-variant-creation

As for the error, I can't tell anything without the full stack trace -- but that may or may not be from ImageOptimize

[turnstylerj]

Ah, there we go. Thanks for clarifying!

[almeric]

@khalwat not to hijack this issue but we also see the getPathInfo() error when running php craft resave/assets from the console.

It's called here: https://github.com/nystudio107/craft-imageoptimize/blob/c07f0297fa1fafd66780f915a71485bd06033580/src/fields/OptimizedImages.php#L243

But a craft\console\Request apparently doesn't have that method.

Stack trace:

2022-03-11 12:14:21 [-][-][-][error][yii\base\UnknownMethodException] yii\base\UnknownMethodException: Calling unknown method: craft\console\Request::getPathInfo() in /app/vendor/yiisoft/yii2/base/Component.php:300
Stack trace:
#0 /app/vendor/nystudio107/craft-imageoptimize/src/fields/OptimizedImages.php(243): yii\base\Component->__call('getPathInfo', Array)
#1 /app/vendor/craftcms/cms/src/base/Element.php(4142): nystudio107\imageoptimize\fields\OptimizedImages->afterElementSave(Object(craft\elements\Asset), false)
#2 /app/vendor/craftcms/cms/src/elements/Asset.php(2035): craft\base\Element->afterSave(false)
#3 /app/vendor/craftcms/cms/src/services/Elements.php(2701): craft\elements\Asset->afterSave(false)
#4 /app/vendor/craftcms/cms/src/services/Elements.php(993): craft\services\Elements->_saveElementInternal(Object(craft\elements\Asset), true, true, false)
#5 /app/vendor/craftcms/cms/src/console/controllers/ResaveController.php(458): craft\services\Elements->resaveElements(Object(craft\elements\db\AssetQuery), true, true, false)
#6 /app/vendor/craftcms/cms/src/console/controllers/ResaveController.php(333): craft\console\controllers\ResaveController->_resaveElements(Object(craft\elements\db\AssetQuery))
#7 /app/vendor/craftcms/cms/src/console/controllers/ResaveController.php(229): craft\console\controllers\ResaveController->resaveElements('craft\\elements\\...', Array)
#8 [internal function]: craft\console\controllers\ResaveController->actionAssets()
#9 /app/vendor/yiisoft/yii2/base/InlineAction.php(57): call_user_func_array(Array, Array)
#10 /app/vendor/yiisoft/yii2/base/Controller.php(178): yii\base\InlineAction->runWithParams(Array)
#11 /app/vendor/yiisoft/yii2/console/Controller.php(182): yii\base\Controller->runAction('assets', Array)
#12 /app/vendor/craftcms/cms/src/console/Controller.php(221): yii\console\Controller->runAction('assets', Array)
#13 /app/vendor/yiisoft/yii2/base/Module.php(552): craft\console\Controller->runAction('assets', Array)
#14 /app/vendor/yiisoft/yii2/console/Application.php(180): yii\base\Module->runAction('resave/assets', Array)
#15 /app/vendor/craftcms/cms/src/console/Application.php(89): yii\console\Application->runAction('resave/assets', Array)
#16 /app/vendor/yiisoft/yii2/console/Application.php(147): craft\console\Application->runAction('resave/assets', Array)
#17 /app/vendor/yiisoft/yii2/base/Application.php(384): yii\console\Application->handleRequest(Object(craft\console\Request))
#18 /app/craft(23): yii\base\Application->run()
#19 {main}

[khalwat]

@almeric Addressed in: https://github.com/nystudio107/craft-imageoptimize/commit/18c0695bb20a7ba429f8b4ed3d49bb24e8963944

You can try it now by setting your semver in your composer.json to look like this:

    "nystudio107/craft-imageoptimize": "dev-develop as 1.6.43”,

Then do a composer clear-cache && composer update