Storage template migration - unexpected behaviour

[lmgithu]

The bug

When I set a new template for the folder structure (switching from yyyy-mm-dd to yyyy-mm only) and run the storage template migration job, I see that 1) the job finished running 2) 80% of the folders now have a mix of yyyy-mm-dd and yyyy-mm folders with data in them).

I wanted to check why immich is not capable of transforming all folders and why there are leftovers? My other contern is whether this is breaking viewing the images and/or the database.

Desired outcome:

• all folders on a file level should be in the new format ONLY

The OS that Immich Server is running on

Debian

Version of Immich Server

v1.98.2

Version of Immich Mobile App

v1.98.2

Platform with the issue

• [X] Server
• [ ] Web
• [ ] Mobile

Your docker-compose.yml content

standard

Your .env content

standard

Reproduction steps

1. set a new storage template
2. run the template migration job
3. not all folders are getting migrated, it is absolutely random

Additional information

No response

[lmgithu]

some additional observations:

If i start another migration to migrate thumbnails for assets and faces to the latest folder structure, that job always runs on ALL my assets (100K+)... each time i see the huge number of assets that go down to zero but it seems it is not getting processed in reality as each time when i start the job it shows same high number.

[alextran1502]

Can you help double-check the files in the older format yyyy-mm-dd to see if the path corresponds to the file's path that is shown in the UI?

[lmgithu]

I think the issue is now even bigger, I tried to check a file in that folder and one of them is there on the file system but NOT on the UI.. I don't see these images anymore :( In another case I do see as you do, it is on the UI as well and it still refers to the old folder (attached screenshot below)

[lmgithu]

If you need, I can share the logs and/or the file itself that it did not process. Let me know how I could help.

[alextran1502]

NOT on the UI.

Can you use the search for file name to find that file to see if it returns in the search result. Please attach the screenshot of the file on the file system as well as the search result if you can

[alextran1502]

Also logs, on the server when you run the migration would be helpful, you can increase the log level to debug in the Server settings in the Administration page

[lmgithu]

Will do and report back in an hour. How can I download/extract the log file?

[alextran1502]

From the terminal, run docker logs immich_server or docker logs immich_microservices

[lmgithu]

hi Alex,

Here are the results: searching for the image name on UI: no result. I can't take a screenshot as there is no results, it just shows my photo feed (I confirm it works for other image names). Screenshot of file system/location of the image: attached.

Log:

immich_server_1            | [Nest] 6  - 03/22/2024, 6:25:05 PM   DEBUG [JobService] Handling command: queue=storageTemplateMigration,force=false
immich_redis_1             | 15:C 22 Mar 2024 18:05:08.034 * RDB: 0 MB of memory used by copy-on-write
immich_redis_1             | 1:M 22 Mar 2024 18:05:08.107 * Background saving terminated with success
immich_redis_1             | 1:M 22 Mar 2024 18:10:09.036 * 100 changes in 300 seconds. Saving...
immich_redis_1             | 1:M 22 Mar 2024 18:10:09.037 * Background saving started by pid 16
immich_redis_1             | 16:C 22 Mar 2024 18:10:09.161 * DB saved on disk
immich_redis_1             | 16:C 22 Mar 2024 18:10:09.162 * RDB: 0 MB of memory used by copy-on-write
immich_redis_1             | 1:M 22 Mar 2024 18:10:09.239 * Background saving terminated with success
immich_redis_1             | 1:M 22 Mar 2024 18:15:10.025 * 100 changes in 300 seconds. Saving...
immich_redis_1             | 1:M 22 Mar 2024 18:15:10.025 * Background saving started by pid 17
immich_redis_1             | 17:C 22 Mar 2024 18:15:10.052 * DB saved on disk
immich_redis_1             | 17:C 22 Mar 2024 18:15:10.052 * RDB: 0 MB of memory used by copy-on-write
immich_redis_1             | 1:M 22 Mar 2024 18:15:10.126 * Background saving terminated with success
immich_redis_1             | 1:M 22 Mar 2024 18:20:11.071 * 100 changes in 300 seconds. Saving...
immich_redis_1             | 1:M 22 Mar 2024 18:20:11.072 * Background saving started by pid 18
immich_redis_1             | 18:C 22 Mar 2024 18:20:11.098 * DB saved on disk
immich_redis_1             | 18:C 22 Mar 2024 18:20:11.098 * RDB: 0 MB of memory used by copy-on-write
immich_redis_1             | 1:M 22 Mar 2024 18:20:11.172 * Background saving terminated with success
immich_redis_1             | 1:M 22 Mar 2024 18:25:12.096 * 100 changes in 300 seconds. Saving...
immich_redis_1             | 1:M 22 Mar 2024 18:25:12.096 * Background saving started by pid 19
immich_redis_1             | 19:C 22 Mar 2024 18:25:12.120 * DB saved on disk
immich_redis_1             | 19:C 22 Mar 2024 18:25:12.120 * RDB: 0 MB of memory used by copy-on-write
immich_redis_1             | 1:M 22 Mar 2024 18:25:12.196 * Background saving terminated with success
immich_machine-learning_1  | [03/22/24 18:25:17] DEBUG    Checking for inactivity...                         
immich_machine-learning_1  | [03/22/24 18:25:27] DEBUG    Checking for inactivity...                         
immich_machine-learning_1  | [03/22/24 18:25:37] DEBUG    Checking for inactivity...                         
immich_machine-learning_1  | [03/22/24 18:25:47] DEBUG    Checking for inactivity...                         
immich_machine-learning_1  | [03/22/24 18:25:57] DEBUG    Checking for inactivity...                         
immich_machine-learning_1  | [03/22/24 18:26:07] DEBUG    Checking for inactivity...                         
immich_machine-learning_1  | [03/22/24 18:26:17] DEBUG    Checking for inactivity...                         
immich_machine-learning_1  | [03/22/24 18:26:27] DEBUG    Checking for inactivity...                         
immich_machine-learning_1  | [03/22/24 18:26:37] DEBUG    Checking for inactivity...  

[alextran1502]

Just to make sure, can you drag-and-drop that file on the web to see if it successfully upload or it will say duplicated

[lmgithu]

I tried, it said "skipped 1 duplicated asset"

[alextran1502]

So it should be on the server, could it be in the archive view somehow?

[lmgithu]

No, it is not there in the archive :(

[alextran1502]

Can you confirm the visual of the image on that date,not just the file name?

[lmgithu]

Yes I can, I can't see this image visually on the UI, despite it is on the file system.

[alextran1502]

hmm strange, can you show me the search query you put in to search for the file name?

[lmgithu]

here you are:

[lmgithu]

this works for any other filename btw and I can see the results

[alextran1502]

Try 5841? and did you tick on the archive button at the very bottom of the page?

[lmgithu]

Do you know why certain folders/files were ignored? this is how my folders look like:

Interestingly, until 2010 every folder looks OK... after that, it is fully random and mixed.

[alextran1502]

I am not sure, we will have to look at this more in-depth. Thanks for the report

[lmgithu]

Could I somehow force process folders again?

[lmgithu]

or at least "delete" these files somehow so i could reupload them?

[alextran1502]

Can you check the Trash page to see if it is in it?

[lmgithu]

WIN, it was in the Trash! Sorry, this was such a noob issue. Now we "just" have the folder structure issue that it is fully random, I think there are no missing images. Do you have any guess what might be the issue and how to fix it?

[alextran1502]

Do you have any guess what might be the issue and how to fix it?

The issue is that we don't move files that are already in the trash :P So it is working as expect

[lmgithu]

no, it was a SINGLE image... not all of them.

[lmgithu]

99% of the files that were not moved are NOT in the Trash

[alextran1502]

Then we will need to circle back and look at this!

[lmgithu]

Thank you! Is there any way to "force" the migration again?