[BUG] Immich is duplicating files and folder structure when I enable a custom Storage template

[schizovivek]

The bug

After setting a custom storage template {{y}}/{{MM}}-{{MMMM}}/{{y}}-{{MM}}-{{dd}}/{{filename}} I can see two folder structures created and all the files duplicated within the two folders

Eg: for images from jan of 2007 I can see library/admin/2007/2007-01-01 and library/admin/2007/01-January/2007-01-01

The OS that Immich Server is running on

Ubuntu 23.04

Version of Immich Server

v1.114.0

Version of Immich Mobile App

1.114.0

Platform with the issue

• [X] Server
• [ ] Web
• [ ] Mobile

Your docker-compose.yml content

Doesn't seem to be related to compose

Your .env content

Not related to env

Reproduction steps

1. Enable Storage template after importing a bunch of media into server
2. Set a custom storage template {{y}}/{{MM}}-{{MMMM}}/{{y}}-{{MM}}-{{dd}}/{{filename}}
3. Run Storage Migration job ...

Relevant log output

No response

Additional information

[Biepa]

I may have a similar issue: My storage template: {{y}}/{{y}}-{{MM}}-{{dd}}/{{HH}}-{{mm}}-{{ss}} Preview: UPLOAD_LOCATION/admin/2022/2022-02-03/04-56-05jpg

And this is what I got: There are a lot of it and the few couples I checked are having the same hash.

EDIT: I also have duplicate files in the "thumbs" folder

[alextran1502]

Hi @danieldietzler, would you like to take a look at this issue since you are going through some slashes bug

[bo0tzz]

Can you reproduce this on a new instance? Please share the full logs after running the migration job.

[schizovivek]

I was hoping it wouldn't come to this :-D .. Give me sometime.. I'll create a new instance and try replicating it with a small subset of the media.

Couple questions:

• How do i specifically check migration logs? Or are you asking for the immich-server logs when I run the migration job?
• Is there a way to check if the existing logs would have some info? I just tried search for the keyword migration in my current server logs but just see started and completed lines

[danieldietzler]

(1) correct, just the immich server logs while running the migration :) (2) Probably, but it's much easier to have "clean" logs on a minimal instance tbh.

[schizovivek]

Tried with a really smart subset of my media. Unable to replicate it

My question then would be; let's say there was some error at my end that could've caused it; why isn't it showing up anywhere in the UI? I'd have thought it would show up as duplicate images in the UI

I tried looking at the duplicates section and although it's saying there are 205 duplicates there is no way for me to scroll through one at a time to view the duplicates. Wish we could see all at one go.

That being said; what would you suggest I do next? Start from scratch?

[C-Otto]

Is it possible that you ran the migration job twice in parallel (or that the first instance took longer than 24 hours)?

[schizovivek]

Yeh.. That could've happened. When i enabled it, i didn't change the default profile immediately.. Might've changed it later. The run shouldn't have taken more than 24 hours since I haven't loaded my full library into Immich yet. Have just about 10k images and 500+ videos. A total of 121 GB for me and my partner but as you can see with the library folder usage; this is the stat

/mnt/pool/personal/immich_library_test/library/admin❯ du -sh .
347G    .

I'll try setting up a completely new instance and reload everything back into it. Just a pain to do honestly. This is why I was still playing around. Have been a bit busy with work so haven't been able to set things up again. Hopefully tomorrow I can set it up and reload everything back into a new immich instance. I'll try to reload it in batches with the storage template enabled and keep an eye on the file system.

My main concern is even if I was running things parallely it should've at least informed me something was running or queued the 2nd run after the first. The fact that my entire library was duplicated is concerning.

[qu1j0t3buendia]

That happened to me too. I tried to run the storage template migration (from a date folder structure to an album/filename structure) and I got some errors saying that the exif was missing and the file would not be migrated. Then, I stopped the task, started a metadata extraction for all, and restarted the storage migration. Now I have several duplicate files outside the album folder. Those duplicates are slightly different in size. Also, a lot of .mov and .heic files are outside of the album folder and the same picture as .jpeg is inside the album folder.

I don't even know where to start to fix this. Any help would be appreciated.

[schizovivek]

That happened to me too. I tried to run the storage template migration (from a date folder structure to an album/filename structure) and I got some errors saying that the exif was missing and the file would not be migrated. Then, I stopped the task, started a metadata extraction for all, and restarted the storage migration. Now I have several duplicate files outside the album folder. Those duplicates are slightly different in size. Also, a lot of .mov and .heic files are outside of the album folder and the same picture as .jpeg is inside the album folder.

I don't even know where to start to fix this. Any help would be appreciated.

for me the duplicates are exactly the same size. if your new template is enabled best bet would be to ssh into your machine and delete the duplicate files manually I guess. For me since this was a test implementation I have the originals separately.

[schizovivek]

Closing for the time being since with my new instance I'm not seeing any duplication with storage template enabled.