duplicate file ending of movie files in iOS live pictures

[pauleseifert]

The bug

Hi everyone,

I'm back on a fresh install on the newest version of immich. There seems to be a small bug in double assignment of the file extensions in iOS .heic pictures. They consist of a jpeg with the .heic ending and a separate movie file with a .mov ending. When running the storage migration job, this leads to the movie file being named IMG_5196.MOV.mov instead of IMG_5196.MOV.

Other video files are fine.

[Nest] 7  - 05/11/2024, 3:01:16 PM     LOG [MediaService] Attempting to finish incomplete move: upload/upload/5cfed9ba-bee6-4021-9e32-21fe2ba98334/ef/f0/eff00ad2-cf8c-4bdf-9212-4a2583e3c754.mov => upload/library/admin/2024/May/IMG_5196.MOV.mov
[Nest] 7  - 05/11/2024, 3:01:16 PM     LOG [MediaService] Found file at old location
[Nest] 7  - 05/11/2024, 3:01:16 PM   DEBUG [MediaService] Attempting to rename file: upload/upload/5cfed9ba-bee6-4021-9e32-21fe2ba98334/ef/f0/eff00ad2-cf8c-4bdf-9212-4a2583e3c754.mov => upload/library/admin/2024/May/IMG_5196.MOV.mov
[Nest] 7  - 05/11/2024, 3:01:16 PM   DEBUG [MediaService] Unable to rename file. Falling back to copy, verify and delete
[Nest] 7  - 05/11/2024, 3:01:16 PM   DEBUG [MediaService] File size check: 2663722 === 2663722
[Nest] 7  - 05/11/2024, 3:01:16 PM   DEBUG [MediaService] File checksum check: QIgNalqElVniov+2816i1ncBg7o= === QIgNalqElVniov+2816i1ncBg7o=
[Nest] 7  - 05/11/2024, 3:01:16 PM     LOG [MediaService] Attempting to finish incomplete move: upload/upload/5cfed9ba-bee6-4021-9e32-21fe2ba98334/30/ec/30ec6c97-7050-4ed8-b6a7-8a6db2876650.HEIC => upload/library/admin/2024/May/IMG_5196.HEIC
[Nest] 7  - 05/11/2024, 3:01:16 PM     LOG [MediaService] Found file at old location
[Nest] 7  - 05/11/2024, 3:01:16 PM   DEBUG [MediaService] Attempting to rename file: upload/upload/5cfed9ba-bee6-4021-9e32-21fe2ba98334/30/ec/30ec6c97-7050-4ed8-b6a7-8a6db2876650.HEIC => upload/library/admin/2024/May/IMG_5196.HEIC

My storage template is as follows:

The OS that Immich Server is running on

docker

Version of Immich Server

v1.103.1

Version of Immich Mobile App

v1.103

Platform with the issue

• [X] Server
• [ ] Web
• [ ] Mobile

Your docker-compose.yml content

not relevant

Your .env content

not relevant

Reproduction steps

1. upload .heic file
2. run storage migration job

Relevant log output

No response

Additional information

No response

[akash329d]

I just noticed this, did you ever find a resolution? Think I messed up my library a bit now by running that job.

[jrasm91]

I just tested this an the issue no longer exists. I am not 100% if there is "old data" that is incorrect though. Let me know if you still see this problem after running the storage migration again and we can figure out a way to remedy the issue for previous instances of it.

[pauleseifert]

Can confirm that it stopped happening from mid June this year onwards. Any ideas on how to clean up the library? I can run a regex job over the file system but I assume that messes up the database?

[jrasm91]

Can you confirm if asset in the database (asset table, original file name column) has the right value? If it does you can just re-run the storage template, otherwise we can write a SQL migration for next release to find/replace them.

[pauleseifert]

Databaset entries for originalFileName look good. Ran re storage template job but nothing changed. I haven't followed to change logs so closely recently but this isn't expected behaviour in the administrator panel, isn't it? Shouldn't there be an option to modify or activate the storage template?

[jrasm91]

That looks quite strange. Can you try another browser, refresh the page, or look at the developer console?

[pauleseifert]

NM. Storage Template was a content security policy problem. The storage template looks fine to.

Lets pick the one example from above. originalFileName: IMG_5196.HEIC IMG_5196.MOV

originalPath: upload/library/admin/2024/May/IMG_5196.HEIC upload/library/admin/2024/May/IMG_5196.MOV.mov

Are there some more fields of relevance? I currently don't see why the template migration job doesn't clear that up.

[jrasm91]

I assumed that would have fixed it, especially if there were no errors logged in the server related to running the job.

[pauleseifert]

Well rejoiced too soon. After the migration job over night with the now active storage template, it looks the same on the file system. I even changed it back and fourth to another folder structure. No errors from the server container. How are file extensions saved in the database? What also bothers me is the capitalisation: the additional endings are written in lower case without exception.

[jrasm91]

So the problem still persists? I will have to try to reproduce this.

[pauleseifert]

It does. Let me know if I can help you with test or database entries.

[jrasm91]

I just tested with a live photo. My motion path was <path>.MOV.mov and the image at <path>.heic. After I ran the storage template migration, the motion had been changed to <path>.mov as it should. Can you confirm that you ran the job that's second from the bottom called "STORAGE TEMPLATE MIGRATION"?

[jrasm91]

The only thing that might impact this is the originalFileName in the database being wrong, ie having .MOV.mov for the motion asset, but you verified that is not the case. Can you double check that you do not see any assets with a double extension in the originalFileName column?

[pauleseifert]

I can confirm that I run the job. I get some errors from one of the users accounts for unfinished uploads, but nothing related to my user account. Storage template job finishes.

[Nest] 6  - 09/28/2024, 9:02:17 PM     LOG [Microservices:StorageTemplateService] Starting storage template migration
[Nest] 6  - 09/28/2024, 9:02:37 PM     LOG [Microservices:MediaService] Attempting to finish incomplete move: upload/upload/d5177981-80f7-4655-bb2b-d41343c50d9b/37/54/375499ea-a8db-49f9-9ba2-e13e59eb9a7c.mov => upload/library/*redacted*/IMG_4593.MOV.mov
[Nest] 6  - 09/28/2024, 9:02:37 PM    WARN [Microservices:MediaService] Unable to complete move. File does not exist at either location.
[Nest] 6  - 09/28/2024, 9:02:39 PM     LOG [Microservices:StorageTemplateService] Finished storage template migration

I checked the database again with: SELECT "originalFileName" , "type", "originalPath" FROM assets ORDER BY "localDateTime" DESC LIMIT 1500;

This results in the following data base entries:

IMG_5196.HEIC | IMAGE | upload/library/admin/2024/May/IMG_5196.HEIC IMG_5196.MOV | VIDEO | upload/library/admin/2024/May/IMG_5196.MOV.mov

So to me the entry looks fine but the path is still wrong. I just realised that all relevant wrong files were uploaded by the mobile app and at least the file above had at some point an error for an unfinished upload (but it is complete on the drive). Is there a database entry for that to be checked?