immich
immich copied to clipboard
immich-app
[BUG] Downloading image to mobile device that has been uploaded with CLI results in duplicates.
The bug
Hi there,
maybe not the most common workflow, but never the less would be great to fix.
Repro:
a) Fresh install of Immich b) Upload tons of pictures to the server using CLI tool (works great). c) Install mobile app (here iOS) d) Browse photo selection and download one to mobile device (say you want to use it as background image, etc). e) Observe that mobile app will instantly detect this image and re-uploads it as a duplicate to the server.
Upon closer inspection what I can see if that the filename endings differ:
Orig.: IMG_6502.jpg Dupe: IMG_6502.jpg.JPG
On the plus: Deleting the duplicate will only delete the duplicate and not both. 👍
Would be a great fix, if easy.
The OS that Immich Server is running on
Truenas Scale
Version of Immich Server
v1.51.0
Version of Immich Mobile App
v1.51.0
Platform with the issue
- [X] Server
- [ ] Web
- [X] Mobile
Your docker-compose.yml content
Stock.
Your .env content
Stock.
Reproduction steps
See above.
Additional information
No response
mobile app will instantly detect this image and re-uploads it as a duplicate to the server.
Does the image actually show up duplicated in the (web) UI after this?
Also, let's say for simplicity, the iOS photo roll is empty when doing c) from the repro steps.
That's very strange, since we have hash-based deduplication so it should be detected that it's the same file. Is there a chance that it's being picked up by iCloud photos or such and being compressed or otherwise altered?
Don't think so, it happens right away. Also I have iClouds photos disabled.
Is there maybe a difference between how the hashes are computed for CLI uploads and mobile assets?
Does the file name go into the hash? Maybe downloading an image modifies it.
I can confirm, I have a similar issue.
Steps to reproduce:
- Download images to PC and upload them to the web with drag-n-drop
- Download the same images from the same source to iOS and add them to the library
- iOS app uploads the images again and there are duplicates in the Immich library
I did notice that GPhotos also has the same issue, so I wonder if the hash or something else about the images actually does change. 🤔 I also don't use iCloud if that can impact anything.
This issue is related to IOS and how it handles pictures. By default IOS will try and optimise the images on your device to save storage and uploading the original to iCloud. So, when you download the image from immich onto your device. IOS will try and optmise it, and because the file has changed. Immich will pick this up as a different photo. I have tested this, and the files are usually different sizes no matter how many times I download and it gets reuploaded.
I found this while searching
" someone on Reddit said the problem is from the iOS end of things. On the iphone, under Settings, Photos, you need to turn off "optimise iphone storage" and switch instead to "download and keep originals". The first option means that if your phone is low on space, it replaces full-resolution photos with smaller versions and keeps the full sized version in iCloud. I just did this and the problem appears to be solved. " Source
I have tried this. On my side already uploaded images seems to keep being reuploaded. Newer images are working properly and you don't get duplicates
As this is an IOS issue, I believe it will be appropriate to have some documentation around this so people understand. Nothing immich can do besides having to extract the metadata and comparing everything @bo0tzz
@Dilanwizzy isn't that the case only if you have iCloud photos enabled? I do not upload photos to iCloud so I don't have that option available, but I still have the the issue.
@klejejs The uploaded pictures have different sizes. I have just done what you explained and each time the size is different. The size difference is usually by 1 or 2 kb. Here is a file continuously downloaded.
3,292,209 3,291,980 3,291,298 3,292,438
File size each time is different. I have also seen it be the same and the image does not get uploaded. I don't see this with a file taken on my iPhone and downloaded. This happens only when the file is downloaded from a different source and then you save the image.
The issue is 95% looking like its all related to IOS, and how they handle images. I am not seeing this on Android
Instead of looking at sizes it would be best to do a sha1 hash.
You should be able to verify the hash at each step and it should be the same at each step. Immich should not modify it at all, ever.
- Take a picture, verify sha1 hash
- Upload the picture to immich, verify sha1 of file in upload directory
- Download the picture using "download" feature in immich web, verify sha1 of file is the same
- Download the picture from the mobile app, verify the sha1 of the file is the same
- Upload the picture again, verify it's a duplicate and not processed
If the sha1 changes at any step, that would be a problem. It might be related to iOS cloud stuff though.
I used sha1 hash to verify and the file each time the hash is different. The different sizes show that maybe IOS is doing something with the files.
The same issue also happens on Google Photos. I am not getting this issue on Android as mentioned
Do you have some files you can upload that I can inspect? Ideally, an original and then a download-and-modified version.
So, it seems like the way google handles this is by appending the word original to the file. And then treating the file on the server and the one downloaded as 1 file. Currently, immich would treat them as 2 separate images, with one being uploaded on the server and the other showing as not uploaded.
Yeah, also quick note that I have iCloud disabled and thus the option mentioned is not available for me.
Can somebody share/upload a sample image (before and after) that I can investigate with? Otherwise, it is hard to know what is going on.
Here is a zip with a picture uploaded via CLI on Feb/23. I just downloaded onto my phone on Mar/22 and it got uploaded right away as a duplicate. I downloaded them both using the web gui. Note that the zip contains two files with the same name.
HTH and thanks for taking a peek!
I'm seeing similar behavior on Android. My wife (also Android) and I share a user account so that we can see each other's pictures. If I download an image to my device that my wife took, then I end up with a duplicate photo. And the original photo still shows up as not on my device.
Thank you all for the report, we are hoping to address this issue in the next release or so
I set up Immich couple days ago and I’m facing the same issue with my iPhone
@florihupf You reported your server and app versions as v1.61.1, which isn't a valid release. The current release is v1.61.0. Would you mind checking both are up to date?
@florihupf these must be different files (maybe iOS does not give access to the original file or a bit error occurred). There can only be exactly 1 asset with the same checksum per user. Please check whether the files have the same checksum (enable advanced error reporting in Immich app settings, show info of the two duplicates)
I can repro with more detail.
How do I enable advanced error reporting? In the iOS Immich app, I see Log Level: Info and a trouble shooting toogle. What settings should apply?
As for the repro steps:
a) Just click download on any picture that is not downloaded to the phone yet (but uploaded via CLI to the server). b) Upon completion of the download, there will be two images in the mobile app (identical looks) and the second image will be uploaded to the server, creating a duplicate there as well (after a delay).
HTH
Enable trouble shooting. Then when clicking the info icon in the full screen view, you see advanced/raw info about an asset.
I bet the checksums are different. But I don't know yet why.
One solution for the slightly adjusted image to detect as duplicate could be to use phash instead of a another hash algorithm. Source: https://www.phash.org
