[Feature]: Delete Duplicated Media

[swimminsparrow]

Feature detail

A tool for managing and deleting duplicated content could be useful, checking the hash or the filename?

Platform

Web

[panoti]

Hi @swimminsparrow, we've already implemented duplicate identifier with sha1 checksum. A tool for managing and deleting duplicated content is also our further work.

[swimminsparrow]

Hi have this situation:

1. Logged in with a device and sync a folder
2. Logout
3. Logged in with same device and same user and sync same folder
4. A new dir under immich-data has been created and all the previous media have been resynced. The application shows both the "old" content and the new. Even if they are the same content

[alextran1502]

Hi have this situation:

1. Logged in with a device and sync a folder
2. Logout
3. Logged in with same device and same user and sync same folder
4. A new dir under immich-data has been created and all the previous media have been resynced. The application shows both the "old" content and the new. Even if they are the same content

This sounds like a bug. Where did you get the app?

Can you provide more information on your setup and device info?

[swimminsparrow]

Android version: 1.28.0 build.38 Server version: 1.26.0 Server OS: 2022-09-06-raspios-bullseye-arm64

[panoti]

@swimminsparrow your backend version is really old. You'd better upgrade to the latest version 1.28

[alextran1502]

Android version: 1.28.0 build.38 Server version: 1.26.0 Server OS: 2022-09-06-raspios-bullseye-arm64

The dedup feature has been available since 1.27. We are going with the convention that the mobile app and the server must be on the same version to ensure compatibility.

[swimminsparrow]

Hi @alextran1502 @panoti how can i upgrade the server version? thanks

[alextran1502]

You can do docker-compose pull then docker-compose up

[swimminsparrow]

Ok thank you. I suppose the feature does not work retroactively so i should clean the current situation with duplicates and make further check for the new uploads

[alextran1502]

FYI, new upload won't run into problem with duplication since we have the checking mechanism in place. The only edge case is the two identical files with two version, one is compressed and one is original, which will cause the duplication.

[joeShuff]

I'm also seeing some duplicate data after an initial upload via googlephotos takeout, then syncing my device photos. Guessing thats likely the edge case you speak of @alextran1502?

[alextran1502]

I'm also seeing some duplicate data after an initial upload via googlephotos takeout, then syncing my device photos. Guessing thats likely the edge case you speak of @alextran1502?

@joeShuff if the file is compressed then we will see the duplication since the hash value is different with the uncompressed file

[Robin-Sch]

In the android app it shows duplicated files, on the web app it doesn't, after logging out and in again and re-uploading.

For example, if I go to the date 26 nov on the android app, it shows 6 pictures, 3 unique and 3 duplicated. If I go to the date 26 nov on my computer, it shows 3 picutres, 3 unique and 0 duplicated.

After closing and re-opening the app this got fixed, but if I didn't close the app it stayed "duplicated".

[jrasm91]

It's worth noting too, that the sha1 hash is only unique per user.

[Robin-Sch]

In the android app it shows duplicated files, on the web app it doesn't, after logging out and in again and re-uploading.

For example, if I go to the date 26 nov on the android app, it shows 6 pictures, 3 unique and 3 duplicated. If I go to the date 26 nov on my computer, it shows 3 picutres, 3 unique and 0 duplicated.

After closing and re-opening the app this got fixed, but if I didn't close the app it stayed "duplicated".

Also I just noticed another problem, the "Backup up photo's and video's" counter is 0 (because the photo's and video's are already uploaded ig?)

[alextran1502]

In the android app it shows duplicated files, on the web app it doesn't, after logging out and in again and re-uploading.

For example, if I go to the date 26 nov on the android app, it shows 6 pictures, 3 unique and 3 duplicated. If I go to the date 26 nov on my computer, it shows 3 picutres, 3 unique and 0 duplicated.

After closing and re-opening the app this got fixed, but if I didn't close the app it stayed "duplicated".

This might be the state glitch when uploading and updating the timeline. I noticed that sometime the websocket doesn't close properly lead to two uploading events triggered and showing duplication. If closing and reopening the app fix it, this is just cosmetic bug and we will get to it sometimes in the future

[alextran1502]

In the android app it shows duplicated files, on the web app it doesn't, after logging out and in again and re-uploading.

For example, if I go to the date 26 nov on the android app, it shows 6 pictures, 3 unique and 3 duplicated. If I go to the date 26 nov on my computer, it shows 3 picutres, 3 unique and 0 duplicated.

After closing and re-opening the app this got fixed, but if I didn't close the app it stayed "duplicated".

Also I just noticed another problem, the "Backup up photo's and video's" counter is 0 (because the photo's and video's are already uploaded ig?)

If this is not correct, please open a separate issue and provide the steps you have taken to arrive at the problem. I am not sure which operations were done and what is the expectation outcome from your brief report