feat(web): synchronise metadata and album membership between duplicate images

[EinToni]

trafficstars

Description

This is my first time contributing here and I hope it is ok to open a PR without creating an issue. This PR arises from the feature request #10262 and in part from #10206. In both more information is wished as well as the possibility to synchronize metadata between the duplicated images.

PR Content

This PR introduces the ability to automatically:

• Synchronize the albums of duplicated images
  • Meaning: The deduplicated images are all put in all albums any duplicate was part of.
• Synchronize the favorites
  • Meaning: If any of the duplicates was marked as favorite, the deduplicated images are also marked as favorites.
• Synchronize the archive
  • Meaning: If any of the duplicates was archived, also archive the deduplicated images.

These three settings can be individually toggled:

I also added an indicator for the archive status of any image, same as the favorites icon:

And lastly I added the possibility to manually select the date/time, description and location to be synched to the deduplicated kept images. I am not that happy with this solution, but these are all fields that could completely differ from each other so no automatic resolution seemed fitting here. In this example the remaining two images would be marked as favorites, marked as archived, have the location set to Paris, the description to "thisis some test" and the timestamp to 9. Jan. 2023 Mo, 07:15.

Open work

The size of the showed duplicates can now vary and therefore not align with each other. This looks very bad, but I didn't find any nice way to fix this (it's my first time using svelte and I'm not doing web development too often).

[bo0tzz]

Cool stuff, thank you!

The UI is a little unclear to me at the moment, I'm hoping @alextran1502 has some ideas there. My suggestion would be to separately display another card showing what the deduplicated end result will look like. Just a super quick and dirty browser devtools mockup:

[EinToni]

Thank you for the suggestion! I like it and it looks great in your example. But where should we put the "preview" card when there are more duplicates (like in my initial example)? The best would probably be to put it below the rest, but I don't know how clean the look then would be.

Also after looking at this again after almost a week, the red and green highlights really clutter up the UI. My idea was to highlight the metadata in red that gets "lost", as it will be overwritten by other data. But I would say removing the red highlighting in favor for proper lineup of the fields will help the readability quite a lot.

I will give the lineup of the data another try and remove the red highlights. Afterwards I can put in your preview card to see how it feels in action.

[EinToni]

I decided against a separate preview asset because I saw an example on my regular immich instance, where some duplicates are looking visually slightly different. There I want to keep e.g. 2 of 5 images. Showing this in a single preview would only confuse as I would have to pick some image to diplay.

I therefore opted to display the preview directly in the selected assets (and only in the selected ones). Previously I highlighted all metadata that is different from the selected one. Now only the selected assets reflect the changes that will be done. In my opinion this has improved the usability a lot. I also adapted the shape of the highlight to match the rest of the UI.

Usability feels better: https://github.com/user-attachments/assets/c5b02bdb-73e8-4d9f-9e89-7f109552fb07

I would be happy about further feedback :)

[bo0tzz]

Good stuff!

There I want to keep e.g. 2 of 5 images. Showing this in a single preview would only confuse as I would have to pick some image to diplay.

What about displaying both?

[EinToni]

Displaying every selected asset is another "preview" card would clutter up the UI quite quickly. And when selecting/deselecting some duplicates the previews would probably dynamically pop up and vanish, dependent on the amount of selected duplicates. This could lead to a lot of confusion and also distinguishing between the previews to tell which preview relates to which duplicate could then become harder than necessary. Therefore I would say a direct preview of the changes in the already existing displays is better.

[mertalev]

Wow, awesome stuff! My thought is that the syncing for albums, favorites, etc. is relatively straightforward while metadata is harder to display and edit in an intuitive way. What do you say about handling the former in this PR and the latter in a separate PR?

[EinToni]

Thank you! We can of course split it up, but as the backend behavior is pretty equal, the only real difference is the UI. If we can come to an agreement for the UI, I can also just do it all in this PR.

But if that agreement sounds unrealistic any time soon I'll split it. I would leave that decision up to the immich team.

[NicholasFlamy]

I decided against a separate preview asset because I saw an example on my regular immich instance, where some duplicates are looking visually slightly different. There I want to keep e.g. 2 of 5 images. Showing this in a single preview would only confuse as I would have to pick some image to diplay.

I therefore opted to display the preview directly in the selected assets (and only in the selected ones). Previously I highlighted all metadata that is different from the selected one. Now only the selected assets reflect the changes that will be done. In my opinion this has improved the usability a lot. I also adapted the shape of the highlight to match the rest of the UI. [image]

Usability feels better: https://github.com/user-attachments/assets/c5b02bdb-73e8-4d9f-9e89-7f109552fb07

I would be happy about further feedback :)

I will comment, the text is touching the edges of the box in some places, if you could put at least like a 2px margin/padding that would probably do it. (Not too much ofc, we still need content, but just enough that it doesn't touch.)

[EinToni]

@bo0tzz What will be the further process to continue with this MR?

[alextran1502]

@EinToni hey, sorry for leaving this open. I am currently working on refactoring the code base to the new Svelte 5 syntax. Once that is done I will come back to this

[EinToni]

@alextran1502 I rebased onto main and adapted my code for svelte 5. I would be happy to hear your feedback about the feature when you find time for it.

[Gorbush]

A with to also sync:

1. Tags of images
2. Star rating of images

[EinToni]

Hey, after not doing anything for a month I just rebased onto main (214 commits!! crazy fast development). Because I don't see this moving forward in a state like this I decided to do what @mertalev already suggested in the beginning. I split it up into the basic synchronization of favorite/archive/album and the more complex UI part. I will move the UI part into its own branch (unless someone contradicts), so this pull request here will only be for the basic automatic sync part.

I hope with this smaller scope and simpler functionality it will be possible to continue and maybe even merge this in the near future.

Edit: If someone searches the original code for the UI-metadata selection, it lives here now: https://github.com/EinToni/immich/tree/deduplicate-UI-metadata

[Phlogi]

Does this PR also sync metadata when mass-merging is happening through the action "Remove all duplicates"?

[EinToni]

@Phlogi No it does not, currently this is only for the manual deduplication.

[Phlogi]

@Phlogi No it does not, currently this is only for the manual deduplication.

Thanks for clarifying, that would be a killer feature. The mass de-deplucation is almost useless without this.

Is there a tagged docker image with your changes or would I need to build this to help testing? I don't think its built automatically.

[NicholasFlamy]

@Phlogi No it does not, currently this is only for the manual deduplication.

Thanks for clarifying, that would be a killer feature. The mass de-deplucation is almost useless without this.

Is there a tagged docker image with your changes or would I need to build this to help testing? I don't think its built automatically.

Follow these instructions and for the clone the repo do git clone https://github.com/EinToni/immich -b deduplicate-sync-album https://immich.app/docs/developer/setup#server-and-web-apps

[WolfgangDpunkt]

I would be happy if this essential feature will be available soon in Immich. As already mentioned, without the synchronization of the meta data, the mass deduplication is actually not usefully applicable. Thank you!

[NicholasFlamy]

without the synchronization of the meta data, the mass deduplication is actually not usefully applicable.

I think there was an issue with mass dedupe recently, so that may be one of the obstacles before this can be approved.

[EinToni]

I would happy for someone to add the feature in for the mass deduplication!

@mertalev suggested to perform the metadata sync in the mass deduplication in the backend with DB queries. Unfortunately I am not proficient with databases, that's why I did not add it in.

[rui-nar]

just wanted to drop a message saying that this feature is awesome and clearly mandatory. I have more than 2k duplicates with exactly this need.

@EinToni if you provide a fork of immich with an associated docker (or instructions on how to create one) i'll gladly use it :)

[WolfgangDpunkt]

I can only agree with that. Many users use Immich to merge their various photo collections. You upload photos from different sources and inevitably there are many duplicates. However, if the metadata is destroyed or lost when cleaning up the duplicates, then this is not practical.

Therefore, I can only express my gratitude that this feature is being worked on.

[EinToni]

Just rebased onto main, because there have been a lot new commits :D

@mertalev You suggested to do the mass deduplication on the server side and I fully agree, no need to hand all of the data back and forth for fully automated things. But I think this would be quite a stretch for this pull request. I would propose to open a separate issue for the backend mass dedupe with the sync of the metadata.

In the meantime the pull request got cut down to only merge the status of being archived/favorite and the album membership. Also here I would say extending this to more metadata could be done if future issues, or are single big pull requests preferred in immich?

If you don't mind, it would be great to also hear your thought about this @bo0tzz @alextran1502

[alextran1502]

Hello, thank you for rebasing and continuing the work here. I read the options and don't really have a clear understanding of what it does. So the copywriting will need some modification

Can you provide an example explanation for each option?

[EinToni]

Hello, thank you for rebasing and continuing the work here. I read the options and don't really have a clear understanding of what it does. So the copywriting will need some modification

Can you provide an example explanation for each option?

Hey, thanks for the review. Unfortunately there was quite a bit of back and forth, therefore I get that it is confusing right now. Regarding the three options:

• Sync Albums: The selected deduplicated image will be put in all albums where the duplicated were.
  • If image 1 is in album A, image 2 in album B and image 3 is selected as the image to keep, it will be put in album A and B
• Sync Favorites: If any duplicate was marked as a favorite, the selected deduplicated image will be also marked as a favorite.
  • If image 1 is marked as favorite and image 2 is selected as the image to keep, it will be marked as a favorite.
• Sync Archive: If any duplicate was marked as archived, the selected deduplicated image will also be marked as archived.
  • If image 1 is marked as archived and image 2 is selected as the image to keep, it will be marked as archived.

Syncing more metadata would be great when deduplicating, but I guess that needs more discussion in terms of the implementation.

[Spacey-Rat]

Looks great - would love to see this implemented. This missing feature to keep/copy missing album data keeps me away from de-dup'ing 15K images.

[WolfgangDpunkt]

I wanted to emphasise the importance of continuing the great work on this issue. It's crucial for Immich's usefulness. :-)

[cookie-monster1649]

Sad to see this feature falling off the amazing work this project is doing. Please keep this particular one going!

[kacboy]

Would love to see this feature! I imported my Google Photos that have been compressed but would be very handy to replace existing albums with the original quality pics via this feature :)

[WolfgangDpunkt]

The problem seems clear: you have painstakingly sorted your images, then find yourself some of the photos in better quality, and now you don't want to upload and replace the poorer image quality with better quality images. The function should be quite simple: does one of the duplicates already have an album assignment? Then keep this for the new version of the image. If at least this basic function worked, most people would be happy. All other complicated features can be added later, optionally.