immich [BUG] The photo appears twice, there are two identical photos

The bug

There are two identical photos in the album One of the tips This asset is offline. Immich can not access its file location. Please ensure the asset is available and then rescan the library. I rescanned the gallery according to the prompts, but it still exists, and there is no delete button in the web management interface.

It feels like the gallery was scanned but the database was not cleaned.

In addition, even if I delete the photo directly from the folder, the photo will still exist.

The OS that Immich Server is running on

QNAP NAS DOCKER

Version of Immich Server

v1.82.1

Version of Immich Mobile App

no

Platform with the issue

[X] Server
[ ] Web
[ ] Mobile

Your docker-compose.yml content

version: "3.8"

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    command: ["start.sh", "immich"]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
      - /share/猫的仓库/照片:/mnt/media
    env_file:
      - .env
    depends_on:
      - redis
      - database
      - typesense
    restart: always

  immich-microservices:
    container_name: immich_microservices
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    # extends:
    #   file: hwaccel.yml
    #   service: hwaccel
    command: ["start.sh", "microservices"]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
      - /share/猫的仓库/照片:/mnt/media
      
    env_file:
      - .env
    depends_on:
      - redis
      - database
      - typesense
    restart: always

  immich-machine-learning:
    container_name: immich_machine_learning
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    volumes:
      - model-cache:/cache
    env_file:
      - .env
    restart: always

  immich-web:
    container_name: immich_web
    image: ghcr.io/immich-app/immich-web:${IMMICH_VERSION:-release}
    env_file:
      - .env
    restart: always

  typesense:
    container_name: immich_typesense
    image: typesense/typesense:0.24.1@sha256:9bcff2b829f12074426ca044b56160ca9d777a0c488303469143dd9f8259d4dd
    environment:
      - TYPESENSE_API_KEY=${TYPESENSE_API_KEY}
      - TYPESENSE_DATA_DIR=/data
      # remove this to get debug messages
      - GLOG_minloglevel=1
    volumes:
      - tsdata:/data
    restart: always

  redis:
    container_name: immich_redis
    image: redis:6.2-alpine@sha256:70a7a5b641117670beae0d80658430853896b5ef269ccf00d1827427e3263fa3
    restart: always

  database:
    container_name: immich_postgres
    image: postgres:14-alpine@sha256:28407a9961e76f2d285dc6991e8e48893503cc3836a4755bbc2d40bcc272a441
    env_file:
      - .env
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
    volumes:
      - pgdata:/var/lib/postgresql/data
    restart: always

  immich-proxy:
    container_name: immich_proxy
    image: ghcr.io/immich-app/immich-proxy:${IMMICH_VERSION:-release}
    environment:
      # Make sure these values get passed through from the env file
      - IMMICH_SERVER_URL
      - IMMICH_WEB_URL
    ports:
      - 2283:8080
    depends_on:
      - immich-server
      - immich-web
    restart: always

volumes:
  pgdata:
  model-cache:
  tsdata:

Your .env content

[/share/Container/immich] # cat .env 
# You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables

# The location where your uploaded files are stored
UPLOAD_LOCATION=/share/Container/immich

# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=release

# Connection secrets for postgres and typesense. You should change these to random passwords
TYPESENSE_API_KEY=some-random-text
DB_PASSWORD=postgres

# The values below this line do not need to be changed
###################################################################################
DB_HOSTNAME=immich_postgres
DB_USERNAME=postgres
DB_DATABASE_NAME=immich

Reproduction steps

Open the album and that's it

Additional information

No response

Nov 09 '23 07:11 MilkTeaNo7

This is happening for me as well. I believe the cause might have been removing an external library path in the middle of a scan.

Mar 25 '24 18:03 mjarrison

Same issue here. I have a lot of duplicates in my library. I think it happened when I changed my path to my external library. I added a 2nd library, so I had to change the path in my docker-compose file to be unique. I now have 2 external libraries, on for me and one for my wife. After changing the path, the images with the path of the old library are still there but show asset offline.

Old path: /usr/src/app/external New path: /usr/src/app/external/pictures1 New path 2: /usr/src/app/external/pictures2

A lot of the pictures of the old path are still visible in Immich. root@d3e0d7300ac2:/usr/src/app/external# ls -l only shows the 2 folders, there are no 'forgotten' pictures in this folder.

pictures1 and pictures2 resolve to 2 different folders on the NAS.

Also related to https://github.com/immich-app/immich/discussions/6845 and https://github.com/immich-app/immich/discussions/9574

Jul 14 '24 09:07 rvoosterhout

Same issue here. I have a lot of duplicates in my library. I think it happened when I changed my path to my external library. I added a 2nd library, so I had to change the path in my docker-compose file to be unique. I now have 2 external libraries, on for me and one for my wife. After changing the path, the images with the path of the old library are still there but show asset offline.

Old path: /usr/src/app/external New path: /usr/src/app/external/pictures1 New path 2: /usr/src/app/external/pictures2

A lot of the pictures of the old path are still visible in Immich. root@d3e0d7300ac2:/usr/src/app/external# ls -l only shows the 2 folders, there are no 'forgotten' pictures in this folder.

pictures1 and pictures2 resolve to 2 different folders on the NAS.

Also related to #6845 and #9574

Alright, this is how I fixed it.

I noticed that my offline files had a different path from the old external library URL. This is why the remove offline files task doesn't work, because it cant find any offline files on the new path. I had to add the old path back.

Removed the container paths
Restarted Immich
Removed the import path on my external library (don't remove the library!)
Renamed the import path on my wife's library to something nonexistent like /usr/src/app/testlibrary. Just to be sure so that nothing happens to my wife's external library, because I was going to remove all offline files in /usr/src/app/external. Since her files are in a subfolder in /external, I did this step just to be sure.
Added the old library path /usr/src/app/external back to my own external library
Hit the button to remove offline files (3 dots). At this moment all the offline files in Immich that had the /usr/src/app/external/... path in my library were removed. Because I didn't run a full rescan off the library, it didn't touch the files with the /usr/src/app/external/pictures1/... path, only the ones already marked offline before.
Restored the docker container paths
Restarted Immich
Restored the library paths
All is fixed.

Jul 14 '24 10:07 rvoosterhout

Same issue here. I have a lot of duplicates in my library. I think it happened when I changed my path to my external library. I added a 2nd library, so I had to change the path in my docker-compose file to be unique. I now have 2 external libraries, on for me and one for my wife. After changing the path, the images with the path of the old library are still there but show asset offline. Old path: /usr/src/app/external New path: /usr/src/app/external/pictures1 New path 2: /usr/src/app/external/pictures2 A lot of the pictures of the old path are still visible in Immich. root@d3e0d7300ac2:/usr/src/app/external# ls -l only shows the 2 folders, there are no 'forgotten' pictures in this folder. pictures1 and pictures2 resolve to 2 different folders on the NAS. Also related to #6845 and #9574

Alright, this is how I fixed it.

I noticed that my offline files had a different path from the old external library URL. This is why the remove offline files task doesn't work, because it cant find any offline files on the new path. I had to add the old path back.
1. Removed the container paths

2. Restarted Immich

3. Removed the import path on my external library (don't remove the library!)

4. Renamed the import path on my wife's library to something nonexistent like /usr/src/app/testlibrary. Just to be sure so that nothing happens to my wife's external library, because I was going to remove **all** offline files in /usr/src/app/external. Since her files are in a subfolder in /external, I did this step just to be sure.

5. Added the old library path /usr/src/app/external back to my own external library

6. Hit the button to remove offline files (3 dots). At this moment all the offline files in Immich that had the /usr/src/app/external/... path **in my library** were removed. Because I didn't run a full rescan off the library, it didn't touch the files with the /usr/src/app/external/pictures1/... path, only the ones already marked offline before.

7. Restored the docker container paths

8. Restarted Immich

9. Restored the library paths

10. All is fixed.

But all this, is because you moved your external storage?

There are many of us who haven't moved any of the external libraries, and they are all reading offline.

Jul 24 '24 23:07 matticas

Same issue here. I have a lot of duplicates in my library. I think it happened when I changed my path to my external library. I added a 2nd library, so I had to change the path in my docker-compose file to be unique. I now have 2 external libraries, on for me and one for my wife. After changing the path, the images with the path of the old library are still there but show asset offline. Old path: /usr/src/app/external New path: /usr/src/app/external/pictures1 New path 2: /usr/src/app/external/pictures2 A lot of the pictures of the old path are still visible in Immich. root@d3e0d7300ac2:/usr/src/app/external# ls -l only shows the 2 folders, there are no 'forgotten' pictures in this folder. pictures1 and pictures2 resolve to 2 different folders on the NAS. Also related to #6845 and #9574

Alright, this is how I fixed it. I noticed that my offline files had a different path from the old external library URL. This is why the remove offline files task doesn't work, because it cant find any offline files on the new path. I had to add the old path back.
1. Removed the container paths

2. Restarted Immich

3. Removed the import path on my external library (don't remove the library!)

4. Renamed the import path on my wife's library to something nonexistent like /usr/src/app/testlibrary. Just to be sure so that nothing happens to my wife's external library, because I was going to remove **all** offline files in /usr/src/app/external. Since her files are in a subfolder in /external, I did this step just to be sure.

5. Added the old library path /usr/src/app/external back to my own external library

6. Hit the button to remove offline files (3 dots). At this moment all the offline files in Immich that had the /usr/src/app/external/... path **in my library** were removed. Because I didn't run a full rescan off the library, it didn't touch the files with the /usr/src/app/external/pictures1/... path, only the ones already marked offline before.

7. Restored the docker container paths

8. Restarted Immich

9. Restored the library paths

10. All is fixed.
But all this, is because you moved your external storage?

There are many of us who haven't moved any of the external libraries, and they are all reading offline.

+1, I have not moved my external library and the files are still accessible within the container with the correct path.

Jul 25 '24 06:07 psla

Same issue here. I have a lot of duplicates in my library. I think it happened when I changed my path to my external library. I added a 2nd library, so I had to change the path in my docker-compose file to be unique. I now have 2 external libraries, on for me and one for my wife. After changing the path, the images with the path of the old library are still there but show asset offline. Old path: /usr/src/app/external New path: /usr/src/app/external/pictures1 New path 2: /usr/src/app/external/pictures2 A lot of the pictures of the old path are still visible in Immich. root@d3e0d7300ac2:/usr/src/app/external# ls -l only shows the 2 folders, there are no 'forgotten' pictures in this folder. pictures1 and pictures2 resolve to 2 different folders on the NAS. Also related to #6845 and #9574

Alright, this is how I fixed it. I noticed that my offline files had a different path from the old external library URL. This is why the remove offline files task doesn't work, because it cant find any offline files on the new path. I had to add the old path back.
1. Removed the container paths

2. Restarted Immich

3. Removed the import path on my external library (don't remove the library!)

4. Renamed the import path on my wife's library to something nonexistent like /usr/src/app/testlibrary. Just to be sure so that nothing happens to my wife's external library, because I was going to remove **all** offline files in /usr/src/app/external. Since her files are in a subfolder in /external, I did this step just to be sure.

5. Added the old library path /usr/src/app/external back to my own external library

6. Hit the button to remove offline files (3 dots). At this moment all the offline files in Immich that had the /usr/src/app/external/... path **in my library** were removed. Because I didn't run a full rescan off the library, it didn't touch the files with the /usr/src/app/external/pictures1/... path, only the ones already marked offline before.

7. Restored the docker container paths

8. Restarted Immich

9. Restored the library paths

10. All is fixed.
But all this, is because you moved your external storage?

There are many of us who haven't moved any of the external libraries, and they are all reading offline.

You know, i'm actually not sure anymore. It has been a long time that this issue started and I just left it at that since the upload from my phone didn't cause any issues. I believe it happened when I changed my external library path in my docker-compose file, but It could also have happened when i ran a few jobs in Immich. Do you also have duplicates? If so, can you check the file path on the foto using the details button? In my installation the duplicates both had a different file path, so that's how I applied my fix. It might not work for everyone.

Jul 25 '24 07:07 rvoosterhout

Library scanning has been significantly improved since this issue was opened. Doing a fresh scan and removing offline files should solve any issues with duplicates for external libraries.

Sep 04 '24 17:09 jrasm91