immich [BUG] Copying files into synced folder causes duplicates

The bug

If there is a folder that is synced by immich on the android (ex internal storage > DCIM > Camera) and you copy images from an unsynced folder into the synced folder immich will create duplicate image within the immich android application (local only).

The OS that Immich Server is running on

Ubuntu 22.04

Version of Immich Server

v1.59.1

Version of Immich Mobile App

v1.60.0

Platform with the issue

[ ] Server
[ ] Web
[X] Mobile

Your docker-compose.yml content

version: "3.6"

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    command: ["start-server.sh"]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
    env_file:
      - .env
    depends_on:
      - redis
      - database
      - typesense
    networks:
      - immich_lan
    restart: unless-stopped

  immich-microservices:
    container_name: immich_microservices
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    command: ["start-microservices.sh"]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
    env_file:
      - .env
    depends_on:
      - redis
      - database
      - typesense
    networks:
      - immich_lan
    restart: unless-stopped

  immich-machine-learning:
    container_name: immich_machine_learning
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - model-cache:/cache
    env_file:
      - .env
    networks:
      - immich_lan
    restart: unless-stopped

  immich-web:
    container_name: immich_web
    image: ghcr.io/immich-app/immich-web:${IMMICH_VERSION:-release}
    env_file:
      - .env
    networks:
      - immich_lan
    restart: unless-stopped

  typesense:
    container_name: immich_typesense
    image: typesense/typesense:0.24.0
    environment:
      - TYPESENSE_API_KEY=${TYPESENSE_API_KEY}
      - TYPESENSE_DATA_DIR=/data
    logging:
      driver: none
    volumes:
      - tsdata:/data
    networks:
      - immich_lan
    restart: unless-stopped

  redis:
    container_name: immich_redis
    image: redis:6.2
    networks:
      - immich_lan
    restart: unless-stopped

  database:
    container_name: immich_postgres
    image: postgres:14
    env_file:
      - .env
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      PG_DATA: /var/lib/postgresql/data
    volumes:
      - pgdata:/var/lib/postgresql/data
    networks:
      - immich_lan
    restart: unless-stopped

  immich-proxy:
    container_name: immich_proxy
    image: ghcr.io/immich-app/immich-proxy:${IMMICH_VERSION:-release}
    environment:
      # Make sure these values get passed through from the env file
      - IMMICH_SERVER_URL
      - IMMICH_WEB_URL
    ports:
      - 8080
    depends_on:
      - immich-server
    networks:
      - immich_lan
    restart: unless-stopped

volumes:
  pgdata:
  model-cache:
  tsdata:

networks:
  immich_lan:
    external: true

Your .env content

###################################################################################
# Database
###################################################################################

DB_HOSTNAME=immich_postgres
DB_USERNAME=postgres
DB_PASSWORD=redacted 
DB_DATABASE_NAME=immich

# Optional Database settings:
# DB_PORT=5432

###################################################################################
# Redis
###################################################################################

REDIS_HOSTNAME=immich_redis

# Optional Redis settings:

# Note: these parameters are not automatically passed to the Redis Container
# to do so, please edit the docker-compose.yml file as well. Redis is not configured
# via environment variables, only redis.conf or the command line

# REDIS_PORT=6379
# REDIS_DBINDEX=0
# REDIS_PASSWORD=
# REDIS_SOCKET=

###################################################################################
# Upload File Location
#
# This is the location where uploaded files are stored.
###################################################################################

UPLOAD_LOCATION=redacted 


###################################################################################
# Log message level - [simple|verbose]
###################################################################################

LOG_LEVEL=simple

###################################################################################
# Typesense
###################################################################################
# TYPESENSE_ENABLED=false
TYPESENSE_API_KEY=redacted 
# TYPESENSE_HOST: typesense
# TYPESENSE_PORT: 8108
# TYPESENSE_PROTOCOL: http

###################################################################################
# Reverse Geocoding
#
# Reverse geocoding is done locally which has a small impact on memory usage
# This memory usage can be altered by changing the REVERSE_GEOCODING_PRECISION variable
# This ranges from 0-3 with 3 being the most precise
# 3 - Cities > 500 population: ~200MB RAM
# 2 - Cities > 1000 population: ~150MB RAM
# 1 - Cities > 5000 population: ~80MB RAM
# 0 - Cities > 15000 population: ~40MB RAM
####################################################################################

DISABLE_REVERSE_GEOCODING=false
REVERSE_GEOCODING_PRECISION=3

####################################################################################
# WEB - Optional
#
# Custom message on the login page, should be written in HTML form.
# For example:
# PUBLIC_LOGIN_PAGE_MESSAGE="This is a demo instance of Immich.<br><br>Email: <i>[email protected]</i><br>Password: <i>demo</i>"
####################################################################################

PUBLIC_LOGIN_PAGE_MESSAGE="Family Photos and Videos Backup Server"

####################################################################################
# Alternative Service Addresses - Optional
#
# This is an advanced feature for users who may be running their immich services on different hosts.
# It will not change which address or port that services bind to within their containers, but it will change where other services look for their peers.
# Note: immich-microservices is bound to 3002, but no references are made
####################################################################################

IMMICH_WEB_URL=http://immich-web:3000
IMMICH_SERVER_URL=http://immich-server:3001
IMMICH_MACHINE_LEARNING_URL=http://immich-machine-learning:3003

####################################################################################
# Alternative API's External Address - Optional
#
# This is an advanced feature used to control the public server endpoint returned to clients during Well-known discovery.
# You should only use this if you want mobile apps to access the immich API over a custom URL. Do not include trailing slash.
# NOTE: At this time, the web app will not be affected by this setting and will continue to use the relative path: /api
# Examples: http://localhost:3001, http://immich-api.example.com, etc
####################################################################################

#IMMICH_API_URL_EXTERNAL=http://localhost:3001

###################################################################################
# Immich Version - Optional
#
# This allows all immich docker images to be pinned to a specific version. By default,
# the version is "release" but could be a specific version, like "v1.59.0".
###################################################################################

#IMMICH_VERSION=

Reproduction steps

1. Have syncing turned on (I'm not sure if this is required, but it's how mine was set up when I noticed the issue)
2. Download images to a non synced folder (IE internal storage > Downloads)
3. Select Images from step 2 and select "Copy"
4. Navigate to Synced folder (internal storage > DCIM > Camera)
5. Select "Copy here"
6. Open Immich app

Additional information

Sorry it's a bit of a long video, but I took a video of me reproducing the bug. https://drive.google.com/file/d/1HPGy0kh6jvjZtlLfXudrKUmUysQYz7MN/view?usp=sharing

Other note is that the unsynced duplicates can be deleted from the application without any issue. Since they aren't duplicated in internal storage > DCIM > Camera or on the server then it doesn't cause any problems I've seen.

Jun 09 '23 22:06 aglad-eng

I believe copying an image would change its metadata in your case, maybe modified, or the creation date would be different from the uploaded image, and it causes issues as such. Can you go into the setting page and enable the Advanced Debug option and then go into the uploaded image and the duplicated local image and open the info page of each photo, then capture the info and post them here?

Jun 10 '23 02:06 alextran1502

Local image { "remoteId": "N/A", "localId": "1000001218", "deviceId": "2156076524589928155", "ownerId": "-481877277952871358", "livePhotoVideoId": "N/A", "fileCreatedAt": "2023-06-10 18:12:49.000", "fileModifiedAt": "2023-06-10 18:12:49.000", "updatedAt": "2023-06-10 18:12:49.000", "durationInSeconds": 0, "type": "AssetType.image", "fileName": "cool-picture-wolf-art-o0ixt449edz5dgpa.jpg", "isFavorite": false, "isLocal": true, "isRemote: false, "storage": AssetState.local, "width": 1920, "height": 1080, "isArchived": false }

Uploaded image: { "remoteId": "d0d91b66-1699-45e0-ad76-7eeae0cd2b24", "localId": "1000001219", "deviceId": "2156076524589928155", "ownerId": "-481877277952871358", "livePhotoVideoId": "N/A", "fileCreatedAt": "2023-06-10 14:13:24.000", "fileModifiedAt": "2023-06-10 14:12:49.000", "updatedAt": "2023-06-10 18:13:32.892", "durationInSeconds": 0, "type": "AssetType.image", "fileName": "cool-picture-wolf-art-o0ixt449edz5dgpa.jpg", "isFavorite": false, "isLocal": true, "isRemote: true, "storage": AssetState.merged, "width": 1920, "height": 1080, "isArchived": false }

Jun 10 '23 22:06 aglad-eng

You can see the updatedAt property is different, that is why

Jun 10 '23 22:06 alextran1502

Yeah, I can see that difference. So I guess my question is a bit of why that occurs? Correct me if I'm wrong but is what is happening is. Android OS creates a file the same as the original and then modifies the metadata. In the time before the modifies the metadata Immich indexes it as a new file. Once the metadata is updated by the OS immich indexes it again? If so wouldn't immich sync two files to the server? I have zero to no experience with android or immich so I'm just feeling out if this is a fixable bug or just one that comes with dealing with the OS.

Jun 12 '23 01:06 aglad-eng

copying files on Android changes the createdAt and modifiedAt and localId properties stored in the Android-internal database. Immich reads that information, different localId = different file. With the upcoming release where #2592 is merged, files are compared via their checksum. Then, you should no longer see duplicates.

Jun 15 '23 07:06 fyfrey

I forgot to close this, but the update fixed the issue. Thanks for the work you all do :)

Aug 18 '23 17:08 aglad-eng

immich immich copied to clipboard

[BUG] Copying files into synced folder causes duplicates

The bug

The OS that Immich Server is running on

Version of Immich Server

Version of Immich Mobile App

Platform with the issue

Your docker-compose.yml content

Your .env content

Reproduction steps

Additional information

immich
immich copied to clipboard