Items appear in trash and can not be restored

[cwallace]

trafficstars

The bug

I have several hundred items listed under the Trash section in Immich. When attempting to restore these items they appear to restore, but after refreshing the page they reappear. The info button next to the "Trash" says that I have 0 pictures and 0 videos in Trash.

https://github.com/user-attachments/assets/4c9a014a-237f-4c4b-9267-dc980aa14a8d

The OS that Immich Server is running on

Ubuntu 22.04

Version of Immich Server

v1.116.2

Version of Immich Mobile App

v1.116.2

Platform with the issue

• [ ] Server
• [X] Web
• [ ] Mobile

Your docker-compose.yml content

name: immich
services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    extends:
      file: hwaccel.transcoding.yml
      service: nvenc # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding
    volumes:
      # Do not edit the next line. If you want to change the media storage location on your system, edit the value of UPLOAD_LOCATION in the .env file
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    ports:
      - 2283:3001
    depends_on:
      - redis
      - database
    restart: always
    healthcheck:
      disable: false

  immich-machine-learning:
    container_name: immich_machine_learning
    # For hardware acceleration, add one of -[armnn, cuda, openvino] to the image tag.
    # Example tag: ${IMMICH_VERSION:-release}-cuda
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration
      file: hwaccel.ml.yml
      service: cuda # set to one of [armnn, cuda, openvino, openvino-wsl] for accelerated inference - use the `-wsl` version for WSL2 where applicable
    volumes:
      - model-cache:/cache
    env_file:
      - .env
    restart: always
    healthcheck:
      disable: false

  redis:
    container_name: immich_redis
    image: docker.io/redis:6.2-alpine@sha256:2d1463258f2764328496376f5d965f20c6a67f66ea2b06dc42af351f75248792
    healthcheck:
      test: redis-cli ping || exit 1
    restart: always

  database:
    container_name: immich_postgres
    image: docker.io/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      POSTGRES_INITDB_ARGS: '--data-checksums'
    volumes:
      # Do not edit the next line. If you want to change the database storage location on your system, edit the value of DB_DATA_LOCATION in the .env file
      - ${DB_DATA_LOCATION}:/var/lib/postgresql/data
    healthcheck:
      test: pg_isready --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' || exit 1; Chksum="$$(psql --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' --tuples-only --no-align --command='SELECT COALESCE(SUM(checksum_failures), 0) FROM pg_stat_database')"; echo "checksum failure count is $$Chksum"; [ "$$Chksum" = '0' ] || exit 1
      interval: 5m
      start_interval: 30s
      start_period: 5m
    command: ["postgres", "-c", "shared_preload_libraries=vectors.so", "-c", 'search_path="$$user", public, vectors', "-c", "logging_collector=on", "-c", "max_wal_size=2GB", "-c", "shared_buffers=512MB", "-c", "wal_compression=on"]
    restart: always

volumes:
  model-cache:

Your .env content

# You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables

# The location where your uploaded files are stored
UPLOAD_LOCATION=/mnt/storage/pictures
# The location where your database files are stored
DB_DATA_LOCATION=/mnt/storage/pictures/database

# To set a timezone, uncomment the next line and change Etc/UTC to a TZ identifier from this list: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List
TZ=America/Denver

# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=release

# Connection secret for postgres. You should change it to a random password
# Please use only the characters `A-Za-z0-9`, without special characters or spaces
DB_PASSWORD=Hunter2

# The values below this line do not need to be changed
###################################################################################
DB_USERNAME=postgres
DB_DATABASE_NAME=immich

Reproduction steps

1. Delete items
2. Navigate to trash
3. Attempt to restore
4. Refresh the page

Relevant log output

No response

Additional information

No response

[AnimeshKotka]

I have the issue and have started working on it. cc: @jrasm91 @alextran1502

[aviv926]

I tried to reproduce your problem, without success. Can you check if you can reproduce the problem in the demo version? https://demo.immich.app/trash

[cwallace]

I tried to reproduce your problem, without success. Can you check if you can reproduce the problem in the demo version? https://demo.immich.app/trash

So far I have not been able to reproduce this in the demo version. There are a couple of items that I'm unable to test in the demo environment however:

1. Upgrading Immich with items in the Trash
2. Deleting items using duplicate detection

[jrasm91]

I'm guessing this is specific to external library assets that are "offline". They are in the trash for now while we work on making a dedicated page for them, which will be home to the new "system integrity" page.

[cwallace]

I'm guessing this is specific to external library assets that are "offline". They are in the trash for now while we work on making a dedicated page for them, which will be home to the new "system integrity" page.

I don't believe I have any external library assets, especially none that are offline. This is a relatively new instance (created last weekend) and all of my media has been added via immich-go or the immich mobile application. Everything exists on my zfs pool (/mnt/storage/) including media and database.

[jrasm91]

Interesting. The only thing I have seen that could cause this is a timezone issue where "now" is an incorrect date and the database is in another timezone or something. Can you confirm the timezone of the immich server?

[cwallace]

Interesting. The only thing I have seen that could cause this is a timezone issue where "now" is an incorrect date and the database is in another timezone or something. Can you confirm the timezone of the immich server?

Wow! No idea how you determined that, but after uploading most (all?) of my images, I noticed that I had forgot to set the correct timezone on the host VM and ran dpkg-reconfigure tzdata.

I've tried restarting the immich stack as well as setting the timezone back to UTC and restarting the stack, but I'm still having the issue with these stuck in the trash. Any suggestions on how to resolve the issue?

edit: I gave up and simply set deletedAt to null for all assets, which at least appears to have solved having these photos being stuck in the trash. Not sure if you want to leave this open to solve the bigger problem of handling timezone changes, but I'll leave that up to you.

I do have TZ=America/Denver in my .env.

Just checked the docker containers: immich-server is America/Denver immich-machine-learning is America/Denver immich-postgres is UTC immich-redis is UTC

The host machine is America/Denver

[piyush97]

I'm facing the same issue, any solution?

[jrasm91]

You should be able to select the assets individually or hit "select all" and then delete them. I believe the issue is specifically with the "empty trash" button when the database is in a different timezone than the server.

[dpad]

Just wanted to report that I have the same (or similar) issue, but seemingly without the timezone factor?

For me, there were many assets in trash that were not counted in the Trash statistics (showed as "0 photos, 0 videos") and could not be deleted by the "empty trash" button. I had to delete them by selecting all and clicking the delete button instead.

I think that all my containers are running in UTC though:

ubuntu@immich:~$ date
Sun Oct  6 06:20:59 UTC 2024
ubuntu@immich:~$ docker exec immich_server date
Sun Oct  6 06:21:14 UTC 2024
ubuntu@immich:~$ docker exec immich_redis date
Sun Oct  6 06:21:20 UTC 2024
ubuntu@immich:~$ docker exec immich_postgres date
Sun Oct  6 06:21:25 AM UTC 2024
ubuntu@immich:~$ docker exec immich_postgres psql -U postgres -c "SHOW TIMEZONE;"
 TimeZone
----------
 Etc/UTC
(1 row)

In case it helps, I only noticed this after upgrading today from 1.115.0 to 1.117.0 (though it may have been present before, I just didn't try to empty the trash until now).

[alextran1502]

@dpad For additional information, are you using external library?

[dpad]

@alextran1502 Thanks for checking. No, I'm not using an external library.

[dpad]

I'll expand on what I did today in case it helps.

1. I upgraded using docker compose from 1.115.0 -> 1.117.0.
2. After the upgrade, I deleted a couple of items.
3. These items seemingly showed up in the Trash statistics (e.g. "1 video, 5 photos").
4. However, the Trash contained these items I just deleted, plus many hundreds more that I had deleted ~2 weeks ago.
5. When I clicked on the "Empty Trash" button, it deleted the items from today, but left all the other items from 2 weeks ago.
6. When I checked the Trash statistics again, it said "0 videos, 0 photos". I refreshed, tried "Empty Trash" again and it said something like "removed 0 items". All the items deleted 2 weeks ago were still in the Trash.
7. I found this issue and saw the recommendation to select all the items and click delete manually, so I did that. In total, it deleted ~600 items from the Trash.

Based on the above, I would suspect it may be something due to the upgrade from 1.115.0 -> 1.117.0. It's possible that this issue was present before, but I only noticed and tried emptying the trash today after the upgrade.

[dpad]

@alextran1502 By the way, I have database backups going back at least several weeks. I'd prefer not to share them directly, but I'm happy to run some queries on them if you need.

[zionun]

Hello, same problem here. Trash info says "0 video, 0 photo" but there's 185 items in it. They are not in an external library and I do not want to delete them. When I restore them, they soon reappear in the trash. The Info section for each photo says that the resource is no longer on the disk, but the file is exactly at the path shown in the error panel and permissions are the same of all other thousands of pictures that are correctly shown in the timeline. My folder is on a NAS and I tried to mount it both with smb and nfs.

[zionun]

I'd like to add some more info on this issue that keeps happening on my immich installation. I ran some queries on the db in order to restore the 185 items I was talking about in my previous message: I set deletedAt to null and isOffline to false, and they went back to the timeline. I found out that every time I rescan my library, those very same 185 items go back in the trash and there's no way to restore them apart from running those queries on the database. The files have the same exact permissions as all other files. So why does this keep happening to those specific items? Yesterday it started happening on other 14 items that I recently uploaded to the timeline. So, every time I manually scan the library, the same 199 items get trashed.

[etnoy]

I'd like to add some more info on this issue that keeps happening on my immich installation. I ran some queries on the db in order to restore the 185 items I was talking about in my previous message: I set deletedAt to null and isOffline to false, and they went back to the timeline. I found out that every time I rescan my library, those very same 185 items go back in the trash and there's no way to restore them apart from running those queries on the database. The files have the same exact permissions as all other files. So why does this keep happening to those specific items? Yesterday it started happening on other 14 items that I recently uploaded to the timeline. So, every time I manually scan the library, the same 199 items get trashed.

Please check your logs when scanning, it'll say why. Check import paths and exclusion patterns

[zionun]

Please check your logs when scanning, it'll say why. Check import paths and exclusion patterns

I turned on verbose debugging and the log says:

Asset is no longer in an import path, removing: upload/upload/2decf131-b085-4e8e-b3d9-666e5efb63a8/19/41/1941adf9-cc00-4bbe-8f92-a53666ec0a99.jpg

All these items are NOT in an external library: they are assets I uploaded to albums using this plugin. I always use this plugin and I uploaded thousands of photo through it, but it is just this bunch of pictures that is getting this treatment.

[etnoy]

Please check your logs when scanning, it'll say why. Check import paths and exclusion patterns

I turned on verbose debugging and the log says:

Asset is no longer in an import path, removing: upload/upload/2decf131-b085-4e8e-b3d9-666e5efb63a8/19/41/1941adf9-cc00-4bbe-8f92-a53666ec0a99.jpg

All these items are NOT in an external library: they are assets I uploaded to albums using this plugin. I always use this plugin and I uploaded thousands of photo through it, but it is just this bunch of pictures that is getting this treatment.

That's interesting. Please check your external library settings under Administration. Are there any libraries configured there?

[zionun]

That's interesting. Please check your external library settings under Administration. Are there any libraries configured there?

Yes, I have a rather big external library configured in Immich, but the pictures that get trashed don't belong to that.

[etnoy]

That's interesting. Please check your external library settings under Administration. Are there any libraries configured there?

Yes, I have a rather big external library configured in Immich, but the pictures that get trashed don't belong to that.

The only way the log would say "Asset is no longer in an import path, removing" is due to a file being offline from an external library.

I see now that you have done manual db queries, that's always dangerous. Can you check the postgres database to see what the asset with path upload/upload/2decf131-b085-4e8e-b3d9-666e5efb63a8/19/41/1941adf9-cc00-4bbe-8f92-a53666ec0a99.jpg looks like?

[zionun]

Yes, I executed some queries on the db but I did it only after this problem started to appear, not before. After fetching the row from the db I found out that "isExternal" is set to true even if the file is not in the external library. I'm sure that's the root of the problem, now I just need to figure out why it is set so. I suspect it may be related to an auto-stack plugin I'm using to group photos: my configuration consists of having raw files in an external library and post-processed files uploaded on the main timeline; I'm using this auto-stack plugin in order to have only the post processed images visible in the timeline, but I guess there must be some bug that sets isExternal to true on both files instead of leaving one to false and the other one to true.

[etnoy]

Yeah, you seem to be in some kind of broken state. Set isExternal and isOffline to false in the affected assets

[zionun]

I just did it but as soon as I launch the scan process they go back in that state again 😩 The weird thing is that I'm launching the external library scan process and non-external items get updated on the db. I really don't know how to fix this.

[etnoy]

Do the assets have a library id set? If then, remove it. The library scanner will otherwise consider them to be part of the library

[zionun]

YES! That was it! There was a libraryId set and removing it fixed the problem. Thank you very much for helping me! Do you think this would be worth of a further investigation? To summarize it, it seems like items linked together with a stackId but one belonging to an external library and the other one belonging to the standard Immich timeline (might) go into a broken state.

[etnoy]

YES! That was it! There was a libraryId set and removing it fixed the problem. Thank you very much for helping me! Do you think this would be worth of a further investigation? To summarize it, it seems like items linked together with a stackId but one belonging to an external library and the other one belonging to the standard Immich timeline (might) go into a broken state.

I haven't tried that yet, but if you would like to experiment with stacking an internal and external asset and see what happens in the asset table in the db that would be great. If you find a bug, please file a github issue