[BUG] Storage Template links do not work on fileshare

[evanreichard]

The bug

Im pretty sure the following is the issue.

Log:

[Nest] 1  - 05/07/2023, 4:28:32 AM   ERROR [StorageTemplateService] Problem applying storage template
[Nest] 1  - 05/07/2023, 4:28:32 AM   ERROR [StorageTemplateService] Error: EIO: i/o error, link 'upload/upload/034dc848-001a-472d-a925-a4bdc56a60bb/33a8e421-4b9b-4ca4-9102-961c18d4c9ab.jpg' -> 'upload/library/034dc848-001a-472d-a925-a4bdc56a60bb/2022/2022-12-18/19-26-48-MeghanandEvanWedding-0247.jpg'
[Nest] 1  - 05/07/2023, 4:28:32 AM   ERROR [StorageTemplateService] Object:
{
  "id": "18017194-c7cd-48d4-9c57-aeaf3b5d5db5",
  "source": "upload/upload/034dc848-001a-472d-a925-a4bdc56a60bb/33a8e421-4b9b-4ca4-9102-961c18d4c9ab.jpg",
  "destination": "upload/library/034dc848-001a-472d-a925-a4bdc56a60bb/2022/2022-12-18/19-26-48-MeghanandEvanWedding-0247.jpg"
}

Manual attempt:

/usr/src/app # ln -s upload/upload/034dc848-001a-472d-a925-a4bdc56a60bb/33a8e421-4b9b-4ca4-9102-961c18d4c9ab.jpg upload/library/034dc84
8-001a-472d-a925-a4bdc56a60bb/2022/2022-12-18/19-26-48-MeghanandEvanWedding-0247.jpg
/usr/src/app # echo $?
0
/usr/src/app # ls -lah upload/library/034dc848-001a-472d-a925-a4bdc56a60bb/2022/2022-12-18/19-26-48-MeghanandEvanWedding-0247.jpg 
lrwxrwxrwx    1 99       users         91 May  7 04:39 upload/library/034dc848-001a-472d-a925-a4bdc56a60bb/2022/2022-12-18/19-26-48-MeghanandEvanWedding-0247.jpg -> upload/upload/034dc848-001a-472d-a925-a4bdc56a60bb/33a8e421-4b9b-4ca4-9102-961c18d4c9ab.jpg
/usr/src/app # cat upload/library/034dc848-001a-472d-a925-a4bdc56a60bb/2022/2022-12-18/19-26-48-MeghanandEvanWedding-0247.jpg
cat: can't open 'upload/library/034dc848-001a-472d-a925-a4bdc56a60bb/2022/2022-12-18/19-26-48-MeghanandEvanWedding-0247.jpg': No such file or directory

It appears because the link is established from outside of the network share, then the share doesn't know what to do. If I change my working directory to the root of the network share, then it successfully links.

The OS that Immich Server is running on

Kubernetes v1.24.12+rke2r1 on Rocky Linux

Version of Immich Server

v1.54.1

Version of Immich Mobile App

NA

Platform with the issue

• [X] Server
• [ ] Web
• [ ] Mobile

Your docker-compose.yml content

Helm Chart

Your .env content

Helm Chart - No deviations outside of DB_PASSWORD

Reproduction steps

See main report

Additional information

No response

[alextran1502]

Hello, thank you for reporting. Can you elaborate on your message below? I am not understanding the issue very well here.

It appears because the link is established from outside of the network share, then the share doesn't know what to do. If I change my working directory to the root of the network share, then it successfully links.

[evanreichard]

Sure. In the container the working dir is /usr/src/app. Inside that dir is an upload folder which is mounted as an NFS share.

If I try to create a link when my working dir is /usr/src/app, then it won't work because its stored on the NFS share with a relative path that does not exist.

If I change my working dir to a dir that's on the share (e.g. the root dir - /usr/src/app/upload), then the link will work because the path is relative to all folders that the share understands.

---

The above behavior is why I think the following error happens:

[Nest] 1  - 05/07/2023, 4:28:32 AM   ERROR [StorageTemplateService] Error: EIO: i/o error, link 'upload/upload/034dc848-001a-472d-a925-a4bdc56a60bb/33a8e421-4b9b-4ca4-9102-961c18d4c9ab.jpg' -> 'upload/library/034dc848-001a-472d-a925-a4bdc56a60bb/2022/2022-12-18/19-26-48-MeghanandEvanWedding-0247.jpg'

As those paths are not relative to the share, so the share doesn't know how to understand them.

[bo0tzz]

This behaviour should probably fallback to a move or copy if links aren't supported @alextran1502

[truppelito]

I'dlike to add that I'm having what looks like to be the same issue:

[Nest] 1624  - 05/07/2023, 4:46:17 PM   ERROR [StorageTemplateService] Problem applying storage template
[Nest] 1624  - 05/07/2023, 4:46:17 PM   ERROR [StorageTemplateService] Object:
[Nest] 1624  - 05/07/2023, 4:46:17 PM   ERROR [StorageTemplateService] Error: ENOENT: no such file or directory, link '/photos/upload/de5e535e-ae74-4180-a43f-976adcde806a/827585c6-2e25-46d0-9aca-b70830d5a7a9.jpg' -> '/photos/library/de5e535e-ae74-4180-a43f-976adcde806a/2022/01/2022-01-30-12B4FA13-D7B4-4AEF-BF12-3E792C34781C+1.jpg'
{
  "id": "90ed8ed2-6c0a-4b96-9230-023940aae626",
  "source": "/photos/upload/de5e535e-ae74-4180-a43f-976adcde806a/827585c6-2e25-46d0-9aca-b70830d5a7a9.jpg",
  "destination": "/photos/library/de5e535e-ae74-4180-a43f-976adcde806a/2022/01/2022-01-30-12B4FA13-D7B4-4AEF-BF12-3E792C34781C+1.jpg"
}

In my case, the docker volume that maps to the /photos folder (and thus /photos/upload and /photos/library) is an SMB share mounted by docker itself (configured inside my docker compose file) that points to a NAS (another device on my LAN). While uploading from my iPhone (for example), many of the files go from /photos/upload to /photos/library as expected, but some get stuck in /photos/upload and never leave.

[megapearl]

Having the same problem, also remote mounted via NFS .

relevant config:

dockerhost local /etc/fstab:

storage.mydomain:/mnt/storage/Homes     /mnt/homes        nfs4 defaults,sec=sys 0 0

environment: UPLOAD_LOCATION=/mnt/homes/photos

docker config:

  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:release
    entrypoint: ["/bin/sh", "./start-server.sh"]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
    ports:
      - 3007:3001
    env_file:
      - stack.env
    depends_on:
      - typesense
    restart: always

  immich-microservices:
    container_name: immich_microservices
    image: ghcr.io/immich-app/immich-server:release
    entrypoint: ["/bin/sh", "./start-microservices.sh"]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
    env_file:
      - stack.env
    depends_on:
      - typesense
    restart: always

  immich-machine-learning:
    container_name: immich_machine_learning
    image: ghcr.io/immich-app/immich-machine-learning:release
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /home/dockers/immich/model-cache:/cache
    env_file:
      - stack.env
    ports:
      - 3008:3003
    restart: always

  immich-web:
    container_name: immich_web
    image: ghcr.io/immich-app/immich-web:release
    entrypoint: ["/bin/sh", "./entrypoint.sh"]
    env_file:
      - stack.env
    ports:
      - 3006:3000
    restart: always

  typesense:
    container_name: immich_typesense
    image: typesense/typesense:0.24.0
    environment:
      - TYPESENSE_API_KEY=${TYPESENSE_API_KEY}
      - TYPESENSE_DATA_DIR=/data
    logging:
      driver: none
    volumes:
      - /home/dockers/immich/tsdata:/data
    restart: always

immich-microservices logs: Hundreds of:

[Nest] 1  - 05/25/2023, 7:43:25 PM   ERROR [StorageTemplateService] Problem applying storage template
[Nest] 1  - 05/25/2023, 7:43:25 PM   ERROR [StorageTemplateService] Error: ENOENT: no such file or directory, link 'upload/upload/3879145c-08e7-4ab6-a8e3-6ba3dbd57ffd/fb7506ff-0290-4d3c-ae94-6c951e15bc3b.heic' -> 'upload/library/donald/2018/12/2018-12-16_12-03-01.heic'
[Nest] 1  - 05/25/2023, 7:43:25 PM   ERROR [StorageTemplateService] Object:
{
  "id": "46d2ae25-e69e-4492-8fad-d1cd61c299b7",
  "source": "upload/upload/3879145c-08e7-4ab6-a8e3-6ba3dbd57ffd/fb7506ff-0290-4d3c-ae94-6c951e15bc3b.heic",
  "destination": "upload/library/donald/2018/12/2018-12-16_12-03-01.heic"
}
[Nest] 1  - 05/25/2023, 7:43:25 PM   ERROR [StorageTemplateService] Problem applying storage template
[Nest] 1  - 05/25/2023, 7:43:25 PM   ERROR [StorageTemplateService] Error: ENOENT: no such file or directory, link 'upload/upload/3879145c-08e7-4ab6-a8e3-6ba3dbd57ffd/ca4b337d-f083-4114-8d22-3eb8526d3a87.mov' -> 'upload/library/donald/2018/12/2018-12-06_01-37-19.mov'
[Nest] 1  - 05/25/2023, 7:43:25 PM   ERROR [StorageTemplateService] Object:
{
  "id": "f36f4bfc-0c0f-4592-93b3-dc3f85ef5c31",
  "source": "upload/upload/3879145c-08e7-4ab6-a8e3-6ba3dbd57ffd/ca4b337d-f083-4114-8d22-3eb8526d3a87.mov",
  "destination": "upload/library/donald/2018/12/2018-12-06_01-37-19.mov"
}
[Nest] 1  - 05/25/2023, 7:43:25 PM   ERROR [StorageTemplateService] Problem applying storage template
[Nest] 1  - 05/25/2023, 7:43:25 PM   ERROR [StorageTemplateService] Error: ENOENT: no such file or directory, link 'upload/upload/3879145c-08e7-4ab6-a8e3-6ba3dbd57ffd/025b27b8-2caf-4553-b9a6-c2c34470f67f.jpeg' -> 'upload/library/donald/2018/07/2018-07-04_15-50-45.jpeg'
[Nest] 1  - 05/25/2023, 7:43:25 PM   ERROR [StorageTemplateService] Object:
{
  "id": "026b2b4d-f5ac-45b8-98f6-93ad08509465",
  "source": "upload/upload/3879145c-08e7-4ab6-a8e3-6ba3dbd57ffd/025b27b8-2caf-4553-b9a6-c2c34470f67f.jpeg",
  "destination": "upload/library/donald/2018/07/2018-07-04_15-50-45.jpeg"
}

[klejejs]

I have the exact same config as @megapearl and unfortunately the exact same issue as well 😞.

[jrasm91]

Is this still an issue?

[truppelito]

Is this still an issue?

It was very recently (when I last checked). As soon as I find some time, I'll check again.

[megapearl]

I don't think it is fixed, just pulled the latest immich images, and started the 'storage template migration' and still hundreds of errors in the log of immich-microservices:

[Nest] 7  - 07/01/2023, 12:29:14 PM     LOG [NestFactory] Starting Nest application...
[Nest] 7  - 07/01/2023, 12:29:14 PM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +114ms
[Nest] 7  - 07/01/2023, 12:29:14 PM     LOG [InstanceLoader] BullModule dependencies initialized +1ms
[Nest] 7  - 07/01/2023, 12:29:14 PM     LOG [InstanceLoader] ConfigHostModule dependencies initialized +2ms
[Nest] 7  - 07/01/2023, 12:29:14 PM     LOG [InstanceLoader] DiscoveryModule dependencies initialized +0ms
[Nest] 7  - 07/01/2023, 12:29:14 PM     LOG [InstanceLoader] ConfigModule dependencies initialized +14ms
[Nest] 7  - 07/01/2023, 12:29:14 PM     LOG [InstanceLoader] BullModule dependencies initialized +1ms
[Nest] 7  - 07/01/2023, 12:29:14 PM     LOG [InstanceLoader] BullModule dependencies initialized +0ms
[Nest] 7  - 07/01/2023, 12:29:15 PM     LOG [InstanceLoader] TypeOrmCoreModule dependencies initialized +479ms
[Nest] 7  - 07/01/2023, 12:29:15 PM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +1ms
[Nest] 7  - 07/01/2023, 12:29:15 PM     LOG [InstanceLoader] TypeOrmModule dependencies initialized +0ms
[Nest] 7  - 07/01/2023, 12:29:15 PM     LOG [InstanceLoader] InfraModule dependencies initialized +32ms
[Nest] 7  - 07/01/2023, 12:29:15 PM     LOG [InstanceLoader] DomainModule dependencies initialized +1ms
[Nest] 7  - 07/01/2023, 12:29:15 PM     LOG [InstanceLoader] MicroservicesModule dependencies initialized +1ms
[Nest] 7  - 07/01/2023, 12:29:15 PM    WARN [MetadataExtractionProcessor] Reverse geocoding is enabled
[Nest] 7  - 07/01/2023, 12:29:15 PM     LOG [MetadataExtractionProcessor] Initializing Reverse Geocoding
[Nest] 7  - 07/01/2023, 12:29:33 PM     LOG [MetadataExtractionProcessor] Reverse Geocoding Initialized
[Nest] 7  - 07/01/2023, 12:29:33 PM     LOG [NestApplication] Nest application successfully started +57ms
[Nest] 7  - 07/01/2023, 12:29:33 PM     LOG [ImmichMicroservice] Immich Microservices is listening on 3002 [v1.65.0] [PRODUCTION] 
[Nest] 7  - 07/01/2023, 12:32:29 PM   ERROR [StorageTemplateService] Problem applying storage template
[Nest] 7  - 07/01/2023, 12:32:29 PM   ERROR [StorageTemplateService] Error: ENOENT: no such file or directory, rename 'upload/upload/3879145c-08e7-4ab6-a8e3-6ba3dbd57ffd/bceeae10-07bf-454e-9f6d-c9e754e2e764.jpeg' -> 'upload/library/donald/2015/10/2015-10-27_14-39-02.jpeg'
[Nest] 7  - 07/01/2023, 12:32:29 PM   ERROR [StorageTemplateService] Object:
{
  "id": "dc27b59b-84ec-4370-9734-a7908f7430e7",
  "source": "upload/upload/3879145c-08e7-4ab6-a8e3-6ba3dbd57ffd/bceeae10-07bf-454e-9f6d-c9e754e2e764.jpeg",
  "destination": "upload/library/donald/2015/10/2015-10-27_14-39-02.jpeg"
}
[Nest] 7  - 07/01/2023, 12:32:29 PM   ERROR [StorageTemplateService] Problem applying storage template
[Nest] 7  - 07/01/2023, 12:32:29 PM   ERROR [StorageTemplateService] Error: ENOENT: no such file or directory, rename 'upload/upload/3879145c-08e7-4ab6-a8e3-6ba3dbd57ffd/dbe65bdb-9f71-49c0-a966-1896edd3c64c.jpg' -> 'upload/library/donald/2015/01/2015-01-05_11-13-25.jpg'
[Nest] 7  - 07/01/2023, 12:32:29 PM   ERROR [StorageTemplateService] Object:
{
  "id": "18be8ccb-edb0-40c5-9de4-51904d5c708d",
  "source": "upload/upload/3879145c-08e7-4ab6-a8e3-6ba3dbd57ffd/dbe65bdb-9f71-49c0-a966-1896edd3c64c.jpg",
  "destination": "upload/library/donald/2015/01/2015-01-05_11-13-25.jpg"
}
[Nest] 7  - 07/01/2023, 12:32:29 PM   ERROR [StorageTemplateService] Problem applying storage template
[Nest] 7  - 07/01/2023, 12:32:29 PM   ERROR [StorageTemplateService] Error: ENOENT: no such file or directory, rename 'upload/upload/3879145c-08e7-4ab6-a8e3-6ba3dbd57ffd/8791b689-ba5c-40f8-ab19-8dfe217d2186.jpg' -> 'upload/library/donald/2015/01/2015-01-06_12-57-08.jpg'
[Nest] 7  - 07/01/2023, 12:32:29 PM   ERROR [StorageTemplateService] Object:
{
  "id": "1a4af389-db40-4c54-a327-2793da253ce9",
  "source": "upload/upload/3879145c-08e7-4ab6-a8e3-6ba3dbd57ffd/8791b689-ba5c-40f8-ab19-8dfe217d2186.jpg",
  "destination": "upload/library/donald/2015/01/2015-01-06_12-57-08.jpg"
}
[Nest] 7  - 07/01/2023, 12:32:29 PM   ERROR [StorageTemplateService] Problem applying storage template
[Nest] 7  - 07/01/2023, 12:32:29 PM   ERROR [StorageTemplateService] Error: ENOENT: no such file or directory, rename 'upload/upload/3879145c-08e7-4ab6-a8e3-6ba3dbd57ffd/b88c89a3-077d-4801-8e59-c3b93bbca2fc.jpeg' -> 'upload/library/donald/2015/01/2015-01-06_15-01-55.jpeg'
[Nest] 7  - 07/01/2023, 12:32:29 PM   ERROR [StorageTemplateService] Object:
{
  "id": "b8f4c3d9-bca6-4dac-a621-95939e7077bb",
  "source": "upload/upload/3879145c-08e7-4ab6-a8e3-6ba3dbd57ffd/b88c89a3-077d-4801-8e59-c3b93bbca2fc.jpeg",
  "destination": "upload/library/donald/2015/01/2015-01-06_15-01-55.jpeg"
}
[Nest] 7  - 07/01/2023, 12:32:29 PM   ERROR [StorageTemplateService] Problem applying storage template
[Nest] 7  - 07/01/2023, 12:32:29 PM   ERROR [StorageTemplateService] Error: ENOENT: no such file or directory, rename 'upload/upload/3879145c-08e7-4ab6-a8e3-6ba3dbd57ffd/7fc24242-8351-464f-97e5-f4e2c1167393.jpeg' -> 'upload/library/donald/2015/01/2015-01-26_17-38-58.jpeg'
[Nest] 7  - 07/01/2023, 12:32:29 PM   ERROR [StorageTemplateService] Object:
{
  "id": "1d61f711-f46f-42e0-a4db-8c97948d5137",
  "source": "upload/upload/3879145c-08e7-4ab6-a8e3-6ba3dbd57ffd/7fc24242-8351-464f-97e5-f4e2c1167393.jpeg",
  "destination": "upload/library/donald/2015/01/2015-01-26_17-38-58.jpeg"
}

[megapearl]

Just checked and the problem stil persist as of the latest version (v1.74.0), is there anything we can do to fix it or provide more information?

[fivestones]

I'm getting a very similar problem on a new install of 1.79.1

I did "edit user" on the admin user, and changed the "storage label". I then did a storage template migration job, and I'm getting very similar errors.

2023-09-24 00:38:09 {
2023-09-24 00:38:09   "id": "4e3485ce-15af-4eec-8554-87cf1dfab4df",
2023-09-24 00:38:09   "source": "upload/library/admin/2022/2022-10-16/FullSizeRender+6.jpg",
2023-09-24 00:38:09   "destination": "upload/library/david/2022/2022-10-16/FullSizeRender+7.jpg"
2023-09-24 00:38:09 }
2023-09-24 00:38:09 
2023-09-24 00:38:09 [Nest] 7  - 09/24/2023, 5:38:09 AM   ERROR [StorageTemplateService] Problem applying storage template
2023-09-24 00:38:09 [Nest] 7  - 09/24/2023, 5:38:09 AM   ERROR [StorageTemplateService] Error: ENOENT: no such file or directory, rename 'upload/library/admin/2022/2022-10-16/FullSizeRender+5.jpg' -> 'upload/library/david/2022/2022-10-16/FullSizeRender+7.jpg'
2023-09-24 00:38:09 [Nest] 7  - 09/24/2023, 5:38:09 AM   ERROR [StorageTemplateService] Object:
2023-09-24 00:38:09 {
2023-09-24 00:38:09   "id": "e0a56c54-c1b8-4b6b-8c3f-e884ac526f98",
2023-09-24 00:38:09   "source": "upload/library/admin/2022/2022-10-16/FullSizeRender+5.jpg",
2023-09-24 00:38:09   "destination": "upload/library/david/2022/2022-10-16/FullSizeRender+7.jpg"
2023-09-24 00:38:09 }
2023-09-24 00:38:09 
2023-09-24 00:38:09 [Nest] 7  - 09/24/2023, 5:38:09 AM   ERROR [StorageTemplateService] Problem applying storage template
2023-09-24 00:38:09 [Nest] 7  - 09/24/2023, 5:38:09 AM   ERROR [StorageTemplateService] Error: ENOENT: no such file or directory, rename 'upload/library/admin/2022/2022-10-16/FullSizeRender+6.jpg' -> 'upload/library/david/2022/2022-10-16/FullSizeRender+7.jpg'
2023-09-24 00:38:09 [Nest] 7  - 09/24/2023, 5:38:09 AM   ERROR [StorageTemplateService] Object:
2023-09-24 00:38:09 {
2023-09-24 00:38:09   "id": "c30447f8-27b1-4795-8128-295b7c9276ea",
2023-09-24 00:38:09   "source": "upload/library/admin/2022/2022-10-16/FullSizeRender+6.jpg",
2023-09-24 00:38:09   "destination": "upload/library/david/2022/2022-10-16/FullSizeRender+7.jpg"
2023-09-24 00:38:09 }
2023-09-24 00:38:09 
2023-09-24 00:38:09 [Nest] 7  - 09/24/2023, 5:38:09 AM   ERROR [StorageTemplateService] Problem applying storage template
2023-09-24 00:38:09 [Nest] 7  - 09/24/2023, 5:38:09 AM   ERROR [StorageTemplateService] Error: ENOENT: no such file or directory, rename 'upload/library/admin/2022/2022-10-16/FullSizeRender+6.jpg' -> 'upload/library/david/2022/2022-10-16/FullSizeRender+7.jpg'
2023-09-24 00:38:09 [Nest] 7  - 09/24/2023, 5:38:09 AM   ERROR [StorageTemplateService] Object:

I think it made a set of errors like this for each photo in my collection that it needed to migrate.

I'm running on MacOS with docker desktop, and I have the volumes in my docker-compose file set as bind mounts. Not sure if that's the same thing as the NFS shares mentioned above.

[truppelito]

Is this still an issue?

It was very recently (when I last checked). As soon as I find some time, I'll check again.

I can confirm this is still very much an issue on 1.81.

[sonce]

[Nest] 7 - 10/18/2023, 3:00:56 AM ERROR [StorageTemplateService] Problem applying storage template [Nest] 7 - 10/18/2023, 3:00:56 AM ERROR [StorageTemplateService] Error: EIO: i/o error, rename 'upload/library/admin/2022/2022-06-19/IMG_20190808_092546.jpg' -> 'upload/library/admin/2019/2019-08-08/IMG_20190808_092546.jpg' [Nest] 7 - 10/18/2023, 3:00:56 AM ERROR [StorageTemplateService] Object: { "id": "1d687557-b6eb-4c68-ba6f-5efb932e023d", "oldPath": "upload/library/admin/2022/2022-06-19/IMG_20190808_092546.jpg", "newPath": "upload/library/admin/2019/2019-08-08/IMG_20190808_092546.jpg" } same issue on 1.82

[alextran1502]

Hello is this issue still relevance? if not, can you help me close it?

[truppelito]

Yes, it unfortunately is. I still get orphaned files in the upload folder, as I described above.

[megapearl]

Problem still persists as of the latest version.

[evanreichard]

Can confirm that this still is an issue on 1.85.0

[drifter75]

still an issue, using v1.93.3.

[truppelito]

Can confirm (1.93.3)

[gtsteffaniak]

Here is the error I get:

[Nest] 8  - 01/31/2024, 1:55:27 PM     LOG [StorageCore] Attempting to finish incomplete move: upload/upload/3744c11d-3a82-4c0e-b187-66c8f2066556/d7/df/d7dfc991-01f1-4505-a74d-52ddcdc077b1.jpg => upload/library/admin/2021/November/IMG-20210930-WA0014.jpg
[Nest] 8  - 01/31/2024, 1:55:27 PM     LOG [StorageCore] Found file at old location
[Nest] 8  - 01/31/2024, 1:55:27 PM    WARN [StorageCore] Unable to complete move. Error renaming file with code ENOENT and message: ENOENT: no such file or directory, rename 'upload/upload/3744c11d-3a82-4c0e-b187-66c8f2066556/d7/df/d7dfc991-01f1-4505-a74d-52ddcdc077b1.jpg' -> 'upload/library/admin/2021/November/IMG-20210930-WA0014.jpg'
[Nest] 8  - 01/31/2024, 1:55:27 PM     LOG [StorageTemplateService] Finished storage template migration

The actual file location is:

-rwxr-xr-x 1 root root 82K Jan 31 13:33 upload/upload/3744c11d-3a82-4c0e-b187-66c8f2066556/d7/df/d7dfc991-01f1-4505-a74d-52ddcdc077b1.jpg

Which appears to be correct

[gtsteffaniak]

I believe I figured out the root cause of the issue.

Since the upload location gets mounted both to the immich-server and immich-microservices , there is an issue with the sync if the filesystem speed is not fast enough.

In my example I was mounting them via docker-compose native cifs volume, which likely was creating two copies of the volume mount that differed slightly.

I was able to resolve this by using a filesystem path mount to the fileshare I mounted on the host.

I hopes this helps others understand why they are seeing this.

[truppelito]

@gtsteffaniak That may be one of the causes, but I don't think it's the only one. I use the ghcr.io/imagegenius/immich image and so the upload volume, despite also being mounted with cifs in docker, is only mounted once. Yet I still have this issue.

[evanreichard]

So I'm not able to duplicate my initial ticket anymore. It all seems to be working for me now as of v1.100.0.

Would love if anyone else can confirm that its no longer an issue.

---

Note: The below may be a consequence of my initial issue, but it totally might not be. I'm dropping details below in case it is related and others see the same.

The only issue I had (and I'm not sure if its a consequence of my initial issue), is some stale items in the move_history table. I was receiving move errors like:

[Nest] 7  - 04/01/2024, 2:33:13 AM    WARN [StorageCore] Unable to complete move. File does not exist at either location.
[Nest] 7  - 04/01/2024, 2:33:13 AM     LOG [StorageCore] Attempting to finish incomplete move: 
  upload/library034dc848-001a-472d-a925-a4bdc56a60bb/2023/2023-06-13/19-46-21-IMG_3663.HEIC => 
  upload/library/034dc848-001a-472d-a925a4bdc56a60bb/2023/2023-06-13/15-46-21-IMG_3663.HEIC

These items existed in the database (in the assets table), but at different originalPaths than the records noted in the move_history table. So when the mover tried to resume the stale move, it failed (resulting in the above error), because of differing paths (I assume).

I cleaned this up by clearing out the move_history table (PROCEED AT OWN RISK -> DELETE FROM move_history;) and rerunning the migration. The only issues I receive now are due to the following, which is expected:

Asset 6842a05f-747a-41e4-b664-266748429c1b missing exif info [...]

[truppelito]

I had a look and I still had files left over in my upload after updating to 1.100, so I tried DELETE FROM move_history; and running the migration again. I can say I still have files in my upload folder. I may try later to re-install Immich from zero and re-copying all my photos using the CLI to see if the issue in this ticket still happens.

[alextran1502]

@truppelito file in the upload folder can be partial uploaded file i.e. you press cancel while the upload progress is still running or the background worker is killed by the OS.

[truppelito]

@alextran1502 I see, so in that case what should be done with these left over files? If they are not migrated, does that mean they can be deleted from the upload folder? Would it make sense for Immich to handle that clean-up automatically?

[alextran1502]

@truppelito Yes, it can be deleted from the upload folder. However, depending on when you start using Immich, you should copy those files to a different location before deleting them, just in case.

It makes sense for Immich to handle this process automatically. We haven't looked into this yet, as the server doesn't know when the upload process is interrupted.

[megapearl]

I still see hundreds of errors, but maybe I need to clean up the upload folder.

[Nest] 7 - 04/02/2024, 11:25:11 AM ERROR [StorageTemplateService] Problem applying storage template [Nest] 7 - 04/02/2024, 11:25:11 AM ERROR [StorageTemplateService] QueryFailedError: duplicate key value violates unique constraint "UQ_newPath" at PostgresQueryRunner.query (/usr/src/app/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:219:19) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async InsertQueryBuilder.execute (/usr/src/app/node_modules/typeorm/query-builder/InsertQueryBuilder.js:106:33) at async SubjectExecutor.executeInsertOperations (/usr/src/app/node_modules/typeorm/persistence/SubjectExecutor.js:260:42) at async SubjectExecutor.execute (/usr/src/app/node_modules/typeorm/persistence/SubjectExecutor.js:92:9) at async EntityPersistExecutor.execute (/usr/src/app/node_modules/typeorm/persistence/EntityPersistExecutor.js:140:21) at async StorageCore.moveFile (/usr/src/app/dist/cores/storage.core.js:147:20) at async /usr/src/app/dist/services/storage-template.service.js:152:17 at async /usr/src/app/dist/repositories/database.repository.js:187:23 [Nest] 7 - 04/02/2024, 11:25:11 AM ERROR [StorageTemplateService] Object: { "id": "f2430741-3ea2-4cee-8f70-9ba9c0787dec", "oldPath": "upload/upload/bb6665e5-8944-4d93-a39d-829d9b759d29/776184b0-f5e6-481c-9570-7ffc4e3a4fb2.png", "newPath": "upload/library/inoka/2023/06/2023-06-28_10-30-21+2.png" } [Nest] 7 - 04/02/2024, 11:25:11 AM LOG [StorageCore] Attempting to finish incomplete move: upload/library/inoka/2023/06/2023-06-26_14-15-54.JPG => upload/library/inoka/2023/06/2023-06-26_16-15-54+1.JPG [Nest] 7 - 04/02/2024, 11:25:11 AM WARN [StorageCore] Unable to complete move. File does not exist at either location. [Nest] 7 - 04/02/2024, 11:25:11 AM ERROR [StorageTemplateService] Asset f7d9bc97-c088-4ea9-88e1-9e7138fc4fa7 missing exif info, skipping storage template migration [Nest] 7 - 04/02/2024, 11:25:11 AM ERROR [StorageTemplateService] Asset 4714f30a-1352-4b26-8e4c-cc41a27b2a17 missing exif info, skipping storage template migration

[evanreichard]

@megapearl - to me it looks like you have three issues.

1. Duplicate Key on UQ_newPath(move_history) table
2. Failure on incomplete move
3. Missing exif info (unrelated / expected)

I think both 1 and 2 are related to my additional notes in my last comment. 1 appears to be trying to save an item to the move_history table that already exists. 2 appears to be a stale move_history item. 3 is just a file that doesn't have exif data and is unrelated / expected.

I would recommend backing up your database before doing any of this. But I think if you cleared out your move_history table (i.e. failed file moves) your issues may go away.

---

I don't see any mention of "Error: EIO: i/o error, link" as referenced in the main ticket in your log.

[evanreichard]

Probably a different ticket, but there probably should be a way for Immich to automatically clean up stale records in the move_history table (i.e. records where both the newPath and oldPath don't exist at all) - maybe in the Admin "Repair" page?