immich
immich copied to clipboard
immich-app
handleGeneratePersonThumbnail incorrectly crops if thumbnail is regenerated with a different size
The bug
I reduced large thumbnail sizes and noticed that after thumbnails were regenerated the faces detected off the original image no longer generate thumbnails correctly.
Looking at the values for those faces still showed that the imageHeight and imageWidth are still the size of the original thumbnail and the bounding boxes are still relative to that, which makes sense, the face detection was ran against the original thumbnail, but face cropping still tries to happen on a differently sized thumbnail against those parameters.
The border that's drawn on the face when you hover over it in the photo viewer does correctly account for that though by adjusting for the height/width of the image. https://github.com/immich-app/immich/blob/main/web/src/lib/utils/people-utils.ts#L44
Something similar is never done in handleGeneratePersonThumbnail as far as I can tell.
Some of them outright fail, I assume from trying to crop out of bounds.
[Nest] 7 - 03/28/2024, 12:49:39 AM VERBOSE [PersonService] Cropping face for person: 7f3a3e7c-fbcf-427e-8aeb-88373c83a2e9
[Nest] 7 - 03/28/2024, 12:49:39 AM ERROR [JobService] Unable to run job handler (thumbnailGeneration/generate-person-thumbnail): Error: extract_area: bad extract area
[Nest] 7 - 03/28/2024, 12:49:39 AM ERROR [JobService] Error: extract_area: bad extract area
at Sharp.toBuffer (/usr/src/app/node_modules/sharp/lib/output.js:161:17)
at MediaRepository.crop (/usr/src/app/dist/infra/repositories/media.repository.js:36:14)
at PersonService.handleGeneratePersonThumbnail (/usr/src/app/dist/domain/person/person.service.js:389:58)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async /usr/src/app/dist/domain/job/job.service.js:137:36
at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:394:28)
at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24)
Happy to get a development setup up and put in a fix if people think the right approach here is to just normalize against the size of the asset it's actually cropping.
The OS that Immich Server is running on
Debian
Version of Immich Server
v1.99
Version of Immich Mobile App
v1.99
Platform with the issue
- [X] Server
- [ ] Web
- [ ] Mobile
Your docker-compose.yml content
version: '3.8'
# https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml
name: immich
services:
immich-server:
container_name: immich_server
image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
command: ['start.sh', 'immich']
volumes:
- ${DIR_PRIVATE}/rng/Photos/Immich:/usr/src/app/upload
- ${DIR_PRIVATE}/rng/Photos/Lightroom/Catalog:/mnt/media/lightroom:ro
- /etc/localtime:/etc/localtime:ro
env_file:
- ${DOCKER_CONFIG}/compose/immich/immich.env
environment:
- TZ=${TZ}
ports:
- 2283:3001
depends_on:
- redis
- database
user: ${PUID}:${PGID}
restart: unless-stopped
immich-microservices:
container_name: immich_microservices
image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
# extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/hardware-transcoding
# file: hwaccel.transcoding.yml
# service: cpu # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding
command: ['start.sh', 'microservices']
volumes:
- ${DIR_PRIVATE}/rng/Photos/Immich:/usr/src/app/upload
- ${DIR_PRIVATE}/rng/Photos/Lightroom/Catalog:/mnt/media/lightroom:ro
- /etc/localtime:/etc/localtime:ro
env_file:
- ${DOCKER_CONFIG}/compose/immich/immich.env
environment:
- TZ=${TZ}
depends_on:
- redis
- database
user: ${PUID}:${PGID}
restart: unless-stopped
immich-machine-learning:
container_name: immich_machine_learning
# For hardware acceleration, add one of -[armnn, cuda, openvino] to the image tag.
# Example tag: ${IMMICH_VERSION:-release}-cuda
image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
# extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration
# file: hwaccel.ml.yml
# service: cpu # set to one of [armnn, cuda, openvino, openvino-wsl] for accelerated inference - use the `-wsl` version for WSL2 where applicable
volumes:
- ${DOCKER_CONFIG}/immich/model-cache:/cache
env_file:
- ${DOCKER_CONFIG}/compose/immich/immich.env
restart: unless-stopped
redis:
container_name: immich_redis
image: registry.hub.docker.com/library/redis:6.2-alpine@sha256:51d6c56749a4243096327e3fb964a48ed92254357108449cb6e23999c37773c5
restart: unless-stopped
database:
container_name: immich_postgres
image: registry.hub.docker.com/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
environment:
- POSTGRES_PASSWORD=${DB_PASSWORD}
- POSTGRES_USER=${DB_USERNAME}
- POSTGRES_DB=${DB_DATABASE_NAME}
- TZ=${TZ}
volumes:
- ${DOCKER_CONFIG}/immich/pgdata:/var/lib/postgresql/data
restart: unless-stopped
Your .env content
IMMICH_VERSION=release
DB_PASSWORD=*****
DB_HOSTNAME=immich_postgres
DB_USERNAME=*****
DB_DATABASE_NAME=immich
REDIS_HOSTNAME=immich_redis
Reproduction steps
1. Generate thumbnails for an image
2. Run face detection
3. Change large thumbnail resolution
4. Regenerate thumbnail
5. Change feature photo for person in image
Additional information
No response
Great catch! There's a PR (#7513) that scales these, but for a different reason. It's currently blocked because there was a bug raised that I can't reproduce.
Great catch! There's a PR (#7513) that scales these, but for a different reason. It's currently blocked because there was a bug raised that I can't reproduce.
Oh yeah, that looks good! that seems like that would resolve this issue anyway.
Hey @raymondnumbergenerator, could you verify it's solved by #7513? :)
I don't know if it was in logs before but since upgrade to 106 I noticed the issue:
immich_server | [Nest] 7 - 06/13/2024, 5:48:30 PM ERROR [Microservices:JobService] Unable to run job handler (thumbnailGeneration/generate-person-thumbnail): Error: extract_area: bad extract area
immich_server | [Nest] 7 - 06/13/2024, 5:48:30 PM ERROR [Microservices:JobService] Error: extract_area: bad extract area
immich_server | at Sharp.toFile (/usr/src/app/node_modules/sharp/lib/output.js:89:19)
immich_server | at MediaRepository.generateThumbnail (/usr/src/app/dist/repositories/media.repository.js:69:14)
immich_server | at PersonService.handleGeneratePersonThumbnail (/usr/src/app/dist/services/person.service.js:412:36)
immich_server | at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
immich_server | at async /usr/src/app/dist/services/job.service.js:148:36
immich_server | at async Worker.processJob (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:394:28)
immich_server | at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24)
immich_server | [Nest] 7 - 06/13/2024, 5:48:30 PM ERROR [Microservices:JobService] Object:
immich_server | {
immich_server | "id": "d68d6a7f-8567-4622-80d0-c3c867fe31bc"
immich_server | }
Yes 1.106.4 Thumbnail from videos are fixed since yesterday, but I have the extract_area issue.
How many of those errors do you see? Could you share an image that has that issue? Also, is your preview resolution default or did you change it?
You can get some info about the asset for the failed job you shared by running this:
docker compose exec -it database psql -U postgres -c "SELECT jsonb_pretty(jsonb_build_object('id', a.id, 'type', a.type, 'height', e.\"exifImageHeight\", 'width', e.\"exifImageWidth\", 'originalPath', a.\"originalPath\", 'previewPath', a.\"previewPath\", 'personThumbnailPath', p.\"thumbnailPath\")) as pretty FROM assets a INNER JOIN exif e ON a.id = e.\"assetId\" INNER JOIN asset_faces af on a.id = af.\"assetId\" INNER JOIN person p on af.id = p.\"faceAssetId\" WHERE p.id = 'd68d6a7f-8567-4622-80d0-c3c867fe31bc'" immich
I count 14 errors.
Here is one of the pictures:
On the person pages I checked, I noticed that every time it's with portrait pictures but the preview is cropped as paysage:
Even it the thumbnail is portrait.
On the photo information, we can see that other person thumbnail is misplaced:
The query for corresponding person uuid:
$ docker compose exec -it database psql -U postgres -c "SELECT jsonb_pretty(jsonb_build_object('id', a.id, 'type', a.type, 'height', e.\"exifImageHeight\", 'width', e.\"exifImageWidth\", 'originalPath', a.\"originalPath\", 'previewPath', a.\"previewPath\", 'personThumbnailPath', p.\"thumbnailPath\")) as pretty FROM assets a INNER JOIN exif e ON a.id = e.\"assetId\" INNER JOIN asset_faces af on a.id = af.\"assetId\" INNER JOIN person p on af.id = p.\"faceAssetId\" WHERE p.id = '6567188a-8dd9-470f-b371-515e191b0a22'" immich
pretty
----------------------------------------------------------------------------------------------------------------------------------
{ +
"id": "43ed4128-0ef4-4bf6-8abe-95343d53d088", +
"type": "IMAGE", +
"width": 2448, +
"height": 3264, +
"previewPath": "upload/thumbs/d38e2bf4-3df6-420c-bba9-a17731d74132/43/ed/43ed4128-0ef4-4bf6-8abe-95343d53d088-preview.jpeg",+
"originalPath": "/usr/src/app/external/path/to/image/Thaïlande (15).jpg", +
"personThumbnailPath": "" +
}
(1 row)
That's interesting, this thumbnail has apparently never been generated successfully. Do you know around when this asset was added to Immich? You can add 'createdAt', a."createdAt" to the start of jsonb_build_object in that command to know for sure.
It's from external library added when I created my instance of immich. "createdAt": "2024-04-20T22:09:08.543505+00:00"
There is a previewPath and the preview in person view is a webp file (didn't attach it as not supported by GitHub).
Is there a way to reset thumbnail?
I launched a /api/assets/jobs with "name": "regenerate-thumbnail" and one of the image with issue, got the Successfully generated JPEG image preview for asset & Successfully generated WEBP image thumbnail for asset in the logs but it doesn't change a thing.
Info from a GET /api/assets/43ed4128-0ef4-4bf6-8abe-95343d53d088:
{
"id": "43ed4128-0ef4-4bf6-8abe-95343d53d088",
"deviceAssetId": "Thaïlande(15).jpg",
"ownerId": "d38e2bf4-3df6-420c-bba9-a17731d74132",
"owner": {
"id": "d38e2bf4-3df6-420c-bba9-a17731d74132",
"email": "***",
"name": "***",
"profileImagePath": "upload/profile/d38e2bf4-3df6-420c-bba9-a17731d74132/3d155a5f-2e1b-4d06-98b6-50c2d591ebbd.jpg",
"avatarColor": "primary"
},
"deviceId": "Library Import",
"libraryId": "6576fc40-c645-45b5-aff7-8e01c72fdf33",
"type": "IMAGE",
"originalPath": "/usr/src/app/external/path/to/image/Thaïlande (15).jpg",
"originalFileName": "Thaïlande (15).jpg",
"originalMimeType": "image/jpeg",
"resized": true,
"thumbhash": "HAgaDQJ4iJ+Hp3eXeHeIZ8Rvjaz0",
"fileCreatedAt": "2013-01-08T16:35:55.000Z",
"fileModifiedAt": "2021-05-23T11:02:20.496Z",
"localDateTime": "2013-01-08T16:35:55.000Z",
"updatedAt": "2024-04-21T19:52:10.976Z",
"isFavorite": false,
"isArchived": false,
"isTrashed": false,
"duration": "0:00:00.00000",
"exifInfo": {
"make": "Panasonic",
"model": "DMC-TZ10",
"exifImageWidth": 2448,
"exifImageHeight": 3264,
"fileSizeInByte": 2877409,
"orientation": "6",
"dateTimeOriginal": "2013-01-08T16:35:55.000Z",
"modifyDate": "2013-01-08T16:35:55.000Z",
"timeZone": null,
"lensModel": null,
"fNumber": 3.3,
"focalLength": 4.1,
"iso": 125,
"exposureTime": "1/60",
"latitude": null,
"longitude": null,
"city": null,
"state": null,
"country": null,
"description": "",
"projectionType": null
},
"livePhotoVideoId": null,
"tags": [],
"people": [
{
"id": "15a4a281-3870-4176-b290-e0831d9518a4",
"name": "",
"birthDate": null,
"thumbnailPath": "upload/thumbs/d38e2bf4-3df6-420c-bba9-a17731d74132/15/a4/15a4a281-3870-4176-b290-e0831d9518a4.jpeg",
"isHidden": false,
"faces": [
{
"id": "fb8c84bd-b210-444d-a79a-8e1ea43b392c",
"imageHeight": 1920,
"imageWidth": 1440,
"boundingBoxX1": 468,
"boundingBoxX2": 555,
"boundingBoxY1": 1002,
"boundingBoxY2": 1142
}
]
},
{
"id": "6567188a-8dd9-470f-b371-515e191b0a22",
"name": "",
"birthDate": null,
"thumbnailPath": "",
"isHidden": false,
"faces": [
{
"id": "8bbb5fcf-609b-49f3-bbff-c43097e39ba6",
"imageHeight": 1920,
"imageWidth": 1440,
"boundingBoxX1": 1005,
"boundingBoxX2": 1085,
"boundingBoxY1": 997,
"boundingBoxY2": 1101
}
]
}
],
"unassignedFaces": [],
"checksum": "wsW5qd6Mo/RtSQjZLU8zrnZ+WYE=",
"stackCount": null,
"isOffline": false,
"hasMetadata": true,
"duplicateId": null
}
I reproduced the issue with that image. There's a discrepancy between when we flip the dimensions based on orientation and when sharp does. In this case, we flip width and height for this image while sharp doesn't. The fix is simple - we know what sharp will orient it to since we have the preview dimensions, so we can just make the crop dimensions consistent with that. Doing this worked for everything when I ran facial recognition, including that image.
Will there be any procedure to fix the processed images or it will by itself when launching the thumbnail job?
With the fix in the next release, you can change the feature photo for a person (it can be the same image) and it will re-generate that person's thumbnail. Re-running thumbnail generation on all assets will also fix it.