Mispositioned imported face regions and missing person thumbnails

[wonx]

The bug

Yesterday I tested the recently implemented face import feature (this is something I had been waiting for a long time, thanks!), but I noticed some issues. Some face regions appear to be misplaced, so, when you hover over the person's thumbnail in the info sidebar, the face rectangle does not correspond to the person's face. In my case, These faces were tagged in Digikam (and in some cases, in Picasa).

Moreover, face thumbnails from these pictures are not generated, and the affected person won't appear in the People sidebar.

Seems to particularly affect photos taken from iPhones (mostly vertical pictures), in occasions it affects other devices, I just found the issue in pictures taken from a motorola moto g71 5G and a SONY ILCE-5000.

I've seen that behavior before. Digikam also had this bug in the past: https://bugs.kde.org/show_bug.cgi?id=372340 And same for pigallery2: https://github.com/bpatrik/pigallery2/issues/515

I can provide picture samples if needed, although I'd prefer to do that in private.

The OS that Immich Server is running on

Debian 12

Version of Immich Server

v1.114.0

Version of Immich Mobile App

1.113.1 build.157

Platform with the issue

• [ ] Server
• [X] Web
• [ ] Mobile

Your docker-compose.yml content

#
# WARNING: Make sure to use the docker-compose.yml of the current release:
#
# https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml
#
# The compose file on main may not be compatible with the latest release.
#

name: immich

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    # extends:
    #   file: hwaccel.transcoding.yml
    #   service: cpu # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding
    volumes:
      # Do not edit the next line. If you want to change the media storage location on your system, edit the value of UPLOAD_LOCATION in the .env file
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
      - /mnt/md0/Media/Fotos:/Biblioteques/Fotos:ro
      - /mnt/md0/Personal/Marc/Fotos:/Biblioteques/Fotos Marc:ro
      - /mnt/md0/mountpoints/shared_with_marc:/Biblioteques/Fotos Nextcloud Shared Marc:ro
    env_file:
      - stack.env
    ports:
      - 2283:3001
    depends_on:
      - redis
      - database
    restart: always
    healthcheck:
      disable: false

  immich-machine-learning:
    container_name: immich_machine_learning
    # For hardware acceleration, add one of -[armnn, cuda, openvino] to the image tag.
    # Example tag: ${IMMICH_VERSION:-release}-cuda
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration
    #   file: hwaccel.ml.yml
    #   service: cpu # set to one of [armnn, cuda, openvino, openvino-wsl] for accelerated inference - use the `-wsl` version for WSL2 where applicable
    volumes:
      - model-cache:/cache
    env_file:
      - stack.env
    restart: always
    healthcheck:
      disable: false

  redis:
    container_name: immich_redis
    image: docker.io/redis:6.2-alpine@sha256:e3b17ba9479deec4b7d1eeec1548a253acc5374d68d3b27937fcfe4df8d18c7e
    healthcheck:
      test: redis-cli ping || exit 1
    restart: always

  database:
    container_name: immich_postgres
    image: docker.io/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      POSTGRES_INITDB_ARGS: '--data-checksums'
    volumes:
      # Do not edit the next line. If you want to change the database storage location on your system, edit the value of DB_DATA_LOCATION in the .env file
      - ${DB_DATA_LOCATION}:/var/lib/postgresql/data
    healthcheck:
      test: pg_isready --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' || exit 1; Chksum="$$(psql --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' --tuples-only --no-align --command='SELECT COALESCE(SUM(checksum_failures), 0) FROM pg_stat_database')"; echo "checksum failure count is $$Chksum"; [ "$$Chksum" = '0' ] || exit 1
      interval: 5m
      #start_interval: 30s
      start_period: 5m
    command: ["postgres", "-c", "shared_preload_libraries=vectors.so", "-c", 'search_path="$$user", public, vectors', "-c", "logging_collector=on", "-c", "max_wal_size=2GB", "-c", "shared_buffers=512MB", "-c", "wal_compression=on"]
    restart: always

volumes:
  model-cache:

Your .env content

UPLOAD_LOCATION=/opt/appdata/immich/library
DB_DATA_LOCATION=/opt/appdata/immich/postgres
IMMICH_VERSION=release
DB_PASSWORD=<redacted>
DB_USERNAME=postgres
DB_DATABASE_NAME=immich

Reproduction steps

1. Upgraded immich to version v1.114.0
2. Re-imported metadata (Administration, Jobs, Extract Metadata, All)
3. Wait for the import to end.
4. Open pictures with people faces. ...

Relevant log output

No response

Additional information

No response

[mertalev]

Hmmm, I guess we need to handle orientation.

[foux]

I can confirm the issue with face positioning, I have the same problem

[tofutim]

@wonx in picasa-snap on Ubuntu 24, I'm stuck "starting application.." - hoping to do face id there

[Hixxey]

I just tested this and get similar results. The face region from Digikam is in the wrong position.

[wonx]

@tofutim

Mmm, something must have broken the compatibility in Ubuntu 24.04. Do you mind opening a bug report (https://github.com/wonx/picasasnap/issues) and I'll have a look when I find a moment?

[gschintgen]

FWIW I can confirm that face import (which is great to have!) has some issues.

In more detail: I'm using Digikam to do the actual photo management and import my library as external library into Immich. Immich had already scanned the images before the face tag import feature was implemented, so the pictures do have face regions in Immich's database, but most of them stayed anonymous. I've now updated Immich to 1.123.0 and re-run metadata import and as far as I can tell it worked fine to import the names and face regions.

BUT:

1. The faces now have two almost identical regions shown in the pictures' details: one anonymous by Immich, one imported. I suppose I can fix this somehow by disabling the face detection feature and then running the face detection reset job to clear out Immich's detected regions. May be this could be handled a bit nicer? (E.g. recognize the regions as identical when they're overlapping by >80%)
2. This is what's really bothering me: All of the imported faces have broken thumbnails. "Error loading image". Edit: Changing it via "Select featured photo" fixes it, so that's great, but there are lots of people in my library... It seems that Immich neglects to "randomly" assign a suitable thumbnail to imported people.

[mertalev]

The faces now have two almost identical regions shown in the pictures' details: one anonymous by Immich, one imported. I suppose I can fix this somehow by disabling the face detection feature and then running the face detection reset job to clear out Immich's detected regions. May be this could be handled a bit nicer? (E.g. recognize the regions as identical when they're overlapping by >80%)

There is an overlap check, but it's in the face detection code that runs after metadata extraction. It doesn't handle the reverse order (but it ideally should).

I'd have to look into the issue with thumbnails, but my hunch is that the timing of the jobs is off - maybe the person thumbnail job runs before metadata extraction finishes, etc.

[gschintgen]

I'd have to look into the issue with thumbnails, but my hunch is that the timing of the jobs is off

I've now manually started the face detection refresh job and it fixes the missing thumbnails! But it will take quite some time to process all the pictures, so I can't yet tell if it will also merge the overlapping face regions. But at least the eye sore is gone :-)

[Dulanic]

The faces now have two almost identical regions shown in the pictures' details: one anonymous by Immich, one imported. I suppose I can fix this somehow by disabling the face detection feature and then running the face detection reset job to clear out Immich's detected regions. May be this could be handled a bit nicer? (E.g. recognize the regions as identical when they're overlapping by >80%)

There is an overlap check, but it's in the face detection code that runs after metadata extraction. It doesn't handle the reverse order (but it ideally should).

I'd have to look into the issue with thumbnails, but my hunch is that the timing of the jobs is off - maybe the person thumbnail job runs before metadata extraction finishes, etc.

Good to know this is on the radar. I also find that if I add tags in digikam, it can pull in overlapping faces. As a note also, refresh faces still seems to allow the overlap.

[dionorgua]

I've found at least one photo from Samsung S20 phone that Immich shows mispositioned Digikam faces. This photo has Orientation : Rotate 180 in exiftool (originally written by phone).

I'm sure that it's not edited after initial capture.

So it's 100% not Apple-specific

[skatsubo]

As for the mispositioned imported face regions, I've created a repo https://github.com/skatsubo/exif-orientation-vs-face-regions with sample images in various orientations: Rotate 90, Rotate 270, Rotate 180 to serve as test images for the issue.

I see the issue is labeled "good first issue", so I would like to try creating a fix/PR in the following weeks, depending on its difficulty (having virtually zero previous experience with typescript).