Extract metadata job not refreshing pictures date/time

[lgrawet]

The bug

Hello,

Full metadata extraction job or metadata refresh of a single picture doesn't refresh/date time. This is not a caching issue like https://github.com/immich-app/immich/issues/4860 as I'm using private browsing mode on a desktop computer to view the results. In my case, time has to be corrected because of a fix in exiftool-vendord. An fresh import of the picture in another test account doesn't show the problem. See this thread for reference.

Thanks a lot

The OS that Immich Server is running on

Debian 12 Docker Host

Version of Immich Server

v1.120.1

Version of Immich Mobile App

v1.120.1

Platform with the issue

• [X] Server
• [ ] Web
• [ ] Mobile

Your docker-compose.yml content

name: immich

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    extends:
      file: hwaccel.transcoding.yml
    #   service: cpu # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding
      service: quicksync
    volumes:
      # Do not edit the next line. If you want to change the media storage location on your system, edit the value of UPLOAD_LOCATION in the .env file
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    ports:
      - 2283:2283
    networks:
      - immich
      - back
    depends_on:
      - redis
      - database
    restart: always
    healthcheck:
      disable: false

  immich-machine-learning:
    container_name: immich_machine_learning
    # For hardware acceleration, add one of -[armnn, cuda, openvino] to the image tag.
    # Example tag: ${IMMICH_VERSION:-release}-cuda
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration
    #   file: hwaccel.ml.yml
    #   service: cpu # set to one of [armnn, cuda, openvino, openvino-wsl] for accelerated inference - use the `-wsl` version for WSL2 where applicable
    volumes:
      - model-cache:/cache
    env_file:
      - .env
    networks:
      - immich
    restart: always
    healthcheck:
      disable: false

  redis:
    container_name: immich_redis
    image: redis:6.2-alpine@sha256:2ba50e1ac3a0ea17b736ce9db2b0a9f6f8b85d4c27d5f5accc6a416d8f42c6d5
    healthcheck:
      test: redis-cli ping || exit 1
    networks:
      - immich
    restart: always

  database:
    container_name: immich_postgres
    image: docker.io/tensorchord/pgvecto-rs:pg16-v0.2.0@sha256:b33e0490b24fad3925294ee8d9e87b52ae64445e8772ad30f9f80091523795a5
    volumes:
      - type: volume
        source: db-postgresql16-data
        target: /var/lib/postgresql/data
    env_file: .env
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      POSTGRES_INITDB_ARGS: '--data-checksums'
      TZ: Europe/Brussels
    healthcheck:
      test: pg_isready --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' || exit 1; Chksum="$$(psql --dbname='${DB_DATABASE_NAME}' --username='$
{DB_USERNAME}' --tuples-only --no-align --command='SELECT COALESCE(SUM(checksum_failures), 0) FROM pg_stat_database')"; echo "checksum failure count is $$
Chksum"; [ "$$Chksum" = '0' ] || exit 1
      interval: 5m
      start_interval: 30s
      start_period: 5m
    command: ["postgres", "-c", "shared_preload_libraries=vectors.so", "-c", 'search_path="$$user", public, vectors', "-c", "logging_collector=on", "-c", 
"max_wal_size=2GB", "-c", "shared_buffers=512MB", "-c", "wal_compression=on"]
    networks:
      - back
    restart: always

volumes:
  model-cache:
  db-postgresql16-data:

networks:                                                                                                                                  
  immich:                                                             
    external: true                                                                                                                         
  back:                                                                                                                                    
    external: true

Your .env content

# You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables

# The location where your uploaded files are stored
UPLOAD_LOCATION=/storage/immich
# The location where your database files are stored
DB_DATA_LOCATION=./postgres

# To set a timezone, uncomment the next line and change Etc/UTC to a TZ identifier from this list: https://en.wikipedia.org/wiki/List_of_tz_database_time_
zones#List
# TZ=Etc/UTC
TZ=Europe/Brussels

# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=release

# Connection secret for postgres. You should change it to a random password
# Please use only the characters `A-Za-z0-9`, without special characters or spaces
DB_PASSWORD=*************

# The values below this line do not need to be changed
###################################################################################
DB_USERNAME=immich
DB_DATABASE_NAME=immich

Reproduction steps

I don't know how to reproduce this as initial bad picture time stored in the DB was due to an exiftool-vendord bug. You can find the original zipped picture in the aforementioned thread.

Relevant log output

No response

Additional information

No response

[alextran1502]

Hello, did you happen to change the files? Reupload the files? Or what actions did you take prior to regenerating the metadata, and what is the expectation?

[lgrawet]

Hello, the file F from account A has bad time due to a bug in exiftool-vendord. Immich is showing 13:35 UTC+1 but the correct time is 12:35 UTC+1 (look at exif metadata).

1. I upgraded immich. I tried to regenerate the metadada for this specific file from account A but no change (immich still showing 13:35 UTC+1).
2. Then I tried an import of file F on account B (not a partner) where the file was not present. Immich is showing the right time (12:35 UTC+1) so the fix is working.
3. Then I tried a full metadata extraction job as last resort

Results:

• Immich is still showing bad time (13:35) on account A
• Immich is still showing correct time (12:35) in account B

[alextran1502]

@lgrawet did you edit the time on account A by chance, so that it has an XMP file?

[C-Otto]

Do you trigger the job with the user who owns the file? If not, see #13615.

[lgrawet]

@alextran1502 I didn't edit the time on account A and I checked there is no xmp file, as mentioned on the thread. I've just checked again to be sure and there is no xmp file for file F. Wait, just to be sure, it is ${ASSET_ID}.xmp on the same level as ${ASSET_ID}.jpg ?

@C-Otto yes, the photo is owned by account A (not shared and no partnership). Account A triggered the rescan job. I also tried the full metadata rescan of the whole library from admin account

[danieldietzler]

Wait, just to be sure, it is ${ASSET_ID}.xmp on the same level as ${ASSET_ID}.jpg

It could also be ${ASSET_ID}.jpg.xmp in some cases, either way it's right next to the asset though correct. (assuming you don't use a storage template in which case the file name will be different)

[lgrawet]

Hello, I don't use storage template and I can confirm there is no *.xmp file at the same level of the asset.

[Blechlawine]

I think my issue might be related: I have some files in an external read only (i set it to read only when mounting it inside truenas) library, which when scanning the library initially had the wrong timestamp on the file (something went wrong when i copied these files onto my NAS). After the initial scan they showed up with the wrong timestamp in immich. I fixed the timestamps on the files themselves, and checking with the stat command in linux it shows the correct timestamp. After pressing refresh metadata in immich's web interface and waiting for the job to finish, they still show up with the old incorrect timestamp. Checking with pgAdmin in the database they still seem to have the incorrect timestamp. I checked the public.exif table and the columns dateTimeOriginal and modifyDate both have the incorrect timestamp. My immich instance only has one user who owns the external library and all the files affected.

I'm running immich inside TrueNAS Scale, but i don't think that that should affect this problem.

[danieldietzler]

I think my issue might be related: I have some files in an external read only (i set it to read only when mounting it inside truenas) library, which when scanning the library initially had the wrong timestamp on the file (something went wrong when i copied these files onto my NAS). After the initial scan they showed up with the wrong timestamp in immich. I fixed the timestamps on the files themselves, and checking with the stat command in linux it shows the correct timestamp. After pressing refresh metadata in immich's web interface and waiting for the job to finish, they still show up with the old incorrect timestamp. Checking with pgAdmin in the database they still seem to have the incorrect timestamp. I checked the public.exif table and the columns dateTimeOriginal and modifyDate both have the incorrect timestamp. My immich instance only has one user who owns the external library and all the files affected.

I'm running immich inside TrueNAS Scale, but i don't think that that should affect this problem.

stat gives you file system metadata. Immich however preferably uses exif metadata. You can check that with e.g. exiftool – I'm assuming that's wrong for you.

[Blechlawine]

stat gives you file system metadata. Immich however preferably uses exif metadata. You can check that with e.g. exiftool – I'm assuming that's wrong for you.

Ok i checked the same files with exiftool, which says the File Modification Date/Time, File Access Date/Time and File Inode Change Date/Time are the correct values, but Create Date, Modify Date, Track Create Date, Track Modify Date, Media Create Date and Media Modify Date are the incorrect values. Does immich not use the file dates when reading metadata?

[danieldietzler]

Does immich not use the file dates when reading metadata?

It does, but only if nothing else exists. Generally we assume exif data to be more likely to be correct.

[C-Otto]

@lgrawet could you please check if this is still an issue with a recent version?

[lgrawet]

@C-Otto Hello, just tried it and yes, it is still an issue on v1.125.3

[C-Otto]

Alright. Any idea how we can tackle this? I'm clueless, but I'd like to get this fixed.

[lgrawet]

Is there a way to look at the metadata task output ?

[C-Otto]

Yes, you can increase the log level and run the "Refresh metadata" job from the asset's context menu. The log settings are in the admin section (web), in settings, logging. If you set this to "verbose", you should see quite a bit of information in the logs.

[lgrawet]

Hello @C-Otto,

Here is the immich-server log output for the refresh metadata task. I hope it can help.

Laurent

immich-server.log

[C-Otto]

Note that the information is read from a sidecar (XMP) file, not the photo (jpg?) itself!

(Most likely) correct:

• tz: Europe/Brussels (determined via GPS coordinates)
• GPSTimeStamp: 11:35:40 UTC
• SubSecModifyDate: 12:35:41 Europe/Brussels
• GPSDateTime: 11:35:40 UTC
• ModifyDate: 12:35:41 Europe/Brussels

incorrect:

• DateTimeOriginal: 12:35:40 UTC
• FileModifyDate: 13:35:40+01:00

Could you delete the XMP file and try again? Immich (or rather exiftool-vendored) has a list of tags that are read, and the first hit is used to determine the time. In this case the first hit from that list is DateTimeOriginal which, indeed, contains the wrong time.

[lgrawet]

You're right, I removed the xmp and it worked! What is frustrating is I already looked up for an xmp file but at the same level of the picture and it is not (see previous posts). I don"t know where it come from but the xmp file date is also strange to me. The date is several months anterior to the immich install but exactly the same of the jpg asset. Does it come from the smartphone ? There are 7883 jpg files on the account and 5709 xmp files but the pictures come from two different smartphones

[lgrawet]

I think the culprit is immich-go. I have to use it to import Google data but I think I've also used it to import smartphone pictures on that account. There is this bug : https://github.com/simulot/immich-go/issues/488 The dates match my problem. At that time, it used date/time encoded in filename instead of Exif.

[C-Otto]

Alright, looks like this can be closed then. Thanks for the update!