Cannot regenerate transcoded videos of Android MP.JPG files

[slackspace-io]

The bug

I recently had to rebuild my stack due to a disagreement with etcd.

As part I attempted to backup from PVs immich's database and photos. Specifically I rsync'd to the new volume the folders library, upload, and profile.

I am using a storage template.

Per the documentation, I did not keep encoded-videos folder as I thought one could regenerate. (Perhaps MPs are an exception ? :D)

I did successfully restore immich, including database and at first glance all my assets.

I however noticed some untracked files, and offline paths.

While investigating these, I noticed none of my motion photos were working. When attempting to play them on a client, you see file not found in the immich-server container. When running transcoding job, you see errors in micro-services about ffprobe.

I deleted the folder again, and now my most recent photos (today) have the same issue. Which makes me suspect this is reproducable.

(I have not found a good way to resolve this. I manually moved one photo out the sync folder for immich on my phone, then deleted/trash/empty in immich, then moved it back. This fixed that single photo.)

The OS that Immich Server is running on

ghcr container image, ontop an RKE2 cluster with ubuntu 22.04 as base image.

Version of Immich Server

v1.94.1

Version of Immich Mobile App

v1.94.1

Platform with the issue

• [X] Server
• [ ] Web
• [ ] Mobile

Your docker-compose.yml content

I am running kubernetes, I exploded from helm onetime fileset and maintain them now. 

Here is deployment for server;
apiVersion: apps/v1
kind: Deployment
metadata:
  name: immich-server
  labels:
    app.kubernetes.io/instance: immich
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: server
    app.kubernetes.io/version: v1.91.4
    helm.sh/chart: immich-0.3.1
spec:
  revisionHistoryLimit: 3
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app.kubernetes.io/name: server
      app.kubernetes.io/instance: immich
  template:
    metadata:
      labels:
        app.kubernetes.io/name: server
        app.kubernetes.io/instance: immich
    spec:

      serviceAccountName: default
      automountServiceAccountToken: true
      dnsPolicy: ClusterFirst
      enableServiceLinks: true
      containers:
        - name: immich-server
          image: ghcr.io/immich-app/immich-server:v1.94.1
          imagePullPolicy: IfNotPresent
          command:
            - "/bin/sh"
          args:

            - ./start-server.sh
          env:
            - name: DB_DATABASE_NAME
              value: immich
            - name: DB_HOSTNAME
              value: postgres-service
            - name: DB_PASSWORD
              value: supersecureipromise
            - name: DB_USERNAME
              value: immich
            #- name: IMMICH_MACHINE_LEARNING_URL
            #  value: http://immich-machine-learning:3003
            - name: REDIS_HOSTNAME
              value: redis-service
            - name: IMMICH_CONFIG_FILE
              value: /usr/src/app/immich-config.json
          ports:
            - name: http
              containerPort: 3001
              protocol: TCP
          volumeMounts:
            - name: immich-config
              mountPath: /usr/src/app/immich-config.json
              subPath: immich-config.json
            - name: library
              mountPath: /usr/src/app/upload
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /server-info/ping
              port: http
            initialDelaySeconds: 0
            periodSeconds: 10
            timeoutSeconds: 1
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /server-info/ping
              port: http
            initialDelaySeconds: 0
            periodSeconds: 10
            timeoutSeconds: 1
      nodeSelector:
        immich: server
      volumes:
        - name: immich-config
          configMap:
            name: immich-config
        - name: library
          persistentVolumeClaim:
            claimName: immich-pvc

here is microservice
---
# Source: immich/templates/microservices.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: immich-microservices
  labels:
    app.kubernetes.io/instance: immich
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: microservices
    app.kubernetes.io/version: v1.91.4
    helm.sh/chart: immich-0.3.1
spec:
  revisionHistoryLimit: 3
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app.kubernetes.io/name: microservices
      app.kubernetes.io/instance: immich
  template:
    metadata:
      labels:
        app.kubernetes.io/name: microservices
        app.kubernetes.io/instance: immich
    spec:

      serviceAccountName: default
      automountServiceAccountToken: true
      dnsPolicy: ClusterFirst
      enableServiceLinks: true
      containers:
        - name: immich-microservices
          image: ghcr.io/immich-app/immich-server:v1.94.1
          imagePullPolicy: IfNotPresent
          command:
            - "/bin/sh"
          args:

            - ./start-microservices.sh
          env:
            - name: DB_DATABASE_NAME
              value: immich
            - name: DB_HOSTNAME
              value: postgres-service
            - name: DB_PASSWORD
              value: totallysecurepasswordfromearlier
            - name: DB_USERNAME
              value: immich
            - name: IMMICH_MACHINE_LEARNING_URL
              value: http://immich-machine-learning:3003
            - name: REDIS_HOSTNAME
              value: redis-service
            - name: IMMICH_CONFIG_FILE
              value: /usr/src/app/immich-config.json
          volumeMounts:
            - name: immich-config
              mountPath: /usr/src/app/immich-config.json
              subPath: immich-config.json
            - name: library
              mountPath: /usr/src/app/upload

      nodeSelector:
        immich: server
      volumes:
        - name: library
          persistentVolumeClaim:
            claimName: immich-pvc
        - name: immich-config
          configMap:
            name: immich-config


If you want the ML or PVCs let me know.. guessing it's not relevant tho :)

Your .env content

{
  "ffmpeg": {
    "crf": 23,
    "threads": 2,
    "preset": "ultrafast",
    "targetVideoCodec": "h264",
    "acceptedVideoCodecs": [
      "h264"
    ],
    "targetAudioCodec": "aac",
    "acceptedAudioCodecs": [
      "aac"
    ],
    "targetResolution": "720",
    "maxBitrate": "0",
    "bframes": -1,
    "refs": 0,
    "gopSize": 0,
    "npl": 0,
    "temporalAQ": false,
    "cqMode": "auto",
    "twoPass": false,
    "preferredHwDevice": "auto",
    "transcode": "required",
    "tonemap": "hable",
    "accel": "disabled"
  },
  "job": {
    "backgroundTask": {
      "concurrency": 5
    },
    "smartSearch": {
      "concurrency": 6
    },
    "metadataExtraction": {
      "concurrency": 5
    },
    "faceDetection": {
      "concurrency": 6
    },
    "search": {
      "concurrency": 5
    },
    "sidecar": {
      "concurrency": 5
    },
    "library": {
      "concurrency": 5
    },
    "migration": {
      "concurrency": 5
    },
    "thumbnailGeneration": {
      "concurrency": 5
    },
    "videoConversion": {
      "concurrency": 1
    }
  },
  "logging": {
    "enabled": true,
    "level": "log"
  },
  "machineLearning": {
    "enabled": true,
    "url": "http://immich-machine-learning:3003",
    "clip": {
      "enabled": true,
      "modelName": "ViT-B-32__openai"
    },
    "facialRecognition": {
      "enabled": true,
      "modelName": "buffalo_l",
      "minScore": 0.6,
      "maxDistance": 0.5,
      "minFaces": 3
    }
  },
  "map": {
    "enabled": true,
    "lightStyle": "",
    "darkStyle": ""
  },
  "reverseGeocoding": {
    "enabled": true
  },
  "oauth": {
 #stuff
  },
  "passwordLogin": {
    "enabled": true
  },
  "storageTemplate": {
    "enabled": true,
    "hashVerificationEnabled": true,
    "template": "{{y}}/{{y}}-{{MM}}-{{dd}}/{{filename}}"
  },
  "thumbnail": {
    "webpSize": 250,
    "jpegSize": 1440,
    "quality": 80,
    "colorspace": "p3"
  },
  "newVersionCheck": {
    "enabled": true
  },
  "trash": {
    "enabled": true,
    "days": 30
  },
  "theme": {
    "customCss": ""
  },
  "library": {
    "scan": {
      "enabled": true,
      "cronExpression": "0 0 * * *"
    },
    "watch": {
      "enabled": false,
      "usePolling": false,
      "interval": 10000
    }
  },
  "server": {
    "externalDomain": "",
    "loginPageMessage": ""
  }
}

Reproduction steps

I believe to reproduce:

- Take a MP photo, allow sync to occur.
- Confirm playback in a client.
- Delete encoded-video directory (or that specific asset)
- Rerun transcoding job

Confirm playback no longer functions on client.-
Observe in micro-services log errors from ffprobe.
Observe in browser the playback does not function - with errors in immich-server logs

Additional information

No response

[jrasm91]

I believe if you run "all" it should have transcoded it. I will have to double check this.

[slackspace-io]

@jrasm91 I did try 'all' a few times but no success.

I was able to reproduce the situation in a local dev environment and reproducing the issue (Importing a single android MP, then deleting the transcoded file).

I started to look through the code, but uh life happened :) It looked like the handling of MP is a bit different than 'normal' transcodings? I don't think immich checks for presence of file anytime after the initial creation of MP so when it is deleted it is not detected compared to normal transcodings. (If I remember right when trying to walk through the code). That was about as far as I made it.

I could provide an example raw android MP photo that I can recreate the issue with if that be helpful, or if I do manage to make it further in triage will try to be more clear with my findings or even potential fix.

[oryjkov]

I think I ran into the same issue after restoring from a backup (not having encoded-videos backed up).

Immich reports a bunch of "offline files", all of them in the "encoded-videos" directory. Transcoding all videos does not help, in fact the transcoding container shows errors for missing input files itself.

Looked a bit further into what's missing and it seems that out of a handful of assets that I checked by hand all correspond to Motion Photos. Below is an example of a missing asset.

{
  "id": "7c6f0b58-a053-4b9d-90aa-06afbad8bfcb",
  "deviceAssetId": "NONE",
  "ownerId": "4f13d54e-b06a-48dc-8f7e-1d47fffe1425",
  "owner": {},
  "deviceId": "NONE",
  "libraryId": "98d52604-7214-4d76-9b70-9a98edd74ffd",
  "type": "VIDEO",
  "originalPath": "upload/encoded-video/4f13d54e-b06a-48dc-8f7e-1d47fffe1425/7c/6f/7c6f0b58-a053-4b9d-90aa-06afbad8bfcb-MP.mp4",
  "originalFileName": "PXL_20210721_090550242.MP.jpg",
  "resized": false,
  "thumbhash": null,
  "fileCreatedAt": "2021-07-21T09:05:50.242Z",
  "fileModifiedAt": "2021-07-21T09:05:50.000Z",
  "localDateTime": "2021-07-21T11:05:50.242Z",
  "updatedAt": "2024-05-14T16:25:39.635Z",
  "isFavorite": false,
  "isArchived": false,
  "isTrashed": false,
  "duration": "0:00:00.00000",
  "exifInfo": {
    "make": "Google",
    "model": "Pixel 5",
    "exifImageWidth": 3024,
    "exifImageHeight": 4032,
    "fileSizeInByte": 6718630,
    "orientation": "1",
    "dateTimeOriginal": "2021-07-21T09:05:50.242Z",
    "modifyDate": "2021-07-21T09:05:50.242Z",
    "timeZone": "UTC+2",
    "lensModel": null,
    "fNumber": 1.7,
    "focalLength": 4.38,
    "iso": 57,
    "exposureTime": "1/1799",
    "latitude": null,
    "longitude": null,
    "city": null,
    "state": null,
    "country": null,
    "description": "",
    "projectionType": null
  },
  "livePhotoVideoId": null,
  "tags": [],
  "people": [],
  "checksum": "G2ZpFz+A0tujR9Mqz73FiQS+OXQ=",
  "stackCount": null,
  "isOffline": false,
  "isExternal": false,
  "isReadOnly": false,
  "hasMetadata": true
}

[jrasm91]

Can you trying running extract metadata a broken asset? I believe this will fix the issue. If so, running that job for all assets should fix the rest.

[oryjkov]

Yeah, it did solve the issue, sorry I forgot to post here.

But I also noticed that some Motion Photo transcodes that it produced are broken. ffprobe fails to identify them, for instance:

immich_microservices     | [mov,mp4,m4a,3gp,3g2,mj2 @ 0x5a5e0a202c00] Format mov,mp4,m4a,3gp,3g2,mj2 detected only with low score of 1, misdetection possible!
immich_microservices     | [mov,mp4,m4a,3gp,3g2,mj2 @ 0x5a5e0a202c00] moov atom not found
immich_microservices     | upload/encoded-video/bf13aeb1-1ef7-4791-9d68-0fcd1cd8f2f8/55/3d/553dd991-7c3b-461e-b96a-709ec204349d-MP.mp4: Invalid data found when processing input
immich_microservices     |
immich_microservices     | [Nest] 20  - 06/04/2024, 1:42:48 PM   ERROR [ImmichMicroservices] [JobService] Error: ffprobe exited with code 1

Any idea what this is?

[jrasm91]

There were some bugs with the motion video extraction awhile ago. Re-running metadata extraction on those assets should detect and fix corrupt MP videos. Or, it could be a different error.

[oryjkov]

These things get re-created on every metadata rebuild.

Yes, it must be some bug with parsing motion photos. At least in all examples I saw, these photos came from a pixel phone and have ".MP.JPG" in the name. But, strangely, even google photos does not show these files as motion photos.

[jrasm91]

It goes something like this:

• Check if the file has metadata indicating a motion video
• Extract the video in memory and hash it
• If the asset is already linked,
• Verify the hash matches the currently extracted video
• On hash mismatches or no linked file, extract it to a new asset and link it.

Due to the prior bug with corrupt extracted videos we implemented this new process. At some point in the future we may remove the code for the extra checking/automatic fix.

[oryjkov]

Thanks for the summary. I think what's happening is that when it tries to extract the MP4 part out of the image (step 2 in your sequence), it somehow cuts out a part of that file that is not an MP4 and fails to do anything with it.

In any case, re-running metadata extraction fixed the motion photos that are actual motion photos.

As for the rest, I put what i have in https://github.com/immich-app/immich/issues/9993

[jrasm91]

Cool. I think this issue is resolved now then. You can regenerate motion photos by running metadata extraction for all.