immich
immich copied to clipboard
Published
20 hours ago •
immich-app
immich-app
[BUG] Videos uploaded from mobile app coming in as corrupt
The bug
I'm trying to upload all photos/videos on my Android device via mobile app (~7500 files). I'm noticing that when uploading videos from the Android mobile app (using VPN to access, no reverse proxy), in my immich upload folder there's a lot of videos that are corrupt and cannot be played on my Windows server. I'm assuming all videos on my Android device can be replayed, as they were taken from the device. This is preventing me from re-generating thumbnails via the admin jobs to fix the "ERROR: duplicate key value violates unique constraint "UQ_userid_checksum" issue. I can use the CLI, but I opted to try the mobile sync first.
Stack trace
2023-06-25 11:12:02 immich_microservices | [Nest] 1 - 06/25/2023, 6:12:02 PM ERROR [JobService] Error: ffmpeg exited with code 1: upload/upload/d9e45c7b-5730-4dc8-8202-70af260de485/957e98c3-ac8b-4d4d-896b-cf3294e5a00b.mp4: Invalid data found when processing input
2023-06-25 11:12:02 immich_microservices |
2023-06-25 11:12:02 immich_microservices | at ChildProcess.<anonymous> (/usr/src/app/node_modules/fluent-ffmpeg/lib/processor.js:182:22)
2023-06-25 11:12:02 immich_microservices | at ChildProcess.emit (node:events:513:28)
2023-06-25 11:12:02 immich_microservices | at ChildProcess._handle.onexit (node:internal/child_process:291:12)
2023-06-25 11:12:02 immich_microservices | [Nest] 1 - 06/25/2023, 6:12:02 PM ERROR [JobService] Object:
2023-06-25 11:12:02 immich_microservices | {
2023-06-25 11:12:02 immich_microservices | "id": "f528db98-9d05-424d-a76f-eeec3d34237a"
2023-06-25 11:12:02 immich_microservices | }
2023-06-25 11:12:02 immich_microservices |
Given that from previous immich bug posts I've found, immich seems to be focused on using solely the UI, I would think potential solutions could be
- immich reuploading the corrupted videos via mobile app (may not work if this is an ffmpeg issue or original file issue) OR
- identifying to the user the original filename of the corrupted file so that the user can diagnose the video on their end (i.e. translating hash to a filename)
- A means to delete the corrupted file/entry based on hash/id and force immich to resync the file.
The OS that Immich Server is running on
Windows 11
Version of Immich Server
v1.63,0
Version of Immich Mobile App
v1.63.0
Platform with the issue
- [X] Server
- [ ] Web
- [X] Mobile
Your docker-compose.yml content
version: "3.8"
services:
immich-server:
container_name: immich_server
image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
command: [ "start.sh", "immich" ]
volumes:
- ${UPLOAD_LOCATION}:/usr/src/app/upload
env_file:
- .env
depends_on:
- redis
- database
- typesense
restart: always
immich-microservices:
container_name: immich_microservices
image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
command: [ "start.sh", "microservices" ]
volumes:
- ${UPLOAD_LOCATION}:/usr/src/app/upload
env_file:
- .env
depends_on:
- redis
- database
- typesense
restart: always
immich-machine-learning:
container_name: immich_machine_learning
image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
volumes:
- ${UPLOAD_LOCATION}:/usr/src/app/upload
- model-cache:/cache
env_file:
- .env
restart: always
immich-web:
container_name: immich_web
image: ghcr.io/immich-app/immich-web:${IMMICH_VERSION:-release}
env_file:
- .env
restart: always
typesense:
container_name: immich_typesense
image: typesense/typesense:0.24.0
environment:
- TYPESENSE_API_KEY=${TYPESENSE_API_KEY}
- TYPESENSE_DATA_DIR=/data
logging:
driver: none
volumes:
- tsdata:/data
restart: always
redis:
container_name: immich_redis
image: redis:6.2
restart: always
database:
container_name: immich_postgres
image: postgres:14
env_file:
- .env
environment:
POSTGRES_PASSWORD: ${DB_PASSWORD}
POSTGRES_USER: ${DB_USERNAME}
POSTGRES_DB: ${DB_DATABASE_NAME}
PG_DATA: /var/lib/postgresql/data
volumes:
- pgdata:/var/lib/postgresql/data
restart: always
immich-proxy:
container_name: immich_proxy
image: ghcr.io/immich-app/immich-proxy:${IMMICH_VERSION:-release}
environment:
# Make sure these values get passed through from the env file
- IMMICH_SERVER_URL
- IMMICH_WEB_URL
ports:
- 2283:8080
depends_on:
- immich-server
restart: always
backup:
container_name: immich_db_dumper
image: prodrigestivill/postgres-backup-local
env_file:
- .env
environment:
POSTGRES_HOST: database
POSTGRES_DB: ${DB_DATABASE_NAME}
POSTGRES_USER: ${DB_USERNAME}
POSTGRES_PASSWORD: ${DB_PASSWORD}
SCHEDULE: "@daily"
BACKUP_NUM_KEEP: 7
BACKUP_DIR: /db_dumps
volumes:
- ./db_dumps:/db_dumps
depends_on:
- database
volumes:
pgdata:
model-cache:
tsdata:
Your .env content
###################################################################################
# Database
###################################################################################
# NOTE: The following four database variables support Docker secrets by adding a *_FILE suffix to the variable name
# See the docker-compose documentation on secrets for additional details: https://docs.docker.com/compose/compose-file/compose-file-v3/#secrets
DB_HOSTNAME=<REDACT>
DB_USERNAME=<REDACT>
DB_PASSWORD=<REDACT>
DB_DATABASE_NAME=<REDACT>
# Optional Database settings:
# DB_PORT=5432
###################################################################################
# Redis
###################################################################################
REDIS_HOSTNAME=immich_redis
# REDIS_URL will be used to pass custom options to ioredis.
# Example for Sentinel
# {"sentinels":[{"host":"redis-sentinel-node-0","port":26379},{"host":"redis-sentinel-node-1","port":26379},{"host":"redis-sentinel-node-2","port":26379}],"name":"redis-sentinel"}
# REDIS_URL=ioredis://eyJzZW50aW5lbHMiOlt7Imhvc3QiOiJyZWRpcy1zZW50aW5lbDEiLCJwb3J0IjoyNjM3OX0seyJob3N0IjoicmVkaXMtc2VudGluZWwyIiwicG9ydCI6MjYzNzl9XSwibmFtZSI6Im15bWFzdGVyIn0=
# Optional Redis settings:
# Note: these parameters are not automatically passed to the Redis Container
# to do so, please edit the docker-compose.yml file as well. Redis is not configured
# via environment variables, only redis.conf or the command line
# REDIS_PORT=6379
# REDIS_DBINDEX=0
# REDIS_USERNAME=
# REDIS_PASSWORD=
# REDIS_SOCKET=
###################################################################################
# Upload File Location
#
# This is the location where uploaded files are stored.
###################################################################################
UPLOAD_LOCATION=D:\Docker\Applications\immich-app\uploads
###################################################################################
# Typesense
###################################################################################
TYPESENSE_API_KEY=<REDACT>
# TYPESENSE_ENABLED=false
# TYPESENSE_URL uses base64 encoding for the nodes json.
# Example JSON that was used:
# [
# { 'host': 'typesense-1.example.net', 'port': '443', 'protocol': 'https' },
# { 'host': 'typesense-2.example.net', 'port': '443', 'protocol': 'https' },
# { 'host': 'typesense-3.example.net', 'port': '443', 'protocol': 'https' },
# ]
# TYPESENSE_URL=ha://WwogICAgeyAnaG9zdCc6ICd0eXBlc2Vuc2UtMS5leGFtcGxlLm5ldCcsICdwb3J0JzogJzQ0MycsICdwcm90b2NvbCc6ICdodHRwcycgfSwKICAgIHsgJ2hvc3QnOiAndHlwZXNlbnNlLTIuZXhhbXBsZS5uZXQnLCAncG9ydCc6ICc0NDMnLCAncHJvdG9jb2wnOiAnaHR0cHMnIH0sCiAgICB7ICdob3N0JzogJ3R5cGVzZW5zZS0zLmV4YW1wbGUubmV0JywgJ3BvcnQnOiAnNDQzJywgJ3Byb3RvY29sJzogJ2h0dHBzJyB9LApd
###################################################################################
# Reverse Geocoding
#
# Reverse geocoding is done locally which has a small impact on memory usage
# This memory usage can be altered by changing the REVERSE_GEOCODING_PRECISION variable
# This ranges from 0-3 with 3 being the most precise
# 3 - Cities > 500 population: ~200MB RAM
# 2 - Cities > 1000 population: ~150MB RAM
# 1 - Cities > 5000 population: ~80MB RAM
# 0 - Cities > 15000 population: ~40MB RAM
####################################################################################
# DISABLE_REVERSE_GEOCODING=false
# REVERSE_GEOCODING_PRECISION=3
####################################################################################
# WEB - Optional
#
# Custom message on the login page, should be written in HTML form.
# For example:
# PUBLIC_LOGIN_PAGE_MESSAGE="This is a demo instance of Immich.<br><br>Email: <i>[email protected]</i><br>Password: <i>demo</i>"
####################################################################################
PUBLIC_LOGIN_PAGE_MESSAGE=
####################################################################################
# Alternative Service Addresses - Optional
#
# This is an advanced feature for users who may be running their immich services on different hosts.
# It will not change which address or port that services bind to within their containers, but it will change where other services look for their peers.
# Note: immich-microservices is bound to 3002, but no references are made
####################################################################################
IMMICH_WEB_URL=http://immich-web:3000
IMMICH_SERVER_URL=http://immich-server:3001
IMMICH_MACHINE_LEARNING_URL=http://immich-machine-learning:3003
####################################################################################
# Alternative API's External Address - Optional
#
# This is an advanced feature used to control the public server endpoint returned to clients during Well-known discovery.
# You should only use this if you want mobile apps to access the immich API over a custom URL. Do not include trailing slash.
# NOTE: At this time, the web app will not be affected by this setting and will continue to use the relative path: /api
# Examples: http://localhost:3001, http://immich-api.example.com, etc
####################################################################################
#IMMICH_API_URL_EXTERNAL=http://localhost:3001
###################################################################################
# Immich Version - Optional
#
# This allows all immich docker images to be pinned to a specific version. By default,
# the version is "release" but could be a specific version, like "v1.59.0".
###################################################################################
#IMMICH_VERSION=
Reproduction steps
1. Upload 7500+ files via app (hard to repro)
Additional information
No response
Is this still an issue?
I appear to also be encountering this issue on Immich v1.88.2 and have about 2000 videos stuck in a loop of trying to process.
[Nest] 7 - 11/26/2023, 7:16:26 AM ERROR [JobService] Error: ffprobe exited with code 1
ffprobe version 6.0-Jellyfin Copyright (c) 2007-2023 the FFmpeg developers
built with gcc 12 (Debian 12.2.0-14)
configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-version=Jellyfin --disable-doc --disable-ffplay --disable-ptx-compression --disable-static --disable-libxcb --disable-sdl2 --disable-xlib --enable-lto --enable-gpl --enable-version3 --enable-shared --enable-gmp --enable-gnutls --enable-chromaprint --enable-libdrm --enable-libass --enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libopenmpt --enable-libdav1d --enable-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --enable-libfdk-aac --arch=amd64 --enable-libsvtav1 --enable-libshaderc --enable-libplacebo --enable-vulkan --enable-opencl --enable-vaapi --enable-amf --enable-libvpl --enable-ffnvcodec --enable-cuda --enable-cuda-llvm --enable-cuvid --enable-nvdec --enable-nvenc
libavutil 58. 2.100 / 58. 2.100
libavcodec 60. 3.100 / 60. 3.100
libavformat 60. 3.100 / 60. 3.100
libavdevice 60. 1.100 / 60. 1.100
libavfilter 9. 3.100 / 9. 3.100
libswscale 7. 1.100 / 7. 1.100
libswresample 4. 10.100 / 4. 10.100
libpostproc 57. 1.100 / 57. 1.100
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x2a5e6170180] moov atom not found
upload/library/f11b6cbd-8439-49c0-8247-c452aa647657/2021/2021-07-11/20190630_112959 (2).mp4: Invalid data found when processing input
at ChildProcess.<anonymous> (/usr/src/app/node_modules/fluent-ffmpeg/lib/ffprobe.js:233:22)
at ChildProcess.emit (node:events:514:28)
at ChildProcess._handle.onexit (node:internal/child_process:294:12)
[Nest] 7 - 11/26/2023, 7:16:26 AM ERROR [JobService] Object:
{
"id": "86945225-9ca5-40bf-b140-25630d375e58"
}
similar issue here on 1.91.4 (although has been happening for a while, not just only on this version). example log snippet here:
2023-12-21 12:54:33 [Nest] 7 - 12/21/2023, 6:54:33 PM ERROR [JobService] Unable to run job handler (thumbnailGeneration/generate-jpeg-thumbnail): Error: ffprobe exited with code 1
2023-12-21 12:54:33 ffprobe version 6.0-Jellyfin Copyright (c) 2007-2023 the FFmpeg developers
2023-12-21 12:54:33 built with gcc 12 (Debian 12.2.0-14)
2023-12-21 12:54:33 configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-version=Jellyfin --disable-doc --disable-ffplay --disable-ptx-compression --disable-static --disable-libxcb --disable-sdl2 --disable-xlib --enable-lto --enable-gpl --enable-version3 --enable-shared --enable-gmp --enable-gnutls --enable-chromaprint --enable-libdrm --enable-libass --enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libopenmpt --enable-libdav1d --enable-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --enable-libfdk-aac --toolchain=hardened --enable-cross-compile --arch=arm64 --cross-prefix=/usr/bin/aarch64-linux-gnu-
2023-12-21 12:54:33 libavutil 58. 2.100 / 58. 2.100
2023-12-21 12:54:33 libavcodec 60. 3.100 / 60. 3.100
2023-12-21 12:54:33 libavformat 60. 3.100 / 60. 3.100
2023-12-21 12:54:33 libavdevice 60. 1.100 / 60. 1.100
2023-12-21 12:54:33 libavfilter 9. 3.100 / 9. 3.100
2023-12-21 12:54:33 libswscale 7. 1.100 / 7. 1.100
2023-12-21 12:54:33 libswresample 4. 10.100 / 4. 10.100
2023-12-21 12:54:33 libpostproc 57. 1.100 / 57. 1.100
2023-12-21 12:54:33 [mov,mp4,m4a,3gp,3g2,mj2 @ 0x23a9e170180] moov atom not found
2023-12-21 12:54:33 /mnt/media/Videos/2012/02-February/2012-02-19_151038_2011-12-20-19.56.20.mov: Invalid data found when processing input
2023-12-21 12:54:33
I recently added my Intel QuickSync to the docker to transcode my videos aswell and having the same issue with this log produced: (it is also happening with HW Transcoding disabled)
immich_microservices | ffprobe version 6.0.1-Jellyfin Copyright (c) 2007-2023 the FFmpeg developers
immich_microservices | built with gcc 12 (Debian 12.2.0-14)
immich_microservices | configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-version=Jellyfin --disable-doc --disable-ffplay --disable-ptx-compression --disable-static --disable-libxcb --disable-sdl2 --disable-xlib --enable-lto --enable-gpl --enable-version3 --enable-shared --enable-gmp --enable-gnutls --enable-chromaprint --enable-opencl --enable-libdrm --enable-libass --enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libopenmpt --enable-libdav1d --enable-libsvtav1 --enable-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --enable-libfdk-aac --arch=amd64 --enable-libshaderc --enable-libplacebo --enable-vulkan --enable-vaapi --enable-amf --enable-libvpl --enable-ffnvcodec --enable-cuda --enable-cuda-llvm --enable-cuvid --enable-nvdec --enable-nvenc
immich_microservices | libavutil 58. 2.100 / 58. 2.100
immich_microservices | libavcodec 60. 3.100 / 60. 3.100
immich_microservices | libavformat 60. 3.100 / 60. 3.100
immich_microservices | libavdevice 60. 1.100 / 60. 1.100
immich_microservices | libavfilter 9. 3.100 / 9. 3.100
immich_microservices | libswscale 7. 1.100 / 7. 1.100
immich_microservices | libswresample 4. 10.100 / 4. 10.100
immich_microservices | libpostproc 57. 1.100 / 57. 1.100
immich_microservices | [mov,mp4,m4a,3gp,3g2,mj2 @ 0x46c00190180] Format mov,mp4,m4a,3gp,3g2,mj2 detected only with low score of 1, misdetection possible!
immich_microservices | [mov,mp4,m4a,3gp,3g2,mj2 @ 0x46c00190180] moov atom not found
immich_microservices | upload/encoded-video/039d74b2-0fde-4527-b584-b7ca8b0a393a/f7/5e/f75eec9e-95af-4f6d-a69e-9cdb82e55878-MP.mp4: Invalid data found when processing input
immich_microservices |
immich_microservices | at ChildProcess.<anonymous> (/usr/src/app/node_modules/fluent-ffmpeg/lib/ffprobe.js:233:22)
immich_microservices | at ChildProcess.emit (node:events:518:28)
immich_microservices | at ChildProcess._handle.onexit (node:internal/child_process:294:12)
immich_microservices | [Nest] 7 - 04/26/2024, 10:20:19 AM ERROR [JobService] Object:
immich_microservices | {
immich_microservices | "id": "f75eec9e-95af-4f6d-a69e-9cdb82e55878"
immich_microservices | }
immich_microservices |
This files where uploaded through the mobile app and were produced on a Pixel 7 Pro if that matters.
I believe this is not an issue with Immich but the video coming from the device that is somehow corrupted