immich-app
[BUG] Files that failed to upload were not cleared
The bug
There are many files that failed to be transferred in the directory library/upload/121b2969-8159-44fe-a86e-4e1395a5c856/, which takes up space. Moreover, I have a video file of about 200M, which is always uploaded and disconnected, resulting in many files.
The OS that Immich Server is running on
Ubuntu 18.04.6 LTS
Version of Immich Server
v1.78.1
Version of Immich Mobile App
v1.78.1
Platform with the issue
- [X] Server
- [ ] Web
- [X] Mobile
Your docker-compose.yml content
version: "3.8"
services:
immich-server:
container_name: immich_server
image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
command: [ "start.sh", "immich" ]
volumes:
- ${UPLOAD_LOCATION}:/usr/src/app/upload
env_file:
- .env
depends_on:
- redis
- database
- typesense
restart: always
immich-microservices:
container_name: immich_microservices
image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
# extends:
# file: hwaccel.yml
# service: hwaccel
command: [ "start.sh", "microservices" ]
volumes:
- ${UPLOAD_LOCATION}:/usr/src/app/upload
env_file:
- .env
depends_on:
- redis
- database
- typesense
restart: always
immich-machine-learning:
container_name: immich_machine_learning
image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
volumes:
- model-cache:/cache
env_file:
- .env
restart: always
immich-web:
container_name: immich_web
image: ghcr.io/immich-app/immich-web:${IMMICH_VERSION:-release}
env_file:
- .env
restart: always
typesense:
container_name: immich_typesense
image: typesense/typesense:0.24.1@sha256:9bcff2b829f12074426ca044b56160ca9d777a0c488303469143dd9f8259d4dd
environment:
- TYPESENSE_API_KEY=${TYPESENSE_API_KEY}
- TYPESENSE_DATA_DIR=/data
# remove this to get debug messages
- GLOG_minloglevel=1
volumes:
- tsdata:/data
restart: always
redis:
container_name: immich_redis
image: redis:6.2-alpine@sha256:70a7a5b641117670beae0d80658430853896b5ef269ccf00d1827427e3263fa3
restart: always
database:
container_name: immich_postgres
image: postgres:14-alpine@sha256:28407a9961e76f2d285dc6991e8e48893503cc3836a4755bbc2d40bcc272a441
env_file:
- .env
environment:
POSTGRES_PASSWORD: ${DB_PASSWORD}
POSTGRES_USER: ${DB_USERNAME}
POSTGRES_DB: ${DB_DATABASE_NAME}
volumes:
- pgdata:/var/lib/postgresql/data
restart: always
immich-proxy:
container_name: immich_proxy
image: ghcr.io/immich-app/immich-proxy:${IMMICH_VERSION:-release}
environment:
# Make sure these values get passed through from the env file
- IMMICH_SERVER_URL
- IMMICH_WEB_URL
ports:
- 2283:8080
depends_on:
- immich-server
- immich-web
restart: always
volumes:
pgdata:
model-cache:
tsdata:
Your .env content
# You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables
# The location where your uploaded files are stored
UPLOAD_LOCATION=./library
# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=release
# Connection secrets for postgres and typesense. You should change these to random passwords
TYPESENSE_API_KEY=some-random-text
DB_PASSWORD=postgres
# The values below this line do not need to be changed
###################################################################################
DB_HOSTNAME=immich_postgres
DB_USERNAME=postgres
DB_DATABASE_NAME=immich
REDIS_HOSTNAME=immich_redis
Reproduction steps
1. Unable to open background fetch.
2. "immich" Turn on foreground backup.
3. Wite upload my video, 200M. See the progress bar showing that uploading is in progress.
4. Switch other app, and back "immich". See the progress bar return to zero.
5. After repeating this many times, go to the server and check that there are many files in the upload directory.
Additional information
No response
I have very similar situation but I see two issues here. Found some time to collect info and hope that would help identifying the issue.
-
Same issue as above. Backing up videos and failed transcoding keeps backing up again and again while filling up upload folder with multiple copies of uploaded files but not transcoded. there could be cleanup option at least from the web portal were you can find all those under "untracked" but not being able to do anything without accessing host.
-
Where is the limit of transcoding and why it fails? how to debug it. Here are some examples. 4k HVEC/H265 format 28s are transcoded to h264 4k HVEC /H265 format 31s (180MB file size) seem to not being able to process 4k H264 (AAC)format 29 seconds (227MB) stuck and not process even though as per setup screenshot this video was not supposed to be processed as it matches formating target log from server
[Nest] 6 - 01/28/2024, 8:21:12 AM ERROR [RangeNotSatisfiableError: Range Not Satisfiable
at createHttpError (/usr/src/app/node_modules/send/index.js:984:7)
at SendStream.error (/usr/src/app/node_modules/send/index.js:270:31)
at SendStream.send (/usr/src/app/node_modules/send/index.js:666:19)
at onstat (/usr/src/app/node_modules/send/index.js:725:10)
at FSReqCallback.oncomplete (node:fs:202:5)] Failed to serve file
[Nest] 6 - 01/28/2024, 8:21:12 AM ERROR [RangeNotSatisfiableError: Range Not Satisfiable
at createHttpError (/usr/src/app/node_modules/send/index.js:984:7)
at SendStream.error (/usr/src/app/node_modules/send/index.js:270:31)
at SendStream.send (/usr/src/app/node_modules/send/index.js:666:19)
at onstat (/usr/src/app/node_modules/send/index.js:725:10)
at FSReqCallback.oncomplete (node:fs:202:5)] RangeNotSatisfiableError: Range Not Satisfiable
[Nest] 6 - 01/28/2024, 8:21:12 AM WARN [ExpressAdapter] Content-Type doesn't match Reply body, you might need a custom ExceptionFilter for non-JSON responses
[Nest] 6 - 01/28/2024, 8:22:05 AM LOG [CommunicationRepository] Websocket Connect: WyFMmXRxmc2FlVIRAABr
[Nest] 6 - 01/28/2024, 8:22:24 AM LOG [CommunicationRepository] Websocket Disconnect: EM1y0uy8knpqNUE0AABp
log from microservices of repeated uploading at similar timing
[Nest] 7 - 01/28/2024, 8:20:54 AM LOG [MediaService] Start encoding video 51643924-07fc-4a6b-b108-283a54de645c {"inputOptions":["-init_hw_device vaapi=accel:/dev/dri/renderD128","-filter_hw_device accel"],"outputOptions":["-c:v h264_vaapi","-c:a aac","-movflags faststart","-fps_mode passthrough","-map 0:0","-map 0:1","-g 256","-v verbose","-vf format=nv12,hwupload","-compression_level 6","-threads 3","-qp 23","-global_quality 23","-rc_mode 1"],"twoPass":false}
[Nest] 7 - 01/28/2024, 8:21:06 AM LOG [MediaService] Encoding success 51643924-07fc-4a6b-b108-283a54de645c
[Nest] 7 - 01/28/2024, 8:21:06 AM LOG [MediaService] Start encoding video 8a784bd5-9633-4e68-81bd-eda8b9b1423c {"inputOptions":["-init_hw_device vaapi=accel:/dev/dri/renderD128","-filter_hw_device accel"],"outputOptions":["-c:v h264_vaapi","-c:a aac","-movflags faststart","-fps_mode passthrough","-map 0:0","-map 0:1","-g 256","-v verbose","-vf format=nv12,hwupload","-compression_level 6","-threads 3","-qp 23","-global_quality 23","-rc_mode 1"],"twoPass":false}
[Nest] 7 - 01/28/2024, 8:22:42 AM LOG [MediaService] Encoding success 8a784bd5-9633-4e68-81bd-eda8b9b1423c
[Nest] 7 - 01/28/2024, 8:22:42 AM LOG [MediaService] Start encoding video 71f8374e-5b34-4de6-b0f2-6bed89a3187c {"inputOptions":["-init_hw_device vaapi=accel:/dev/dri/renderD128","-filter_hw_device accel"],"outputOptions":["-c:v h264_vaapi","-c:a aac","-movflags faststart","-fps_mode passthrough","-map 0:0","-map 0:1","-g 256","-v verbose","-vf format=nv12,hwupload","-compression_level 6","-threads 3","-qp 23","-global_quality 23","-rc_mode 1"],"twoPass":false}
[Nest] 7 - 01/28/2024, 8:23:44 AM LOG [MediaService] Encoding success 71f8374e-5b34-4de6-b0f2-6bed89a3187c
transcoding setup from webportal
immich server 1.93.3
mobile app 1.93.2 build 119
hwaccel vaapi is used for N100 Intel Alder lake CPU
here is example of system load spykes when upload and transcoding is being performed
ok, my bad. I have actually reviewed uploaded files and they are incomplete. then also reconnected app via local server address instead of cloudflare tunel and it all worked fast and fine. so issue Nr2 is due to proxy limitation I suppose.
so in general I would say there is needed at least better way to handle these failed uploads. A button near untracks file list to flush them would be appreciated.
will dig into finding better replacement of cloudflare tunnel
I'm facing a similar problem,
when uploading (aka backuping) my media from an android (~1500 media) there were 50 upload errors.
Those errors are not visibile from the app and there's not an option to delete all incomplete uploads from the server.
Also I'm unable to find in a fast way the "corrupted" items on smartphone and reupload them. (btw I think all media are ok, it just an upload error)
How should I solve this issue?
I'm facing a similar problem, when uploading (aka backuping) my media from an android (~1500 media) there were 50 upload errors. Those errors are not visibile from the app and there's not an option to delete all incomplete uploads from the server. Also I'm unable to find in a fast way the "corrupted" items on smartphone and reupload them. (btw I think all media are ok, it just an upload error) How should I solve this issue?
I am just coming to this after a family holiday with unreliable internet has exacerbated this issue. I can see a similar situation with hundreds of GB of unplayable (corrupted) uploads staged and some working but wont move.
Just to be clear are we saying that these files are just noise or is there a chance I have hundreds of GB of files on phones that I think are backed up and actually silently are not?
I'm afraid the answer is both. Sometimes the upload retries. Other times it is considered successful even though the uploaded file is just a partial upload (and thus either incomplete or corrupt depending on the file format). See https://github.com/immich-app/immich/issues/9964 for more details.
I wonder if my current issue isn't link to this. I have big files (4 to 5 GB) that uploaded without issue but files from 8GB and up are unsuccessful, no error in the app beside the failed upload, and I don't see any error in the the docker log either.
I run a nginx proxy with client_max_body_size 0 after seeing the same incident here I tried directly upload on the local network bypassing nginx, the result is exactly the same.
I have multiple files in the repair section as well, would it be something corrupted that stop the download ?
