immich icon indicating copy to clipboard operation
immich copied to clipboard

[BUG] Mobile App Crashes When Uploading Lots Of Media

Open Orbitally opened this issue 1 year ago • 38 comments

The bug

While uploading images to the server through the mobile app (Latest version of the iOS app) it'll crash at seemingly random times.

The OS that Immich Server is running on

Ubuntu 22.04 Server

Version of Immich Server

1.74.0

Version of Immich Mobile App

1.74.0

Platform with the issue

  • [ ] Server
  • [ ] Web
  • [X] Mobile

Your docker-compose.yml content

version: "3.8"

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    command: [ "start.sh", "immich" ]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
    env_file:
      - .env
    depends_on:
      - redis
      - database
      - typesense
    restart: always

  immich-microservices:
    container_name: immich_microservices
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    # extends:
    #   file: hwaccel.yml
    #   service: hwaccel
    command: [ "start.sh", "microservices" ]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
    env_file:
      - .env
    depends_on:
      - redis
      - database
      - typesense
    restart: always

#  immich-machine-learning:
#    container_name: immich_machine_learning
#    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
#    volumes:
#      - model-cache:/cache
#    env_file:
#      - .env
#    restart: always

  immich-web:
    container_name: immich_web
    image: ghcr.io/immich-app/immich-web:${IMMICH_VERSION:-release}
    env_file:
      - .env
    restart: always

  typesense:
    container_name: immich_typesense
    image: typesense/typesense:0.24.1@sha256:9bcff2b829f12074426ca044b56160ca9d777a0c488303469143dd9f8259d4dd
    environment:
      - TYPESENSE_API_KEY=${TYPESENSE_API_KEY}
      - TYPESENSE_DATA_DIR=/data
      # remove this to get debug messages
      - GLOG_minloglevel=1
    volumes:
      - tsdata:/data
    restart: always

  redis:
    container_name: immich_redis
    image: redis:6.2-alpine@sha256:70a7a5b641117670beae0d80658430853896b5ef269ccf00d1827427e3263fa3
    restart: always

  database:
    container_name: immich_postgres
    image: postgres:14-alpine@sha256:28407a9961e76f2d285dc6991e8e48893503cc3836a4755bbc2d40bcc272a441
    env_file:
      - .env
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
    volumes:
      - pgdata:/var/lib/postgresql/data
    restart: always

  immich-proxy:
    container_name: immich_proxy
    image: ghcr.io/immich-app/immich-proxy:${IMMICH_VERSION:-release}
    environment:
      # Make sure these values get passed through from the env file
      - IMMICH_SERVER_URL
      - IMMICH_WEB_URL
    ports:
      - 2283:8080
    depends_on:
      - immich-server
      - immich-web
    restart: always

volumes:
  pgdata:
  model-cache:
  tsdata:

Your .env content

###################################################################################
# Database
###################################################################################

DB_HOSTNAME=immich_postgres
DB_USERNAME=postgres
DB_PASSWORD=beepbooppasswordthings
DB_DATABASE_NAME=immich

# Optional Database settings:
# DB_PORT=5432

###################################################################################
# Redis
###################################################################################

REDIS_HOSTNAME=immich_redis

# Optional Redis settings:

# Note: these parameters are not automatically passed to the Redis Container
# to do so, please edit the docker-compose.yml file as well. Redis is not configured
# via environment variables, only redis.conf or the command line

# REDIS_PORT=6379
# REDIS_DBINDEX=0
# REDIS_PASSWORD=
# REDIS_SOCKET=

###################################################################################
# Upload File Location
#
# This is the location where uploaded files are stored.
###################################################################################

UPLOAD_LOCATION=/media/immich-media


###################################################################################
# Log message level - [simple|verbose]
###################################################################################

LOG_LEVEL=simple

###################################################################################
# Typesense
###################################################################################
# TYPESENSE_ENABLED=false
TYPESENSE_API_KEY=beepboopapikeyhere
# TYPESENSE_HOST: typesense
# TYPESENSE_PORT: 8108
# TYPESENSE_PROTOCOL: http

###################################################################################
# Reverse Geocoding
#
# Reverse geocoding is done locally which has a small impact on memory usage
# This memory usage can be altered by changing the REVERSE_GEOCODING_PRECISION variable
# This ranges from 0-3 with 3 being the most precise
# 3 - Cities > 500 population: ~200MB RAM
# 2 - Cities > 1000 population: ~150MB RAM
# 1 - Cities > 5000 population: ~80MB RAM
# 0 - Cities > 15000 population: ~40MB RAM
####################################################################################

# DISABLE_REVERSE_GEOCODING=false
# REVERSE_GEOCODING_PRECISION=3

####################################################################################
# WEB - Optional
#
# Custom message on the login page, should be written in HTML form.
# For example:
# PUBLIC_LOGIN_PAGE_MESSAGE="This is a demo instance of Immich.<br><br>Email: <i>[email protected]</i><br>Password: <i>demo</i>"
####################################################################################

PUBLIC_LOGIN_PAGE_MESSAGE="Backing up camera rolls without needing to pay [REDACTED]. Ha!"

####################################################################################
# Alternative Service Addresses - Optional
#
# This is an advanced feature for users who may be running their immich services on different hosts.
# It will not change which address or port that services bind to within their containers, but it will change where other services look for their peers.
# Note: immich-microservices is bound to 3002, but no references are made
####################################################################################

IMMICH_WEB_URL=http://immich-web:3000
IMMICH_SERVER_URL=http://immich-server:3001
IMMICH_MACHINE_LEARNING_URL=http://192.168.209.35:3003

####################################################################################
# Alternative API's External Address - Optional
#
# This is an advanced feature used to control the public server endpoint returned to clients during Well-known discovery.
# You should only use this if you want mobile apps to access the immich API over a custom URL. Do not include trailing slash.
# NOTE: At this time, the web app will not be affected by this setting and will continue to use the relative path: /api
# Examples: http://localhost:3001, http://immich-api.example.com, etc
####################################################################################

#IMMICH_API_URL_EXTERNAL=http://localhost:3001

###################################################################################
# Immich Version - Optional
#
# This allows all immich docker images to be pinned to a specific version. By default,
# the version is "release" but could be a specific version, like "v1.59.0".
###################################################################################

#IMMICH_VERSION=

Reproduction steps

1. Start an upload in the mobile app with a bunch of pictures
2. Let it run for a little while
3. The app will crash after about 5-10 min (Sometimes earlier)

Additional information

I'm currently on the iOS 17 Beta

Orbitally avatar Aug 22 '23 19:08 Orbitally

Hello, would it be possible for you to test on ios16 since it is stable?

alextran1502 avatar Aug 22 '23 20:08 alextran1502

Hey! Sure, I installed the beta my main day to day devices but I can see if I have an old phone lying around which I can install iOS 16 on to see how that works. I'll see what I can do and keep you posted.

Orbitally avatar Aug 22 '23 20:08 Orbitally

Turns out the device I was thinking of was running the latest version of iOS 15 but the issue is still there on that where it'll upload a couple things and then the app will crash. Are there any logs or anything I could send which would help with this?

Orbitally avatar Aug 23 '23 15:08 Orbitally

Just had the crash happen while I was watching the server logs and this is what popped up:

immich_postgres       | 2023-08-23 15:40:15.803 UTC [2655] DETAIL:  Key ("ownerId", checksum)=(e275a2cf-9991-4587-90f8-e74424bf42da, \x71727b07c21243f56c136ec36d54effe312623fd) already exists.
immich_postgres       | 2023-08-23 15:40:15.803 UTC [2655] STATEMENT:  INSERT INTO "assets"("id", "deviceAssetId", "ownerId", "deviceId", "type", "originalPath", "resizePath", "webpPath", "thumbhash", "encodedVideoPath", "createdAt", "updatedAt", "fileCreatedAt", "fileModifiedAt", "isFavorite", "isArchived", "isReadOnly", "checksum", "duration", "isVisible", "livePhotoVideoId", "originalFileName", "sidecarPath") VALUES (DEFAULT, $1, $2, $3, $4, $5, $6, $7, $8, $9, DEFAULT, DEFAULT, $10, $11, $12, $13, $14, $15, $16, $17, DEFAULT, $18, $19) RETURNING "id", "webpPath", "encodedVideoPath", "createdAt", "updatedAt", "isFavorite", "isArchived", "isReadOnly", "isVisible"
immich_server         | [Nest] 7  - 08/23/2023, 3:40:37 PM     LOG [CommunicationGateway] New websocket connection: ppQ6CTtt10NayOhuAAA0
immich_redis          | 1:M 23 Aug 2023 15:41:03.007 * 100 changes in 300 seconds. Saving...
immich_redis          | 1:M 23 Aug 2023 15:41:03.007 * Background saving started by pid 190
immich_redis          | 190:C 23 Aug 2023 15:41:03.097 * DB saved on disk
immich_redis          | 190:C 23 Aug 2023 15:41:03.097 * RDB: 0 MB of memory used by copy-on-write
immich_redis          | 1:M 23 Aug 2023 15:41:03.108 * Background saving terminated with success
immich_server         | [Nest] 7  - 08/23/2023, 3:41:18 PM   ERROR [ExpressAdapter] Premature close
immich_server         | Error [ERR_STREAM_PREMATURE_CLOSE]: Premature close
immich_server         |     at new NodeError (node:internal/errors:405:5)
immich_server         |     at ServerResponse.onclose (node:internal/streams/end-of-stream:154:30)
immich_server         |     at ServerResponse.emit (node:events:526:35)
immich_server         |     at emitCloseNT (node:_http_server:996:10)
immich_server         |     at Socket.onServerResponseClose (node:_http_server:278:5)
immich_server         |     at Socket.emit (node:events:526:35)
immich_server         |     at TCP.<anonymous> (node:net:323:12)
immich_server         | [Nest] 7  - 08/23/2023, 3:41:18 PM   ERROR [ExpressAdapter] Premature close
immich_server         | Error [ERR_STREAM_PREMATURE_CLOSE]: Premature close
immich_server         |     at new NodeError (node:internal/errors:405:5)
immich_server         |     at ServerResponse.onclose (node:internal/streams/end-of-stream:154:30)
immich_server         |     at ServerResponse.emit (node:events:526:35)
immich_server         |     at emitCloseNT (node:_http_server:996:10)
immich_server         |     at Socket.onServerResponseClose (node:_http_server:278:5)
immich_server         |     at Socket.emit (node:events:526:35)
immich_server         |     at TCP.<anonymous> (node:net:323:12)
immich_server         | [Nest] 7  - 08/23/2023, 3:41:18 PM     LOG [CommunicationGateway] Client ppQ6CTtt10NayOhuAAA0 disconnected from Websocket

Not sure if that helps and please let me know if you'd like anything else!

Orbitally avatar Aug 23 '23 15:08 Orbitally

My app keeps crashing for me upon upload - I'm having to babysit the upload.

https://1drv.ms/f/s!AkYD3wMYcl2kz7406Mpe05cOH3kJPg

Rustymage avatar Feb 05 '24 09:02 Rustymage

Same here

Bart1909 avatar Feb 08 '24 09:02 Bart1909

Also getting this issue exactly as @Rustymage 's video

Adzter avatar Feb 09 '24 17:02 Adzter

Also getting same issue as @Rustymage Unless I "cancel and restart" the upload it will also start over at the first starting point after the crash. Hence need to cancel and restart every now and then to update the progress.

NegaDuck420 avatar Feb 10 '24 11:02 NegaDuck420

Also getting same issue as @Rustymage Unless I "cancel and restart" the upload it will also start over at the first starting point after the crash. Hence need to cancel and restart every now and then to update the progress.

That's how I babysat it. Cancel after a little bit to lock in the upload, then restarted.

Rustymage avatar Feb 10 '24 11:02 Rustymage

+1 Exact sam issue hier on iOS 17.2.1

FelixArndt avatar Feb 16 '24 09:02 FelixArndt

Hello everyone, did you manage to overcome the problem somehow? Faced the same thing

owerrride avatar Feb 16 '24 15:02 owerrride

Can you guys help provide the mobile app log and the server log?

alextran1502 avatar Feb 16 '24 16:02 alextran1502

Ребята, можете ли вы помочь предоставить журнал мобильного приложения и журнал сервера? server logs (ios crash).txt Immich_log_2024-02-16T18_35_28.110792.csv Sreen record from iPhone (iOS 17.2.1) - https://streamable.com/1hmnzc

owerrride avatar Feb 16 '24 16:02 owerrride

Hello everyone, did you manage to overcome the problem somehow? Faced the same thing

I babysat it and watched it the whole time.

Rustymage avatar Feb 16 '24 16:02 Rustymage

@alextran1502 I am running into this as well. I could not find anything interesting in the app logs but the crash report looked interesting

  "exception" : {"codes":"0x0000000000000000, 0x0000000000000000","rawCodes":[0,0],"type":"EXC_CRASH","signal":"SIGABRT"},
  "termination" : {"flags":0,"code":6,"namespace":"SIGNAL","indicator":"Abort trap: 6","byProc":"Runner","byPid":2743},
  "ktriageinfo" : "VM - (arg = 0x3) mach_vm_allocate_kernel failed within call to vm_map_enter\nVM - (arg = 0x3) mach_vm_allocate_kernel failed within call to vm_map_enter\nVM - (arg = 0x3) mach_vm_allocate_kernel failed within call to vm_map_enter\nVM - (arg = 0x3) mach_vm_allocate_kernel failed within call to vm_map_enter\nVM - (arg = 0x3) mach_vm_allocate_kernel failed within call to vm_map_enter\n",
  "asi" : {"libsystem_c.dylib":["abort() called"]},

Iphone 12 Pro IOS 17.2.1 Immich App version 1.94.1 Immich Server version 1.91.4

I can share server and client logs and full crash report if you think they would be helpful

surdy avatar Feb 21 '24 21:02 surdy

My immich ios app also crashed every time I tried to use the foreground / background upload.

It sometimes even led to my phone getting unresponsive or rebooting. I tried solving it by dumping my icloud onto my server and using the immich cli to upload all the files locally.

But the local upload on the server crashed as well. So I inspected the upload on the server and found that it kept on crashing while trying to upload one big 22GB video file. The file itself was perfectly fine and played when opened on my computer.

However, after removing the video from my icloud (and my iphones gallery), both upload tools (app and cli) worked again.

Maybe auto-exclusion of big files from the upload with a warning would be helpful.

PS: I love using immich, great work and vision!

Edit

Got the random crashes again, just took a bit of time for one to appear, so I thought the big files were the problem.

leolurch avatar Feb 24 '24 22:02 leolurch

@alextran1502 Here are two iOS crash logs. These happened on server version 1.94.1 and 1.95.1. App version is 1.95.0. I haven't been able to recreate it on my device (pulled these off a friend's device who is seeing multiple crashes). This is his initial backup of a moderate sized library. It doesn't seem to fail on any specific image/video that he can identify, and sometimes goes for 15 minutes without a crash, but other times crashes within a minute.

https://gist.github.com/adambeck7/3bb6104ab558994c98e303e63154f1b8 https://gist.github.com/adambeck7/a5c2b0665354dff3d12488e32077cfb9

adambeck7 avatar Feb 27 '24 18:02 adambeck7

I have the same issue. The app will crash randomly while uploading. I am on 1.97.0 build 1.41 on iPhone 15 Pro iOS 17.4 (21E217) and 1.97.0 server. Issue is that logs don't show anything on phone or the server.

miqdadraza avatar Mar 01 '24 09:03 miqdadraza

Same for me (iPhone 14 Pro / iOS 17.4 / server 1.98.2 / mobile 1.98.1 ) - uploaded only 300 photos in foreground mode and had 5 app crashes. Just a note - I have all my photos stored on iCloud so Immich app downloads fullres version of every photo from iCloud.

baloo2 avatar Mar 12 '24 06:03 baloo2

I have this issue too. I was hoping to migrate to immich but 120k photos at 100/crash might be a bit ambitious...

josiahw avatar Mar 20 '24 05:03 josiahw

@josiahw i assume you are using iCloud for photos. I think there is a bug when reading iCloud asset that cause memory leaked that lead to the crash. I will need to investigate this. For the work around, you can toggle ignore iCloud file for the time being

alextran1502 avatar Mar 20 '24 06:03 alextran1502

@alextran1502 but what if all my pics go to iCloud? I have iCloud backup also on so pics do get uploaded on there. If I toggle iCloud off in Immich, Immich will possibly not upload any image?

miqdadraza avatar Mar 20 '24 06:03 miqdadraza

@josiahw i assume you are using iCloud for photos. I think there is a bug when reading iCloud asset that cause memory leaked that lead to the crash. I will need to investigate this. For the work around, you can toggle ignore iCloud file for the time being

Well, that's not a solution as all of my photos are on iCloud. I hope you can find a solution for this bug. Thanks a lot.

baloo2 avatar Mar 20 '24 07:03 baloo2

As a workaround you can setup an automation in Shortcuts to automatically launch Immich after crash. Just be sure to add some delay so you don't end up in a loop.

IMG_9632

IMG_9633

maksg avatar Mar 30 '24 10:03 maksg

I just enable iCloud this week, so hopefully I can find the cause for this behavior. How long into the upload when you guys start seeing crash?

alextran1502 avatar Mar 30 '24 12:03 alextran1502

For me almost instantly - after 200 photos.

baloo2 avatar Mar 30 '24 14:03 baloo2

It's random for me. Sometimes it will go for 20 mins before crashing. Other times it will crash 1 minute into uploading. Haven't seen it go longer than 20 mins before crashing.

mithirich avatar Apr 03 '24 16:04 mithirich

Same issue for me. Crashes every 15-20 minutes. On latest iOS (just upgraded last night).

nothing2obvi avatar Apr 04 '24 03:04 nothing2obvi

Any progress on this? Immich is not usable like this unfortunatly

mvenus2 avatar Apr 17 '24 17:04 mvenus2

@mvenus2 unfortunately not yet, I have heard that PhotoSync app also has this issue when downloading photos and videos from iCloud

alextran1502 avatar Apr 17 '24 17:04 alextran1502