[BUG] iOS Not Backing up Majority of Photos

[srwareham]

The bug

Zero of my photos previously taken via my iPhone's camera app (or other apps that save to camera roll) are being backed up anymore. Only screenshots and photos that were sent to me (e.g., via WhatsApp and then manually saved) are backed up to immich. The iOS app reports 3012 assets in "Remainder." Clicking sync again does not change this number. However, it will show "Uploading file info" and clicking the icon shows a file preview for each of the thousands of photos that aren't being synced.

If I take a new photo, it will not sync. If I take a new screenshot, it will sync. I have tried a clean install on my server and uninstalling/reinstalling the iOS app to no avail.

The OS that Immich Server is running on

Arch Linux

Version of Immich Server

v1.50 (also occured on v1.49)

Version of Immich Mobile App

v1.49 build.87

Platform with the issue

• [ ] Server
• [ ] Web
• [X] Mobile

Your docker-compose.yml content

version: "3.8"
name: immich
services:
  immich-server:
    container_name: immich_server
    image: altran1502/immich-server:release
    entrypoint: ["/bin/sh", "./start-server.sh"]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
    env_file:
      - .env
    environment:
      - NODE_ENV=production
    depends_on:
      - redis
      - database
    restart: unless-stopped

  immich-microservices:
    container_name: immich_microservices
    image: altran1502/immich-server:release
    entrypoint: ["/bin/sh", "./start-microservices.sh"]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
    env_file:
      - .env
    environment:
      - NODE_ENV=production
    depends_on:
      - redis
      - database
    restart: unless-stopped

  immich-machine-learning:
    container_name: immich_machine_learning
    image: altran1502/immich-machine-learning:release
    command: ["python", "src/main.py"]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - "/mnt/homeserver/service_data/immich/data/cache:/cache"
    env_file:
      - .env
    environment:
      - NODE_ENV=production
    restart: unless-stopped

  immich-web:
    container_name: immich_web
    image: altran1502/immich-web:release
    entrypoint: ["/bin/sh", "./entrypoint.sh"]
    env_file:
      - .env
    restart: unless-stopped

  redis:
    container_name: immich_redis
    image: redis:6.2
    restart: unless-stopped

  database:
    container_name: immich_postgres
    image: postgres:14
    env_file:
      - .env
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      PG_DATA: /var/lib/postgresql/data
    volumes:
      - "/mnt/homeserver/service_data/immich/data/pgdata:/var/lib/postgresql/data"
    restart: unless-stopped

  immich-proxy:
    container_name: immich_proxy
    image: altran1502/immich-proxy:release
    environment:
      IMMICH_SERVER_URL: "http://immich-server:3001"
      IMMICH_WEB_URL: "http://immich-web:3000"
    ports:
      - "9102:8080"
    logging:
      driver: none
    depends_on:
      - immich-server
    restart: unless-stopped

Your .env content

###################################################################################
# Database
###################################################################################

DB_HOSTNAME=immich_postgres
DB_USERNAME="postgres"
DB_PASSWORD=redacted
DB_DATABASE_NAME="immich"

# Optional Database settings:
# DB_PORT=5432

###################################################################################
# Redis
###################################################################################

REDIS_HOSTNAME=immich_redis

# Optional Redis settings:
# REDIS_PORT=6379
# REDIS_DBINDEX=0
# REDIS_PASSWORD=
# REDIS_SOCKET=

###################################################################################
# Upload File Location
#
# This is the location where uploaded files are stored.
###################################################################################

UPLOAD_LOCATION="/mnt/homeserver/service_data/immich/data/upload_location"

###################################################################################
# JWT SECRET
#
# This JWT_SECRET is used to sign the authentication keys for user login
# You should set it to a long randomly generated value
# You can use this command to generate one: openssl rand -base64 128
###################################################################################

JWT_SECRET=redacted

###################################################################################
# Reverse Geocoding
#
# Reverse geocoding is done locally which has a small impact on memory usage
# This memory usage can be altered by changing the REVERSE_GEOCODING_PRECISION variable
# This ranges from 0-3 with 3 being the most precise
# 3 - Cities > 500 population: ~200MB RAM
# 2 - Cities > 1000 population: ~150MB RAM
# 1 - Cities > 5000 population: ~80MB RAM
# 0 - Cities > 15000 population: ~40MB RAM
####################################################################################

# DISABLE_REVERSE_GEOCODING=false
# REVERSE_GEOCODING_PRECISION=3

####################################################################################
# WEB - Optional
#
# Custom message on the login page, should be written in HTML form.
# For example:
# PUBLIC_LOGIN_PAGE_MESSAGE="This is a demo instance of Immich.<br><br>Email: <i>[email protected]</i><br>Password: <i>demo</i>"
####################################################################################

PUBLIC_LOGIN_PAGE_MESSAGE="Homeserver Immich"

####################################################################################
# Alternative Service Addresses - Optional
#
# This is an advanced feature for users who may be running their immich services on different hosts.
# It will not change which address or port that services bind to within their containers, but it will change where other services look for their peers.
# Note: immich-microservices is bound to 3002, but no references are made
####################################################################################

IMMICH_WEB_URL="http://immich-web:3000"
IMMICH_SERVER_URL="http://immich-server:3001"
IMMICH_MACHINE_LEARNING_URL="http://immich-machine-learning:3003"

Reproduction steps

1. Delete iOS app & clean install immich server
2. Install iOS app and connect to immich server
3. Click Start Backup to run manual sync (using default Recents All album)

...

Additional information

Immich is being run and accessed in a LAN only. Previously, everything was working perfectly. I'm not sure when this changed. Probably sometime in the past 2-3 weeks. Within the past week, I upgraded from iOS 16.1(?) to 16.3.

Running the server with docker-compose up (without -d flag), I can see "POST /api/asset/upload HTTP/1.0" for my screenshots that are uploading. For the photos that aren't uploading, there is nothing in the server logs.

iOS device

• iPhone SE on iOS 16.3.1

• iCloud "Optimize iPhone Storage" is not enabled, it is set to "Download and Keep Originals"

• Settings -> Immich-> shows immich has access to Photos->All Photos

• iOS logs below after 1) full closing iOS app, 2) Open app, 3) Start Backup

Thank you for the fantastic product. Please let me know if/how I can help debug this.

[alextran1502]

Can you help me take a screen capture when you click the start upload button on the backup screen? Can you also monitor the log of the server at the uploading time?

[alextran1502]

Can you also help me take the screenshot of the album that you are selected to be uploaded?

[srwareham]

Video of Start Upload

https://user-images.githubusercontent.com/3392952/222052010-56920612-9cab-47bc-9a6a-c362a7983be2.mp4

Screenshot of server log not changing at all during Start Upload session

The one POST is from previously testing taking a screenshot and clicking backup. That worked as described in my bug report above.

[alextran1502]

hmm this is very strange, I cannot reproduce this. Can you upload the files from the web interface?

Can you create a new instance and try to upload the files to the test instance?

[srwareham]

I created a completely new instance today to try to debug this--so what I've shared above is just that. Right now, just tried uploading one of the photos via web UI and that worked fine.

I'd guess this is some sort of iOS permissions or weird iCloud issue. But the odd thing is that clicking the icon next to the "Uploading file info" actually does in fact show a preview of the relevant images. So its at least able to access a thumbnail, at a minimum

[alextran1502]

I assume the iCloud photos aren't actually on your device. If you tried to disable iCloud Photos, what would happen if you take a photo and then run the upload sequence?

[srwareham]

Just disabled iCloud photo sync and took a picture. Didn't upload unfortunately. Is there a way to increase logging verbosity in the iOS app?

[JustAmply]

I have a similar issue, and it seems to be related to #1122 It has to do with importing pictures via CLI, that are also on the iPhone. It doesn't deduplicate properly, as it errors out. Don't know if it's the same issue though

[espy-engineering]

I am also experiencing this issue. I was gonna dig into the code before posting but haven't had the chance. Mine was working perfectly fine until about beginning of January. Not sure if it was an app update or an iOS update.

[srwareham]

I was hoping v1.52.0 would resolve this but unfortunately not. Observing exact same behavior, even after a clean install on both server and iOS

[alextran1502]

@srwareham Can you help me confirm again that you don't have iCloud Photo and optimized storage turned on?

[srwareham]

@alextran1502 yes, can confirm

[alextran1502]

@srwareham hmm I assume that somehwere in shared album and iCloud photo that is causing the issue you are seeing since Immich has not been designed to work with those scenarios yet

[srwareham]

@alextran1502 is there a way to increase logging verbosity in the iOS app to try to help diagnose this?

[espy-engineering]

So for me it was user error. After finally having some time to debug it, I realized that I was getting 413s back from the server. Since I have my instance running behind my own Nginx reverse proxy, my request were getting denied because nginx has default limit of 1MB uploads in order to protect against DDOS. I modified the nginx config for that server and all is well. All photos are being uploaded. Previous only screenshots were being uploaded because screenshots were under the 1MB limit. So that was my specific issue.

[srwareham]

@espy-engineering you are my absolute hero. That fixed my problem! I hadn't remembered that at some point I started using a ngnix reverse proxy and was no longer hard coding my immich server's IP address in the mobile app.

@alextran1502 is there a way for the mobile app's log to display when it gets 413 (and presumably other HTTP) error messages? This would help tremendously for this occasion and I'm sure others.

For anyone finding this later, the fix for me was editing /etc/nginx/nginx.conf to add a user-defined value for client_max_body_size. I picked 20G as an arbitrarily large number that should be bigger than any of my videos

http {
    include       mime.types;
    default_type  application/octet-stream;
    client_max_body_size 20G;
...
}

[srwareham]

Looks like more verbose mobile logging was just released in v1.54. Thanks so much!