[BUG] iOS background sync does not work

[easez88]

The bug

Enabling background sync on iOS doesn't upload any pictures. Enabled both background processes and out the app in the background. Took new photos to test, connected phone to Wi-Fi and charger. Also, tried disabling foreground upload as well. No matter how long I wait, no new photos are uploaded.

The OS that Immich Server is running on

Fedora 37

Version of Immich Server

v1.49.0

Version of Immich Mobile App

v1.48..0 build.87

Platform with the issue

• [X] Server
• [ ] Web
• [X] Mobile

Your docker-compose.yml content

version: "3.8"

services:
  immich-server:
    container_name: immich_server
    image: altran1502/immich-server:release
    entrypoint: [ "/bin/sh", "./start-server.sh" ]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
    env_file:
      - .env
    environment:
      - NODE_ENV=production
    depends_on:
      - redis
      - database
    restart: always

  immich-microservices:
    container_name: immich_microservices
    image: altran1502/immich-server:release
    entrypoint: [ "/bin/sh", "./start-microservices.sh" ]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
    env_file:
      - .env
    environment:
      - NODE_ENV=production
    depends_on:
      - redis
      - database
    restart: always

  immich-machine-learning:
    container_name: immich_machine_learning
    image: altran1502/immich-machine-learning:release
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - model-cache:/cache
    env_file:
      - .env
    environment:
      - NODE_ENV=production
    restart: always

  immich-web:
    container_name: immich_web
    image: altran1502/immich-web:release
    entrypoint: [ "/bin/sh", "./entrypoint.sh" ]
    env_file:
      - .env
    restart: always

  redis:
    container_name: immich_redis
    image: redis:6.2
    restart: always

  database:
    container_name: immich_postgres
    image: postgres:14
    env_file:
      - .env
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      PG_DATA: /var/lib/postgresql/data
    volumes:
      - pgdata:/var/lib/postgresql/data
    restart: always

  immich-proxy:
    container_name: immich_proxy
    image: altran1502/immich-proxy:release
    environment:
      # Make sure these values get passed through from the env file
      - IMMICH_SERVER_URL
      - IMMICH_WEB_URL
    ports:
      - 2283:8080
    logging:
      driver: none
    depends_on:
      - immich-server
    restart: always

volumes:
  pgdata:
  model-cache:

Your .env content

immich_postgres
DB_USERNAME=postgres
DB_PASSWORD=postgres
DB_DATABASE_NAME=immich
REDIS_HOSTNAME=immich_redis
UPLOAD_LOCATION="/mnt/data"
PUBLIC_LOGIN_PAGE_MESSAGE=
IMMICH_WEB_URL=http://immich-web:3000
IMMICH_SERVER_URL=http://immich-server:3001
IMMICH_MACHINE_LEARNING_URL=http://immich-machine-learning:3003

Reproduction steps

1. Set up server
2. Set up background fetch in iOS app
3. Start initial backup from iOS app
4. Put app in background

Additional information

No response

[alextran1502]

Can you read this info section https://immich.app/docs/features/automatic-backup#ios

And let me know if you have it enabled and if you have everything enabled in that screen

[easez88]

Yes, I can confirm. Here are some screenshots

[alextran1502]

How many apps do you have with background refresh enabled? The more you have, the more it will compete for the timeslot to run in the background. Maybe try to disable the apps that you don't need background refresh to run and see if it changes the behavior

[easez88]

Ive turned off as many as I can. Reset background check and will report back.

[easez88]

I followed these instructions and also turned background sync off and on again. Also have background sync turned off for most apps. I can see that the under background sync last sync does update. Right now the last sync was about 1 hour ago. However, no new assets have been uploaded to the server.

[alextran1502]

What method have you been using to identify if there is no new asset uploaded yet?

[lxtzfr]

Hello, I have the similar issue with iPhone 13 running iOS 16.3. I enabled "automatic background backup" and "background app refresh". In my case, I still have items to send to the server. When I quit the application and there is a transfer in progress, the transfer in progress stops. If I immediately reopen the app or wait longer the transfer starts over from the beginning.

[alextran1502]

Hello,

I have the similar issue with iPhone 13 running iOS 16.3.

I enabled "automatic background backup" and "background app refresh".

In my case, I still have items to send to the server.

When I quit the application and there is a transfer in progress, the transfer in progress stops.

If I immediately reopen the app or wait longer the transfer starts over from the beginning.

This is the intended behavior, the background and foreground backup are two different mechanisms. So whenever a mechanism start, it has to re-calculate which files have not been backup and pickup from there

[lxtzfr]

This is the intended behavior, the background and foreground backup are two different mechanisms. So whenever a mechanism start, it has to re-calculate which files have not been backup and pickup from there

I see, when the application is open, it is the foreground backup that is in action. And when the app is closed, background backup takes over. However, when the app is closed, nothing happens. No files are sent to the server, the number of files remaining to be sent does not change.

[easez88]

I think this worked for me. I also had to repulo the containers. Now it works exactly as explained above. I get the notification of upload. However even after 24 hours with background notifications and not opening the app, the server stays in the web ui have not increased. I still have the same number of photos, videos, and storage.

[alextran1502]

@alexandrewurtzfr the background backup runs based on the user's usage behavior of Immich and other apps. It depends on the native OS's background scheduler to specify when the background task can run. It might take 10 minutes or 30 minutes, or a few hours. The goal is to improve this aspect, to let the scheduler somehow grant the app better background run time.

@easez88 How do you connect to your server, DNS or local IP?

[lxtzfr]

OK I understand. I'll keep an eye on it.

We can perhaps add a note in the application to inform that the background scheduler is executed according to the behavior of the OS.

[easez88]

I have Immich behind nginx and let’s encrypt.

[easez88]

I think a note about the background scheduler would help. Understanding what you said I’ve been opening the app every couple of hours and kept background turned off for apps that don’t need it. The result is that now the scheduler runs several times a day.

My other issue where the scheduler was running but nothing was making it to the server was because nginx was missing the client_max_body_size directive. Without it the pictures were getting rejected due to size. Must have some kind of default.

[lxtzfr]

@alextran1502 I waited a whole day and no files were transferred in the background. I have less than 10 apps that use "Background App Refresh".

About the file upload size limitation, you have to be careful when you go through a service such as Cloudflare for example. You can send a maximum file size of 100mb when proxy mode is enabled. An article explains that: https://gridpane.com/kb/cloudflares-cdn-and-upload-limitations/

[alextran1502]

@alexandrewurtzfr hello, do you see anything change in the back up screen regarding the time it was last synced?

[lxtzfr]

@alextran1502 I hadn't taken the first reference time, but it seems to have been updated. The last synchronization time was at the end of the day. I can do a test again on it.

The app must be in the background for the backup worker to start running. From: https://immich.app/docs/features/automatic-backup#background-backup

Does that mean the app should be visible in the list of recent apps? Shouldn't the app be killed by swiping up?

[alextran1502]

@alexandrewurtzfr The app should be visible in the recent apps, you shouldn't kill the app since that would terminate everything

[lxtzfr]

I think it's good, it works. My mistake is to have killed the application. I should have paid attention to this detail.

However, I think I found another bug. Some files are not uploaded to the server. I will investigate and if necessary create a new topic.

[alextran1502]

Closing this issue as the reported problems are resolved by waiting for the iOS's background scheduler to run. The future improvement is to get the scheduler favors Immich more

[twitsforbrains]

I believe I'm experiencing this issue.

I initially used foreground upload to some selected photos and then decided I wanted to test background upload so I disabled foreground upload right when there were 12 photos outstanding. I then selected background upload and minimized the app. I'm on ios and my immich server is behind traefik. I also disabled background app refresh for all apps outside of immich to test this, so there really is only 1 app that has background app refresh enabled. I'm on my home wifi with 1Gb upload right now as well.

What I noticed is that when I open the immich app, my immich-server logs immediately say "web socket connected" and "websocket disconnected". This corresponds with a screen popup in the UI that says "do you want to select more photos". I select no, then another websocket connected log is created. When I minimize the immich app, I then see the websocket is disconnected log.

So in short, once my app is minimized, there are no websocket connections to immich. I don't see how background app refresh is able to upload if there is no connection. My traefik access logs suggest there are no additional PUT, POST, etc requests to immich so I suspect the upload happens through this websocket api?

How might I better be able to debug this?

[alextran1502]

@twitsforbrains Can you show the permission page that you have for Immich? From what you describe, it seems like you only allow Immich to access to a few photos, correct?

[twitsforbrains]

Yes, I've given it access to basically the last 3-4 months of photos. My bulk upload was from 2020 to Nov 2023. After which I selected all photos after Nov 2023. By permission page, which page are you referring to? Are you referring to my IOS Privacy & Security -> Photos -> Immich page?

[alextran1502]

@twitsforbrains I suggest you to allow the app the access to the whole gallery. The app is not designed to work well with a group of selected assets so it might be the cause for background backup not working for you.

When did you start enabling the background backup on iOS?

To answer your question earlier about websocket, it doesn't related to background function of the app. Websocket is for updating the timeline when a new asset is finished processing on the server

[twitsforbrains]

I just tested this on my partners phone with entire gallery access and still don't see any uploads. Is the endpoint that should be called the POST /asset/upload? I don't see that in my traefik access logs. I think that suggests the issue is with the ios app?

[alextran1502]

@twitsforbrains do you have background app refresh enabled on your partner's phone?

[twitsforbrains]

Yes, but hold this thought, I noticed the microservices container is failing restarting every few seconds. Looks like I am suffering a partial download of the reverse geocode data again. Not sure if that's related to this. I'm creating a fix in the upstream library and when that is merged will revisit if this is still an issue