Background Upload Fails on Slow Connection

[kevincox]

trafficstars

I have searched the existing issues, both open and closed, to make sure this is not a duplicate report.

• [x] Yes

The bug

I am currently traveling in China. Most hotels have very slow wifi (at least when connecting outside of the firewall) so backing up images can take quite a while. For some reason this results in automatic background upload never finishing. This occurs on both my phone (Pixel 7a running Lineage OS) and my partner's phone (Pixel 6 running Google Android).

I believe that this mostly happens when uploading a large asset (usually videos). I have turned on the per-item upload notification and when I am watching this appears to be the case. It gets partway through a video then the notification just goes away and network activity stops. When opening the app ever the backup page just spins forever.

(It says no folders selected, but that is just a separate UI bug where lots of stuff on the backup icon and page don't differentiate between "loading" and "no results".)

I have checked the logs and there seems to be nothing, the process just exits silently. I have saved copies and will review + post the logs when I have more time (likely in a few weeks when I get home). My current theory is that the background process is just getting killed. This is either just happening after a while or particularly when no asset finishes in a while. Furthermore there appears to be no form of auto-restart so it never completes and be app UI is broken waiting on the background process until it is force killed and restarted.

Foreground upload does work. If I open the backup page (after force-killing the app to fix the backup page) and leave the screen on and focused on the app it reliably completes, so the problem doesn't appear to be the network in general, just the background process failing in some way.

The wifi isn't that slow. Usually in the 1-5Mbps range. I have seen this for assets as small as 256 MiB. So that should only take about 15min. Definitely not fast, but this could also be hit by people uploading properly large files on a decent connection.

The OS that Immich Server is running on

NixOS

Version of Immich Server

1.133.1

Version of Immich Mobile App

1.134.0 build.200

Platform with the issue

• [ ] Server
• [ ] Web
• [x] Mobile

Your docker-compose.yml content

None

Your .env content

None

Reproduction steps

1. Take a "large" video.
2. Connect to a slow network.
3. Fulfill your auto-backup conditions

Relevant log output

Additional information

No response

[kevincox]

I just ran another backup and it failed about about 10min 80% of the way through an almost 500MB file (so about 5.3 Mbps average). There were 0 logs generated.

[kevincox]

I tried again and failed again after about 10min with no logs. This time it got to 38% before stopping.

So quite possibly a 10min timeout somewhere. IDK if in Immich or in Android.

[kevincox]

Another attempt uploaded a few images first then failed on the large file after ~10min and 60%. This suggests that the timeout is per-file, but I wasn't paying enough attention to how long the initial uploads took to be sure that they aren't <1min.

[bo0tzz]

You're probably hitting the request timeout on your reverse proxy.

[kevincox]

Interesting idea. I checked the logs and it does look like the request is somewhat closed by nginx

{
  "cache_status": "",
  "connection_id": 110845,
  "connection_request_count": 39,
  "duration_s": 613.521,
  "protocol": "HTTP/1.1",
  "remote_addr": "REDACTED",
  "req_bytes": 271664546,
  "req_content_length": "507212944",
  "req_content_type": "multipart/form-data; boundary=dart-http-boundary-oSRTtwRfdVcnCLrMq9DM4C84YfMi.74jWoz586GwxNa8gTC4fWh",
  "req_host": "immich.example",
  "req_method": "POST",
  "req_pipelined": ".",
  "req_referrer": "",
  "req_uri": "/api/assets",
  "req_via": "",
  "req_user": "",
  "req_user_agent": "Dart/3.7 (dart:io)",
  "res_bytes": 0,
  "res_bytes_body": 0,
  "res_content_length": "",
  "res_content_type": "",
  "res_status": 408,
  "res_gzip_ratio": "",
  "ssl_session_id": "REDACTED",
  "ssl_session_reused": ".",
  "time": "2025-06-08T06:32:38+00:00",
  "http2": "",
  "uri": "https://immich.example/api/assets"
}

I didn't see any other logs about closed connections or similar.

However there are some strange things here:

1. nginx only supports a headers timeout and a read timeout. I have both of these at the default of 60s. I would assume that by this time the body was being read and it seems that traffic continued flowing so I wouldn't expect either timeout to be hit.
2. The request was 613s long. It is possible that with the slow connection headers took 13s but that seems unlikely as the request was pipelined. So this timeout isn't a clean number from nginx's point of view.
3. Slow uploads succeed in the foreground even though they take longer than 10min. nginx doesn't (directly) know about foreground vs background upload tasks.

So my current theory is that this 408 response is just a reaction to the client quitting. But I need to do more investigation to be sure. If the client got a 408 response would it log it? Because I am not seeing any logs.

[kevincox]

So me and my partner both took loads of pictures and many large videos yesterday but when we went to sleep we started a foreground upload. Both finished without any issues.

So there is definitely some difference between foreground and background uploads. There may be some server-side settings at play but it seems unlikely to be a major issue since foreground backup works perfectly.

I think there are a handful of contributing issues here:

1. Something stops background uploads (seemingly a timeout after about 10min)
2. When the background upload fails it is never resumed. (Other than killing and restarting the app.)
3. A single failed upload fails the background upload (rather than proceeding with other files). (This may not actually be relevant I'd the whole background process is dying, but there seems to be some sort of per-aseet behaviour)
4. There is no resumable upload support so files that fail due to a timeout are highly likely to continue failing even after killing and restarting the app.

[jrasm91]

Yup, that is all correct. Background workers have a 10 minute timeout. We've changed the code to use foreground upload for large assets, so I think that issue will be resolved in the new beta timeline. The other issues have been reported a bunch of time and there are probably several open issues for each one of those items before. We're actively working to make the backup algorithm work better, especially when it comes to failures and retries. I'm going to close this, since the information has already been reported in other open issues.

[kevincox]

Hi @jrasm91 what issues should I be following to know when this is resolved? Being able to background in the background is pretty important to me even if it takes more than 10 minutes for an individual asset.

[jrasm91]

I believe that specific issue has already been fixed (the ten minute timeout issue). Have you run into it again on the latest release?

[kevincox]

Ah ok. I will try to reproduce and report back if I hit it.