iOS Background updates eventually hang

iOS device model, version and app version

Model Name: iPhone 15 Pro Software Version: 17.4 App version: 2024.2

Home Assistant Core Version 2024.3.0

Describe the bug Eventually background sensor updates stop happening, running the "Update sensors" shortcut times out. Force closing the app brings everything back to life and the shortcut starts working again.

To Reproduce Have iOS app pushing sensor updates in background for a ~period of time.

Expected behavior App continues to function without repeatedly force closing.

Mar 10 '24 07:03 lwis

When you open the app, does the sensors get updated correctly? I mean, when it is in foreground all works? Do you have "local push" enabled?

Mar 11 '24 15:03 bgoncal

It does not update when the app is in the foreground.

I do not have local push enabled, or an internal URL configured.

Mar 11 '24 15:03 lwis

Did you observe a pattern for when does it stop reporting? In background I know for sure the sensor updates wont be live without local push enabled but foreground should be working properly.

Mar 11 '24 15:03 bgoncal

Unfortunately there doesn't seem to be a pattern to it, I have no way to reproduce. Is there anything useful I can capture the next time it happens?

Mar 11 '24 16:03 lwis

Your logs. You can export going in Debugging > export

Mar 11 '24 17:03 bgoncal

Can you check the latest testflight beta? It may have fixed this

Mar 20 '24 13:03 bgoncal

I'll update to the latest beta and continue to monitor! 🤞🏻

Mar 20 '24 16:03 lwis

Still happening unfortunately, this is what I see when the shortcut fails;

Mar 20 '24 19:03 lwis

Home Assistant_2024-03-21_181706.txt Home Assistant_2024-03-21_192239.txt Home Assistant_2024-03-21_200921.txt Home Assistant_2024-03-21_191428.txt

Mar 22 '24 00:03 dmulcahey

POSIXErrorCode(rawValue: 28): No space left on device) Is thrown constantly by the app when the WS connection breaks

was something changed in the test flight version related to URLsessions or any kind of networking introspection libraries?

Mar 22 '24 00:03 dmulcahey

@bgoncal let me know if there is anything you need to assist in debugging this.

Mar 22 '24 23:03 dmulcahey

https://stackoverflow.com/questions/67318867/error-domain-nsposixerrordomain-code-28-no-space-left-on-device-userinfo-kcf

This looks related, I'll investigate next week

Mar 22 '24 23:03 bgoncal

RC still has the issue. I’ll also note that when this happens and you open the app after it has been in the background it doesn’t connect to HA at all. It fails and HA logs the following:

Logger: homeassistant.components.websocket_api.http.connection
Source: components/websocket_api/http.py:461
integration: Home Assistant WebSocket API (documentation, issues)
First occurred: March 24, 2024 at 6:32:56 PM (6 occurrences)
Last logged: 7:21:09 AM

[281473247231552] from 192.168.1.68 (Home Assistant/2024.3 (io.robbie.HomeAssistant; build:2024.603; iOS 17.4.1)): Disconnected: Did not receive auth message within 10 seconds
[281472302263104] from fe80::1460:ed53:6ed7:34ed (Home Assistant/2024.3 (io.robbie.HomeAssistant; build:2024.603; iOS 17.4.1)): Disconnected: Did not receive auth message within 10 seconds
[281472222859200] from 192.168.1.68 (Home Assistant/2024.3 (io.robbie.HomeAssistant; build:2024.603; iOS 17.4.1)): Disconnected: Did not receive auth message within 10 seconds
[281471861314368] from fe80::1460:ed53:6ed7:34ed (Mozilla/5.0 (iPhone; CPU iPhone OS 17_4_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Home Assistant/2024.3 (io.robbie.HomeAssistant; build:2024.608; iOS 17.4.1) Mobile/HomeAssistant, like Safari): Disconnected: Did not receive auth message within 10 seconds
[281471753716160] from fe80::1460:ed53:6ed7:34ed (Mozilla/5.0 (iPhone; CPU iPhone OS 17_4_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Home Assistant/2024.3 (io.robbie.HomeAssistant; build:2024.608; iOS 17.4.1) Mobile/HomeAssistant, like Safari): Disconnected: Did not receive auth message within 10 seconds

hard closing the app and reopening it resolves the issue. I have been running test flight versions since they started getting published again and this just started with this cycle of work: after CarPlay release version so maybe 2 or 3 test flight pushes back.

Mar 25 '24 11:03 dmulcahey

I just thought of something else. The timing could possibly coincide with the release of iOS 17.4

Mar 25 '24 11:03 dmulcahey

Can you consistently reproduce the issue in some way? Like rebooting HA with the App open for example. I would like to have the same issue happening on my device to investigate it better, for now I couldn't reproduce

Mar 25 '24 12:03 bgoncal

Having a Shortcut automation which fires on power connect and disconnect to "Update Sensors" in the background is the way this manifests most often for me.

Mar 25 '24 12:03 lwis

Can you consistently reproduce the issue in some way? Like rebooting HA with the App open for example. I would like to have the same issue happening on my device to investigate it better, for now I couldn't reproduce

Just leaving the app alone in the background for like 30 mins is all it takes for me

Mar 25 '24 12:03 dmulcahey

@dmulcahey Im checking your logs, when they happened were you somewhere http://192.168.1.78:8123 was accessible? (your home probably) or were you using external URL?

Mar 25 '24 13:03 bgoncal

I work from home and am generally here. I just opened the app from the background again. Last open was around the time of my last reply and it happened again. I’ll get new logs for you in a few. Can you give me somewhere to send an iCloud link so I can get you the full bundle from the app?

Mar 25 '24 14:03 dmulcahey

Are you in discord? You can ping me there @bgoncal2

Mar 25 '24 14:03 bgoncal

Im preparing a testflight build with a possible fix , when you see 2024.612 in testflight you can test it

Mar 26 '24 11:03 bgoncal

Im preparing a testflight build with a possible fix , when you see 2024.612 in testflight you can test it

Just saw this update and installed it. Will report back

Mar 26 '24 22:03 dmulcahey

Im preparing a testflight build with a possible fix , when you see 2024.612 in testflight you can test it

Initial testing looks good. Let’s see how it does overnight

Mar 27 '24 00:03 dmulcahey

Everything is still working. What was the issue?

Mar 27 '24 10:03 dmulcahey

I'm still investigating but initially I believe the issue is that StarScream lib we use for websocket interaction creates one URLSession per request, which in a retry loop goes over iOS limits. I believe this wasn't happening in some updates ago because we were still supporting iOS 12 at that time and this limit was not applied. I created a PR to starscream https://github.com/daltoniam/Starscream/pull/1011

Mar 27 '24 11:03 bgoncal

Heads up that this fix is not on our main branch yet, so if you see new builds coming in testflight, they may revert this fix

Mar 27 '24 11:03 bgoncal

@lwis can you also check if this works for you?

Mar 27 '24 14:03 bgoncal

Updated this morning, so maybe too early to tell, but I've not run into any issues yet.

Mar 27 '24 15:03 lwis

Feel free to reopen this issue if needed, I'm closing this issue for now and planning the next release to include the fix

Apr 03 '24 13:04 bgoncal

Bad news; it happened again this morning. I'm unable to run the shortcut directly at all. The app is fully functional. Closing the app allows the shortcut to run again.

I grabbed logs before closing the app.

2024.624

Apr 06 '24 05:04 lwis