element-x-ios
element-x-ios copied to clipboard
element-hq
Push broke about 10 days ago, and never recovered.
Steps to reproduce
- I no longer receive APNS push from Apple, despite the app theoretically setting a pusher on launch correctly.
Outcome
What did you expect?
push
What happened instead?
no push
Your phone model
No response
Operating system version
No response
Application version
226
Homeserver
No response
Will you send logs?
Yes
matrix.org synapse doesn't have a pusher for my current EIX device (SUMRMOAXDJ). it does have one for a stale EIX device from May 12th (RENYEZTIXC). Push broke on May 19th apparently. but why?
Turns out that Apple are rejecting APNS for the device with 410 Unregistered (which causes the pusher to be deleted on synapse):
May 30 17:00:52 corus sygnal-matrixorg2[2499435]: 2023-05-30 17:00:51,940 [2499435] sygnal.apnspushkin 280 - INFO - [37cbef36-6633-41ca-ab62-44743cc0c821] APNs token d6bf797ef43003f5543a6f188f61eff604c67bf1324d6c19893e3228fe7cda5f for pushkin io.element.elementx.nightly.ios.prod was rejected
: 410 Unregistered
May 30 17:00:52 corus sygnal-matrixorg2[2499435]: 2023-05-30 17:00:51,941 [2499435] sygnal.http 286 - INFO - [37cbef36-6633-41ca-ab62-44743cc0c821] Successfully delivered notifications with 1 rejected pushkeys
May 30 17:00:52 corus sygnal-matrixorg2[2499435]: 2023-05-30 17:00:51,941 [2499435] sygnal.access 357 - INFO - Handled request: "::ffff:x.x.x.x" - - [30/May/2023:17:00:51 +0000] "POST /_matrix/push/v1/notify HTTP/1.1" 200 62 "-" "Synapse/1.85.0rc1 (b=matrix-org-hotfixes,4427151a21)"
The reason is likely given by https://stackoverflow.com/a/48649312 - we only call registerForRemoteNotifications when the user grants push notif permission to the app, rather than on every launch. So the devicetoken may end up being wrong if we switch the app between push profiles (e.g. xcode v. TF), or presumably if we restore apps from backup or something.
So, presumably we should try to reregister with Apple for push on every cold launch.
(EI also reregisters for push if the user manually toggles push on & off in settings in the app, which EIX doesn't have yet. However, in general, we should surely spot if push is broken and nag the user to enable it if we can't automatically re-register for it, as per #992)
(i also had very similar symptoms on EI, come to think of it, after lending my EIR to Andy U a month or two back to debug something - so i suspect the same bug exists there.)
The associated rageshake shows that the app is setting up a pusher on startup:
2023-05-28T11:22:15.794279Z DEBUG send{homeserver="https://matrix-client.matrix.org/" server_versions=[V1_0, V1_1, V1_2, V1_3, V1_4, V1_5] config=RequestConfig { timeout: 30s } request_id="REQ-2" request_size="520 B" path="/_matrix/client/v3/pushers/set"}: matrix_sdk::http_client: crates/matrix-sdk/src/http_client.rs:347: Sending request
2023-05-28T11:22:16.093630Z DEBUG send{homeserver="https://matrix-client.matrix.org/" server_versions=[V1_0, V1_1, V1_2, V1_3, V1_4, V1_5] config=RequestConfig { timeout: 30s } request_id="REQ-2" request_size="520 B" path="/_matrix/client/v3/pushers/set" status=200 response_size="2 B"}: matrix_sdk::http_client: crates/matrix-sdk/src/http_client.rs:353: Got response
2023-05-28T11:22:16.094137Z INFO root: elementx: NotificationManager.swift:147: [NotificationManager] set pusher succeeded
The register function returns the granted == true even if the UI is not shown. Which means that as long as the user granted us permission either in the past or in that moment we always register for pushes.
However I am starting to wonder if the correct approach would be to unregister on logouts or on env changes
Okay did some investigations and tried to reproduce the issue in the same way it happened initially (all the following steps where done on both the normal release and the nightly release):
- Installed the TF version
- Used the app, tested pushes
- Then built the app with Xcode
- Used the app, tested pushes
- Repeat these steps for a while
I could not reproduce the issue sadly (or maybe luckily?). Probably then the dev to release env swap was not the cause. Since on the client side everything looks ok for both El-X and El-iOS and we experienced the same issues for both the app so suddenly, it could also be a temporary disservice by Apple but ofc can't be sure about that 100%.
I would say that the best course of action is to keep this issue tracked, and if it presents again, we should send the rageshake logs immediatelly, because the latest ones had an unlogged time gap between NSE stopping receiving APNS and the app logs (so we couldn't really see what happened during the registration process at the moment APNS stopped sending the pushes).
Also I added a specific log for when the device token registration fails in the app, so we can track that if it happens.
Given we're re-registering on every launch, it sounds like we're going to need a callback from the backend to say that the key was rejected so we can remove it and request a new one.
@ara4n I'm going to close this issue for now as we haven't seen it recently and cannot reproduce right now. Please reopen if you are still experiencing it.
Haven't seen this in quite some time now, closing until hearing different