UnifiedPush (ntfy) not working

[MrRulf]

Steps to reproduce

1. Set up/start Element from f-droid
2. Set notification method to unifiedPush (ntfy in my case)
3. Close Element
4. Ask somebody to write u a message
5. Check if there is a notification (in my case, none)

My guess is that the problem is on Elements side because I'm only having this issue since the last update (1.4.34)

Outcome

What did you expect?

To get notifications

What happened instead?

I get no notifications

Your phone model

Fairphone 4

Operating system version

Android 12

Application version and app store

Element - Secure Messenger version 1.4.34 from f-droid

Homeserver

matrix.org

Will you send logs?

No

Are you willing to provide a PR?

No

[MrRulf]

Forgot to add, using the background sync of Element still works.

[MrRulf]

If somebody explains me how I can send useful logs I'm totally willing to send them, I just picked No because I think they won't help much, since the notification isn't broken or anything, just delayed until I open the app. About the PR, if I can help in any way, same as with the logs, I just don't think I'd be able to find the error in the code and fix it, why I picked No to give no wrong expectations.

[blacklight]

I can confirm the issue. ntfy push notifications used to work before the latest update, they stopped working after updating, and background sync is still working.

Are these things properly tested before pushing out a new stable release?

[p1gp1g]

176.126.240.158 got banned by ntfy.sh after too many requests, and looks like a matrix.org IP (https://github.com/matrix-org/synapse/issues?q=is%3Aissue+176.126.240.158).

Is there a list of matrix.org IP to whitelist them on ntfy.sh ?

[p1gp1g]

We need the ones for push requests (outgoing connections)

[blacklight]

Hmm this is weird. I use my own ntfy server for push notifications, and that's what my Android app is registered to. Does the IP blacklist on ntfy.sh impact also personal servers?

[p1gp1g]

It is possible: check the doc https://ntfy.sh/docs/config/#rate-limiting

[binwiederhier]

This should be resolved now. I whitelisted the matrix.org IP address using this https://github.com/binwiederhier/ntfy/commit/ed4cc86c5c28633e4a8809e5aa1af8be7bc903f0. Thanks @p1gp1g for the pointers.

[p1gp1g]

If you want to do the same setup than ntfy.sh:

/etc/ntfy/server.yml

visitor-request-limit-exempt-hosts: "176.126.240.158"

/etc/nginx/nginx.conf

        geo $limited {
                default 1;
                176.126.240.158/32 0;
        }
        map $limited $limitkey {
                1 $binary_remote_addr;
                0 "";
        }
        limit_req_zone $limitkey zone=one:10m rate=1r/s;

You should add ipv6 too your server supports it (2a00:1098:84:1c8::158)

Please not these IPs were discovered with the requests from the matrix.org server

[huguesdk]

i have the same problem. i’m using my own ntfy server. could it be a problem related to rate limiting when fluffychat notifications (that use the same ntfy server) continue to work?

i’ve noticed quite slow sync times recently. i don’t know whether it’s a client or server problem, though. i run my own synapse homeserver.

to be sure to get notifications on android, i use element, fluffychat and schildichat at the same time. i usually get 1 or 2 notifications, and not always from the same clients. it’s sadly funny.

people trying to use element instead of whatsapp or signal keep complaining to me that they often don’t receive notifications. would it be possible to have a safety measure that would ensure to at least perform a sync every hour or so, to catch messages for which notifications failed? i know android is quite aggressive regarding battery optimizations, but maybe it’s nevertheless possible?

[binwiederhier]

@huguesdk I recommend enabling debug (or trace mode) in ntfy for a while to see where the issue is. It's really verbose so you should be able to spot a problem if there is one in ntfy. Don't forget to turn it off though, because otherwise your logs will fill up.

[kojid0]

Any news on this? Element is not an instant messenger anymore... This should get maximum priority.

[bahur142]

I've never had any problems since I get using ntfy. I self host both - synapse and ntfy. Using schildichat fork instead element. Non of my friends complains about notifications. Are you sure your ntfy app shows silent notification "Listening for incoming notifications"? This should be enabled. Because if this silent notification is suppressed, you could miss some notifications on element. Also you should set ntfy app in android settings "battery optimization" to "Not optimized" to ensure the app will stay awake and not killed by the os.

[DeepBlueBubble]

Mine is also not working. I am using the FCM distributor one. When the App is in the background the notification will never arrive. It can receive when in the foreground. However when I switch to the Nextpush it works nomral.

[dessalines]

Same, I just switched from google's fcm to ntfy. The element subscription shows up in ntfy android, but no notifications come through until I manually open element.

[binwiederhier]

For anyone using the ntfy.sh server and not a selfhosted instance: I am seeing lots and lots of rate limited (aka rejected) requests from various matrix instances. There are very few whitelisted Matrix servers (matrix.org being one of them), so it may very well be the case that you are just a victim of rate limiting on the ntfy.sh server.

This only affects the public ntfy.sh instance. If you are selfhosting ntfy, i suggest you turn on debugging/tracing and look at the logs.

(I am the ntfy maintainer)

[dessalines]

I'm using the matrix.org homeserver, and I don't get notifications until I open up the element app. In the ntfy android app, it says ntfy.sh/[MY_ID_HERE] .... im.vector.app (UnifiedPush), so I assume everything is set up correctly, but this is my first time trying it out.

Is anyone correctly getting ntfy notifs?

[binwiederhier]

I'm using the matrix.org homeserver, and I don't get notifications until I open up the element app. In the ntfy android app, it says im.vector.app (UnifiedPush), so I assume everything is set up correctly, but this is my first time trying it out.

Is anyone correctly getting ntfy notifs?

You can enable logging in ntfy, and see if ntfy receives and forwards the notifications. Go to Settings -> Record logs, and then after you should have received a notification, you can Copy or Upload the logs. That should tell you more.

[krair]

This could be an entirely separate issue, but wanted to add my findings for posterity. I was also having this issue:

Setup:

• AlmaLinux 9
• Rootless Podman 4.2.0
• Synapse 1.76
• ntfy 1.30.1
• Element Android 1.5.22 (F-Droid)

Annoyingly the Element android app was able to receive the test notification from the "Troubleshooting Notifications" test. But I was not receiving message notifications. Looking through the homeserver.log makes me wonder if this test uses both my ntfy gateway and the matrix.org gateway https://matrix.org/_matrix/push/v1/notify.

I found my log was filled with:

synapse.http.client - 469 - INFO - httppush.process-2 - Error sending request to  POST https://ntfy.mydomain.tld/_matrix/push/v1/notify: DNSLookupError no results for hostname lookup: ntfy.mydomain.tld

For me the solution was to both:

• Add ntfy.mydomain.tld as an extra host to my synapse container, pointing to the internal IP address
• Add my internal IP range to the synapse config ip_range_whitelist as discussed in the docs.

Restarted the synapse container and started receiving instant notifications on the android app.

[FrederikSchack]

I'm using Element and Fluffychat and NTFY with the default server ntfy.sh. It's running on LineageOS on my Poco X3 Pro. It worked great for about a month, until I updated Element, but now I have the same issue, I don't always get notifications and friends are complaining that they don't receive notifications.

It further seems to me that there's no open source application that can get notifications reliably through (also Tox, Jami, XMPP and many others).

I'm not a developer, but I suspect that LineageOS is suffering under the same energy optimizations as Android: https://github.com/vector-im/element-android/blob/develop/docs/notifications.md#how-does-a-matrix-client-get-a-message-from-a-homeserver?

[binwiederhier]

ntfy.sh + Element works for me and many others. Try re-subscribing in Element by re-selecting the Notification method.

Also: I updated the Matrix code in ntfy a little which will make it less likely that notifications stop working. This was rolled out less than 24h ago, so it's worth a shot to retry it.

(FYI: I am the ntfy maintainer)

[kojid0]

Try re-subscribing in Element by re-selecting the Notification method

Unfortunately this didn't work for me and 2 friends (all Google Pixel phones with Android 13). Ntfy is still "working" in the background and is not being killed by the battery manager

[teohhanhui]

Unfortunately this didn't work for me and 2 friends (all Google Pixel phones with Android 13).

I'm on the Pixel 6a. What worked for me was to run "Troubleshoot Notifications" in Element. It detected a problem in "Endpoint Registration" and gave me a button to re-register that. Notifications have been working fine since then.

[casta]

Hello,

It seems I face the same issue. Using the Fdroid Element build. In my case, I’m using NextPush with personal matrix gateway configured. Messages correctly flows from matrix homeserver (matrix.im) to my push server, and from my push server to my android device.

Just to be sure that my NextPush client has not been shut-down by android, I restart it, then I send a message. Here is the logcat I get:

03-10 12:29:58.716 27157 28746 D RestartWorker: Restarting
03-10 12:29:58.717 27157 28746 D StartServiceCompanion: Starting the Listener
03-10 12:29:58.717 27157 28746 D StartServiceCompanion: Service is started: false
03-10 12:29:58.717 27157 28746 D StartServiceCompanion: nFails: 0
03-10 12:29:58.721 27157 27179 I WM-WorkerWrapper: Worker result SUCCESS for Work [ id=da8b502d-9a38-488e-b9be-b99af2faf3b1, tags={ org.unifiedpush.distributor.nextpush.services.RestartWorker } ]
03-10 12:29:58.727 27157 27157 I StartService: StartService created
03-10 12:29:58.733 27157 27157 D StartService: onStartCommand
03-10 12:29:58.734 27157 27157 D RestartNetworkCallback: Registering new ConnectivityManager
03-10 12:29:58.735  1833  3394 D ConnectivityService: requestNetwork for uid/pid:10327/27157 activeRequest: null callbackRequest: 1292 [NetworkRequest [ REQUEST id=1293, [ Capabilities: INTERNET&NOT_RESTRICTED&TRUSTED&NOT_VCN_MANAGED Uid: 10327 RequestorUid: 10327 RequestorPkg: org.unifiedpush.distributor.nextpush] ]] callback flags: 0 priority: 2147483647
03-10 12:29:58.738 27157 27254 D RestartNetworkCallback: Network is CONNECTED
03-10 12:29:58.738 27157 27254 D RestartNetworkCallback: Network Capabilities changed
03-10 12:29:58.745 27157 27157 D Api     : cSync done.
03-10 12:29:58.974 27157 27176 D OpenGLRenderer: endAllActiveAnimators on 0x77548176c0 (MenuPopupWindow$MenuDropDownListView) with handle 0x76647ee2e0
03-10 12:29:58.983 27157 19737 D FailureHandler: newEvent/Eventsource: okhttp3.internal.sse.RealEventSource@563fe3f
03-10 12:29:58.987 27157 19737 D SSEListener: onOpen: 200
03-10 12:29:58.987 27157 19737 D SSEListener: New SSE message event: start
03-10 12:29:58.992 27157 19737 D SSEListener: New SSE message event: keepalive
03-10 12:29:58.996 27157 19737 D SSEListener: New keepalive: 300
03-10 12:30:33.014 27157 19737 D SSEListener: New SSE message event: message
03-10 12:30:33.030 27157 19737 D Distributor: Message forwarded

Here is the corresponding message flowing through my NextPush server:

2a03:4000:60:d87:a494:8ff:fe4d:99b6 - - [10/Mar/2023:12:30:33 +0100] "POST /_matrix/push/v1/notify HTTP/1.1" 200 35 "-" "Synapse/1.77.0"

You can see the socket is restarting is setting up with initial messages (correctly setting keeplive to 300), then the 2 last lines, the NextPush client receive a message (from my matrix home server through my personal NextPush server)

The last logcat line is the forward of the message to the application. Then… nothing happens. No reaction from Element. But:

• If I start Element, notifications appears
• If I let Element running in background, notifications appears
• If I swipe Element from the tasks, notifications still appears for some time, but at some point, do not appears any more. Probably after Android started dozing Element.

If I uses the Playstore version with FCM push (I have microg on my phone), there are no such issues. So I would suspect something around the UnifiedPush listener activity that prevent Element to wake up ?

[p1gp1g]

If I swipe Element from the tasks, notifications still appears for some time, but at some point, do not appears any more. Probably after Android started dozing Element.

Can you check https://dontkillmyapp.com/ ?

This looks like the behavior of some vendor's aggressive battery "optimization". Some of the vendors even set applications in user force-stopped state which prevent other apps from starting it. Preventing user force-stopped application from starting is a good thing when the user has force stopped it :facepalm:

[casta]

I should not be affected by this since I’m running LineageOS 19 with microg, and I have disabled battery optimization for Element (which should not be necessary).

And remember: I do not have this behavior with the PlayStore build with FCM. Element is perfectly woken-up by the event from FCM (but I would like to avoid using FCM…)

[p1gp1g]

MicroG is a system app and is allowed to wake up app even in force-stopped state.

It would be interesting if you could get logcat of nextpush and element + nextcloud log when you don't receive notifications. Which version of nextpush/element do you use ?

Edit: As @binwiederhier said: we should move the discussion related to NextPush to https://matrix.to/#/#unifiedpush:matrix.org

[binwiederhier]

It feels like this issue has evolved and is no longer about ntfy not working, right? I would suggest to move troubleshooting of NextPush to the UP matrix room, and troubleshooting of ntfy to the ntfy Discord.

(I am the ntfy maintainer, and I'm happy to help with ntfy issues. But this is the Element GitHub issue tracker, so I feel like this discussion is misplaced. I'm going to unfollow the issue.)

[kojid0]

I'm on the Pixel 6a. What worked for me was to run "Troubleshoot Notifications" in Element. It detected a problem in "Endpoint Registration" and gave me a button to re-register that. Notifications have been working fine since then.

Unfortunately this didn't help. I still don't get any notifications in Element (F-Droid build). I'm on CalyxOS with MicroG, running Android 13. Ntfy standard server, both Element and ntfy are excluded from battery optimizations. Endpoint registration seems to be fine (Element notification troubleshooting).

[casta]

Not sure it’s specific to NextPush. I think I have the same behaviour with ntfy and UP-FCM (https://unifiedpush.org/users/distributors/fcm/).

After digging a bit more, it seems that the notification is broken as soon as Element is not running. For example just after booting the phone, on after killing the Element process. Then I have a strange behavior: “sometime”, the message is lost, and “sometime”, the broadcast intent is caught and element is starting, and the notification appear, with this logs:

03-10 15:24:49.179  4447  4930 D Distributor: Message forwarded
03-10 15:24:49.180  1849  1943 D CompatibilityChangeReporter: Compat change id reported: 143937733; UID 10326; state: ENABLED
03-10 15:24:49.200  1849  1943 I ActivityManager: Start proc 6180:im.vector.app/u0a326 for broadcast {im.vector.app/im.vector.app.core.pushers.VectorUnifiedPushMessagingReceiver}

For me it could be more like Element not intercepting the UnifiedPush broadcast intent under some conditions, or something like this.