How to debug push notification issues ?

[Notmarrco]

trafficstars

Background information

• Dendrite version or git SHA: 0.9.1
• Monolith or Polylith?: monolyth
• SQLite3 or Postgres?: postgres
• Running in Docker?: no
• go version: 1.18
• Client used (if applicable): element

Description

• What is the problem: when debugging notifications with element, element is waiting for the push notification that doesn't come
• Who is affected: me
• How is this bug manifesting: I have no push notification on element
• When did this first appear: dendrite 0.8.9 I think, before that it worked well

Steps to reproduce

• open element
• debug notification
• element get google fcm token etc., every tick is green
• wait for "push notification" from dendrite

There isn't much logs related to this issue.

time="2022-08-12T13:59:59.810507178Z" level=info msg="Cleaning old notifications" func=github.com/matrix-org/dendrite/userapi.NewInternalAPI.func1 file="/opt/dendrite/userapi/userapi.go:101"
time="2022-08-12T14:00:10.311907582Z" level=debug msg="Responding to sync since client gave up or timeout was reached" func="github.com/matrix-org/dendrite/syncapi/sync.(*RequestPool).OnIncomingSyncRequest.func1" file="/opt/dendrite/syncapi/sync/requestpool.go:268" device_id=1Ih6kxxx limit=20 req.id=6Xp2MXvqxxxl req.method=GET req.path=/_matrix/client/r0/sync since=s568244_0_197_610_567943_568246_55_107770_0 timeout=30s user_id="@user:dendrite.server.example"
time="2022-08-12T14:00:23.227203919Z" level=debug msg="Database returned 1 pushers" func="github.com/matrix-org/dendrite/userapi/storage/postgres.(*pushersStatements).SelectPushers" file="/opt/dendrite/userapi/storage/postgres/pusher_table.go:139"
time="2022-08-12T14:00:40.529283666Z" level=debug msg="Responding to sync since client gave up or timeout was reached" func="github.com/matrix-org/dendrite/syncapi/sync.(*RequestPool).OnIncomingSyncRequest.func1" file="/opt/dendrite/syncapi/sync/requestpool.go:268" device_id=1Ih6kxxx limit=20 req.id=PAC15XNmxxxx req.method=GET req.path=/_matrix/client/r0/sync since=s568244_0_197_610_567943_568246_55_107770_0 timeout=30s user_id="@user:dendrite.server.example"

I also found this log, just after receiving a message :

time="2022-08-12T13:42:03.664811412Z" level=info msg=PerformPusherCreation func="github.com/matrix-org/dendrite/userapi/internal.(*UserInternalAPI).PerformPusherSet" file="/opt/dendrite/userapi/internal/api.go:699" display_name=Element localpart=marc pushkey="eTpEjS2aS2eabktg1jqSUe:APA91bxxx" req.id=C6l4Needxxx req.method=POST req.path=/_matrix/client/r0/pushers/set user_id="@user:dendrite.server.example"

Is there a way to better debug issues with push notifications ? Everything seems fine, but element cannot receive push notifications.

[BobWs]

Having the same problem with Dendrite monolith server 0.94 and Elements MacOS dekstop app and also with iOS FluffyChat. No push notifications.

time="2022-08-24T13:30:00.310545728Z" level=warning msg="Deleting pushers rejected by the HTTP push gateway" app_id0=chat.fluffy.fluffychat localpart=USER num_devices=1
time="2022-08-24T13:35:52.221466466Z" level=warning msg="Deleting pushers rejected by the HTTP push gateway" app_id0=chat.fluffy.fluffychat localpart=USER num_devices=1
time="2022-08-24T13:35:52.789982190Z" level=warning msg="Deleting pushers rejected by the HTTP push gateway" app_id0=chat.fluffy.fluffychat localpart=USER num_devices=1
time="2022-08-24T13:35:53.145954769Z" level=warning msg="Deleting pushers rejected by the HTTP push gateway" app_id0=chat.fluffy.fluffychat localpart=USER num_devices=1

[Notmarrco]

Fwiw I still have this issue with v0.9.5.

[BobWs]

Fwiw I still have this issue with v0.9.5.

Guess it is not a priority for development! Testing also matrix/conduit and there push notifications is working fine with Fluffychat and Elements apps on iOS devices. This was fixed after reporting it.

[S7evinK]

@Notmarrco: To clarify: "wait for "push notification" from dendrite" - do you mean one of the steps in Element Android when troubleshooting notifications?

Having the same problem with Dendrite monolith server 0.94 and Elements MacOS dekstop app and also with iOS FluffyChat. No push notifications.

time="2022-08-24T13:30:00.310545728Z" level=warning msg="Deleting pushers rejected by the HTTP push gateway" app_id0=chat.fluffy.fluffychat localpart=USER num_devices=1
time="2022-08-24T13:35:52.221466466Z" level=warning msg="Deleting pushers rejected by the HTTP push gateway" app_id0=chat.fluffy.fluffychat localpart=USER num_devices=1
time="2022-08-24T13:35:52.789982190Z" level=warning msg="Deleting pushers rejected by the HTTP push gateway" app_id0=chat.fluffy.fluffychat localpart=USER num_devices=1
time="2022-08-24T13:35:53.145954769Z" level=warning msg="Deleting pushers rejected by the HTTP push gateway" app_id0=chat.fluffy.fluffychat localpart=USER num_devices=1

Unfortunately the push gateway API doesn't provide any information about "why" a push was rejected, just that something was rejected.

[Notmarrco]

@S7evinK yes exactly, when troubleshooting notifications, the 10th step

[S7evinK]

Looking at the Element Android code, this seems to be connecting to the push gateway directly, so Dendrite isn't involved in this. (Also didn't see any connections to my server while running the test) Probably best to create an issue for Element Android or checking that an adblocker/a firewall doesn't block the connection to the push gateway (default would be https://matrix.org/_matrix/push/v1/notify)

[Notmarrco]

sorry I don't understand how element could work without contacting dendrite :thinking: What is the "push gateway" ? some google platform to send push notifications ? but it works well with other apps

Afaik I'm the only one to have this issue with element + dendrite (in BobWs case its with fulffychat), so I'm willing to think it's something I do or didn't do during versions upgrade, but I don't know what.

@S7evinK Do you have any pointer on what I could do to help debug please ?

I tried curl https://my_dendrite/_matrix/push/v1/notify?access_token=xxxx but I get a 404 error. (with this token I successfully sent messages to a matrix room)

edit:

well it seems to indeed be an element issue : https://github.com/vector-im/element-android/issues/6896

I'll wait for this release and close the issue if it's really linked !

[Notmarrco]

Answering my own questions and closing this issue since it's not dendrite related.

A very good explanation lives here : https://github.com/vector-im/element-android/blob/main/docs/notifications.md sadly I didn't found this page at first (and this one with api examples : https://spec.matrix.org/unstable/push-gateway-api/)

In short dendrite must send the notification to the push gateway ("new vector"-owned sygnal server with their developper token) that can use FCM (google push platform) that sends the notification to the client device.

To debug I have to verify each step of the notification, and POST-ing to the sygnal server (https://spec.matrix.org/unstable/push-gateway-api/#post_matrixpushv1notify) as suggested by @S7evinK from the server running dendrite can be used to verify that this server can contact the push gateway.

In my case, it was element that couldn't get the push notification from FCM because of a bad token (as stated in the linked issue) and reinstalling the app generates a new valid token, therefore activating push notifications again.

Thanks