dendrite icon indicating copy to clipboard operation
dendrite copied to clipboard

Published 20 hours ago • verified publisher icon matrix-org

DAG split with Synapse in room version 10

Open Mikaela opened this issue 2 years ago • 5 comments

Background information

  • Dendrite version or git SHA: 0.9.5 and ed79e862 and earlier.
  • Monolith or Polylith?: unknown
  • SQLite3 or Postgres?: unknown
  • Running in Docker?: unknown
  • go version: unknown
  • Client used (if applicable): N/A

Description

  • What is the problem: In private room !DuAMUOvPwiaiqWgChA:pikaviestin.fi all Dendrite users are unable to see @aminda.matterbridge:jae.fi (PL100) thus unable to see messages from IRC, XMPP, Telegram...
  • Who is affected: Directly all Dendrite users (rose.place, kumma.juttu.asia, pain.agency), indirectly everyone trying to communicate in the room.
  • How is this bug manifesting: invisible user, logs on auth-chain failure (below)
  • When did this first appear: It was brought to my attention on 2022-08-26 around 17 UTC+3.

Steps to reproduce

Unknown. Additionally I am not hosting a Dendrite by myself, but I am unable to get actual Dendrite users to report it and a Matrix.org/Vector.im developer expressed a wish for bugs to be reported using proper channels and this is the best I can do.

Belows are the logs grepping for jae.fi from rose.place:

Aug 27 00:05:37 MooMatrix dendrite[36730]: time="2022-08-26T14:05:37.868130310Z" level=warning msg="Error authing soft-failed event" context=missing error="eventauth: sender \"@aminda.matterbridge:jae.fi\" not in room" event_id="$Ielgb5UY9PwxLehGzJyX2n84FGeFZdnuAsGlcJgYoUE" kind=KindNew origin=jae.fi room_id="!DuAMUOvPwiaiqWgChA:pikaviestin.fi" type=m.room.message
Aug 27 00:05:37 MooMatrix dendrite[36730]: time="2022-08-26T14:05:37.905364358Z" level=warning msg="Stored rejected event" context=missing error="eventauth: sender \"@aminda.matterbridge:jae.fi\" not in room" event_id="$Ielgb5UY9PwxLehGzJyX2n84FGeFZdnuAsGlcJgYoUE" kind=KindNew origin=jae.fi room_id="!DuAMUOvPwiaiqWgChA:pikaviestin.fi" type=m.room.message
Aug 27 00:05:37 MooMatrix dendrite[36730]: time="2022-08-26T14:05:37.905611736Z" level=warning msg="Roomserver rejected event" error="eventauth: sender \"@aminda.matterbridge:jae.fi\" not in room" event_id="$Ielgb5UY9PwxLehGzJyX2n84FGeFZdnuAsGlcJgYoUE" room_id="!DuAMUOvPwiaiqWgChA:pikaviestin.fi" type=m.room.message
Aug 27 00:07:42 MooMatrix dendrite[36730]: time="2022-08-26T14:07:42.951133062Z" level=warning msg="Error authing soft-failed event" context=missing error="eventauth: sender \"@aminda.matterbridge:jae.fi\" not in room" event_id="$5iXqga1sYPscLe4YmctXE_5zh7PE7ooDxEk_UNJRVfo" kind=KindNew origin=jae.fi room_id="!DuAMUOvPwiaiqWgChA:pikaviestin.fi" type=m.room.message
Aug 27 00:07:42 MooMatrix dendrite[36730]: time="2022-08-26T14:07:42.976506313Z" level=warning msg="Stored rejected event" context=missing error="eventauth: sender \"@aminda.matterbridge:jae.fi\" not in room" event_id="$5iXqga1sYPscLe4YmctXE_5zh7PE7ooDxEk_UNJRVfo" kind=KindNew origin=jae.fi room_id="!DuAMUOvPwiaiqWgChA:pikaviestin.fi" type=m.room.message
Aug 27 00:07:42 MooMatrix dendrite[36730]: time="2022-08-26T14:07:42.981363503Z" level=warning msg="Roomserver rejected event" error="eventauth: sender \"@aminda.matterbridge:jae.fi\" not in room" event_id="$5iXqga1sYPscLe4YmctXE_5zh7PE7ooDxEk_UNJRVfo" room_id="!DuAMUOvPwiaiqWgChA:pikaviestin.fi" type=m.room.message
Aug 27 00:08:15 MooMatrix dendrite[36730]: time="2022-08-26T14:08:15.707442958Z" level=warning msg="Error authing soft-failed event" context=missing error="eventauth: sender \"@aminda.matterbridge:jae.fi\" not in room" event_id="$j3GeBjy3RGEgFLMX1om0zIV7NiWNE4QSe5u5ZXAZb6Q" kind=KindNew origin=jae.fi room_id="!DuAMUOvPwiaiqWgChA:pikaviestin.fi" type=m.room.message
Aug 27 00:08:15 MooMatrix dendrite[36730]: time="2022-08-26T14:08:15.757836042Z" level=warning msg="Stored rejected event" context=missing error="eventauth: sender \"@aminda.matterbridge:jae.fi\" not in room" event_id="$j3GeBjy3RGEgFLMX1om0zIV7NiWNE4QSe5u5ZXAZb6Q" kind=KindNew origin=jae.fi room_id="!DuAMUOvPwiaiqWgChA:pikaviestin.fi" type=m.room.message
Aug 27 00:08:15 MooMatrix dendrite[36730]: time="2022-08-26T14:08:15.758032311Z" level=warning msg="Roomserver rejected event" error="eventauth: sender \"@aminda.matterbridge:jae.fi\" not in room" event_id="$j3GeBjy3RGEgFLMX1om0zIV7NiWNE4QSe5u5ZXAZb6Q" room_id="!DuAMUOvPwiaiqWgChA:pikaviestin.fi" type=m.room.message
Aug 27 00:08:21 MooMatrix dendrite[36730]: time="2022-08-26T14:08:21.749985330Z" level=warning msg="Error authing soft-failed event" context=missing error="eventauth: sender \"@aminda.matterbridge:jae.fi\" not in room" event_id="$Sh5ycDghD3jPtPRd8kFNFaR2TwbHHzgi-SpljXql7J8" kind=KindNew origin=jae.fi room_id="!DuAMUOvPwiaiqWgChA:pikaviestin.fi" type=m.room.message
Aug 27 00:08:21 MooMatrix dendrite[36730]: time="2022-08-26T14:08:21.782408713Z" level=warning msg="Stored rejected event" context=missing error="eventauth: sender \"@aminda.matterbridge:jae.fi\" not in room" event_id="$Sh5ycDghD3jPtPRd8kFNFaR2TwbHHzgi-SpljXql7J8" kind=KindNew origin=jae.fi room_id="!DuAMUOvPwiaiqWgChA:pikaviestin.fi" type=m.room.message
Aug 27 00:08:21 MooMatrix dendrite[36730]: time="2022-08-26T14:08:21.782719203Z" level=warning msg="Roomserver rejected event" error="eventauth: sender \"@aminda.matterbridge:jae.fi\" not in room" event_id="$Sh5ycDghD3jPtPRd8kFNFaR2TwbHHzgi-SpljXql7J8" room_id="!DuAMUOvPwiaiqWgChA:pikaviestin.fi" type=m.room.message
Aug 27 00:08:39 MooMatrix dendrite[36730]: time="2022-08-26T14:08:39.696240725Z" level=info msg="Signalling other goroutines waiting for this goroutine to generate the thumbnail." Height=16 MediaID=9576df037f6c01fa16dbc022d9b17a5fbbd4c029 Origin=jae.fi ResizeMethod=crop Width=16 req.id=hedvbv8SfmiE req.method=GET req.path=/_matrix/media/r0/thumbnail/jae.fi/9576df037f6c01fa16dbc022d9b17a5fbbd4c029
Aug 27 00:08:50 MooMatrix dendrite[36730]: time="2022-08-26T14:08:50.895167474Z" level=warning msg="Error authing soft-failed event" context=missing error="eventauth: sender \"@aminda.matterbridge:jae.fi\" not in room" event_id="$2E2gwwQsaZ5LsMzRVzIp4AXnTll3muxqddl7U1trM4c" kind=KindNew origin=jae.fi room_id="!DuAMUOvPwiaiqWgChA:pikaviestin.fi" type=m.room.message
Aug 27 00:08:50 MooMatrix dendrite[36730]: time="2022-08-26T14:08:50.925154497Z" level=warning msg="Stored rejected event" context=missing error="eventauth: sender \"@aminda.matterbridge:jae.fi\" not in room" event_id="$2E2gwwQsaZ5LsMzRVzIp4AXnTll3muxqddl7U1trM4c" kind=KindNew origin=jae.fi room_id="!DuAMUOvPwiaiqWgChA:pikaviestin.fi" type=m.room.message
Aug 27 00:08:50 MooMatrix dendrite[36730]: time="2022-08-26T14:08:50.925338218Z" level=warning msg="Roomserver rejected event" error="eventauth: sender \"@aminda.matterbridge:jae.fi\" not in room" event_id="$2E2gwwQsaZ5LsMzRVzIp4AXnTll3muxqddl7U1trM4c" room_id="!DuAMUOvPwiaiqWgChA:pikaviestin.fi" type=m.room.message
Aug 27 00:09:45 MooMatrix dendrite[36730]: time="2022-08-26T14:09:45.717676590Z" level=warning msg="Error authing soft-failed event" context=missing error="eventauth: sender \"@aminda.matterbridge:jae.fi\" not in room" event_id="$YSMjFbr0IzjcYS6zILRO1S7LpXEae1MYTo2Id8RRHCs" kind=KindNew origin=jae.fi room_id="!DuAMUOvPwiaiqWgChA:pikaviestin.fi" type=m.room.message
Aug 27 00:09:45 MooMatrix dendrite[36730]: time="2022-08-26T14:09:45.747887927Z" level=warning msg="Stored rejected event" context=missing error="eventauth: sender \"@aminda.matterbridge:jae.fi\" not in room" event_id="$YSMjFbr0IzjcYS6zILRO1S7LpXEae1MYTo2Id8RRHCs" kind=KindNew origin=jae.fi room_id="!DuAMUOvPwiaiqWgChA:pikaviestin.fi" type=m.room.message
Aug 27 00:09:45 MooMatrix dendrite[36730]: time="2022-08-26T14:09:45.748582899Z" level=warning msg="Roomserver rejected event" error="eventauth: sender \"@aminda.matterbridge:jae.fi\" not in room" event_id="$YSMjFbr0IzjcYS6zILRO1S7LpXEae1MYTo2Id8RRHCs" room_id="!DuAMUOvPwiaiqWgChA:pikaviestin.fi" type=m.room.message
Aug 27 00:09:46 MooMatrix dendrite[36730]: time="2022-08-26T14:09:46.390589426Z" level=warning msg="Error authing soft-failed event" context=missing error="eventauth: sender \"@aminda.matterbridge:jae.fi\" not in room" event_id="$5X0co4Aj9ZqGyWG20ux8JZd-IXp0-9k_vGtYOT_34e4" kind=KindNew origin=jae.fi room_id="!DuAMUOvPwiaiqWgChA:pikaviestin.fi" type=m.room.message
Aug 27 00:09:46 MooMatrix dendrite[36730]: time="2022-08-26T14:09:46.439700737Z" level=warning msg="Stored rejected event" context=missing error="eventauth: sender \"@aminda.matterbridge:jae.fi\" not in room" event_id="$5X0co4Aj9ZqGyWG20ux8JZd-IXp0-9k_vGtYOT_34e4" kind=KindNew origin=jae.fi room_id="!DuAMUOvPwiaiqWgChA:pikaviestin.fi" type=m.room.message
Aug 27 00:09:46 MooMatrix dendrite[36730]: time="2022-08-26T14:09:46.451772265Z" level=warning msg="Roomserver rejected event" error="eventauth: sender \"@aminda.matterbridge:jae.fi\" not in room" event_id="$5X0co4Aj9ZqGyWG20ux8JZd-IXp0-9k_vGtYOT_34e4" room_id="!DuAMUOvPwiaiqWgChA:pikaviestin.fi" type=m.room.message
Aug 27 00:10:09 MooMatrix dendrite[36730]: time="2022-08-26T14:10:09.773209236Z" level=warning msg="Error authing soft-failed event" context=missing error="eventauth: sender \"@aminda.matterbridge:jae.fi\" not in room" event_id="$0LdK3LtTNCmrR_AQ2utE0ch1oMPA1Pa2fPrZ1TyUSEY" kind=KindNew origin=jae.fi room_id="!DuAMUOvPwiaiqWgChA:pikaviestin.fi" type=m.room.message
Aug 27 00:10:09 MooMatrix dendrite[36730]: time="2022-08-26T14:10:09.807152350Z" level=warning msg="Stored rejected event" context=missing error="eventauth: sender \"@aminda.matterbridge:jae.fi\" not in room" event_id="$0LdK3LtTNCmrR_AQ2utE0ch1oMPA1Pa2fPrZ1TyUSEY" kind=KindNew origin=jae.fi room_id="!DuAMUOvPwiaiqWgChA:pikaviestin.fi" type=m.room.message
Aug 27 00:10:09 MooMatrix dendrite[36730]: time="2022-08-26T14:10:09.807364679Z" level=warning msg="Roomserver rejected event" error="eventauth: sender \"@aminda.matterbridge:jae.fi\" not in room" event_id="$0LdK3LtTNCmrR_AQ2utE0ch1oMPA1Pa2fPrZ1TyUSEY" room_id="!DuAMUOvPwiaiqWgChA:pikaviestin.fi" type=m.room.message
Aug 27 00:10:30 MooMatrix dendrite[36730]: time="2022-08-26T14:10:30.726527954Z" level=warning msg="Error authing soft-failed event" context=missing error="eventauth: sender \"@aminda.matterbridge:jae.fi\" not in room" event_id="$3FUWnl-_AQotkAVtsST-wvQ5HqbbVQ5KVAsEYdCd3lY" kind=KindNew origin=jae.fi room_id="!DuAMUOvPwiaiqWgChA:pikaviestin.fi" type=m.room.message
Aug 27 00:10:30 MooMatrix dendrite[36730]: time="2022-08-26T14:10:30.769516734Z" level=warning msg="Stored rejected event" context=missing error="eventauth: sender \"@aminda.matterbridge:jae.fi\" not in room" event_id="$3FUWnl-_AQotkAVtsST-wvQ5HqbbVQ5KVAsEYdCd3lY" kind=KindNew origin=jae.fi room_id="!DuAMUOvPwiaiqWgChA:pikaviestin.fi" type=m.room.message
Aug 27 00:10:30 MooMatrix dendrite[36730]: time="2022-08-26T14:10:30.769728499Z" level=warning msg="Roomserver rejected event" error="eventauth: sender \"@aminda.matterbridge:jae.fi\" not in room" event_id="$3FUWnl-_AQotkAVtsST-wvQ5HqbbVQ5KVAsEYdCd3lY" room_id="!DuAMUOvPwiaiqWgChA:pikaviestin.fi" type=m.room.message
Aug 27 00:11:05 MooMatrix dendrite[36730]: time="2022-08-26T14:11:05.233184215Z" level=warning msg="Error authing soft-failed event" context=missing error="eventauth: sender \"@aminda.matterbridge:jae.fi\" not in room" event_id="$Rw_YjpVYMYqJ38h6stW_XlHm4kmjKhmahcEJP6a8yc4" kind=KindNew origin=jae.fi room_id="!DuAMUOvPwiaiqWgChA:pikaviestin.fi" type=m.room.message
Aug 27 00:11:05 MooMatrix dendrite[36730]: time="2022-08-26T14:11:05.296729067Z" level=warning msg="Stored rejected event" context=missing error="eventauth: sender \"@aminda.matterbridge:jae.fi\" not in room" event_id="$Rw_YjpVYMYqJ38h6stW_XlHm4kmjKhmahcEJP6a8yc4" kind=KindNew origin=jae.fi room_id="!DuAMUOvPwiaiqWgChA:pikaviestin.fi" type=m.room.message
Aug 27 00:11:05 MooMatrix dendrite[36730]: time="2022-08-26T14:11:05.296950833Z" level=warning msg="Roomserver rejected event" error="eventauth: sender \"@aminda.matterbridge:jae.fi\" not in room" event_id="$Rw_YjpVYMYqJ38h6stW_XlHm4kmjKhmahcEJP6a8yc4" room_id="!DuAMUOvPwiaiqWgChA:pikaviestin.fi" type=m.room.message
Aug 27 00:11:14 MooMatrix dendrite[36730]: time="2022-08-26T14:11:14.245570209Z" level=warning msg="Error authing soft-failed event" context=missing error="eventauth: sender \"@aminda.matterbridge:jae.fi\" not in room" event_id="$qwTRBD_nE07H9BA29T6vAkBzZAWDADX-GzOFgKatImw" kind=KindNew origin=jae.fi room_id="!DuAMUOvPwiaiqWgChA:pikaviestin.fi" type=m.room.message
Aug 27 00:11:14 MooMatrix dendrite[36730]: time="2022-08-26T14:11:14.311632002Z" level=warning msg="Stored rejected event" context=missing error="eventauth: sender \"@aminda.matterbridge:jae.fi\" not in room" event_id="$qwTRBD_nE07H9BA29T6vAkBzZAWDADX-GzOFgKatImw" kind=KindNew origin=jae.fi room_id="!DuAMUOvPwiaiqWgChA:pikaviestin.fi" type=m.room.message
Aug 27 00:11:14 MooMatrix dendrite[36730]: time="2022-08-26T14:11:14.311799728Z" level=warning msg="Roomserver rejected event" error="eventauth: sender \"@aminda.matterbridge:jae.fi\" not in room" event_id="$qwTRBD_nE07H9BA29T6vAkBzZAWDADX-GzOFgKatImw" room_id="!DuAMUOvPwiaiqWgChA:pikaviestin.fi" type=m.room.message

I will request the actual Dendrite users to check this in hopes of them filling the information I was unable to give.

Mikaela avatar Aug 30 '22 05:08 Mikaela

I forgot to say that the invisible-user in question has been in the room about since /upgraderoom 10 give or take a few minutes, having been in the previous version. I don't remember whether it had access with restricted or was invited directly. join_rule is knock_restricted.

Additionally this room isn't performing weird powerlevels unlike some that have previously experienced DAG splitting, powers range between 0 (unset) to 100.

Mikaela avatar Aug 30 '22 05:08 Mikaela

MooMatrix is my host; ill fill in missing details.

  • Monolith or Polylith?: Monolith

  • SQLite3 or Postgres?: Postgres

  • Running in Docker?: No, (Deployed on Archlinux on native libraries)

  • go version: go1.19 linux/amd64

Problem appeared at around 0.9.5, server has been operating since 0.8.7.

EndlessEden avatar Aug 30 '22 06:08 EndlessEden

Can confirm still exists on latest commit

EndlessEden avatar Sep 16 '22 15:09 EndlessEden

Can you please try commit a50556d (or any newer one after it)?

You might not see things correct straight away (unless you leave all of your server users from the room and re-join) but it should hopefully stop things from fragmenting or resetting like before.

neilalexander avatar Sep 23 '22 12:09 neilalexander

@neilalexander - Currently on commit, problem persists.

Sep 23 23:25:48 MooMatrix dendrite[32426]: time="2022-09-23T13:25:48.983758559Z" level=warning msg="Error authing soft-failed event" context=missing error="eventauth: sender "@aminda.matterbridge:jae.fi" not in room" event_id="$V6dWLwV-EwUcGbxUDzQ-i4-DHSMThi7-Yo865mudjtc" kind=KindNew origin=jae.fi room_id="!DuAMUOvPwiaiqWgChA:pikaviestin.fi" type=m.room.message

Sep 23 23:25:49 MooMatrix dendrite[32426]: time="2022-09-23T13:25:49.904796643Z" level=warning msg="Stored rejected event" context=missing error="eventauth: sender "@aminda.matterbridge:jae.fi" not in room" event_id="$V6dWLwV-EwUcGbxUDzQ-i4-DHSMThi7-Yo865mudjtc" kind=KindNew origin=jae.fi room_id="!DuAMUOvPwiaiqWgChA:pikaviestin.fi" type=m.room.message

Sep 23 23:25:49 MooMatrix dendrite[32426]: time="2022-09-23T13:25:49.905043004Z" level=warning msg="Roomserver rejected event" error="eventauth: sender "@aminda.matterbridge:jae.fi" not in room" event_id="$V6dWLwV-EwUcGbxUDzQ-i4-DHSMThi7-Yo865mudjtc" room_id="!DuAMUOvPwiaiqWgChA:pikaviestin.fi" type=m.room.message

then after leave/rejoin;

Sep 23 23:38:05 MooMatrix dendrite[32426]: time="2022-09-23T13:38:05.292998919Z" level=warning msg="Error authing soft-failed event" context=missing error="eventauth: sender "@aminda:pikaviestin.fi" not in room" event_id="$TOZEIoaK7GzNQHR2Uz9ZF2qp4zUaMm8RBbhitZKn_9Q" kind=KindNew origin=pikaviestin.fi room_id="!sVpGiywFpzCrNoXMDY:pikaviestin.fi" type=m.room.encrypted

Sep 23 23:38:05 MooMatrix dendrite[32426]: time="2022-09-23T13:38:05.384038188Z" level=warning msg="Stored rejected event" context=missing error="eventauth: sender "@aminda:pikaviestin.fi" not in room" event_id="$TOZEIoaK7GzNQHR2Uz9ZF2qp4zUaMm8RBbhitZKn_9Q" kind=KindNew origin=pikaviestin.fi room_id="!sVpGiywFpzCrNoXMDY:pikaviestin.fi" type=m.room.encrypted

Sep 23 23:38:05 MooMatrix dendrite[32426]: time="2022-09-23T13:38:05.384411090Z" level=warning msg="Roomserver rejected event" error="eventauth: sender "@aminda:pikaviestin.fi" not in room" event_id="$TOZEIoaK7GzNQHR2Uz9ZF2qp4zUaMm8RBbhitZKn_9Q" room_id="!sVpGiywFpzCrNoXMDY:pikaviestin.fi" type=m.room.encrypted

EndlessEden avatar Sep 23 '22 13:09 EndlessEden

I think this is resolved although I don't remember what was the solution in this case as the room had other issues too.

Mikaela avatar Nov 23 '22 09:11 Mikaela