state resets still happen in v2 rooms

[richvdh]

I just upgraded #synapse:matrix.org because I got fed up of losing my ops bit. It was a v4 room.

EDIT 2022-08-03 by @DMRobertson: if you suspect you have seen a state reset, please provide:

• the room_id of the room with the reset
• the type and state_key that suffered a reset
• the event_id of the piece of state after the reset
• ideally, the event_id of the piece of state before the reset
  • failing that, a rough time range before and after the reset occurred (including timezone information)

To confirm if this really is a server-side state reset (rather than e.g. matrix-org/matrix-spec#1209), server administrators can run the following queries.

SELECT *
FROM current_state_delta_stream 
WHERE
	room_id ='<room id>'
	and type='<type of event>'' 
    and state_key='<state key>'
ORDER BY stream_id asc;

SELECT *
FROM current_state_events
WHERE
	room_id ='<room id>'
	and type='<type of event>'' 
    and state_key='<state key>';

The last row in the event_id columns from these two queries should match. It represents the event which currently describes the relevant piece of state from your server's point of view.

[BillCarsonFr]

Just to clarify, that means that it still happens in v2+ (3...9) versions right?

[Mikaela]

Yes, since the state resolution algorithm hasn't been changed since room version 2

• https://spec.matrix.org/v1.5/rooms/#complete-list-of-room-versions

[solus-hq]

Hello!

I'm experiencing this bug as well. Every month I'm being added to #dendrite room (which is a v1 room, I suppose) and I can't even leave this room because nothing happens when I click "Leave".

[jae1911]

Hey there, Every now and then in Furry Tech (!vRGLvqJYlFvzpThbxI:matrix.org; room v6) we are getting a state reset that mostly resets profile pictures and usernames.

[ptman]

I was under the impression that state res v2 was studied and deemed sound. https://matrix.org/blog/2020/06/16/matrix-decomposition-an-independent-academic-analysis-of-matrix-state-resolution . Did the research group finish work on a formal proof?

[ara4n]

There's a large project within the core team currently working through a new formal expression of state res v2 to try to characterise the remaining issues. Given the risk of discovering security issues, it's being treated as an internal security project (juggled alongside our other security work). There's been some progress in identifying the root cause here though; will update as soon we can.

[DMRobertson]

Hey there, Every now and then in Furry Tech (!vRGLvqJYlFvzpThbxI:matrix.org; room v6) we are getting a state reset that mostly resets profile pictures and usernames.

Hi @jae1911. I would like to look into the details here. Could you share

• an example of a user whose profile was reset
• a time range before and after the reset (with timezone, if relevant)
• (if possible) the event id of that user's m.room.member event before and after the reset?

[jae1911]

Hey there, Every now and then in Furry Tech (!vRGLvqJYlFvzpThbxI:matrix.org; room v6) we are getting a state reset that mostly resets profile pictures and usernames.

Hi @jae1911. I would like to look into the details here. Could you share

* an example of a user whose profile was reset

* a time range before and after the reset (with timezone, if relevant)

* (if possible) the event id of that user's `m.room.member` event before and after the reset?

Hey there, sure!
I will update this comment when a state reset reoccurs.
For now, I will take my own user @me:jae.fi, that is getting reset every now and then, taken 2022/08/01 (Helsinki Time):

{
  "type": "m.room.member",
  "sender": "@me:jae.fi",
  "content": {
    "membership": "join",
    "displayname": "Jae",
    "avatar_url": "mxc://jae.fi/c5e04109ec88975b4a4b9b3fc73c06716a5a235e"
  },
  "state_key": "@me:jae.fi",
  "origin_server_ts": 1659273166743,
  "unsigned": {
    "replaces_state": "$3MkegYwrP0_HgS1_2sSw9W_WZc8gL-DvFMH3-CzX_dg",
    "prev_content": {
      "membership": "join",
      "displayname": "Jae",
      "avatar_url": "mxc://jae.fi/d31d8316f1342c4e44aa599262d0500dbcde0f67"
    },
    "prev_sender": "@me:jae.fi",
    "age": 61275344
  },
  "event_id": "$GXG0iXSO6ICGZ_XoLr_rSR_nuDmrKjBGfbAliDhrDic",
  "room_id": "!vRGLvqJYlFvzpThbxI:matrix.org"
}

Ping @DMRobertson ; state reset occurred in the night of the 2022/08/03; here is the m.room.member of my user at 06:12;

{
  "type": "m.room.member",
  "sender": "@me:jae.fi",
  "content": {
    "membership": "join",
    "displayname": "Garlic Queen (JAE)",
    "avatar_url": "mxc://jae.fi/0fbaa7c82e7ad07507eb8bbf3f0d5302ed207c73"
  },
  "state_key": "@me:jae.fi",
  "origin_server_ts": 1650141793560,
  "unsigned": {
    "replaces_state": "$OPzjkNQTMMcIQ4fOaa-bm7yp5VgeAAdA1P21rpeKnWU",
    "age": 9354472102
  },
  "event_id": "$MhTBxLMe9Jc2mYv3L0xiiXsAolXFx_urSyH3dCMGTEM",
  "room_id": "!vRGLvqJYlFvzpThbxI:matrix.org"
}

[DMRobertson]

Ping @DMRobertson ; state reset occurred in the night of the 2022/08/03; here is the m.room.member of my user at 06:12;

@jae1911 I assume this is from the perspective of jae.fi? From the matrix.org perspective, your membership is currently $RD1ZT-xYbd8WqjqqktsvKyyRlI8dnzWf3kVL7ojOAwA (created by jae.fi at 2022-08-02 15:11:21+01).

Are you able to run sql queries on the Synapse database hosting jae.fi? If so, please reach out to @dmrobertson:matrix.org---I'd like to ask you to run a few SELECTs to establish what prompted the reset.

[DMRobertson]

Thanks for getting in touch @jae1911. From the queries we ran, we saw that from the point of view of jae.fi:

• your membership of the room recently changed from $GXG0iXSO6ICGZ_XoLr_rSR_nuDmrKjBGfbAliDhrDic to $SH9zmA3rnQutHvL2GGxfJmEsnaatPB-7Y-cTzqL80Pc to $RD1ZT-xYbd8WqjqqktsvKyyRlI8dnzWf3kVL7ojOAwA
• the current state of the room has your membership determined by $RD1ZT-xYbd8WqjqqktsvKyyRlI8dnzWf3kVL7ojOAwA
• a new account also saw that your membership was determined by $RD1ZT-xYbd8WqjqqktsvKyyRlI8dnzWf3kVL7ojOAwA

The first two bullets are identical to what I see on matrix.org.

Therefore, I don't think this is an example of a server-side state reset; rather, I think this is an instance of matrix-org/matrix-spec#1209 (or something like it) where clients can have their view of state differ from that of the server. Debugging that problem will be fiddly and should hopefully be avoided by using sliding sync rather than that current sync API.

With that said, please do let me know if you see another instance of state that appears to have reset unexpectedly.

I will edit the description of this to include the relevant sql queries for any future reports.

[Mikaela]

This happened to be in the Furry Tech room. Looking from Nheko on pikaviestin.fi my avatar and name went back in time to Pikaminda ⚡ from Dea Mortis. jae.fi sees the expected profile though.

[DMRobertson]

This happened to be in the Furry Tech room. Looking from Nheko on pikaviestin.fi my avatar and name went back in time to Pikaminda ⚡ from Dea Mortis. jae.fi sees the expected profile though.

I don't see the reset in matrix.org's database. Are you able to run the SQL queries in the description?

[Mikaela]

I am not a server admin anywhere

[JokerGermany]

This Message was deleted but can still be seen on matrix.org https://matrix.to/#/!POnsZmdSAzRrxMaTOj:matrix.kraut.space/$JokvE8q-ggX02mw-r4Lv3PvW2Tkh431WBrXxfJ5CFWo?via=behm.me&via=matrix.org&via=tchncs.de

matrix.org user:
{
  "content": {
    "body": "Allesamt ins [...].",
    "msgtype": "m.text"
  },
  "origin_server_ts": 1663225039525,
  "sender": "@christinaf9:matrix.org",
  "type": "m.room.message",
  "unsigned": {
    "age": 1234720
  },
  "event_id": "$JokvE8q-ggX02mw-r4Lv3PvW2Tkh431WBrXxfJ5CFWo",
  "room_id": "!POnsZmdSAzRrxMaTOj:matrix.kraut.space"
}

asra.gr user:

{
  "content": {},
  "origin_server_ts": 1663225039525,
  "room_id": "!POnsZmdSAzRrxMaTOj:matrix.kraut.space",
  "sender": "@christinaf9:matrix.org",
  "type": "m.room.message",
  "unsigned": {
    "redacted_by": "$O4n91n4jwoQZH7nB0w-7ySZ_PFQVx5OlM4ljcH56wnY",
    "redacted_because": {
      "content": {},
      "origin_server_ts": 1663230084270,
      "redacts": "$JokvE8q-ggX02mw-r4Lv3PvW2Tkh431WBrXxfJ5CFWo",
      "room_id": "!POnsZmdSAzRrxMaTOj:matrix.kraut.space",
      "sender": "@joerg:alea.gnuu.de",
      "type": "m.room.redaction",
      "unsigned": {},
      "event_id": "$O4n91n4jwoQZH7nB0w-7ySZ_PFQVx5OlM4ljcH56wnY",
      "user_id": "@joerg:alea.gnuu.de"
    },
    "age": 5202172
  },
  "event_id": "$JokvE8q-ggX02mw-r4Lv3PvW2Tkh431WBrXxfJ5CFWo",
  "user_id": "@christinaf9:matrix.org",
  "age": 5202172,
  "redacted_because": {
    "content": {},
    "origin_server_ts": 1663230084270,
    "redacts": "$JokvE8q-ggX02mw-r4Lv3PvW2Tkh431WBrXxfJ5CFWo",
    "room_id": "!POnsZmdSAzRrxMaTOj:matrix.kraut.space",
    "sender": "@joerg:alea.gnuu.de",
    "type": "m.room.redaction",
    "unsigned": {},
    "event_id": "$O4n91n4jwoQZH7nB0w-7ySZ_PFQVx5OlM4ljcH56wnY",
    "user_id": "@joerg:alea.gnuu.de"
  }
}

Occouring approximatly 8:58 CEST

Message was send after the user was banned.

€dit: They are in sync now, the message is deleted.

[DMRobertson]

This Message was deleted but can still be seen on matrix.org They are in sync now, the message is deleted.

Sounds like there was some delay in the redaction event $O4n91n4jwoQZH7nB0w-7ySZ_PFQVx5OlM4ljcH56wnY sent by alea.gnuu.de reading matrix.org. Given that this resolved itself I don't think this is related to a state reset.

[vivia]

I had created #randomstuffhere:gnome.org and set up a few co-admins together with myself. Now I just realised we've all been demoted to moderators, and I don't have the admin rights to fix that myself. I asked for help on #element-web:matrix.org and was told to write here.

[DMRobertson]

I had created #randomstuffhere:gnome.org and set up a few co-admins together with myself. Now I just realised we've all been demoted to moderators, and I don't have the admin rights to fix that myself. I asked for help on #element-web:matrix.org and was told to write here.

Thanks for the report. I took at look at this today. It seems that matrix.org is joined to the room. From its perspective, I couldn't see any evidence of a state reset: there are 17 power levels in that room, and each of occurred exactly once in the current state of the room. (A reset would mean that at least one event was in the current state of the room on two or more separate occasions.)

I could see that two users' power levels dropped from 50 to either NULL or 0. However, three other users' power levels never dropped from 50, and the room creator's power level never dropped from 100. (I've been somewhat vague here: the room is private, and I don't want to publish any room, event or user IDs.)

Of note: the two users who saw a drop were both appservice-managed users---in this case, users managed by an IRC bridge to irc.oftc.net. I'm not sure if this is expected behaviour (is the bridge meant to demote users when they disconnect from IRC?) or a bug on in the IRC bridge, but I can't see anything that suggests this behaviour is caused in a bug by Matrix-the-protocol.

If you're using element web, I'd recommend clearing its caches on the off-chance there is a client-side problem reporting an older set of power levels for some reason.

Now I just realised we've all been demoted to moderators, and I don't have the admin rights to fix that myself.

This doesn't match what I see in the matrix.org database (five users currently at power level 50, the room creator at power level 100).

[vivia]

Thanks, this actually adds up with what I remember happening. We had a different room in the past, and at some point several months ago, the IRC bridge was acting up and required bridge admin intervention. When trying to fix this, I moved us all to this room that had been created by that specific creator (staying intentionally vague here, but I'm sure you will understand exactly what I mean). Apparently in the process our power levels became Moderators instead, and I never noticed until now because that happens to give me enough privileges to manage a low-traffic room with just friends. I guess it can stay like this then, people can get moderator status by opping themselves on the IRC side, where I can manage ChanServ.

Thanks for your investigation!

[Mikaela]

We at #fluffychat:matrix.org are getting reports of the room being DAG split and thus view of avatars (including the room avatar) depending on which homeserver is being used.

Can this situation be confirmed somehow and that room upgrade is absolutely necessary or is knowledge of it at all helpful? I understand that DAG splits are also tracked in this issue.

[DMRobertson]

Can this situation be confirmed somehow

We'd need more details (see also the top comment, which I edited some months back with some suggestions). Whose avatar is being split? Which servers see it differently; ideally, which event ID do they report as being current?

[Mikaela]

• Room ID is !IwPxifXSjBLghpVEMh:matrix.org
• Room avatar is missing from some servers
• Users such as @krille:janian.de and @mxidupwitch:the-apothecary.club have avatars missing depending on server. I cannot see the later or their display name on pikaviestin.fi using Nheko
• at least matrix.org and pikaviestin.fi and the-apothecary.club and arcticfoxes.net are reported seeing different state and the last has been claiming multiple DAG splits past few months and upgraded most if not all of their rooms.

I don't know how to see current event ID. I have no access to any database personally.

I am going by assumption that it's better that issues like this are reported rather than letting them happen silently and migrating to protocols where they don't happen.

DAG split was claimed in event https://matrix.to/#/%23fluffychat%3Amatrix.org/%24DN3Au8zP9FMwDXULg_ZJZmXkw5BcKVDCL-BHx3-ARvM?via=pikaviestin.fi&via=matrix.org&via=the-apothecary.club&via=envs.net

[wj25czxj47bu6q]

I have documented a new case of state reset at https://github.com/matrix-org/synapse/issues/15987#issuecomment-1655485693.

[wj25czxj47bu6q]

I have documented a new case of state reset at https://github.com/matrix-org/synapse/issues/16629.