An event was sent twice down `/sync` (and down application service txns?) consistently across clients; ghost notifications

Open matrixbot opened this issue 2 years ago • 3 comments

This issue has been migrated from #14539.

Description

Ghost notifications

Since synapse v1.70 or v1.71, I've noticed randomly occurring "ghost notification", happening on element-web, element-desktop and element-android, hydrogen. They usually come in small, fast bursts of two to four notifications (I hear an audible sound telling me about new events in a room). When I check for new events, there aren't any. Nothing has changed. I have the options Developer mode and Show hidden events in timeline enabled in element, so I would assume, I'd see any changes to a room.

Event duplication

Around the same Time, I've noticed some events, some from weeks or months ago, from being randomly duplicated and pushed as new ones. This happens independently of the "ghost notification issue". At first, I've only noticed this in one particular room, where the same messages were duplicated a few times over a couple of days (Example: see "Image 1" below), possibly leading to an unstable room (see "Possible implications). Today, the same thing happened in our mjolnir control room, where mjolnir (configured as a client and not as an appservice) executed the duplicated command (Example: see "Image 2" below). Actually, I'm actually not certain, if my homeserver is the one, the issues originate from, as the duplicated events are visible on other instances too, including matrix.org and maybe others as well.

Image 1: Duplicated messages in the python room

event_duplicated_python_room

Image 2: Mjolnir bot reacts to duplicated events

The "failed to leave" error from mjolnir is unrelated (and already fixed). The duplication errors happened before the room, it was trying to leave, had issues, too, and might have led to the issue, where the room was becoming unstable on my homeserver. event_duplicated_mjolnir

Possible implications

After this happened a few times, in the Python room #python:matrix.org, the room became unstable on my end, and I was unable to send events to that room any more because of missing state events. I also wasn't able to remove the room with the admin API, so, I manually had to remove the room from the database to gain access again:

BEGIN; SET CONSTRAINTS ALL DEFERRED;

 DELETE FROM appservice_room_list WHERE room_id = '!roomid:matrix.org';
 DELETE FROM batch_events WHERE room_id = '!roomid:matrix.org';
 DELETE FROM current_state_events WHERE room_id = '!roomid:matrix.org';
 DELETE FROM blocked_rooms WHERE room_id = '!roomid:matrix.org';
 DELETE FROM current_state_delta_stream WHERE room_id = '!roomid:matrix.org';
 DELETE FROM destination_rooms WHERE room_id = '!roomid:matrix.org';
 DELETE FROM device_lists_changes_in_room WHERE room_id = '!roomid:matrix.org';
 DELETE FROM e2e_room_keys WHERE room_id = '!roomid:matrix.org';
 DELETE FROM event_auth_chain_to_calculate WHERE room_id = '!roomid:matrix.org';
 DELETE FROM event_auth WHERE room_id = '!roomid:matrix.org';
 DELETE FROM event_backward_extremities WHERE room_id = '!roomid:matrix.org';
 DELETE FROM event_failed_pull_attempts WHERE room_id = '!roomid:matrix.org';
 DELETE FROM event_json WHERE room_id = '!roomid:matrix.org';
 DELETE FROM event_forward_extremities WHERE room_id = '!roomid:matrix.org';
 DELETE FROM event_edges WHERE room_id = '!roomid:matrix.org';
 DELETE FROM event_labels WHERE room_id = '!roomid:matrix.org';
 DELETE FROM event_push_actions WHERE room_id = '!roomid:matrix.org';
 DELETE FROM event_push_summary WHERE room_id = '!roomid:matrix.org';
 DELETE FROM event_reports WHERE room_id = '!roomid:matrix.org';
 DELETE FROM event_search WHERE room_id = '!roomid:matrix.org';
 DELETE FROM event_txn_id WHERE room_id = '!roomid:matrix.org';
 DELETE FROM event_edges WHERE event_id IN (SELECT event_id FROM events WHERE room_id='!roomid:matrix.org');
 DELETE FROM event_auth_chains WHERE event_id IN (SELECT event_id FROM events WHERE room_id = '!roomid:matrix.org');
 DELETE FROM event_relations WHERE event_id IN (SELECT event_id FROM events WHERE room_id = '!roomid:matrix.org');
 DELETE FROM events WHERE room_id = '!roomid:matrix.org';
 DELETE FROM federation_inbound_events_staging WHERE room_id = '!roomid:matrix.org';
 DELETE FROM insertion_event_edges WHERE room_id = '!roomid:matrix.org';
 DELETE FROM insertion_event_extremities WHERE room_id = '!roomid:matrix.org';
 DELETE FROM insertion_events WHERE room_id = '!roomid:matrix.org';
 DELETE FROM local_current_membership WHERE room_id = '!roomid:matrix.org';
 DELETE FROM partial_state_events WHERE room_id = '!roomid:matrix.org';
 DELETE FROM partial_state_rooms_servers WHERE room_id = '!roomid:matrix.org';
 DELETE FROM pusher_throttle WHERE room_id = '!roomid:matrix.org';
 DELETE FROM receipts_linearized WHERE room_id = '!roomid:matrix.org';
 DELETE FROM room_account_data WHERE room_id = '!roomid:matrix.org';
 DELETE FROM room_aliases WHERE room_id = '!roomid:matrix.org';
 DELETE FROM room_depth WHERE room_id = '!roomid:matrix.org';
 DELETE FROM room_memberships WHERE room_id = '!roomid:matrix.org';
 DELETE FROM room_retention WHERE room_id = '!roomid:matrix.org';
 DELETE FROM room_stats_current WHERE room_id = '!roomid:matrix.org';
 DELETE FROM room_stats_earliest_token WHERE room_id = '!roomid:matrix.org';
 DELETE FROM room_stats_state WHERE room_id = '!roomid:matrix.org';
 DELETE FROM state_groups_state WHERE room_id = '!roomid:matrix.org';
 DELETE FROM rooms WHERE room_id = '!roomid:matrix.org';
 DELETE FROM user_directory WHERE room_id = '!roomid:matrix.org';
 DELETE FROM room_tags WHERE room_id = '!roomid:matrix.org';
 DELETE FROM room_tags_revisions WHERE room_id = '!roomid:matrix.org';
 DELETE FROM state_groups WHERE room_id = '!roomid:matrix.org';
 DELETE FROM state_events WHERE room_id = '!roomid:matrix.org';
 DELETE FROM users_in_public_rooms WHERE room_id = '!roomid:matrix.org';
 DELETE FROM users_who_share_private_rooms WHERE room_id = '!roomid:matrix.org';
 DELETE FROM stream_ordering_to_exterm WHERE room_id = '!roomid:matrix.org';
 DELETE FROM receipts_graph WHERE room_id = '!roomid:matrix.org';
 DELETE FROM threads WHERE room_id = '!roomid:matrix.org';
 DELETE FROM partial_state_rooms WHERE room_id = '!roomid:matrix.org';
COMMIT;

REINDEX (verbose) database synapse;
REINDEX (verbose) system synapse;
VACUUM FULL VERBOSE ANALYZE;

(Because I had to roll back a few times, while I was trying to figure out, how I can remove the room, I accidentally deleted the logs as well.)

Steps to reproduce

Do nothing
Duplications / ghost events occur

Homeserver

matrix.org; michaelsasser.org

Synapse Version

matrixdotorg/synapse; tags: v1.71.0, v1.72.0, possibly v1.70.0

Installation Method

Docker (matrixdotorg/synapse)

Database

postgres:15.0-alpine on same machine, It was recently restored to upgrade to v15, not sure, if it happeded before; recently restored while trying to removing room, issues happened before

Workers

Multiple workers

Platform

Environment: CX31 Hetzner Cloud Using the Ansible playbook: spantaleev/matrix-docker-ansible-deploy Debian: Linux matrix 5.10.0-19-amd64 vector-im/element-web#1 SMP Debian 5.10.149-2 (2022-10-21) x86_64 GNU/Linux Docker: Docker version 20.10.21, build baeda1f Containerd: containerd containerd.io 1.6.10 770bd0108c32f3fb5c73ae1264f7e503fe7b2661

Configuration

Playbook

Using the Ansible playbook: [spantaleev/matrix-docker-ansible-deploy], contains multiple files (result below):

Playbook/config

---
# hookshot
matrix_hookshot_enabled: true  # default
matrix_hookshot_gitlab_enabled: true
matrix_hookshot_permissions:
  # Allow all users to send commands to existing services
  - actor: "*"
    services:
      - service: gitlab
        level: commands
  # Allow users on this domain to log in to gitlab.
  - actor: "!foo:michaelsasser.org"
    services:
      - service: gitlab
        level: login
  # Allow this specific user to do any action
  - actor: "@foo:michaelsasser.org"
    services:
      - service: "*"
        level: admin
---
# Mjolnir
matrix_bot_mjolnir_enabled: true
matrix_bot_mjolnir_access_token: "foo"
matrix_bot_mjolnir_management_room: "!foo:michaelsasser.org"
matrix_bot_mjolnir_configuration_extension_yaml: |
  backgroundDelayMS: 50
  fasterMembershipChecks: true
  recordIgnoredInvites: true
  commands:
    additionalPrefixes:
        - mjolnir
        - bot
  automaticallyRedactForReasons:
    - spam
    - advertising
    - advertisement
    - ad
    - scam
  protections:
    wordlist:
      words:
        - foo
        - more foo
      minutesBeforeTrusting: 1234  # 1234 is not the actual time
---
# ssl

matrix_ssl_lets_encrypt_support_email: [email protected]
---
# element

# set integration manager
matrix_client_element_integrations_ui_url: "https://scalar.vector.im/"
matrix_client_element_integrations_rest_url: "https://scalar.vector.im/api"
matrix_client_element_integrations_widgets_urls:
  ["https://scalar.vector.im/api"]

# set server room directory
matrix_client_element_roomdir_servers:
  - michaelsasser.org
  - matrix.org

# Extra themes
matrix_client_element_themes_enabled: false

# Default Template
matrix_client_element_default_theme: "dark"

matrix_client_element_configuration_extension_json: |
  {
     "sso_redirect_options": {
        "immediate": true
     }
  }

---
# hydrogen
#
matrix_client_hydrogen_enabled: true
---

matrix_grafana_enabled: true
matrix_grafana_anonymous_access: false
matrix_grafana_default_admin_user: foo # Do not change
matrix_grafana_default_admin_password: foo

matrix_grafana_container_extra_arguments:
  - -e GF_AUTH_GENERIC_OAUTH_NAME="Single Sign-On"
  - -e GF_AUTH_GENERIC_OAUTH_ENABLED="true"
  - -e GF_AUTH_GENERIC_OAUTH_CLIENT_ID="matrix-grafana"
  - -e GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET="foo"
  - -e GF_AUTH_GENERIC_OAUTH_SCOPES="openid"
  - -e GF_AUTH_GENERIC_OAUTH_AUTH_URL="https://foo.michaelsasser.org/realms/MichaelSasser/protocol/openid-connect/auth"
  - -e GF_AUTH_GENERIC_OAUTH_TOKEN_URL="https://foo.michaelsasser.org/realms/MichaelSasser/protocol/openid-connect/token"
  - -e GF_AUTH_GENERIC_OAUTH_API_URL="https://foo.michaelsasser.org/realms/MichaelSasser/protocol/openid-connect/userinfo"
  - -e GF_AUTH_GENERIC_OAUTH_ALLOW_SIGN_UP="true"
  - -e GF_AUTH_GENERIC_OAUTH_ROLE_ATTRIBUTE_PATH="contains(roles[*], 'Admin') && 'Admin' || contains(roles[*], 'Editor') && 'Editor' || contains(roles[*], 'Viewer') && 'Viewer'|| ''"
  - -e GF_AUTH_GENERIC_OAUTH_ROLE_ATTRIBUTE_STRICT="true"
  # Disable Login Form
  - -e GF_AUTH_DISABLE_LOGIN_FORM="true"
  - -e GF_AUTH_OAUTH_AUTO_LOGIN="true"
  - -e GF_AUTH_BASIC_ENABLED="false"
---
# email

matrix_mailer_sender_address: "[email protected]"
matrix_mailer_relay_use: true
matrix_mailer_relay_host_name: "foo.michaelsasser.org"
matrix_mailer_relay_host_port: 587
matrix_mailer_relay_auth: true
matrix_mailer_relay_auth_username: "[email protected]"
matrix_mailer_relay_auth_password: "foo"
---

matrix_synapse_rust_synapse_compress_state_find_rooms_command_wait_time: 1200
matrix_synapse_rust_synapse_compress_state_compress_room_time: 68400
matrix_synapse_rust_synapse_compress_state_psql_import_time: 68400

# Base Domain Serving
matrix_nginx_proxy_base_domain_serving_enabled: true

matrix_nginx_proxy_connect_timeout: 120
matrix_nginx_proxy_send_timeout: 120
matrix_nginx_proxy_read_timeout: 120
matrix_nginx_send_timeout: 120

# Fix issue where the admin api is not reacheable anymore
matrix_nginx_proxy_proxy_matrix_client_redirect_root_uri_to_domain: ""
matrix_nginx_proxy_proxy_matrix_client_api_forwarded_location_synapse_admin_api_enabled: true
---

matrix_postgres_connection_password: "foo"
matrix_postgres_container_postgres_bind_port: 5432

matrix_postgres_process_extra_arguments:
  [
    "-c max_connections=120",
    "-c shared_buffers=512MB",
    "-c effective_cache_size=1536MB",
    "-c maintenance_work_mem=128MB",
    "-c checkpoint_completion_target=0.9",
    "-c wal_buffers=16MB",
    "-c default_statistics_target=100",
    "-c random_page_cost=1.1",
    "-c effective_io_concurrency=200",
    "-c work_mem=2184kB",
    "-c min_wal_size=1GB",
    "-c max_wal_size=4GB",
    "-c max_worker_processes=2",
    "-c max_parallel_workers_per_gather=1",
    "-c max_parallel_workers=2",
    "-c max_parallel_maintenance_workers=1"
  ]

# Postgres Backups

matrix_postgres_backup_enabled: true
matrix_postgres_backup_keep_months: 12
---
# grafana prometheus

matrix_prometheus_enabled: true
matrix_prometheus_node_exporter_enabled: true
matrix_prometheus_postgres_exporter_enabled: false
matrix_prometheus_postgres_exporter_database_username: "matrix_prometheus_postgres_exporter"
matrix_prometheus_postgres_exporter_database_password: "foo"


matrix_prometheus_node_exporter_process_extra_arguments:
  - "--collector.disable-defaults"
  - "--collector.cpu"
  - "--collector.filesystem"
  - "--collector.os"
  - "--collector.stat"
  - "--collector.meminfo"
  - "--collector.loadavg"
  - "--collector.time"
  - "--collector.uname"  # Mandatory, otherwide the data vanishes from grafana
---
# synapse

# Federation
matrix_synapse_federation_enabled: true
matrix_synapse_allow_public_rooms_over_federation: true

# Controls whether accessing the server's public rooms directory can be done
# without authentication.
# For private servers, you most likely wish to require authentication,
# unless you know what list of rooms you're publishing to the world and
# explicitly want to do it.
matrix_synapse_allow_public_rooms_without_auth: true

# Matrix cache size (ram) default: 0.5
matrix_synapse_caches_global_factor: 1.5 # 0.5, is default

matrix_synapse_event_cache_size: "100K"

# OpenID (Keycloak) (rest see below in matrix_synapse_configuration_extension_yaml)
matrix_nginx_proxy_proxy_matrix_client_api_forwarded_location_synapse_oidc_api_enabled: true

# Extra Synapse homeserver.yaml config
matrix_synapse_configuration_extension_yaml: |
  suppress_key_server_warning: true  # defaults to matrix.org

  ####################
  # room complexity
  ####################
  limit_remote_rooms:
    enabled: true
    complexity: 0.5
    complexity_error: "This room is too complex."
    admins_can_join: true


  ####################
  # blacklist
  ####################
  ip_range_blacklist:
    - "123.123.123.123/32"   # foo.xyz
    - "123.123.123.123/32"  # matrix.foo.com
    - "123.123.123.123/32"    # matrix.foo.sh
    - "123.123.123.123/32"   # matrix.foo.net
    - "123.123.123.123/32"    # matrix.foo.biz (CNAME)

  ####################
  # retention
  ####################
  redaction_retention_period: 30d
  retention:
    enabled: true
    default_policy:
      min_lifetime: 4w
      max_lifetime: 8w
    allowed_lifetime_min: 1d
    allowed_lifetime_max: 24w
  purge_jobs:
   - longest_max_lifetime: 3d
     interval: 12h
   - shortest_max_lifetime: 3d
     interval: 1d

  ####################
  # server notice
  ####################
  server_notices:
    system_mxid_localpart: server
    system_mxid_display_name: "Server Notices"
    system_mxid_avatar_url: "mxc://{{ matrix_domain }}/foo"
    room_name: "Server Notices"

  ####################
  # OIDC
  ####################
  oidc_providers:
    - idp_id: keycloak
      idp_name: "Single Sign-On"
      issuer: "https://foo.michaelsasser.org/realms/MichaelSasser"
      client_id: "synapse"
      client_secret: "foo"
      scopes: ["openid", "profile"]
      allow_existing_users: true
      user_mapping_provider:
        config:
          localpart_template: "{% raw %}{{ user.preferred_username }}{% endraw %}"
          display_name_template: "{% raw %}{{ user.name }}{% endraw %}"

# Media retention
matrix_synapse_media_retention_local_media_lifetime: 8w
matrix_synapse_media_retention_remote_media_lifetime: 1w

# workers
matrix_synapse_workers_enabled: true

matrix_synapse_workers_preset: michaelsasser_org

# https://github.com/matrix-org/synapse/blob/develop/docs/workers.md
matrix_synapse_workers_presets:
  michaelsasser_org:
    generic_workers_count: 1
    pusher_workers_count: 1
    federation_sender_workers_count: 1
    media_repository_workers_count: 1
    appservice_workers_count: 1
    user_dir_workers_count: 1
    background_workers_count: 1
    stream_writer_events_stream_workers_count: 0
    stream_writer_typing_stream_workers_count: 0
    stream_writer_to_device_stream_workers_count: 0
    stream_writer_account_data_stream_workers_count: 0
    stream_writer_receipts_stream_workers_count: 0
    stream_writer_presence_stream_workers_count: 0

# A secret used to protect access keys issued by the server.
# You can put any string here, but generating a strong one is preferred
# (e.g. `pwgen -s 64 1`).
matrix_synapse_macaroon_secret_key: "foo"
matrix_homeserver_generic_secret_key: "{{ matrix_synapse_macaroon_secret_key }}"
matrix_synapse_max_upload_size_mb: 256

matrix_synapse_auto_join_rooms:
  - "#support:{{ matrix_domain }}"
# matrix_synapse_use_presence: true  # Disable on high load

# Shared Secret Provider
matrix_synapse_ext_password_provider_shared_secret_auth_enabled: false # disabled to make sure no m.password authentication is available
matrix_synapse_ext_password_provider_shared_secret_auth_m_login_password_support_enabled: false # disabled to make sure no m.password authentication is available
matrix_synapse_ext_password_provider_shared_secret_auth_shared_secret: "foo"
---
devture_timesync_ntpd_package: systemd-timesyncd
devture_timesync_ntpd_service: systemd-timesyncd
# A shared secret (between Coturn and Synapse) used for authentication.
# You can put any string here, but generating a strong one is preferred
# (e.g. `pwgen -s 64 1`).
matrix_coturn_turn_static_auth_secret: "foo"

---

# Enable location sharing
matrix_well_known_matrix_client_configuration_extension_json: |-
  {
    "org.matrix.msc3488.tile_server": {
      "map_style_url": "https://api.maptiler.com/maps/streets/style.json?key=foo"
    },
    "m.tile_server": {
      "map_style_url": "https://api.maptiler.com/maps/streets/style.json?key=foo"
    },
    "setting_defaults": {
      "MessageComposerInput.showStickersButton": false
    },
    "features": {
      "feature_latex_maths": true,
      "feature_pinning": true,
      "feature_thread": true,
      "feature_html_topic": true
    },
    "admins": [{
      "matrix_id": "@foo:michaelsasser.org",
      "email_address": "[email protected]",
      "role": "admin"
    },
    {
      "email_address": "[email protected]",
      "role": "security"
    }]
  }

Results in:

homeserver.yaml

account_threepid_delegates:
  msisdn: ''
alias_creation_rules:
- action: allow
  alias: '*'
  room_id: '*'
  user_id: '*'
allow_guest_access: false
allow_public_rooms_over_federation: true
allow_public_rooms_without_auth: true
allowed_avatar_mimetypes:
- image/png
- image/jpeg
- image/gif
app_service_config_files:
- /hookshot-registration.yml
auto_join_rooms:
- '#foo:michaelsasser.org'
autocreate_auto_join_rooms: true
background_updates: null
caches:
  global_factor: 1.5
  per_cache_factors: null
cas_config: null
daemonize: false
database:
  args:
    cp_max: 10
    cp_min: 5
    database: synapse
    host: matrix-postgres
    password: foo
    port: 5432
    user: synapse
  name: psycopg2
  txn_limit: 0
default_room_version: '9'
email:
  app_name: Matrix
  client_base_url: https://element.michaelsasser.org
  enable_notifs: true
  invite_client_location: https://app.element.io
  notif_for_new_users: true
  notif_from: Matrix <[email protected]>
  require_transport_security: false
  smtp_host: matrix-mailer
  smtp_port: 8025
enable_media_repo: false
enable_metrics: true
enable_registration: false
enable_registration_captcha: false
enable_registration_without_verification: false
enable_room_list_search: true
encryption_enabled_by_default_for_room_type: 'off'
event_cache_size: 100K
federation_rr_transactions_per_room_per_second: 50
federation_sender_instances:
- matrix-synapse-worker-federation-sender-0
form_secret: foo
include_profile_data_on_invite: true
instance_map: {}
ip_range_blacklist:
- 123.123.123.123/32  # not actually 123...
- 123.123.123.123/32
- 123.123.123.123/32
- 123.123.123.123/32
- 123.123.123.123/32
limit_profile_requests_to_users_who_share_rooms: false
limit_remote_rooms:
  admins_can_join: true
  complexity: 0.5
  complexity_error: This room is too complex.
  enabled: true
listeners:
- bind_addresses:
  - 0.0.0.0
  port: 9100
  type: metrics
- bind_addresses:
  - '::'
  port: 8008
  resources:
  - compress: false
    names:
    - client
  tls: false
  type: http
  x_forwarded: true
- bind_addresses:
  - '::'
  port: 8048
  resources:
  - compress: false
    names:
    - federation
  tls: false
  type: http
  x_forwarded: true
- bind_addresses:
  - 0.0.0.0
  port: 9093
  resources:
  - names:
    - replication
  type: http
log_config: /data/matrix.michaelsasser.org.log.config
macaroon_secret_key: foo
manhole_settings: null
max_spider_size: 10M
max_upload_size: 256M
media_instance_running_background_jobs: matrix-synapse-worker-media-repository-0
media_retention:
  local_media_lifetime: 8w
  remote_media_lifetime: 1w
media_storage_providers: []
media_store_path: /matrix-media-store-parent/media-store
metrics_flags: null
modules: []
notify_appservices_from_worker: matrix-synapse-worker-appservice-0
oembed: null
oidc_providers:
- allow_existing_users: true
  client_id: synapse
  client_secret: foo
  idp_id: keycloak
  idp_name: Single Sign-On
  issuer: https://foo.michaelsasser.org/realms/MichaelSasser
  scopes:
  - openid
  - profile
  user_mapping_provider:
    config:
      display_name_template: '{{ user.name }}'
      localpart_template: '{{ user.preferred_username }}'
old_signing_keys: null
opentracing: null
password_config:
  localdb_enabled: false
  pepper: ''
  policy: null
pid_file: /homeserver.pid
presence:
  enabled: true
public_baseurl: https://matrix.michaelsasser.org/
purge_jobs:
- interval: 12h
  longest_max_lifetime: 3d
- interval: 1d
  shortest_max_lifetime: 3d
push:
  include_content: true
pusher_instances:
- matrix-synapse-worker-pusher-0
rc_admin_redaction:
  burst_count: 50
  per_second: 1
rc_federation:
  concurrent: 3
  reject_limit: 50
  sleep_delay: 500
  sleep_limit: 10
  window_size: 1000
rc_invites:
  per_issuer:
    burst_count: 10
    per_second: 0.3
  per_room:
    burst_count: 10
    per_second: 0.3
  per_user:
    burst_count: 5
    per_second: 0.003
rc_joins:
  local:
    burst_count: 10
    per_second: 0.1
  remote:
    burst_count: 10
    per_second: 0.01
rc_login:
  account:
    burst_count: 3
    per_second: 0.17
  address:
    burst_count: 3
    per_second: 0.17
  failed_attempts:
    burst_count: 3
    per_second: 0.17
rc_message:
  burst_count: 10
  per_second: 0.2
rc_registration:
  burst_count: 3
  per_second: 0.17
recaptcha_private_key: ''
recaptcha_public_key: ''
redaction_retention_period: 30d
redis:
  enabled: true
  host: matrix-redis
  password: null
  port: 6379
registration_requires_token: false
registration_shared_secret: foo
report_stats: false
require_auth_for_profile_requests: false
retention:
  allowed_lifetime_max: 24w
  allowed_lifetime_min: 1d
  default_policy:
    max_lifetime: 8w
    min_lifetime: 4w
  enabled: true
room_list_publication_rules:
- action: allow
  alias: '*'
  room_id: '*'
  user_id: '*'
room_prejoin_state: null
run_background_tasks_on: matrix-synapse-worker-background-0
saml2_config:
  sp_config: null
  user_mapping_provider:
    config: null
send_federation: false
server_name: michaelsasser.org
server_notices:
  room_name: Server Notices
  system_mxid_avatar_url: mxc://michaelsasser.org/foo
  system_mxid_display_name: Server Notices
  system_mxid_localpart: server
signing_key_path: /data/matrix.michaelsasser.org.signing.key
spam_checker: []
sso: null
start_pushers: false
stats: null
stream_writers: {}
suppress_key_server_warning: true
templates: null
tls_certificate_path: null
tls_private_key_path: null
trusted_key_servers:
- server_name: matrix.org
turn_allow_guests: false
turn_shared_secret: foo
turn_uris:
- turn:matrix.michaelsasser.org?transport=udp
- turn:matrix.michaelsasser.org?transport=tcp
ui_auth: null
update_user_directory_from_worker: matrix-synapse-worker-user-dir-0
url_preview_accept_language:
- en-US
- en
url_preview_enabled: true
url_preview_ip_range_blacklist:
- 127.0.0.0/8
- 10.0.0.0/8
- 172.16.0.0/12
- 192.168.0.0/16
- 100.64.0.0/10
- 192.0.0.0/24
- 169.254.0.0/16
- 192.88.99.0/24
- 198.18.0.0/15
- 192.0.2.0/24
- 198.51.100.0/24
- 203.0.113.0/24
- 224.0.0.0/4
- ::1/128
- fe80::/10
- fc00::/7
- 2001:db8::/32
- ff00::/8
- fec0::/10
user_directory: null
user_ips_max_age: 28d
worker_app: synapse.app.homeserver

Workers

# $cat worker.matrix-synapse-worker-*

################################################


worker_app: synapse.app.generic_worker
worker_name: matrix-synapse-worker-appservice-0

worker_daemonize: false
worker_log_config: /data/matrix.michaelsasser.org.log.config

worker_replication_host: matrix-synapse
worker_replication_http_port: 9093


################################################


worker_listeners:
  - type: metrics
    bind_addresses: ['0.0.0.0']
    port: 19300
worker_app: synapse.app.generic_worker
worker_name: matrix-synapse-worker-background-0

worker_daemonize: false
worker_log_config: /data/matrix.michaelsasser.org.log.config

worker_replication_host: matrix-synapse
worker_replication_http_port: 9093


################################################


worker_listeners:
  - type: metrics
    bind_addresses: ['0.0.0.0']
    port: 19700
worker_app: synapse.app.federation_sender
worker_name: matrix-synapse-worker-federation-sender-0

worker_daemonize: false
worker_log_config: /data/matrix.michaelsasser.org.log.config

worker_replication_host: matrix-synapse
worker_replication_http_port: 9093


################################################


worker_listeners:
  - type: metrics
    bind_addresses: ['0.0.0.0']
    port: 19400
worker_app: synapse.app.generic_worker
worker_name: matrix-synapse-worker-generic-0

worker_daemonize: false
worker_log_config: /data/matrix.michaelsasser.org.log.config

worker_replication_host: matrix-synapse
worker_replication_http_port: 9093

worker_main_http_uri: http://matrix-synapse:8008


################################################

worker_listeners:
  - type: http
    bind_addresses: ['::']
    x_forwarded: true
    port: 18111
    resources:
      - names: ["client", "federation"]
  - type: metrics
    bind_addresses: ['0.0.0.0']
    port: 19111
worker_app: synapse.app.media_repository
worker_name: matrix-synapse-worker-media-repository-0

worker_daemonize: false
worker_log_config: /data/matrix.michaelsasser.org.log.config

worker_replication_host: matrix-synapse
worker_replication_http_port: 9093


################################################


worker_listeners:
  - type: http
    bind_addresses: ['::']
    x_forwarded: true
    port: 18551
    resources:
      - names: ["media"]
  - type: metrics
    bind_addresses: ['0.0.0.0']
    port: 19551
worker_app: synapse.app.pusher
worker_name: matrix-synapse-worker-pusher-0

worker_daemonize: false
worker_log_config: /data/matrix.michaelsasser.org.log.config

worker_replication_host: matrix-synapse
worker_replication_http_port: 9093


################################################


worker_listeners:
  - type: metrics
    bind_addresses: ['0.0.0.0']
    port: 19200
worker_app: synapse.app.generic_worker
worker_name: matrix-synapse-worker-user-dir-0

worker_daemonize: false
worker_log_config: /data/matrix.michaelsasser.org.log.config

worker_replication_host: matrix-synapse
worker_replication_http_port: 9093


################################################


worker_listeners:
  - type: http
    bind_addresses: ['::']
    x_forwarded: true
    port: 18661
    resources:
      - names: ["client"]
  - type: metrics
    bind_addresses: ['0.0.0.0']
    port: 19661

Relevant log output

# Synapse + workers during the time the duplication, see "Image 2" happened:
-- Journal begins at Sat 2022-11-12 20:25:09 CET, ends at Wed 2022-11-23 18:42:30 CET. --
Nov 23 11:33:23 matrix matrix-synapse[1029358]: 2022-11-23 10:33:23,569 - synapse.handlers.device - 1132 - WARNING - _maybe_retry_device_resync-770 - Failed to handle device list update for @kosyne:kaosteam.net: Failed to send request: DNSLookupError: DNS lookup failed: no results for hostname lookup: matrix.kaosteam.net.
Nov 23 11:35:23 matrix matrix-synapse[1029358]: 2022-11-23 10:35:23,700 - synapse.handlers.device - 1132 - WARNING - _maybe_retry_device_resync-774 - Failed to handle device list update for @dries.staelens:rubicon.biz: Failed to send request: DNSLookupError: DNS lookup failed: no results for hostname lookup: rubicon.biz.
Nov 23 11:39:17 matrix matrix-synapse-worker-federation-sender-0[1030801]: 2022-11-23 10:39:17,029 - synapse.federation.sender.per_destination_queue - 368 - WARNING - federation_transaction_transmission_loop-21408 - TX [sameteem.com] Failed to send transaction: Failed to send request: TimeoutError: User timeout caused connection failure.
Nov 23 11:40:07 matrix matrix-synapse-worker-federation-sender-0[1030801]: 2022-11-23 10:40:07,211 - synapse.federation.sender.per_destination_queue - 368 - WARNING - federation_transaction_transmission_loop-21582 - TX [chat.jacobsen.app] Failed to send transaction: Failed to send request: TimeoutError: User timeout caused connection failure.
Nov 23 11:40:22 matrix matrix-synapse-worker-federation-sender-0[1030801]: 2022-11-23 10:40:22,671 - synapse.federation.sender.per_destination_queue - 368 - WARNING - federation_transaction_transmission_loop-21588 - TX [chat.perafree.ml] Failed to send transaction: Failed to send request: TimeoutError: User timeout caused connection failure.
Nov 23 11:45:53 matrix matrix-synapse[1029358]: 2022-11-23 10:45:53,782 - synapse.handlers.device - 1132 - WARNING - _maybe_retry_device_resync-795 - Failed to handle device list update for @dries.staelens:rubicon.biz: Failed to send request: DNSLookupError: DNS lookup failed: no results for hostname lookup: rubicon.biz.

Anything else that would be useful to know?

Full log during the time the duplication, see "Image 2" happened: message_duplication.log

All times & dates are UTC+1.

Dec 20 '23 19:12 matrixbot

@DMRobertson

I haven't had any issues which I described as "ghost notifications" for quite some time now. So I think that was fixed somehow.

The situation with the "Event duplication" relaxed a lot for a long time, but increased noticeably in the past couple of weeks. The events that were "duplicated" all seem to come from a particular time span, as before, where we experienced a distributed join spam attack with >35k users.

On Dec 16 2023 we had an interesting "conversation" with our bot (mjolnir). We told it to ban a user, just as usual, and it told us that it isn't in the room anymore:

Dec 16 19:02:16 matrix matrix-bot-mjolnir[189389]: Sat, 16 Dec 2023 19:02:16 GMT [ERROR] [MatrixHttpClient] (REQ-46674) [Error: Error during MatrixClient request GET /_matrix/client/v3/rooms/!iuyQXswfjgxQMZGrfQ%3Amatrix.org/state/m.room.server_acl/: 404 Not Found -- {"errcode":"M_NOT_FOUND","error":"Event not found."}]
Dec 16 19:02:16 matrix matrix-bot-mjolnir[189389]: Sat, 16 Dec 2023 19:02:16 GMT [ERROR] [MatrixHttpClient] (REQ-46676) [Error: Error during MatrixClient request PUT /_matrix/client/v3/rooms/!iuyQXswfjgxQMZGrfQ%3Amatrix.org/state/m.room.server_acl/: 403 Forbidden -- {"errcode":"M_FORBIDDEN","error":"User @bot.mjolnir:michaelsasser.org not in room !iuyQXswfjgxQMZGrfQ:matrix.org"}]

So I first checked the room member list in element-web (with sliding-sync) on my instance, and it was still there, but it had the avatar and display name from over a year ago. On the matrix.org instance using the app.vector.im element client, it wasn't in the room list anymore. When I cleared my instance's client, it was gone too. When I checked the developer tools (on my instance) under room state -> m.room.member, the bot wasn't shown at all, as if it would have never been joined to the room. So I checked the logs where nothing indicated anything interesting. Then I went to the database and found something disturbing. The current membership state event of the bot is not the one it should be. It was actually an historical one from the time span mentioned above.

The last join the bot logged happened at 2023-12-07 11:02 and the next join took place at 2023-12-07 11:34:00 (which was not logged anymore by the bot). So, whatever went sideways happened, with high certainty between 2023-12-07 11:02:00 and 2023-12-07 11:34:00 (UTC+1).

Asking the database what the current state event of the bot is

-- Query the current membership state event for the bot
SELECT json::JSONB, event_id
FROM event_json
WHERE event_id IN
    (SELECT event_id
     FROM current_state_events
     WHERE room_id='!iuyQXswfjgxQMZGrfQ:matrix.org'
       AND TYPE='m.room.member'
       AND state_key='@bot.mjolnir:michaelsasser.org'
    );

// The response to that query was an old state event which should not be the
// current one. Though, from the origin_server_ts it seems to be consistent
// with the old avatar and displayname I initially saw in the room user
// directory on element-web which sliding-sync enabled.

// $cb_QySiCLd5FLanxgrgnYzPLG83U2zO69PHwGj4Cxu8
{
  type: "m.room.member",
  depth: 45711,
  hashes: {
    sha256: "Y4IQHedNuOc1PQ/bZAEUnZw3asz/dcUr8vINr3hhPN0",
  },
  origin: "michaelsasser.org",
  sender: "@bot.mjolnir:michaelsasser.org",
  content: {
    membership: "leave",
  },
  room_id: "!iuyQXswfjgxQMZGrfQ:matrix.org", // Python Room: @python:matrix.org
  unsigned: {
    replaces_state: "$qU1rX95f1WpJKa6oHtwAPIyp0dmVbvP6DqDQ7gQtEI8",
  },
  state_key: "@bot.mjolnir:michaelsasser.org",
  prev_state: [],
  signatures: {
    "michaelsasser.org": {
      "ed25519:a_QNjX": "L3HB/fwK4lpQRMcBcXSJQkc9PnY1zxucjJ8w25jAwLwHINT1HWPt4d1IHOg7ldIg3/+MbW/L8ax+RRRg0aYmCA",
    },
  },
  auth_events: [
    "$qU1rX95f1WpJKa6oHtwAPIyp0dmVbvP6DqDQ7gQtEI8",
    "$Pq-FciS-IXzHqwLHHu6EZG9u7Y_hDes8nD3p4VwzfJo",
    "$ZW_yTzbdiRuFrpN5SNhnHCYSmzObqCsP0ADKGF8bsz4",
  ],
  prev_events: ["$6DeuJJjdF7lkemjqQOxCJavCfg0_-cixJ_vYyTaShrw"],
  origin_server_ts: 1643233431682, // Wednesday, January 26, 2022 9:43:51.682 PM
}

Normally, it should be or more precisely, it should have responded to with this state event:

-- Query the latest (by time) membership state event for the bot
SELECT json::JSONB, event_id
FROM event_json
WHERE event_id IN
    (SELECT event_id
     FROM state_events
     WHERE room_id='!iuyQXswfjgxQMZGrfQ:matrix.org'
       AND TYPE='m.room.member'
       AND state_key='@bot.mjolnir:michaelsasser.org'
    )
ORDER BY json::JSONB->'origin_server_ts' DESC
LIMIT 1;

// $_6N5ltJ9hg35sk4j8Do7PTNNzlMjcZSk7p8qlJ51fFA
{
  type: "m.room.member",
  depth: 118579,
  hashes: {
    sha256: "YZ/qfnBF+RlFUO8vaXpPep6UVcAgc9sVbue8oUDVaAU",
  },
  origin: "michaelsasser.org",
  sender: "@bot.mjolnir:michaelsasser.org",
  content: {
    avatar_url: "mxc://michaelsasser.org/kPMJgudCvuUzCIrOeJuUrPTn",
    membership: "join",
    displayname: "Administrator",
  },
  room_id: "!iuyQXswfjgxQMZGrfQ:matrix.org",
  unsigned: {
    age_ts: 1669213202762,
    replaces_state: "$rST-rCiCfl8z0zZVSlBWZEeuVX8oy40SFOkrlNoJ5RY",
  },
  state_key: "@bot.mjolnir:michaelsasser.org",
  signatures: {
    "michaelsasser.org": {
      "ed25519:a_QNjX": "gfeD/i2blF1OSX7A3D59lGsDMBeQx0zdfmmP6ZK7D/VKcwjI1tTm9zDLS+//Nq07qKUsGx9KJWF9WuhUTdg7CQ",
    },
  },
  auth_events: [
    "$rST-rCiCfl8z0zZVSlBWZEeuVX8oy40SFOkrlNoJ5RY",
    "$Ur-nGignA9Bqyyv-YRC5oH8UkGHzqgoqevuzM_m8Jn8",
    "$Pq-FciS-IXzHqwLHHu6EZG9u7Y_hDes8nD3p4VwzfJo",
    "$tAy72IFE7UsohvKVrhH8jbolxKq0N0qM1P7cClN9xIE",
  ],
  prev_events: ["$_rx-nDqmJuZW5Tu5qRP3xdLbuFSm3wby970RG3JP1wU"],
  origin_server_ts: 1669213202762, // Wednesday, November 23, 2022 2:20:02.762 PM
}

Just for reference:

Asking the database about all the membership state events of the bot

-- Get all membership state events for the bot and sort them by `depth`
SELECT json::JSONB, event_id
FROM event_json
WHERE event_id IN
    (SELECT event_id
     FROM state_events
     WHERE room_id='!iuyQXswfjgxQMZGrfQ:matrix.org'
       AND TYPE='m.room.member'
       AND state_key='@bot.mjolnir:michaelsasser.org'
    )
ORDER BY json::JSONB->'depth';

// Result
[
  // $Mq-oxPctUHcgjuhnqv3HCcK9q-shzshhjrVVHkb-wv4
  {
    type: "m.room.member",
    depth: 14212,
    hashes: {sha256: "9xr2Ni1HNNarZySoGM26qaVElKgUEA1FAky3zwr2iWE"},
    origin: "michaelsasser.org",
    sender: "@michael:michaelsasser.org",
    content: {
      avatar_url: "mxc://michaelsasser.org/XGOSHXVNMogAChjWByqRsYTT",
      membership: "invite",
      displayname: "Abuse Management",
    },
    room_id: "!iuyQXswfjgxQMZGrfQ:matrix.org",
    unsigned: {
      age: 48290275580,
      invite_room_state: [
        {
          type: "m.room.avatar",
          sender: "@michael:michaelsasser.org",
          content: {
            url: "mxc://michaelsasser.org/LQdedDsAKKrwxEFWZvRLuSfa",
          },
          state_key: "",
        },
        {
          type: "m.room.canonical_alias",
          sender: "@michaelsasser:matrix.org",
          content: {alias: "#Python3:matrix.org"},
          state_key: "",
        },
        {
          type: "m.room.join_rules",
          sender: "@michaelsasser:matrix.org",
          content: {join_rule: "public"},
          state_key: "",
        },
        {
          type: "m.room.member",
          sender: "@michael:michaelsasser.org",
          content: {
            avatar_url: "mxc://michaelsasser.org/gHeydUcioJByDfoMSwRRfNHc",
            membership: "join",
            displayname: "Michael",
          },
          state_key: "@michael:michaelsasser.org",
        },
        {
          type: "m.room.name",
          sender: "@michaelsasser:matrix.org",
          content: {name: "Python"},
          state_key: "",
        },
      ],
    },
    state_key: "@bot.mjolnir:michaelsasser.org",
    prev_state: [],
    signatures: {
      "michaelsasser.org": {
        "ed25519:a_QNjX": "lzQWtRz17YC9EONtraKyCSbO/EZtN4y9FJ/YZfM0Tc8xfUuqB6VdzpSRH0Jyl31tCTeRtyKwmrNmsi4MkKrkAQ",
      },
    },
    auth_events: [
      "$nrVVXusAZIdWQ2aODw5XXs-A5F7iUvvEEytQzheVuY0",
      "$wslk8IDCsILP43g8VqaAH8TsThP8_zsC4Vzr_Fi6GYw",
      "$FQrAS1-_g3idI8GoWcKDBy-DRuhqVqS8zmMnylsJ9Z0",
      "$Pq-FciS-IXzHqwLHHu6EZG9u7Y_hDes8nD3p4VwzfJo",
    ],
    prev_events: ["$Fd0Y1VSUzbxMenKzpWZ_sm08VGVMuBmVje8u--ftM3c"],
    origin_server_ts: 1620665527678, // Monday, May 10, 2021 4:52:07.678 PM
  },

  // $CosN_FgJRmvn-xqxywY5R3XVsdv5Wig6yetgygioAcw
  {
    type: "m.room.member",
    depth: 14213,
    hashes: {sha256: "WRC7Z8HHRTvHYQ/KLZDqBzh8Ipo+aOzLepXlwjxe6N0"},
    origin: "michaelsasser.org",
    sender: "@bot.mjolnir:michaelsasser.org",
    content: {
      avatar_url: "mxc://michaelsasser.org/XGOSHXVNMogAChjWByqRsYTT",
      membership: "join",
      displayname: "Abuse Management",
    },
    room_id: "!iuyQXswfjgxQMZGrfQ:matrix.org",
    unsigned: {
      age: 48290275158,
      replaces_state: "$Mq-oxPctUHcgjuhnqv3HCcK9q-shzshhjrVVHkb-wv4",
    },
    state_key: "@bot.mjolnir:michaelsasser.org",
    prev_state: [],
    signatures: {
      "michaelsasser.org": {
        "ed25519:a_QNjX": "6BX/DbQRP6jNgLvxevgOPp0w1RJoRl+w6cT8Tls1CXUTTYQpszbXdGDFBpINK6qBYq+crnxNgGuA+nMhaZkXCA",
      },
    },
    auth_events: [
      "$Mq-oxPctUHcgjuhnqv3HCcK9q-shzshhjrVVHkb-wv4",
      "$Pq-FciS-IXzHqwLHHu6EZG9u7Y_hDes8nD3p4VwzfJo",
      "$wslk8IDCsILP43g8VqaAH8TsThP8_zsC4Vzr_Fi6GYw",
      "$FQrAS1-_g3idI8GoWcKDBy-DRuhqVqS8zmMnylsJ9Z0",
    ],
    prev_events: ["$Mq-oxPctUHcgjuhnqv3HCcK9q-shzshhjrVVHkb-wv4"],
    origin_server_ts: 1620665528100, // Monday, May 10, 2021 4:52:08.100 PM
  },

  // $kj0hopGM88K3qvvIJXTFrTZP7kaLtFdDCqBNg__RVXg
  {
    type: "m.room.member",
    depth: 14314,
    hashes: {sha256: "vCrtMIDJwJ0b31ZDJXnfwblMJBED/ysnhpn8duqVKFs"},
    origin: "michaelsasser.org",
    sender: "@bot.mjolnir:michaelsasser.org",
    content: {
      avatar_url: "mxc://michaelsasser.org/YdLGvNWOrJiZNQvYWxpdyith",
      membership: "join",
      displayname: "Abuse Management",
    },
    room_id: "!iuyQXswfjgxQMZGrfQ:matrix.org",
    unsigned: {
      age: 48095980698,
      replaces_state: "$CosN_FgJRmvn-xqxywY5R3XVsdv5Wig6yetgygioAcw",
    },
    state_key: "@bot.mjolnir:michaelsasser.org",
    prev_state: [],
    signatures: {
      "michaelsasser.org": {
        "ed25519:a_QNjX": "4pBBdTMrUXURB41sgE2hHr4PKfaZlTlMWrQLNhQ505K8say2vkeCiMuNd6tsNgmeQ8vwtwNalbN3S9EmLygzAw",
      },
    },
    auth_events: [
      "$CosN_FgJRmvn-xqxywY5R3XVsdv5Wig6yetgygioAcw",
      "$FQrAS1-_g3idI8GoWcKDBy-DRuhqVqS8zmMnylsJ9Z0",
      "$Pq-FciS-IXzHqwLHHu6EZG9u7Y_hDes8nD3p4VwzfJo",
      "$ZW_yTzbdiRuFrpN5SNhnHCYSmzObqCsP0ADKGF8bsz4",
    ],
    prev_events: ["$uP758I6_pK_L_LqUvLzNPB18ZRCpkg7XtbDRV1oFkiQ"],
    origin_server_ts: 1620859822560, // Wednesday, May 12, 2021 10:50:22.560 PM
  },

  // $-n0TlZQ3ae9mXQKeAVFdBfxbO_-0wIwY4bIguMkaAgU
  {
    type: "m.room.member",
    depth: 45689,
    hashes: {sha256: "YqYhi1uyc9Bii1PF9M+vEg5vNhWhaD2kKCgS50vIa7U"},
    origin: "michaelsasser.org",
    sender: "@bot.mjolnir:michaelsasser.org",
    content: {membership: "leave"},
    room_id: "!iuyQXswfjgxQMZGrfQ:matrix.org",
    unsigned: {
      replaces_state: "$kj0hopGM88K3qvvIJXTFrTZP7kaLtFdDCqBNg__RVXg",
    },
    state_key: "@bot.mjolnir:michaelsasser.org",
    prev_state: [],
    signatures: {
      "michaelsasser.org": {
        "ed25519:a_QNjX": "NZCj21eFB0BUe5y8XRhwzPd4yr7woH1bYUXOidmljzxF3YMvLLBbYES2VfA3OOHkJL4Viws3Y7RbNEwHNfhzDA",
      },
    },
    auth_events: [
      "$kj0hopGM88K3qvvIJXTFrTZP7kaLtFdDCqBNg__RVXg",
      "$Pq-FciS-IXzHqwLHHu6EZG9u7Y_hDes8nD3p4VwzfJo",
      "$ZW_yTzbdiRuFrpN5SNhnHCYSmzObqCsP0ADKGF8bsz4",
    ],
    prev_events: ["$cgxH6iDVSpvPSZXsQSEWZB-ZJGIzMeFd9uL2SDGihvU"],
    origin_server_ts: 1643233112395, // Wednesday, January 26, 2022 9:38:32.395 PM
  },

  // $qU1rX95f1WpJKa6oHtwAPIyp0dmVbvP6DqDQ7gQtEI8
  {
    type: "m.room.member",
    depth: 45706,
    hashes: {sha256: "/uCHlAoGBl3Ge9ZrTRRuubvdgMExJj6eVBIQT2M9KTU"},
    origin: "michaelsasser.org",
    sender: "@bot.mjolnir:michaelsasser.org",
    content: {
      avatar_url: "mxc://michaelsasser.org/YdLGvNWOrJiZNQvYWxpdyith",
      membership: "join",
      displayname: "Abuse Management",
    },
    room_id: "!iuyQXswfjgxQMZGrfQ:matrix.org",
    unsigned: {
      replaces_state: "$-n0TlZQ3ae9mXQKeAVFdBfxbO_-0wIwY4bIguMkaAgU",
    },
    state_key: "@bot.mjolnir:michaelsasser.org",
    prev_state: [],
    signatures: {
      "michaelsasser.org": {
        "ed25519:a_QNjX": "X4Nm2EtObxA5L4jEs40uLJA90lq61RCZB0CW7d0pY0EFfCAnB2Ur69n81FW6fJLHkaNOr42p6NKSFGFvLB/lAA",
      },
    },
    auth_events: [
      "$-n0TlZQ3ae9mXQKeAVFdBfxbO_-0wIwY4bIguMkaAgU",
      "$Pq-FciS-IXzHqwLHHu6EZG9u7Y_hDes8nD3p4VwzfJo",
      "$FQrAS1-_g3idI8GoWcKDBy-DRuhqVqS8zmMnylsJ9Z0",
      "$ZW_yTzbdiRuFrpN5SNhnHCYSmzObqCsP0ADKGF8bsz4",
    ],
    prev_events: ["$rmOzmrZbOut2UK43mAASkCHZBeL2duUte5PbjgvUH0E"],
    origin_server_ts: 1643233352820, // Wednesday, January 26, 2022 9:42:32.820 PM
  },

  // $cb_QySiCLd5FLanxgrgnYzPLG83U2zO69PHwGj4Cxu8
  {
    type: "m.room.member",
    depth: 45711,
    hashes: {sha256: "Y4IQHedNuOc1PQ/bZAEUnZw3asz/dcUr8vINr3hhPN0"},
    origin: "michaelsasser.org",
    sender: "@bot.mjolnir:michaelsasser.org",
    content: {membership: "leave"},
    room_id: "!iuyQXswfjgxQMZGrfQ:matrix.org",
    unsigned: {
      replaces_state: "$qU1rX95f1WpJKa6oHtwAPIyp0dmVbvP6DqDQ7gQtEI8",
    },
    state_key: "@bot.mjolnir:michaelsasser.org",
    prev_state: [],
    signatures: {
      "michaelsasser.org": {
        "ed25519:a_QNjX": "L3HB/fwK4lpQRMcBcXSJQkc9PnY1zxucjJ8w25jAwLwHINT1HWPt4d1IHOg7ldIg3/+MbW/L8ax+RRRg0aYmCA",
      },
    },
    auth_events: [
      "$qU1rX95f1WpJKa6oHtwAPIyp0dmVbvP6DqDQ7gQtEI8",
      "$Pq-FciS-IXzHqwLHHu6EZG9u7Y_hDes8nD3p4VwzfJo",
      "$ZW_yTzbdiRuFrpN5SNhnHCYSmzObqCsP0ADKGF8bsz4",
    ],
    prev_events: ["$6DeuJJjdF7lkemjqQOxCJavCfg0_-cixJ_vYyTaShrw"],
    origin_server_ts: 1643233431682, // Wednesday, January 26, 2022 9:43:51.682 PM
  },

  // $bycEakqs8rCaoj4R6p5d8SMEw3u-X95a5cPcTkhjQUo
  {
    type: "m.room.member",
    depth: 45759,
    hashes: {sha256: "lUbjxLBtAMdFsP2eCzehw+/q61eewB2lUd6Db2O/RRM"},
    origin: "michaelsasser.org",
    sender: "@michael:michaelsasser.org",
    content: {
      avatar_url: "mxc://michaelsasser.org/YdLGvNWOrJiZNQvYWxpdyith",
      membership: "invite",
      displayname: "Abuse Management",
    },
    room_id: "!iuyQXswfjgxQMZGrfQ:matrix.org",
    unsigned: {
      replaces_state: "$cb_QySiCLd5FLanxgrgnYzPLG83U2zO69PHwGj4Cxu8",
      invite_room_state: [
        {
          type: "m.room.avatar",
          sender: "@michael:michaelsasser.org",
          content: {
            url: "mxc://michaelsasser.org/LQdedDsAKKrwxEFWZvRLuSfa",
          },
          state_key: "",
        },
        {
          type: "m.room.canonical_alias",
          sender: "@michaelsasser:matrix.org",
          content: {alias: "#python:matrix.org", alt_aliases: []},
          state_key: "",
        },
        {
          type: "m.room.create",
          sender: "@michaelsasser:matrix.org",
          content: {
            creator: "@michaelsasser:matrix.org",
            room_version: "6",
          },
          state_key: "",
        },
        {
          type: "m.room.join_rules",
          sender: "@michaelsasser:matrix.org",
          content: {join_rule: "public"},
          state_key: "",
        },
        {
          type: "m.room.member",
          sender: "@michael:michaelsasser.org",
          content: {
            avatar_url: "mxc://michaelsasser.org/gHeydUcioJByDfoMSwRRfNHc",
            membership: "join",
            displayname: "Michael",
          },
          state_key: "@michael:michaelsasser.org",
        },
        {
          type: "m.room.name",
          sender: "@michaelsasser:matrix.org",
          content: {name: "Python"},
          state_key: "",
        },
      ],
    },
    state_key: "@bot.mjolnir:michaelsasser.org",
    prev_state: [],
    signatures: {
      "michaelsasser.org": {
        "ed25519:a_QNjX": "YUR6B4c3y7gvB3NW00Aq4Q3D1aUhr/1OqBmKTGvgxpFfUWMeqnYZV2PnQgGy/dfhVRno3iEuKnKKh/WfJoXBDw",
      },
    },
    auth_events: [
      "$cb_QySiCLd5FLanxgrgnYzPLG83U2zO69PHwGj4Cxu8",
      "$nrVVXusAZIdWQ2aODw5XXs-A5F7iUvvEEytQzheVuY0",
      "$ZW_yTzbdiRuFrpN5SNhnHCYSmzObqCsP0ADKGF8bsz4",
      "$FQrAS1-_g3idI8GoWcKDBy-DRuhqVqS8zmMnylsJ9Z0",
      "$Pq-FciS-IXzHqwLHHu6EZG9u7Y_hDes8nD3p4VwzfJo",
    ],
    prev_events: ["$wbhx9QDWZzbc60qvGmSpfpr8F-WGA6_hF98lU03O7Rc"],
    origin_server_ts: 1643273377451, // Thursday, January 27, 2022 8:49:37.451 AM
  },

  // $AwKIg7YRtAfl1wbsrm40_gqsXJsibib3FpJX7I7CRrI
  {
    type: "m.room.member",
    depth: 45760,
    hashes: {sha256: "tw+taBgFb6YyFQfuK1jIYpixXGDVdci7kn7Fzv3AAlc"},
    origin: "michaelsasser.org",
    sender: "@bot.mjolnir:michaelsasser.org",
    content: {
      avatar_url: "mxc://michaelsasser.org/YdLGvNWOrJiZNQvYWxpdyith",
      membership: "join",
      displayname: "Abuse Management",
    },
    room_id: "!iuyQXswfjgxQMZGrfQ:matrix.org",
    unsigned: {
      replaces_state: "$bycEakqs8rCaoj4R6p5d8SMEw3u-X95a5cPcTkhjQUo",
    },
    state_key: "@bot.mjolnir:michaelsasser.org",
    prev_state: [],
    signatures: {
      "michaelsasser.org": {
        "ed25519:a_QNjX": "ufky5xnc8k6NfeT/pii6NGqeh3EexIY3ftygEwZrldlbVIwiIt2nY56J8CySWEN+mRNjl8C8uTJp9w66Ja4tCQ",
      },
    },
    auth_events: [
      "$bycEakqs8rCaoj4R6p5d8SMEw3u-X95a5cPcTkhjQUo",
      "$Pq-FciS-IXzHqwLHHu6EZG9u7Y_hDes8nD3p4VwzfJo",
      "$FQrAS1-_g3idI8GoWcKDBy-DRuhqVqS8zmMnylsJ9Z0",
      "$ZW_yTzbdiRuFrpN5SNhnHCYSmzObqCsP0ADKGF8bsz4",
    ],
    prev_events: ["$bycEakqs8rCaoj4R6p5d8SMEw3u-X95a5cPcTkhjQUo"],
    origin_server_ts: 1643273378028, // Thursday, January 27, 2022 8:49:38.028 AM
  },
  {
    type: "m.room.member",
    depth: 50867,
    hashes: {sha256: "Yeo4hdZuxBzZElV1uQ93Sl1YryUPg3LqdcGbSp/V2tA"},
    origin: "michaelsasser.org",
    sender: "@bot.mjolnir:michaelsasser.org",
    content: {
      avatar_url: "mxc://michaelsasser.org/YdLGvNWOrJiZNQvYWxpdyith",
      membership: "join",
      displayname: "Administrator",
    },
    room_id: "!iuyQXswfjgxQMZGrfQ:matrix.org",
    unsigned: {
      replaces_state: "$AwKIg7YRtAfl1wbsrm40_gqsXJsibib3FpJX7I7CRrI",
    },
    state_key: "@bot.mjolnir:michaelsasser.org",
    prev_state: [],
    signatures: {
      "michaelsasser.org": {
        "ed25519:a_QNjX": "kKrHKoi/r6mhu2VdRZzlRDpEFfKImyLGuy0AL6NcLekhctEk/xAW73HLR0531K+fQmjvcyoCOuy9C9KxuMzBCQ",
      },
    },
    auth_events: [
      "$AwKIg7YRtAfl1wbsrm40_gqsXJsibib3FpJX7I7CRrI",
      "$tUYE1mFeqtmd09uaK36r5eW_m1KFEWzYa4ji7ga2EKg",
      "$FQrAS1-_g3idI8GoWcKDBy-DRuhqVqS8zmMnylsJ9Z0",
      "$Pq-FciS-IXzHqwLHHu6EZG9u7Y_hDes8nD3p4VwzfJo",
    ],
    prev_events: ["$xKb1Fofi9ztN23BeejTeFYOBxETkwrly0zkTXYYKPzg"],
    origin_server_ts: 1647259553837, // Monday, March 14, 2022 12:05:53.837 PM
  },

  // $NfdoJih78gBbTys_V3FeE9tv9-JZG69Lbqqz8fWjeoA
  {
    type: "m.room.member",
    depth: 59399,
    hashes: {sha256: "45CskOjxJmqcDc+6mvUV6g0d2a4M+59nweH9eIqvCv8"},
    origin: "michaelsasser.org",
    sender: "@bot.mjolnir:michaelsasser.org",
    content: {
      avatar_url: "mxc://michaelsasser.org/FtcRMfFgpJnpLzWZoFKQNMMd",
      membership: "join",
      displayname: "Administrator",
    },
    room_id: "!iuyQXswfjgxQMZGrfQ:matrix.org",
    unsigned: {},
    state_key: "@bot.mjolnir:michaelsasser.org",
    prev_state: [],
    signatures: {
      "michaelsasser.org": {
        "ed25519:a_QNjX": "2IAZoyf4yTlYoCjtFX1IOKV5xLWhTC+NmY/3axu0sWYn9V21m3sXiU7928p1gqf/AYRG2nNMIpg0L/+XeiOqCg",
      },
    },
    auth_events: [
      "$NfdoJih78gBbTys_V3FeE9tv9-JZG69Lbqqz8fWjeoA",
      "$Pq-FciS-IXzHqwLHHu6EZG9u7Y_hDes8nD3p4VwzfJo",
      "$hUZdUKqNn6maEll77yE0z2467P4HQgVg95wTygi5Gvg",
      "$9sNpU5w38YrbcRBzx6IKGRVfv-T8-D6IS_lTVBEckro",
    ],
    prev_events: [
      "$BnWnDOK5G73hBFDCoR3wmOy2zgFfmGtUp_AzLvEBFzs",
      "$s5L-e6Vmbo3R3DVLXNYEncx_GWOH3DEEFoVGgkFeGEg",
    ],
    origin_server_ts: 1651987205907, // Sunday, May 8, 2022 5:20:05.907 AM
  },

  // $b0o1xm8AIuTFDAz7f_lC89f1JEvMGO5rjEdyNffd2Wg
  {
    type: "m.room.member",
    depth: 59400,
    hashes: {sha256: "R1HILYpnatdcckdcCm+WdkqswRCi0ZKaYqOVAegO7LU"},
    origin: "michaelsasser.org",
    sender: "@bot.mjolnir:michaelsasser.org",
    content: {
      avatar_url: "mxc://michaelsasser.org/kPMJgudCvuUzCIrOeJuUrPTn",
      membership: "join",
      displayname: "Administrator",
    },
    room_id: "!iuyQXswfjgxQMZGrfQ:matrix.org",
    unsigned: {},
    state_key: "@bot.mjolnir:michaelsasser.org",
    prev_state: [],
    signatures: {
      "michaelsasser.org": {
        "ed25519:a_QNjX": "p4Lae6lJ060FkmXB5TLk2hgKLwAapYculSa2tyRgk5DTp8d4WIFr9T6TZZ8yU1w0u1ybsziqPGjymf60hB36DA",
      },
    },
    auth_events: [
      "$qX_SKHxi7hCBj7OL4PEhT8rM4f3_MX33Asx95z5kCJQ",
      "$Pq-FciS-IXzHqwLHHu6EZG9u7Y_hDes8nD3p4VwzfJo",
      "$hUZdUKqNn6maEll77yE0z2467P4HQgVg95wTygi5Gvg",
      "$9sNpU5w38YrbcRBzx6IKGRVfv-T8-D6IS_lTVBEckro",
    ],
    prev_events: ["$qX_SKHxi7hCBj7OL4PEhT8rM4f3_MX33Asx95z5kCJQ"],
    origin_server_ts: 1651990924046, // Sunday, May 8, 2022 6:22:04.046 AM
  },

  // $SKK1_K51jxGTJcUueDFVoHajkXGTRFMa-IMoIrmz6q4
  {
    type: "m.room.member",
    depth: 118048,
    hashes: {sha256: "5rDK7MwQnO8NWFkWyf1tzvPAXwkLUXcp3iYaM8rFHuk"},
    origin: "matrix.org",
    sender: "@michaelsasser:matrix.org",
    content: {membership: "leave"},
    room_id: "!iuyQXswfjgxQMZGrfQ:matrix.org",
    unsigned: {
      age: 8635818,
      replaces_state: "$b0o1xm8AIuTFDAz7f_lC89f1JEvMGO5rjEdyNffd2Wg",
    },
    state_key: "@bot.mjolnir:michaelsasser.org",
    signatures: {
      "matrix.org": {
        "ed25519:a_RXGa": "nI72s0AUVy1v6k/la2SeDUSdxvdqp+SZuURrOAA5dU40kIS+BMtDpd8ft39iqT6iw+ryXD6MQ4ZletG8TPMGCA",
      },
    },
    auth_events: [
      "$Pq-FciS-IXzHqwLHHu6EZG9u7Y_hDes8nD3p4VwzfJo",
      "$A-7fXhjCI7UazqdD0cz_1Srzg-493XoCBsq2abAkSNc",
      "$b0o1xm8AIuTFDAz7f_lC89f1JEvMGO5rjEdyNffd2Wg",
      "$rGDjDeraGK57FgbQOhS6nfpiJ-Y-4jKiKEcpYtVQcUI",
    ],
    prev_events: ["$A-7fXhjCI7UazqdD0cz_1Srzg-493XoCBsq2abAkSNc"],
    origin_server_ts: 1668947167440, // Sunday, November 20, 2022 12:26:07.440 PM
  },

  // $iexffkStREHb4KcvmBLS4QiOlxvUouKvWKUdp6vuojk
  {
    type: "m.room.member",
    depth: 118151,
    hashes: {sha256: "V7lanv+6Ib+1vB/8gG3tiH4pB58gUTU3Qe/9gzanhM0"},
    origin: "michaelsasser.org",
    sender: "@bot.mjolnir:michaelsasser.org",
    content: {
      avatar_url: "mxc://michaelsasser.org/kPMJgudCvuUzCIrOeJuUrPTn",
      membership: "join",
      displayname: "Administrator",
    },
    room_id: "!iuyQXswfjgxQMZGrfQ:matrix.org",
    unsigned: {
      age_ts: 1668989318053,
      replaces_state: "$SKK1_K51jxGTJcUueDFVoHajkXGTRFMa-IMoIrmz6q4",
    },
    state_key: "@bot.mjolnir:michaelsasser.org",
    signatures: {
      "michaelsasser.org": {
        "ed25519:a_QNjX": "LCJBRa5oliq/RfahJa8NxmSAGrfbdzdss3AyG3XABSx7MbBUoHzsu2Ati78mv4EgdoeRvX9LS+UN5a8X9s2GBQ",
      },
    },
    auth_events: [
      "$tAy72IFE7UsohvKVrhH8jbolxKq0N0qM1P7cClN9xIE",
      "$Ur-nGignA9Bqyyv-YRC5oH8UkGHzqgoqevuzM_m8Jn8",
      "$Pq-FciS-IXzHqwLHHu6EZG9u7Y_hDes8nD3p4VwzfJo",
      "$SKK1_K51jxGTJcUueDFVoHajkXGTRFMa-IMoIrmz6q4",
    ],
    prev_events: ["$UChrHiRIGdgPpDGD4jFa4G5oB-umimKO3yjr8YrfFyo"],
    origin_server_ts: 1668989318053, // Monday, November 21, 2022 12:08:38.053 AM
  },

  // $iexffkStREHb4KcvmBLS4QiOlxvUouKvWKUdp6vuojk
  {
    type: "m.room.member",
    depth: 118570,
    hashes: {sha256: "Dhd8n5Jq0HAY+AgWIseGV5BVCmXDGJLSHRcFLC79Gis"},
    origin: "michaelsasser.org",
    sender: "@bot.mjolnir:michaelsasser.org",
    content: {membership: "leave"},
    room_id: "!iuyQXswfjgxQMZGrfQ:matrix.org",
    unsigned: {
      age_ts: 1669200418456,
      replaces_state: "$iexffkStREHb4KcvmBLS4QiOlxvUouKvWKUdp6vuojk",
    },
    state_key: "@bot.mjolnir:michaelsasser.org",
    signatures: {
      "michaelsasser.org": {
        "ed25519:a_QNjX": "6KBJbMEQ4cbEV7O1I/5sjg/i4+8gmf4xzBoOHlSi5fkgU/WtOuFsszZXzjOaO9+F3tVufQON/OSXw9YBhD6mBg",
      },
    },
    auth_events: [
      "$iexffkStREHb4KcvmBLS4QiOlxvUouKvWKUdp6vuojk",
      "$Ur-nGignA9Bqyyv-YRC5oH8UkGHzqgoqevuzM_m8Jn8",
      "$Pq-FciS-IXzHqwLHHu6EZG9u7Y_hDes8nD3p4VwzfJo",
    ],
    prev_events: ["$U7DFEXnnyuZ2ez6znqUaLeJodQTAcTN1hsDDX6f5ZTg"],
    origin_server_ts: 1669200418456, // Wednesday, November 23, 2022 10:46:58.456 AM
  },

  // $_6N5ltJ9hg35sk4j8Do7PTNNzlMjcZSk7p8qlJ51fFA
  {
    type: "m.room.member",
    depth: 118579,
    hashes: {sha256: "YZ/qfnBF+RlFUO8vaXpPep6UVcAgc9sVbue8oUDVaAU"},
    origin: "michaelsasser.org",
    sender: "@bot.mjolnir:michaelsasser.org",
    content: {
      avatar_url: "mxc://michaelsasser.org/kPMJgudCvuUzCIrOeJuUrPTn",
      membership: "join",
      displayname: "Administrator",
    },
    room_id: "!iuyQXswfjgxQMZGrfQ:matrix.org",
    unsigned: {
      age_ts: 1669213202762,
      replaces_state: "$rST-rCiCfl8z0zZVSlBWZEeuVX8oy40SFOkrlNoJ5RY",
    },
    state_key: "@bot.mjolnir:michaelsasser.org",
    signatures: {
      "michaelsasser.org": {
        "ed25519:a_QNjX": "gfeD/i2blF1OSX7A3D59lGsDMBeQx0zdfmmP6ZK7D/VKcwjI1tTm9zDLS+//Nq07qKUsGx9KJWF9WuhUTdg7CQ",
      },
    },
    auth_events: [
      "$rST-rCiCfl8z0zZVSlBWZEeuVX8oy40SFOkrlNoJ5RY",
      "$Ur-nGignA9Bqyyv-YRC5oH8UkGHzqgoqevuzM_m8Jn8",
      "$Pq-FciS-IXzHqwLHHu6EZG9u7Y_hDes8nD3p4VwzfJo",
      "$tAy72IFE7UsohvKVrhH8jbolxKq0N0qM1P7cClN9xIE",
    ],
    prev_events: ["$_rx-nDqmJuZW5Tu5qRP3xdLbuFSm3wby970RG3JP1wU"],
    origin_server_ts: 1669213202762, // Wednesday, November 23, 2022 2:20:02.762 PM
  },
]

//edit 2024-01-09 I see the same thing happening now with power levels m.room.power_levels i.e. moderators are suddenly no longer moderators and so on.

Environment: CX31 Hetzner Cloud

NAME    CORES   CPU TYPE    ARCHITECTURE   MEMORY     DISK     STORAGE TYPE   TRAFFIC
cx31    2       shared      x86            8.0 GB     80 GB    local          20 TB

The matrix parts were deployed using the Ansible playbook: spantaleev/matrix-docker-ansible-deploy

Debian: Linux matrix 6.1.0-13-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.55-1 (2023-09-29) x86_64GNU/Linux
Docker: Docker version 24.0.7, build afdd53b
Containerd: containerd containerd.io 1.6.24 61f9fd88f79f081d64d6fa3bb1a0dc71ec870523
Synapse: ghcr.io/matrix-org/synapse:v1.98.0
Sliding-Sync: ghcr.io/matrix-org/sliding-sync:v0.99.13
Element: vectorim/element-web:v1.11.51
Postgres: postgres:15.5-alpine

Jan 03 '24 05:01 MichaelSasser

Thanks for the update and your thorough investigation. I'm no longer working on Matrix, but I'm sure the Synapse team will take a look at this as part of issue review.

Jan 03 '24 12:01 DMRobertson

Since the last edit (2024-01-09), where power levels reverted suddenly, we have seen something like this happening a few times, Topics revert to a previous state, Users are suddenly no longer joined to the room, Homeservers who have joined the room before and have not purged the room from the homeserver can no longer join the room (we asked for logs, or more information but none of them were able to provide them).

Apr 07 '24 16:04 MichaelSasser