Partial Message Decryption Failure with SSO Authentication on Matrix Server

[gitayam]

Describe the bug Users across different devices (desktop and mobile) experience partial loss of message decryption. The issue persists across different Single Sign-On (SSO) methods, specifically with SSO-only authentication enabled (without password). A portion of messages remain encrypted and inaccessible, particularly those before the recent login sessions.

To Reproduce

Steps to Reproduce

1. Users log in using SSO (authentik and GitHub).
2. Attempt to access message history.
3. Observe that messages before the most recent login remain encrypted and unreadable.

Context and Environment

• The issue was observed in both authentik and GitHub SSO accounts.
• A direct clone of the public persona element qube does not exhibit the same issue.
• Recovery keys process reports successful recovery (e.g., 12 keys) but does not resolve the issue for certain message sections. The last successful login and message access was around Friday, 0350 UTC 12/29/2023.

Configuration Details

• Traefik Reverse-Proxy Configuration with playbook-managed-traefik.
• SSO enabled without password.
• Synapse Admin, Sliding Sync, and other relevant configurations are active.
• Various bots enabled (Signal Bot, Mjolnir Moderation Bot, etc.).
• OIDC configurations for GitHub and IrregularChat SSO.

My vars.yml file looks like this:

# Traefik Reverse-Proxy Configuration
matrix_playbook_reverse_proxy_type: playbook-managed-traefik
matrix_playbook_ssl_enabled: true
matrix_well_known_matrix_server_enabled: false
devture_systemd_service_manager_up_verification_delay_seconds: 30
matrix_synapse_admin_enabled: true # Synapse Admin will be accessible at: https://matrix.DOMAIN/synapse-admin/


# Devture Traefik Configuration
# Source Guide: https://appelman.se/matrix-on-cloudflare/
devture_traefik_config_entrypoint_web_secure_enabled: false
devture_traefik_container_web_host_bind_port: "127.0.0.1:81"
devture_traefik_config_entrypoint_web_forwardedHeaders_insecure: true
devture_traefik_additional_entrypoints_auto:
  - name: matrix-federation
    port: 8449
    host_bind_port: "127.0.0.1:8449"
    config: {}

# Coturn Server Configuration
matrix_coturn_enabled: false

# Sliding Sync Configuration
matrix_sliding_sync_enabled: true
devture_traefik_config_certificatesResolvers_acme_email: [email protected]
enable_set_displayname: true
devture_systemd_service_manager_up_verification_delay_seconds: 30

# Homeserver and Registration Configuration
matrix_homeserver_implementation: synapse
matrix_registration_enabled: true
matrix_synapse_admin_enabled: true
matrix_domain: domain.com
matrix_synapse_max_upload_size_mb: 150
matrix_synapse_media_retention_local_media_lifetime: 1 y
matrix_synapse_media_retention_remote_media_lifetime: 3 m
matrix_synapse_report_stats: true
matrix_synapse_user_directory_prefer_local_users: true
matrix_synapse_admin_enabled: true
# wether to accept passwords or not. True is yes. false is no.
matrix_synapse_password_config_enabled: false

# Synapse Auto Compressor Configuration
matrix_synapse_auto_compressor_enabled: true

# Synapse Workers Configuration
matrix_synapse_workers_enabled: true

# Mautrix Signal Bot Configuration

##################################
## BOTS
##################################

## Signal Bot
matrix_mautrix_signal_enabled: true
matrix_mautrix_signal_relaybot_enabled: false
matrix_synapse_ext_password_provider_shared_secret_auth_enabled: true
matrix_mautrix_signal_bridge_encryption_allow: true
matrix_mautrix_signal_configuration_extension_yaml: |
  bridge:
    encryption:
      allow: true
      default: true


# Mjolnir Moderation Bot Configuration
matrix_bot_mjolnir_enabled: true

# Anti-Spam Configuration
matrix_synapse_ext_spam_checker_mjolnir_antispam_enabled: true
matrix_synapse_ext_spam_checker_mjolnir_antispam_config_block_invites: true
matrix_synapse_ext_spam_checker_mjolnir_antispam_config_block_messages: true
matrix_synapse_ext_spam_checker_mjolnir_antispam_config_block_usernames: false
matrix_synapse_ext_spam_checker_mjolnir_antispam_config_ban_lists: []

# Maubot Configuration
matrix_bot_maubot_enabled: true

# Discord Bot Configuration
matrix_mautrix_discord_enabled: false

# ChatGPT Bot Configuration
matrix_bot_chatgpt_enabled: false
matrix_bot_chatgpt_matrix_bot_username_localpart: 'chatgpt'
matrix_bot_chatgpt_matrix_encryption: true
matrix_bot_chatgpt_matrix_bot_prompt_prefix: 'Instructions:\nYou are (Sentient AI Command (SAC)), a large language model trained by OpenAI. along with answering the prompt, provide a funny tech joke first'

matrix_synapse_auto_join_rooms: 
  - "#entry-public:domain.com"
  - "#announcements:domain.com"
  - "#chatgpt:domain.com


##################################
##################################
# OIDC Configuration
## roles/custom/matrix-synapse/templates/synapse/homeserver.yaml.j2
# https://github.com/spantaleev/matrix-docker-ansible-deploy/commit/9427f9408dfded216d7c29027c234b9762a26727
##################################
# matrix_synapse_oidc_enabled: true
# matrix_synapse_oidc_providers: 
matrix_synapse_configuration_extension_yaml: |
  oidc_providers:
    - idp_id: Github-Login
      idp_name: "domain Login: Github"
      idp_icon: mxc://domain.com/xxx
      discover: false
      issuer: "https://github.com/"
      client_id: "_redacted_"
      client_secret: "_redacted_"
      client_auth_method: client_secret_post
      scopes: [user:email]
      allow_existing_users: true
      authorization_endpoint: "https://github.com/login/oauth/authorize"
      token_endpoint: "https://github.com/login/oauth/access_token"
      userinfo_endpoint: "https://api.github.com/user"
      user_mapping_provider:
        config:
          subject_claim: "id"
          localpart_template: "{% raw %}{{ user.login }}{% endraw %}"
          display_name_template: "{% raw %}{{ user.name }}{% endraw %}"
          email_template: "{% raw %}{{ user.email }}{% endraw %}"
    - idp_id: domain-sso
      idp_name: "domain SSO"
      idp_icon: mxc://domain.com/xxx
      discover: true
      issuer: "https://sso.domain.com/application/o/element-messenger/"
      client_id: "_redacted_"
      client_secret: "_redacted_"
      client_auth_method: client_secret_post
      scopes:
          - "openid"
          - "profile"
      allow_existing_users: true
      backchannel_logout_enabled: true
      user_mapping_provider:
          config:
              localpart_template: "{% raw %}{{ user.preferred_username }}{% endraw %}"
              display_name_template: "{% raw %}{{ user.preferred_username|capitalize }}{% endraw %}"
  delete_stale_devices_after: 1y
  retention:
    enabled: true
    purge_jobs:
      - longest_max_lifetime: 1h
        interval: 30m  # Purge more frequently to handle the 1-hour max lifetime
      - shortest_max_lifetime: 1h
        longest_max_lifetime: 1d
        interval: 6h  # Moderate frequency for messages lasting up to a day
      - shortest_max_lifetime: 1d
        interval: 12h  # For messages older than a day, purging twice a day

Expected behavior All messages should be decrypted successfully upon logging in, especially after using recovery keys.

Actual behavior

• Messages remain encrypted and unreadable before the most recent login despite the successful recovery of some keys.
• The issue seems to affect some accounts but not others.

The Decryption Issue looks like this:

{
  "type": "m.room.message",
  "content": {
    "msgtype": "m.bad.encrypted",
    "body": "** Unable to decrypt: DecryptionError: The sender's device has not sent us the keys for this message. **"
  }
}

Matrix Server:

• OS: Linux debian 6.1.0-16-amd64 https://github.com/spantaleev/matrix-docker-ansible-deploy/issues/1 SMP PREEMPT_DYNAMIC Debian 6.1.67-1 (2023-12-12) x86_64 GNU/Linux
• Architecture amd64

Client:

• Multiple, mobile and desktop.

Additional context

• Some users reported the resolution of similar issues, while others, like the current reporter, continue to face decryption problems.
• It's unclear why the issue impacts only specific accounts or environments.

[spantaleev]

This is unlikely to be a problem with the Ansible deployment method, but rather with the specific Matrix clients you're using. This issue likely does not belong here.

[gitayam]

@spantaleev trying to ensure nothing in my config is causing this

[gitayam]

Another error when seeing the source message:

{
  "type": "m.room.message",
  "content": {
    "msgtype": "m.bad.encrypted",
    "body": "** Unable to decrypt: DecryptionError: OLM.UNKNOWN_MESSAGE_INDEX **"
  }
}

[gitayam]

Did not resolve the messages lost

SOLVED the issue of users being logged out by removing some of the cloudflare security for the entire subdomain.