matrix-synapse service fails to start

[drwahl]

Playbook Configuration:

My vars.yml file looks like this:

matrix_domain: drwahl.me
matrix_ssl_lets_encrypt_support_email: <redacted>
matrix_coturn_turn_static_auth_secret:  <redacted>
matrix_synapse_macaroon_secret_key:  <redacted>
devture_postgres_connection_password: <redacted>
matrix_prometheus_enabled: true
matrix_prometheus_node_exporter_enabled: true
matrix_grafana_enabled: true
matrix_grafana_anonymous_access: false
matrix_grafana_default_admin_user: <redacted>
matrix_grafana_default_admin_password: <redacted>
matrix_synapse_admin_enabled: true
matrix_synapse_ext_password_provider_shared_secret_auth_enabled: true
matrix_synapse_ext_password_provider_shared_secret_auth_shared_secret: <redacted>
matrix_registration_enabled: true
matrix_registration_admin_secret: <redacted>
matrix_dimension_enabled: true
matrix_dimension_admins:
  - "@<redacted>:{{ matrix_domain }}"
matrix_dimension_access_token: <redacted>
matrix_mautrix_whatsapp_version: latest
matrix_mautrix_whatsapp_enabled: true
matrix_mautrix_googlechat_enabled: true
matrix_mautrix_signal_enabled: true
matrix_mautrix_signal_relaybot_enabled: true
matrix_mx_puppet_discord_enabled: false
matrix_mx_puppet_slack_enabled: true
matrix_appservice_webhooks_enabled: true
matrix_appservice_webhooks_api_secret: <redacted>
matrix_appservice_webhooks_log_level: 'info'
matrix_mx_puppet_instagram_enabled: true
matrix_mx_puppet_steam_enabled: true
matrix_bot_matrix_reminder_bot_enabled: true
matrix_bot_matrix_reminder_bot_matrix_user_password: <redacted>
matrix_bot_matrix_reminder_bot_reminders_timezone: MST
matrix_s3_media_store_enabled: true
matrix_s3_media_store_bucket_name: "drwahl-matrix"
matrix_s3_media_store_aws_access_key: <redacted>
matrix_s3_media_store_aws_secret_key: <redacted>
matrix_s3_media_store_region: "us-east-1"
matrix_s3_media_store_custom_endpoint_enabled: true
matrix_s3_media_store_custom_endpoint: <redacted>
matrix_jitsi_enabled: true
matrix_jitsi_jicofo_auth_password: <redacted>
matrix_jitsi_jvb_auth_password: <redacted>
matrix_jitsi_jibri_recorder_password: <redacted>
matrix_jitsi_jibri_xmpp_password: <redacted>
matrix_jitsi_web_custom_config_extension: |
  config.enableLayerSuspension = true;
  config.disableAudioLevels = false;
  // Limit the number of video feeds forwarded to each client
  config.channelLastN = 4;
matrix_homeserver_generic_secret_key: "{{ matrix_synapse_macaroon_secret_key }}"
matrix_mautrix_facebook_enabled: true
matrix_mautrix_facebook_configuration_extension_yaml: |
  bridge:
    encryption:
      allow: true
      default: true
    community_template: "unofficial/facebook_{localpart}={server}"
matrix_synapse_configuration_extension_yaml: |
  enable_group_creation: true
  group_creation_prefix: "unofficial/"
matrix_etherpad_enabled: true
matrix_hookshot_enabled: true
matrix_hookshot_gitlab_enabled: true
matrix_hookshot_gitlab_instances:
  git.drwahl.me:
    url: https://git.drwahl.me
matrix_hookshot_gitlab_webhook_secret: <redacted>
matrix_synapse_log_level: "INFO"
matrix_synapse_storage_sql_log_level: "INFO"
matrix_synapse_root_log_level: "INFO"

Matrix Server:

• OS: Ubuntu 20.04
• Architecture: amd64

Problem description:

After deploying matrix with setup.yaml, the matrix-synapse service/container fails to start. When running with service matrix-synapse start, systemd continuously restarts the docker container with very little in the logs (simply saying it's starting, then says stopped, then starting again). If I copy the command out of the systemd service file and paste it into my shell, the container launches successfully works as expected, so this seems specific to systemd somehow.

Additional context This deployment was working for quite some time. At some point within the last 6 months or so, I started to notice this behaviour. My workaround has just been to run the launch command from a root shell, which launches the container and everything "just works" at that point.

[meri-leeworthy]

I'm having this problem too :(

[spantaleev]

Maybe you're running out of memory. Check dmesg.

[drwahl]

It doesn't seem to be memory pressure. dmesg doesn't show any OOMkiller messages, and when I do run synapse manually (and it stays running), I still have over 3GB of memory available.

I should also mention that setting logs to "INFO" doesn't seem to really turn the logging up any (at least from sysctemctl). I do see the config file (in /matrix/synapse/config/matrix.drwahl.me.log.config) shows "INFO", but the logs in the journal are nothing more than "Starting" then "Stopped".

[meri-leeworthy]

@drwahl would you please share how you manually start the service? I have tried looking at the systemd template and using the docker run command there but struggling with permissions on homeserver.yaml / not sure if this is how you were getting it to work

[aaronraimist]

@meri-leeworthy sudo systemctl start matrix-synapse

[drwahl]

@meri-leeworthy I simply did sudo cat /etc/systemd/system/matrix-synapse.service, and then copied the ExecStart command into my clipboard. Then I did sudo su - into root, and pasted the command. Once I verified it worked, I added the -d flag to daemonize it. This is how I'm currently running things, as doing it through systemctl isn't working for me.

I've also modified the systemctl files for the other services that depend on synapse so they won't fail when it isn't running. This way, I'm able to still use the systemctl service files for the all the other services and just manually run synapse.

[baloan]

I just had a message about a failure of synapse-dimension startup after upgrading the installation. In fact systemctl confirmed the process is up and running. Had to increase devture_systemd_service_manager_up_verification_delay_seconds from 15 to 25 seconds to get a clean startup.

skipping: [matrix.zntrl.de] => (item=matrix-synapse.service)
failed: [matrix.zntrl.de] (item=matrix-dimension.service) => changed=false
  ansible_loop_var: item
  item: matrix-dimension.service
  msg: matrix-dimension.service was not detected to be running. It's possible that there's a configuration problem or another service on your server interferes with it (uses the same ports, etc.). Try running `systemctl status matrix-dimension.service` and `journalctl -fu matrix-dimension.service` on the server to investigate. If you're on a slow or overloaded server, it may be that services take a longer time to start and that this error is a false-positive. You can consider raising the value of the `devture_systemd_service_manager_up_verification_delay_seconds` variable. See `/root/matrix-docker-ansible-deploy/roles/galaxy/com.devture.ansible.role.systemd_service_manager/defaults/main.yml` for more details about that.

root@relay:~# systemctl status matrix-dimension.service
● matrix-dimension.service - Matrix Dimension
   Loaded: loaded (/etc/systemd/system/matrix-dimension.service; enabled; vendor preset: enabled)
   Active: active (running) since Sat 2023-02-11 18:45:07 CET; 3min 58s ago
  Process: 9800 ExecStartPre=/usr/bin/env sh -c /usr/bin/env docker rm matrix-dimension 2>/dev/null || true (code=exited, status=0/SUCCESS)
  Process: 9760 ExecStartPre=/usr/bin/env sh -c /usr/bin/env docker kill matrix-dimension 2>/dev/null || true (code=exited, status=0/SUCCESS)
 Main PID: 9822 (docker)
    Tasks: 9 (limit: 4655)
   CGroup: /system.slice/matrix-dimension.service
           └─9822 docker run --rm --name matrix-dimension --log-driver=none --user=1000:1000 --cap-drop=ALL --network=traefik --mount type=bind,src=/matrix/dimension,dst=/data docker.io/turt2live/matrix-dimension:latest

Feb 11 18:48:43 relay matrix-dimension[9822]: Sat, 11 Feb 2023 17:48:43 GMT [DEBUG] [MatrixHttpClient (REQ-27)] GET http://matrix-nginx-proxy:12080/_matrix/client/r0/sync