Traefik fronted with nginx returns "404 page not found" for matrix.<my-domain>

[shade-belisar]

Describe the bug I followed this guide to setup my matrix server with nginx fronting for traefik. https://github.com/spantaleev/matrix-docker-ansible-deploy/blob/master/examples/nginx/README.md When accessing matrix.my-domain, I only get the text "404 page not found" (no actual 404 though). However, the web client on element.my-domain is delivered correctly, it just can't connect to the home server. Neither can my Element apps on Linux or Android.

To Reproduce My vars.yml file looks like this:

# The bare domain name which represents your Matrix identity.
# Matrix user ids for your server will be of the form (`@user:<matrix-domain>`).
#
# Note: this playbook does not touch the server referenced here.
# Installation happens on another server ("matrix.<matrix-domain>").
#
# If you've deployed using the wrong domain, you'll have to run the Uninstalling step,
# because you can't change the Domain after deployment.
#
# Example value: example.com
matrix_domain: <my-domain>

# The Matrix homeserver software to install.
# See `roles/matrix-base/defaults/main.yml` for valid options.
matrix_homeserver_implementation: synapse

# A secret used as a base, for generating various other secrets.
# You can put any string here, but generating a strong one is preferred (e.g. `pwgen -s 64 1`).
matrix_homeserver_generic_secret_key: '<secret-key>'

# This is something which is provided to Let's Encrypt when retrieving SSL certificates for domains.
#
# In case SSL renewal fails at some point, you'll also get an email notification there.
#
# If you decide to use another method for managing SSL certificates (different than the default Let's Encrypt),
# you won't be required to define this variable (see `docs/configuring-playbook-ssl-certificates.md`).
#
# Example value: [email protected]
matrix_ssl_lets_encrypt_support_email: '<my-email>'

# A Postgres password to use for the superuser Postgres user (called `matrix` by default).
#
# The playbook creates additional Postgres users and databases (one for each enabled service)
# using this superuser account.
devture_postgres_connection_password: '<postgres-password>'

matrix_nginx_proxy_enabled: false

matrix_ssl_retrieval_method: none

matrix_synapse_admin_enabled: true

matrix_registration_enabled: true

matrix_registration_admin_secret: "<admin-secret>"

matrix_coturn_turn_udp_max_port: 49200

# matrix_synapse_federation_enabled: false

matrix_ma1sd_enabled: true

matrix_synapse_auto_join_rooms: ["#main:<my-domain>"]

# Use traefik as the internal webserver for matrix
matrix_playbook_reverse_proxy_type: playbook-managed-traefik

# Ensure that public urls use https
matrix_playbook_ssl_enabled: true

# Disable the web-secure (port 443) endpoint, which also disables SSL certificate retrieval
devture_traefik_config_entrypoint_web_secure_enabled: false

# If your reverse-proxy runs on another machine, consider using `0.0.0.0:81`, just `81` or `SOME_IP_ADDRESS_OF_THIS_MACHINE:81`
devture_traefik_container_web_host_bind_port: '127.0.0.1:81'

devture_traefik_config_log_level: DEBUG

# We bind to `127.0.0.1` by default (see above), so trusting `X-Forwarded-*` headers from
# a reverse-proxy running on the local machine is safe enough.
devture_traefik_config_entrypoint_web_forwardedHeaders_insecure: true

devture_traefik_additional_entrypoints_auto:
  - name: matrix-federation
    port: 8449
    host_bind_port: '127.0.0.1:8449'
    config: {}

My nginx config looks like this:

server {
	listen 8448 ssl http2 default_server;
	listen [::]:8448 ssl http2 default_server;

	server_name matrix.bielefeldt.berlin;

	location / {
		proxy_pass http://localhost:8449;
		proxy_set_header X-Forwarded-For $remote_addr;
		proxy_set_header X-Forwarded-Proto $scheme;
		proxy_set_header Host $host;

		access_log /var/log/nginx/matrix.access.log;
		error_log /var/log/nginx/matrix.error.log;

		client_max_body_size 50M;
	}

	ssl_certificate /etc/letsencrypt/live/bielefeldt.berlin/fullchain.pem; # managed by Certbot
	ssl_certificate_key /etc/letsencrypt/live/bielefeldt.berlin/privkey.pem; # managed by Certbot
	include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
	ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}

server {
    if ($host = element.bielefeldt.berlin) {
        return 301 https://$host$request_uri;
    } # managed by Certbot


    if ($host = matrix.bielefeldt.berlin) {
        return 301 https://$host$request_uri;
    } # managed by Certbot


	listen 80;
	listen [::]:80;

	server_name matrix.bielefeldt.berlin element.bielefeldt.berlin;
    return 404; # managed by Certbot
}

I install matrix and setup nginx according to the guide, then try to access matrix.my-domain

Expected behavior I would expect the home server to be accessible at the matrix subdomain.

Matrix Server:

• OS: Debian GNU/Linux 11 (bullseye)
• Architecture amd64

[lukasnoebauer]

I have exactly the same problem.

[shade-belisar]

As a workaround I disabled Traefik and configured nginx directly (as described here), but since that is not future-proof I'd like to solve the original problem.

[NablaMetal]

I think I'm facing the same issue (#2651) I tried the "direct" solution but I don't understand well how the different containers interact with each other. There is a lot of networks and stuffs (matrix-synapse-reverse-proxy-companion ?).

If I just put a redirection to the synapse container on 8008 I'm able to load the "It works! Synapse is running" page. But what about element ? and coturn ?

My matrix.conf :

server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;
    server_name matrix.* element.*;
    include /config/nginx/ssl.conf; 

    location / {
        include /config/nginx/proxy.conf;
        include /config/nginx/resolver.conf;

        proxy_pass http://matrix-synapse:8008;
        proxy_set_header X-Forwarded-For $remote_addr;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        client_max_body_size 50M;
    }
}

@Adrian-Bielefeldt could you kindly post your matrix.conf ?

[spantaleev]

That's why you should leave Traefik in place in local-only mode. It will reverse-proxy to the correct container. You only need to bother with reverse-proxying to Traefik itself and can forget about all other containers, networks, etc.

[NablaMetal]

Correct that was my initial goal, the documentation is clear about that. But because of #2651 and because I don't know much about traefik, I am having a hard time finding the answer. I looked at traefik logs, but nothing suspicious except that error line (dynamic.HTTPConfiguration) which is more a warning than an error. The request managed by nginx front which is sent to traefik does not seem to trigger any log in traefik container.

[ganyuke]

I believe it has to do with matrix_nginx_proxy_enabled being set to false. I managed to fix mine after commenting that line out in vars.yml.

[brizee]

Yeah I had set matrix_nginx_proxy_enabled to false as it was still creating an nginx docker container which was binding to my port before the traefik one and preventing traefik initialising.

I was curious why element. works but matrix. does not, and found these lines:

matrix_client_element_container_network: "{{ matrix_nginx_proxy_container_network if matrix_playbook_reverse_proxy_type == 'playbook-managed-nginx' else 'matrix-client-element' }}"
matrix_client_element_container_additional_networks: "{{ [matrix_playbook_reverse_proxyable_services_additional_network] if matrix_playbook_reverse_proxyable_services_additional_network else [] }}"

in https://github.com/spantaleev/matrix-docker-ansible-deploy/blob/14f7eed9324b58f4acb264f0cab3b15bfd10ac07/group_vars/matrix_servers#L3313

It seems if you don't use nginx then it drops the correct docker network from being the primary network for the container but will include it in the additional networks, the equivalent logic for matrix_synapse_container_network and matrix_synapse_container_additional_networks don't include this addition, so I modified this line, in my local fork: https://github.com/spantaleev/matrix-docker-ansible-deploy/blob/14f7eed9324b58f4acb264f0cab3b15bfd10ac07/roles/custom/matrix-synapse/defaults/main.yml#L122

to be this, in my local fork

matrix_synapse_container_additional_networks: "{{ [matrix_playbook_reverse_proxyable_services_additional_network] if matrix_playbook_reverse_proxyable_services_additional_network else [] }}"

and this seems to fix it without falling back to nginx - though I'll concede I don't understand too much of the various config techs at work here.

[Amcio]

I encountered the same issue, however none of the above fixed it. element.domain.com works fine, matrix.domain.com returns 404, for all endpoints. Both .well_known and /_matrix return 404.

traefik can connect to matrix-container-socket-proxy and matrix-synapse-reverse-proxy-companion I'm not sure what else traefik requires for operation as I admit I do not have a lot of experience with it.

I will attach any additional information required, if anyone has some advice what to do next

After rebooting the server, I noticed that matrix-traefik has no way of talking to matrix-synapse-reverse-proxy-companion, I have no idea how these are supposed to communicate

[parisni]

I had the same goal / 404 troubles and yes, commenting all matrix_nginx_proxy_* magically fixed it. BTW that's confusing for me why we need to enable a nginx proxy when we rely on traefik. Any explanation apreciated.

NOTE: i wanted traefik to allow elementx to work with sliding, and I can confirm it works, even if proxied from a user managed nginx -> matrix-managed-traefik

[IF-Adin]

I have the same problem. Does anyone have more insight about this?

[IF-Adin]

@spantaleev I hate to bother you with this, because you are already doing a lot, but is there any chance you could explain this? I would like to follow your recommendation and switch to Traefik, but this bug is not letting me.

Could you please explain what you meant by your earlier comment?

[spantaleev]

I'm not sure I follow. What is your configuration like?

[IF-Adin]

matrix_domain: --
matrix_homeserver_implementation: synapse
matrix_synapse_federation_enabled: true
matrix_synapse_allow_public_rooms_over_federation: false
jitsi_enabled: true
etherpad_enabled: true
matrix_bot_matrix_reminder_bot_enabled: true
matrix_bot_matrix_reminder_bot_matrix_user_password: --
matrix_bot_matrix_reminder_bot_reminders_timezone: --
jitsi_jicofo_component_secret: "--"
jitsi_jicofo_auth_password: "--"
jitsi_jvb_auth_password: "--"
jitsi_jibri_recorder_password: "--"
jitsi_jibri_xmpp_password: "--"
matrix_dimension_access_token: "--"
matrix_dimension_enabled: true
matrix_dimension_admins:
- "@--:{{ matrix_domain }}"
integrations_ui_url: https://dimension.--.eu/element
integrations_rest_url: https://dimension.--.eu/api/v1/scalar
integrations_widgets_urls: [ "https://dimension.--.eu/_matrix/integrations/v1",
    "https://dimension.--.eu/api",
    "https://dimension.--.eu/scalar/api",
    "https://dimension.--.eu/widgets"
]
integrations_jitsi_widget_url: https://dimension.--.eu/widgets/jitsi
devture_postgres_connection_password: '--'

matrix_synapse_container_client_api_host_bind_port: '--:8008'
matrix_synapse_container_federation_api_plain_host_bind_port: '--:8048'
matrix_synapse_container_metrics_api_host_bind_port: '--:9100'
matrix_client_element_container_http_host_bind_port: '--:8765'
matrix_dimension_container_http_host_bind_port: '--:8184'
matrix_corporal_container_http_gateway_host_bind_port: '--:41080'
matrix_corporal_container_http_api_host_bind_port: '--:41081'
matrix_appservice_irc_container_http_host_bind_port: '--:9999'
matrix_appservice_discord_container_http_host_bind_port: '--:9005'
matrix_nginx_proxy_container_http_host_bind_port: '--:80'
matrix_nginx_proxy_container_https_host_bind_port: '--:443'
matrix_nginx_proxy_container_federation_host_bind_port: '--:8448'

matrix_ssl_domains_to_obtain_certificates_for:
- '{{ matrix_server_fqn_matrix }}'
- '{{ matrix_server_fqn_element }}'
- '{{ matrix_server_fqn_etherpad }}'
- '{{ matrix_server_fqn_jitsi }}'
- '{{ matrix_server_fqn_grafana }}'
- '{{ matrix_server_fqn_dimension }}'
- '{{ matrix_domain }}'
matrix_bot_maubot_enabled: true
matrix_bot_maubot_admins:
- admin: --
matrix_synapse_max_upload_size_mb: 1000
matrix_synapse_media_retention_local_media_lifetime: 90d
matrix_synapse_media_retention_remote_media_lifetime: 90d
matrix_synapse_metrics_enabled: false
matrix_synapse_metrics_port: 9092
matrix_playbook_reverse_proxy_type: playbook-managed-nginx
devture_traefik_config_certificatesResolvers_acme_email: --
matrix_ssl_lets_encrypt_support_email: --
matrix_nginx_proxy_base_domain_serving_enabled: true
matrix_nginx_proxy_base_domain_homepage_enabled: false
matrix_coturn_turn_static_auth_secret: "--"
matrix_synapse_macaroon_secret_key: "--"
matrix_mautrix_telegram_enabled: true
matrix_mautrix_telegram_api_id: --
matrix_mautrix_telegram_api_hash: --
matrix_mx_puppet_discord_enabled: false
matrix_mx_puppet_discord_client_id: "--"
matrix_mx_puppet_discord_client_secret: "--"
matrix_synapse_admin_enabled: true
matrix_registration_enabled: true
matrix_registration_admin_secret: "--"
devture_postgres_backup_enabled: true
devture_postgres_backup_data_path: '--'
devture_postgres_backup_schedule: '@daily'
devture_postgres_backup_keep_days: 2
devture_postgres_backup_keep_weeks: 0
devture_postgres_backup_keep_months: 0
devture_postgres_process_extra_arguments: [
    "-c max_connections=100",
    "-c shared_buffers=2GB",
    "-c effective_cache_size=6GB",
    "-c maintenance_work_mem=512MB",
    "-c checkpoint_completion_target=0.9",
    "-c wal_buffers=16MB",
    "-c default_statistics_target=100",
    "-c random_page_cost=1.1",
    "-c effective_io_concurrency=200",
    "-c work_mem=5242kB",
    "-c min_wal_size=1GB",
    "-c max_wal_size=4GB",
    "-c max_worker_processes=4",
    "-c max_parallel_workers_per_gather=2",
    "-c max_parallel_workers=4",
    "-c max_parallel_maintenance_workers=2",
]
prometheus_enabled: false
prometheus_node_exporter_enabled: false
grafana_enabled: false
grafana_anonymous_access: false
grafana_default_admin_user: "admin"
grafana_default_admin_password: "--"
matrix_homeserver_generic_secret_key: "--"
matrix_client_element_themes_enabled: true

Currently, like this. If i switch it to traefik, i get the same error the op get @spantaleev

[IF-Adin]

@shade-belisar Were you ever able to fix this?

[Marx1st]

Same here. .well-known/matrix just gives 404 when proxied from our main domain. We could only get it to work by disabling traefik...

[IF-Adin]

@Marx1st Yeah no matter what i try, it always does this. It seems to be unable to communicate with synapse.

[shade-belisar]

@IF-Adin Unfortunately not. My matrix server is still running under a directly configured nginx.

[WowSuchRicky]

I get the 404 too, but I did realize that my /matrix/static_files directory on the matrix host itself is totally empty so in my case it may be a legitimate 404. I am currently trying to dig into why the well_known files don't get generated and if I find anything that is related to the issue I will report back.

[IF-Adin]

@WowSuchRicky Thank you so much!

[WowSuchRicky]

@IF-Adin I don't believe my issue was related unfortunately but it was a combo of a few different things, can still share what I did in case your setup is similar.

• I run SWAG-nginx as my actual frontend for my base domain. I have a proxy config in that to forward matrix.* and element.* to the managed Traefik instance that gets set up with this ansible playbook.
• I had noticed that my static_files directory was totally empty, which was also leading me to get a 404 upon trying to load those files since they were indeed missing. I had to add the var: matrix_well_known_matrix_server_enabled: true into my vars.yml and it did appear to re-generate the well-known dir for me.
• Issue solved.

For us to help you with yours more though, have you verified that you can reach traefik directly (i.e. curl-ing or navigating to localhost:8449 in your case) to make sure it works? See anything interesting in the nginx or traefik access/error logs that might show what's going on?

Hope you can get your issue resolved soon, would be happy to try and rubber duck it either here or via matrix (@r:ricky.sh).

[IF-Adin]

@WowSuchRicky Yeah Traefik seemed to be running just fine, i was able to reach it and the frontend that comes with it and it did not complain about anything.

I was not able to reach matrix. on my domain, but i could reach element.

[Happyfeet01]

Hello,

have exactly the same Problem with an external Nginx Proxy

[spantaleev]

People hitting this problem are liking not having their other reverse-proxy set the appropriate Host header, so Traefik cannot route the request correctly and ultimately returns "404 not found".

Our Fronting the integrated reverse-proxy webserver with another reverse-proxy documentation section has been updated to mention this.

[IF-Adin]

@spantaleev This isn't quite my problem, sadly.

I get the 404 if i simply configure the playbook to use traefik as the reverse proxy, i do not wish to do anything else. I am current still running nginx as my reverse proxy, but as per your instructions, i tried to switch to traefik for the new syncing.

[IF-Adin]

@spantaleev After the latest update, this issue was resolved for me.