Enabling TLS with Let's Encrypt yields "502 Bad Gateway"

[caugner]

I have followed the Production Docker Deployment guide and the Configuring TLS on Mattermost Server instructions afterwards.

However, once I configure Listen Address, Connection Security, Forward port 80 to 443 and Use Let’s Encrypt and restart Mattermost (docker-compose restart), accessing the instance via HTTP/80 yields "502 Bad Gateway" and I cannot connect via HTTPS/443 either.

Should this normally work or is this a known limitation with the Docker setup?

[caugner]

Sorry, it looks like the port HTTPS/443 was not properly forwarded (and already used by another application).

[caugner]

Now I have the same issue on a cloud server (before I was using an internal VM with port forwarding from the router).

It looks like Mattermost just doesn't listen on Port 443, even though I configured it accordingly.

[caugner]

I wonder whether these lines really have to be uncommented, since it isn't mentioned in the Production Docker Deployment guide: https://github.com/mattermost/mattermost-docker/blob/492ecd5ca3c761e309834ba0389dd13e7bc431d4/docker-compose.yml#L62-L64

[potaito]

I'm running into the same issue.

How to reproduce:

1. Follow the guide

   sudo apt-get install git
   git clone https://github.com/mattermost/mattermost-docker.git
   cd mattermost-docker
   docker-compose build
   mkdir -pv ./volumes/app/mattermost/{data,logs,config,plugins,client-plugins}
   sudo chown -R 2000:2000 ./volumes/app/mattermost/
   docker-compose up -d
   

2. Create user and login to system console

3. Under Environment -> Web Server, make the following changes

   • Change listen address from 8000 to 443
   • Forward port 80 to 443: true
   • Connection Security: TLS
   • Use Let's Encrypt: true

Restart with docker-compose, and then when connecting I get 502 Bad Gateway.

The problem

502 Bad Gateway happens, because the nginx proxy is not aware of the SSL certificate configured inside the mattermost app container. When using the standard docker-compose.yaml, it loads the default web server. That server is just a nginx docker container, which loads one of these two configs on startup:

1. /etc/nginx/sites-available/mattermost
2. /etc/nginx/sites-available/mattermost-ssl

Looking at the web container's entrypoint, we can see that the SSL nginx config is only loaded when the certificate and private key are found in the right place: https://github.com/mattermost/mattermost-docker/blob/1427315ef927914e0ee03fe230cc4a1b045ebdb7/web/entrypoint.sh#L7-L13

The bad gateway error is a result of changing the listen port in the web UI of the app without reconfiguring nginx, which will point to port :8000 no matter what:

https://github.com/mattermost/mattermost-docker/blob/1427315ef927914e0ee03fe230cc4a1b045ebdb7/web/entrypoint.sh#L5

Solution

Still looking into this. But I guess there are two options:

1. Use the mattermost web UI to configure SSL + letsencrypt, but then add APP_PORT_NUMBER to the docker-compose file under web, such that the web proxy points at port 443 instead of 8000.
2. Don't use the mattermost web UI to configure SSL. Instead, do it with the nginx config files and by putting the certificates in the right place, i.e.

cp /etc/letsencrypt/live/<site>/fullchain.pem ./volumes/web/cert/cert.pem
cp /etc/letsencrypt/live/<site>/privkey.pem ./volumes/web/cert/key-no-password.pem

as described here: https://github.com/mattermost/mattermost-docker#install-with-ssl-certificate

I wonder whether these lines really have to be uncommented, since it isn't mentioned in the Production Docker Deployment guide:

https://github.com/mattermost/mattermost-docker/blob/492ecd5ca3c761e309834ba0389dd13e7bc431d4/docker-compose.yml#L62-L64

As far as I can tell this environment variable is not used anywhere and is probably a leftover from the past.

Update: SSL works without issues when copying the certificate and private key to ./volumes/web/cert/ and then making sure, that the symlink is actually applied (https://github.com/mattermost/mattermost-docker/pull/454). No SSL configuration through the mattermost web UI was necessary.

[caugner]

SSL works without issues when copying the certificate and private key to ./volumes/web/cert/

@potaito Where do the certificate and private key come from? And how can the certifcate be automatically renewed if I have to copy it manually?

[potaito]

@caugner I ran the letsencrypt certbot outside of docker and set it up on the host, then manually copied the certificates

cp /etc/letsencrypt/live/mysite/fullchain.pem ./volumes/web/cert/cert.pem
cp /etc/letsencrypt/live/mysite/privkey.pem ./volumes/web/cert/key-no-password.pem

Now I'm going to either renew the certificate automatically on the host and copy the new certificates with the cronjob, or setup another docker container that fetches and copies the letsencrypt certificate. I think the second option is the cleanest, since it allows to configure the letsencrypt task outside of mattermost (or any other app in that regard) and is therefore independent of the application. But at the same time the settings can be version-controlled easily by committing the modified docker-compose config file.

[caugner]

@potaito In general, I would expect Mattermost to offer automatic renewal of LE certificates out of the box.

Specifically, I don't see how the LE certificate could be renewed without stopping Mattermost temporarily, since the ACME request would normally be passed to Mattermost, except if we stop it and allow certbot to start a webserver on the port usually used by Mattermost.

[potaito]

@caugner LE certbot can be run with --webroot Place files in a server's webroot folder for authentication which should allow to keep the mattermost application running, though I have not tried it yet. With --webroot no additional web server is started for the verification.

[caugner]

@potaito The webroot is inside the web container, so I doubt that certbot can place the files there.

[potaito]

@caugner Can't we mount webroot as read-only in the nginx container, and mount the same webroot as read-write in the letsencrypt container?

[caugner]

@potaito You're right, that should work.

But I guess we can agree that it would be better if LE worked without any workaround like this?

Maybe @ventz or @cpanato can tell if LE is supposed to work out of the box?

[potaito]

Did you try the first possible solution I mentioned?

Solution

Still looking into this. But I guess there are two options:

1. Use the mattermost web UI to configure SSL + letsencrypt, but then add `APP_PORT_NUMBER` to the docker-compose file under `web`, such that the web proxy points at port `443` instead of `8000`.

2. Don't use the mattermost web UI to configure SSL. Instead, do it with the nginx config files and by putting the certificates in the right place, i.e.

I'm pretty sure LE works inside mattermost as well, but the web proxy is not automatically modified (changing the port) since it lives outside the mattermost application.

EDIT: I tried, but did not manage to get it to work with the internal tools.

[teodorkasap]

hi @potaito , I am struggling with the second option. after copying the cert files into the web/cert folder. I start all containers but when I try to connect, the connection times out... is there any config I need to change in docker-compose.yml file?

without certificates in the mentioned folder all works fine.. I am using 5.21 version BTW thank you in advance for your help

[potaito]

Hey @teodorkasap

1. Did you copy the key and the chain to the right place?
2. Did you copy the correct files?

   cp /etc/letsencrypt/live/<site>/fullchain.pem ./volumes/web/cert/cert.pem
   cp /etc/letsencrypt/live/<site>/privkey.pem ./volumes/web/cert/key-no-password.pem
   

3. Did you use a passphrase when creating the ssl key? The configuration expects no password, else it won't work.

The way SSL works in this setup is that upon starting with docker-compose, the entrypoint checks if the certificate exists:

https://github.com/mattermost/mattermost-docker/blob/7fd2eb47a91a975a855270c0ace8efadb7cea0fc/web/entrypoint.sh#L7-L12

If the files /cert/cert.pem and /cert/key-no-password.pem exist, the ssl config for NGINX is automatically loaded. Your terminal should print somehwere

found certificate and key, linking ssl config

it this step was succesful, or

linking plain config

if the plain configuration without SSL is loaded for nginx.

[teodorkasap]

Thank you for the speedy response. I've copied files to the right folder. However I don't see either output in the terminal when I do docker-compose up - d

Without certificate files in the mentioned folder therefore without encryption it works fine though I don't see the line printed in terminal as you mentioned.

On Fri, Apr 17, 2020, 18:10 alessandro [email protected] wrote:

Hey @teodorkasap https://github.com/teodorkasap

1. Did you copy the key and the chain to the right place?
2. Did you copy the correct files?

cp /etc/letsencrypt/live//fullchain.pem ./volumes/web/cert/cert.pem cp /etc/letsencrypt/live//privkey.pem ./volumes/web/cert/key-no-password.pem

3. Did you use a passphrase when creating the ssl key? The configuration expects no password, else it won't work.

The way SSL works in this setup is that upon starting with docker-compose, the entrypoint checks if the certificate exists:

https://github.com/mattermost/mattermost-docker/blob/7fd2eb47a91a975a855270c0ace8efadb7cea0fc/web/entrypoint.sh#L7-L12

If the files /cert/cert.pem and /cert/key-no-password.pem exist, the ssl config for NGINX is automatically loaded. Your terminal should print somehwere

"found certificate and key, linking ssl config"

it this step was succesful, or

"linking plain config"

if the plain configuration without SSL is loaded for nginx.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/mattermost/mattermost-docker/issues/450#issuecomment-615299352, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB4XXBOMBYNLRRCMGZVNRGTRNBWPRANCNFSM4KX5J6HQ .

[teodorkasap]

I've followed these steps exactly https://forum.mattermost.org/t/mattermost-recipe-using-lets-encrypt-for-tls-certificates-with-mattermost-docker/7596

[potaito]

However I don't see either output in the terminal when I do docker-compose up - d

Use docker compose up without the -d --detach option. Otherwise you won't see the output. You can shut it down with CTRL+c later.

[caugner]

Or just type: docker compose logs -f

[teodorkasap]

I am a noob when it comes to docker as you can tell :) Anyway ... I have done as you said. The output is below according to these. actually web container can see the certificates and do what is necessary. I can see the requests, my phone app (android 9, oneplus A3003) is making. However I can neither connect through this app nor from desktop browser...

Starting mattermost-docker_app_1 ... done
Starting mattermost-docker_db_1  ... done
Starting mattermost-docker_web_1 ... done
Attaching to mattermost-docker_db_1, mattermost-docker_web_1, mattermost-docker_app_1
app_1  | Using existing config file /mattermost/config/config.json
app_1  | Using existing database connection
db_1   | AWS_ACCESS_KEY_ID is required for Wal-E but not set. Skipping Wal-E setup.
db_1   | AWS_SECRET_ACCESS_KEY is required for Wal-E but not set. Skipping Wal-E setup.
db_1   | WALE_S3_PREFIX is required for Wal-E but not set. Skipping Wal-E setup.
db_1   | AWS_REGION is required for Wal-E but not set. Skipping Wal-E setup.
db_1   | 
db_1   | PostgreSQL Database directory appears to contain a database; Skipping initialization
db_1   | 
db_1   | LOG:  database system was interrupted; last known up at 2020-04-19 10:16:07 UTC
web_1  | found certificate and key, linking ssl config
db_1   | LOG:  database system was not properly shut down; automatic recovery in progress
db_1   | LOG:  redo starts at 0/1D55040
db_1   | LOG:  record with zero length at 0/1D58A58
db_1   | LOG:  redo done at 0/1D57468
db_1   | LOG:  last completed transaction was at log time 2020-04-19 10:16:11.87017+00
db_1   | LOG:  MultiXact member wraparound protections are now enabled
db_1   | LOG:  database system is ready to accept connections
db_1   | LOG:  autovacuum launcher started
app_1  | Starting mattermost
db_1   | ERROR:  relation "idx_teams_description" does not exist
db_1   | STATEMENT:  SELECT $1::regclass
db_1   | ERROR:  duplicate key value violates unique constraint "users_username_key"
db_1   | DETAIL:  Key (username)=(surveybot) already exists.
db_1   | STATEMENT:  insert into "users" ("id","createat","updateat","deleteat","username","password","authdata","authservice","email","emailverified","nickname","firstname","lastname","position","roles","allowmarketing","props","notifyprops","lastpasswordupdate","lastpictureupdate","failedattempts","locale","timezone","mfaactive","mfasecret") values ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16,$17,$18,$19,$20,$21,$22,$23,$24,$25);
web_1  | 176.40.251.220 - - [19/Apr/2020:13:19:58 +0300] "GET /api/v4/system/ping?time=1587291597550 HTTP/1.1" 301 162 "-" "okhttp/3.13.1" "-"
web_1  | 127.0.0.1 - - [19/Apr/2020:13:19:59 +0300] "GET / HTTP/1.1" 301 162 "-" "curl/7.66.0" "-"
web_1  | 176.40.251.220 - - [19/Apr/2020:13:19:59 +0300] "POST /api/v4/channels/members/me/view HTTP/1.1" 301 162 "-" "Mozilla/5.0 (Linux; Android 9; ONEPLUS A3003 Build/PKQ1.181203.001; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/80.0.3987.162 Mobile Safari/537.36" "-"
web_1  | 176.40.251.220 - - [19/Apr/2020:13:20:01 +0300] "GET /api/v4/system/ping?time=1587291597550 HTTP/1.1" 301 162 "-" "okhttp/3.13.1" "-"
web_1  | 176.40.251.220 - - [19/Apr/2020:13:20:10 +0300] "GET /api/v4/system/ping?time=1587291609089 HTTP/1.1" 301 162 "-" "okhttp/3.13.1" "-"
web_1  | 176.40.251.220 - - [19/Apr/2020:13:20:13 +0300] "GET /api/v4/system/ping?time=1587291609089 HTTP/1.1" 301 162 "-" "okhttp/3.13.1" "-"
web_1  | 176.40.251.220 - - [19/Apr/2020:13:20:21 +0300] "GET /api/v4/system/ping?time=1587291620415 HTTP/1.1" 301 162 "-" "okhttp/3.13.1" "-"
web_1  | 176.40.251.220 - - [19/Apr/2020:13:20:24 +0300] "GET /api/v4/system/ping?time=1587291620415 HTTP/1.1" 301 162 "-" "okhttp/3.13.1" "-"
web_1  | 127.0.0.1 - - [19/Apr/2020:13:20:30 +0300] "GET / HTTP/1.1" 301 162 "-" "curl/7.66.0" "-"
web_1  | 176.40.251.220 - - [19/Apr/2020:13:20:32 +0300] "GET /api/v4/system/ping?time=1587291631681 HTTP/1.1" 301 162 "-" "okhttp/3.13.1" "-"
web_1  | 176.40.251.220 - - [19/Apr/2020:13:20:35 +0300] "GET /api/v4/system/ping?time=1587291631681 HTTP/1.1" 301 162 "-" "okhttp/3.13.1" "-"
web_1  | 176.40.251.220 - - [19/Apr/2020:13:20:44 +0300] "GET /api/v4/system/ping?time=1587291642963 HTTP/1.1" 301 162 "-" "okhttp/3.13.1" "-"
web_1  | 176.40.251.220 - - [19/Apr/2020:13:20:46 +0300] "POST /api/v4/users/logout HTTP/1.1" 301 162 "-" "Mozilla/5.0 (Linux; Android 9; ONEPLUS A3003 Build/PKQ1.181203.001; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/80.0.3987.162 Mobile Safari/537.36" "-"
web_1  | 176.40.251.220 - - [19/Apr/2020:13:20:47 +0300] "GET /api/v4/system/ping?time=1587291642963 HTTP/1.1" 301 162 "-" "okhttp/3.13.1" "-"
web_1  | 176.40.251.220 - - [19/Apr/2020:13:20:51 +0300] "POST /api/v4/users/logout HTTP/1.1" 301 162 "-" "Mozilla/5.0 (Linux; Android 9; ONEPLUS A3003 Build/PKQ1.181203.001; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/80.0.3987.162 Mobile Safari/537.36" "-"
web_1  | 176.40.251.220 - - [19/Apr/2020:13:20:58 +0300] "GET /api/v4/system/ping?time=1587291657288 HTTP/1.1" 301 162 "-" "Mozilla/5.0 (Linux; Android 9; ONEPLUS A3003 Build/PKQ1.181203.001; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/80.0.3987.162 Mobile Safari/537.36" "-"
web_1  | 127.0.0.1 - - [19/Apr/2020:13:21:01 +0300] "GET / HTTP/1.1" 301 162 "-" "curl/7.66.0" "-"
web_1  | 176.40.251.220 - - [19/Apr/2020:13:21:03 +0300] "GET /api/v4/system/ping?time=1587291657288 HTTP/1.1" 301 162 "-" "Mozilla/5.0 (Linux; Android 9; ONEPLUS A3003 Build/PKQ1.181203.001; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/80.0.3987.162 Mobile Safari/537.36" "-"
web_1  | 176.40.251.220 - - [19/Apr/2020:13:21:06 +0300] "GET /api/v4/system/ping?time=1587291665346 HTTP/1.1" 301 162 "-" "okhttp/3.13.1" "-"
web_1  | 176.40.251.220 - - [19/Apr/2020:13:21:09 +0300] "GET /api/v4/system/ping?time=1587291665346 HTTP/1.1" 301 162 "-" "okhttp/3.13.1" "-"
web_1  | 176.40.251.220 - - [19/Apr/2020:13:21:17 +0300] "GET /api/v4/system/ping?time=1587291676531 HTTP/1.1" 301 162 "-" "okhttp/3.13.1" "-"
web_1  | 176.40.251.220 - - [19/Apr/2020:13:21:20 +0300] "GET /api/v4/system/ping?time=1587291676531 HTTP/1.1" 301 162 "-" "okhttp/3.13.1" "-"
web_1  | 127.0.0.1 - - [19/Apr/2020:13:21:31 +0300] "GET / HTTP/1.1" 301 162 "-" "curl/7.66.0" "-"

[teodorkasap]

guys I am very embarrased. our admin forgot to enable the 443 port this was the reason it did not work. Sorry I wasted your time. Thank you all for your time.

[potaito]

@teodorkasap no worries :smile: