easyengine
easyengine copied to clipboard
EasyEngine
Site down after updating to v4.0.12
After running ee cli update to upgrade from version 4.0.10 to 4.0.12 the site is no longer accessible. No response from the server to either the url https://dev.brightray.com/ or the direct IP address. Both the URL and IP address respond to ping but the web service doesn't seem to be responding.
No visible errors during the upgrade. It came back saying successfully upgraded. After a reboot the server came back up without issue but no response to web requests. I can telnet into the system, attempte to restart the services, watched the logs. No messages show up on the logs when trying to access either the site or the direct IP address.
System Information
- [x] ee cli info
+-------------------+----------------------------------------------------------+
| OS | Linux 4.15.0-47-generic #50-Ubuntu SMP Wed Mar 13 10:44: |
| | 52 UTC 2019 x86_64 |
| Shell | /bin/bash |
| PHP binary | /usr/bin/php7.2 |
| PHP version | 7.2.17-1+ubuntu18.04.1+deb.sury.org+3 |
| php.ini used | /etc/php/7.2/cli/php.ini |
| EE root dir | phar://ee.phar |
| EE vendor dir | phar://ee.phar/vendor |
| EE phar path | /root |
| EE packages dir | |
| EE global config | /opt/easyengine/config/config.yml |
| EE project config | |
| EE version | 4.0.12 |
+-------------------+----------------------------------------------------------+
- [x] lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.2 LTS
Release: 18.04
Codename: bionic
- [x] docker version
Client:
Version: 18.09.4
API version: 1.39
Go version: go1.10.8
Git commit: d14af54266
Built: Wed Mar 27 18:35:44 2019
OS/Arch: linux/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 18.09.4
API version: 1.39 (minimum version 1.12)
Go version: go1.10.8
Git commit: d14af54
Built: Wed Mar 27 18:01:48 2019
OS/Arch: linux/amd64
Experimental: false
- [x] docker-compose version
docker-compose version 1.23.2, build 1110ad01
docker-py version: 3.6.0
CPython version: 3.6.7
OpenSSL version: OpenSSL 1.1.0f 25 May 2017
It looks like the easyengine/nginx-proxy:v4.0.2 container keeps restarting every min after doing the update.
I checked the docker logs. This is the error the log is reporting.
forego | sending SIGTERM to nginx.1
forego | sending SIGTERM to dockergen.1
Custom dhparam.pem file found, generation skipped
forego | starting dockergen.1 on port 5000
forego | starting nginx.1 on port 5100
nginx.1 | 2019/04/12 14:42:54 [emerg] 25#25: PEM_read_bio_DHparams("/etc/nginx/dhparam/dhparam.pem") failed (SSL: error:0906D06C:PEM routines:PEM_read_bio:no start line:Expecting: DH PARAMETERS)
nginx.1 | nginx: [emerg] PEM_read_bio_DHparams("/etc/nginx/dhparam/dhparam.pem") failed (SSL: error:0906D06C:PEM routines:PEM_read_bio:no start line:Expecting: DH PARAMETERS)
forego | starting nginx.1 on port 5200
forego | sending SIGTERM to nginx.1
forego | sending SIGTERM to dockergen.1
Because the error made a reference to an SSL error I attempted to renew my cert and get the below error for each of the commands I attempted.
ee site ssl dev.brightray.com
Starting SSL verification.
Warning: Failed to verify SSL: cURL error 7: Failed to connect to dev.brightray.com port 80: Connection refused (see http://curl.haxx.se/libcurl/c/libcurl-errors.html)
Warning: Check logs and retry `ee site ssl dev.brightray.com` once the issue is resolved.
ee site ssl-renew dev.brightray.com
Starting SSL cert renewal
Starting SSL verification.
PHP Fatal error: Uncaught GuzzleHttp\Exception\ConnectException: cURL error 7: Failed to connect to dev.brightray.com port 80: Connection refused (see http://curl.haxx.se/libcurl/c/libcurl-errors.html) in phar:///usr/local/bin/ee/vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php:185
Stack trace:
#0 phar:///usr/local/bin/ee/vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php(149): GuzzleHttp\Handler\CurlFactory::createRejection(Object(GuzzleHttp\Handler\EasyHandle), Array)
#1 phar:///usr/local/bin/ee/vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php(102): GuzzleHttp\Handler\CurlFactory::finishError(Object(GuzzleHttp\Handler\CurlHandler), Object(GuzzleHttp\Handler\EasyHandle), Object(GuzzleHttp\Handler\CurlFactory))
#2 phar:///usr/local/bin/ee/vendor/guzzlehttp/guzzle/src/Handler/CurlHandler.php(43): GuzzleHttp\Handler\CurlFactory::finish(Object(GuzzleHttp\Handler\CurlHandler), Object(GuzzleHttp\Handler\EasyHandle), Object(GuzzleHttp\Handler\CurlFactory))
#3 phar:///usr/local/bin/ee/vendor/guzzlehttp/guzzle/src/Handler/P in phar:///usr/local/bin/ee/vendor/guzzlehttp/guzzle/src/Handler/CurlFactory.php on line 185
Warning: An Error occurred. Initiating clean-up.
Warning: Exiting gracefully after rolling back. This may take some time.
Success: Rollback complete. Exiting now.
I was looking online for the error I posted above and came across an issue posted to the nginx-proxy repository. Issue with recent container update and SSL #1226
In case anyone is looking a solution (cross post from the EasyEngine Community)
in /opt/easyengine/services there is a docker-compose.yml where I was able to change the nginx-proxy version from 4.0.3 to 4.1.2. . Bring the EE services down, rebuild the containers (fetch the prebuilt images), bring the containers up and you should be good.
cd /opt/easyengine/services
docker-compose stop
vim docker-compose.yml #change the nginx-proxy version number to 4.1.2
docker-compose build
docker-compose up
# Still wasn't working for me so I restarted the server in case there was a magic order for EE to launch
reboot
#Sites and services were back up but not nginx-proxy, force start
ee service restart nginx-proxy
I am not sure why these images are not updated automatically and there is no information from the EasyEngine team on how to force it. I was hesitant not knowing how intereconnected EE made their containers and if anything would break. In this case, I was lucky enough to get it to work. Your mileage may vary.
I hope this can help someone save quite a few hours of troubleshooting.
