nginx returns 500 on everything, hard-coded in nginx-config

[cpury]

Mup version (mup --version): 1.4.5

Mup config

{
  "servers": {
    "one": {
      "host": "1.2.3.4",
      "username": "ubuntu",
      "pem": "~/.ssh/pem"
    }
  },
  "app": {
    "name": "my-app",
    "path": "../",
    "servers": {
      "one": {}
    },
    "buildOptions": {
      "serverOnly": true
    },
    "env": {
      "ROOT_URL": "https://subdomain.host.com",
      "MONGO_URL": "...",
      "MONGO_OPLOG_URL": "...",
      "VIRTUAL_HOST": "sub.domain.com",
      "HTTPS_METHOD": "redirect",
      "HTTP_FORWARDED_COUNT": 1,
      "LETSENCRYPT_HOST": "sub.domain.com",
      "LETSENCRYPT_EMAIL": "[email protected]"
    },
    "docker": {
      "image": "abernix/meteord:node-8.11.2-base",
      "stopAppDuringPrepareBundle": true
    },
    "enableUploadProgressBar": true,
    "type": "meteor"
  },
  "proxy": {
    "domains": "sub.domain.com",
    "ssl": {
      "letsEncryptEmail": "[email protected]"
    }
  }
}

I've been stuck on this for days. Sometimes it produces a working nginx-config and the server works as expected. Then I change one line (e.g. I'm trying to have multiple domains in the proxy), and suddenly I'm getting 500s for every route. From then on, no amount of reconfiguring etc. helps any more, I need to start a new instance and hope it's correct there.

The direct reason is very obvious when looking at mup proxy nginx-config: When it's working, I get entries such as these

server {
	server_name sub.domain.com;
	listen 443 ssl http2 ;
	access_log /var/log/nginx/access.log vhost;
	ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3;
	ssl_ciphers 'blablabla';
	ssl_prefer_server_ciphers on;
	ssl_session_timeout 5m;
	ssl_session_cache shared:SSL:50m;
	ssl_session_tickets off;
	ssl_certificate /etc/nginx/certs/sub.domain.com.crt;
	ssl_certificate_key /etc/nginx/certs/sub.domain.com.key;
	ssl_dhparam /etc/nginx/certs/sub.domain.com.dhparam.pem;
	ssl_stapling on;
	ssl_stapling_verify on;
	ssl_trusted_certificate /etc/nginx/certs/sub.domain.com.chain.pem;
	include /etc/nginx/vhost.d/sub.domain.com;
	location / {
		proxy_pass http://sub.domain.com;
	}
}

When the 500 thing happens, those entries are replaced with this:

server {
	server_name sub.domain.com;
	listen 443 ssl http2 ;
	access_log /var/log/nginx/access.log vhost;
	return 500;
	ssl_certificate /etc/nginx/certs/default.crt;
	ssl_certificate_key /etc/nginx/certs/default.key;
}

So the 500 is hard-wired! How can that ever happen? How is that a useful nginx config to generate? I've also seen cases where sub.domain.com servers are never configured, just _ that returns 503.

Because of this, Let's Encrypt also fails because it's getting 500s for its challenge.

I tried overwriting these entries with custom nginx configs, but so far have not succeeded.

[ddaydd]

hi,

I don't know if it's the same but I had similar problems, but thanks to the command ssh docker container ls I have seen that I have several container.

reboot, mup deploy and that was good

[suezzo]

@cpury I've got the same problem. Did you find a solution?

[cpury]

@suezzo I didn't find a fix, but have since switched to https://github.com/zodern/mup-aws-beanstalk which works great so far

[suezzo]

Using AWS was not an option for me. I find out that port 80 on my machine was not opened. Letsencrypt uses it to generate SSL certificate. After getting the certificate I closed this port again. The application works properly now.

[aspirinchaos]

Any updates? i've got the same bad nginx-config:

server {
        server_name domain.app.com;
        listen 443 ssl http2 ;
        access_log /var/log/nginx/access.log vhost;
        return 500;
        ssl_certificate /etc/nginx/certs/default.crt;
        ssl_certificate_key /etc/nginx/certs/default.key;
}

[vpalos]

The problem is that the certificate generation from Let's Encrypt fails! This, in turn leads to improper generation of NGINX configs, i.e. the return 500; anomaly.

Add this to the mup.js configuration:

shared: {
  envLetsEncrypt: {
    DEBUG: true
  }
}

Then run mup proxy reconfig-shared to update the configuration.

Then, if you run these...

• mup proxy logs and...
• mup proxy logs-le

You'll get the verbose errors that makes Let's Encrypt crash.

---

For me, the problem was that I was using CloudFlare to redirect from HTTP to HTTPS. This was a problem for Let's Encrypt generator tool, because it expected to be able to connect to my domain via HTTP during the generation step (but CloudFlare kept redirecting it to HTTPS instead, which of-course wasn't working yet).

Hope this helps!

[vpalos]

I guess the bottom line (in my case) was:

If you're using Let'sEncrypt certificate generation, make sure there's nothing in front of your server that redirects from HTTP to HTTPS (like CloudFlare), because that will break the generation process and lead to NGINX hard-coded errors.

Instead, use the ssl: { forceSSL: true } option in mup.js and let NGINX make this redirection.

This should be added to the documentation as a warning, I think.

[sayandcode]

The problem is that the certificate generation from Let's Encrypt fails! This, in turn leads to improper generation of NGINX configs, i.e. the return 500; anomaly.

Thanks for the insight @vpalos! In my case, the EC2 was configured to whitelist traffic from only certain IPs. As a result, Let's Encrypt probably couldn't access my port 80 for verification purposes. I whitelisted port 80, now meteor app is deployed as expected.