passenger-config restart-app failing with non-specific 500 error code

[SeanSith]

Question 1: What is the problem?

• What is the expected behavior? passenger-config restart-app /path/to/deployed/application --ignore-app-not-running should restart the running application, resulting in the message

Restarting /path/to/deployed/application/current (production)

alternatively

*** Cleaning stale instance directory /tmp/passenger.Db42Pn4
There are no Phusion Passenger(R)-served applications running whose paths begin with '/path/to/deployed/application'.

• What is the actual behavior? The application should restart, but we receive the following error message and the Passenger processes do not seem to have received a restart command (start time remains in the distant past).

*** An error occured while querying the Phusion Passenger(R) server:
Status: 500 Internal Server Error
Content-Length: 22
Content-Type: text/plain
Connection: close

Internal server error

This problem seems to go away for a while after restarting Nginx. It does return after a while. No logs are produced in /var/log/nginx/error.log.

The only other errors I'm seeing are in here passenger-error-Rgik6F.html.gz We do not have an application group configured for /etc/nginx/tls, as /etc/nginx/tls/www is the directory we utilize for ACME challenges when renewing TLS certificates and is served statically.

Question 2: Passenger version and integration mode:

nginx-common                 1.14.0-0ubuntu1.9
nginx-extras                 1.14.0-0ubuntu1.9
libnginx-mod-http-passenger  1:6.0.11-1~bionic1
passenger                    1:6.0.11-1~bionic1

Integration Mode: Nginx module

Question 3: OS or Linux distro, platform (including version):

Ubuntu 18.04.6 LTS, x86_64, 5.4.0-1049-aws

Question 4: Passenger installation method

Phusion APT repo

Question 5: Your app's programming language (including any version managers) and framework (including versions):

NodeJS 14.16.0, Nuxt.js 2.15.8

Question 6: Are you using a PaaS and/or containerization? If so which one?

No. The application is on an AWS EC2 instance. Relevant configuration snippets:

http {
  ...
  passenger_root /usr/lib/ruby/vendor_ruby/phusion_passenger/locations.ini;
  passenger_ruby /usr/bin/passenger_free_ruby;
  passenger_max_pool_size 6;
  passenger_spawn_method smart-lv2;
  passenger_buffer_response on;
  passenger_min_instances 1;
  passenger_max_instances_per_app 0;
  passenger_pool_idle_time 300;
  passenger_max_requests 0;
  passenger_show_version_in_header off;
  passenger_nodejs /usr/local/bin/node;
}

server {
  ... 
  root /path/to/deployed/application/current/public;
  passenger_enabled on;
  passenger_app_type node;
  passenger_startup_file server/index.js;
}

Question 7: Anything else about your setup that we should know?

Not to my knowledge. Willing to provide information when requested.

---

We strive for quality and appreciate you taking the time to submit a report! Please note that if you want guaranteed response times and priority issue support we encourage you to join our enterprise customer base. They also provide us with the means to continue our high level of open source support!

[TBadyl]

@SeanSith

Error page you linked to shows this line:

Cannot lstat("/etc/nginx/tls/server/index.js"): No such file or directory (errno=2)

It looks like Passenger is looking for your passenger_startup_file in wrong location. Probably app_root autodetection failed for some reason here.

You should define passenger_app_root explicitly in your nginx config like this

server {
    ...
    passenger_app_root path/to/deployed/application/current;
}

[CamJN]

Meteor is known to change the working directory for the app which sometimes causes issues like this. Is this a meteor app?

[SeanSith]

It's a Nuxt.js app. Without talking to the application team, I'm going to assume that Meteor isn't in play.

Changes:

• I set the passenger_app_root as suggested above on November 12
• The application team last deployed on November 17. The person who deployed is out-of-office this week, so I don't know if they had issues during the deploy.
• The server is now running Passenger 6.0.12 due to automated patching.

Attempting to restart the app by hand as the application user resulted in the following error message:

*** Cleaning stale instance directory /tmp/passenger.jvXMGqJ
    Warning: Operation not permitted @ rb_file_chown - /tmp/passenger.jvXMGqJ/.
*** An error occured while querying the Phusion Passenger(R) server:
Status: 500 Internal Server Error
Content-Length: 22
Content-Type: text/plain
Connection: close

Internal server error

And checking the permissions of /tmp/passenger.jvXMGqJ showed that it's owned by 'root' instead of the application user which the Passenger NodeApp processes are running (and deployed) under. It would be my expectation that this would remain owned by the application user for the purposes of cleanup.

These are the current permissions on /tmp/passenger.jvXMGqJ:

drwxr-xr-x   5 root root  4096 Nov 19 17:05 .
drwxrwxrwt 157 root root 36864 Nov 23 20:33 ..
drwxr-xr-x   2 root root  4096 Nov 19 17:05 agents.s
drwx-wx-wt   2 root root  4096 Nov 19 17:05 apps.s
-rw-r--r--   1 root root     4 Nov 19 17:05 core.pid
-rw-r--r--   1 root root     0 Nov 19 17:05 creation_finalized
-rw-------   1 root root    24 Nov 19 17:05 full_admin_password.txt
-rw-r--r--   1 root root     0 Nov 19 17:05 lock
-rw-r--r--   1 root root   394 Nov 19 17:05 properties.json
-rw-------   1 root root    24 Nov 19 17:05 read_only_admin_password.txt
-rw-r--r--   1 root root     4 Nov 19 17:05 watchdog.pid
drwxr-xr-x   2 root root  4096 Nov 19 17:05 web_server_info

Not sure if any of this gets us further down the road on this issue.