Disabled workers growing without any limitations

[dmitry]

trafficstars

While working in thread concurrency model number of workers are growing, and unable to be disabled even after memory limitation should get out of the traffic stream. Example below is not perfect, as I've copied it in the third wave when tried to choke the fire.

Also that means there could be dozen of disabled workers, which are still in working state. Means we should prepare for 8 (passenger_max_instances) * 800 (passenger_memory_limit + 100mb) * 2 (disabled and enabled) memory or even more, as number of disabled could be grown to unlimited number of workers.

Please note passenger_max_request_time is zero, as limitation in thread concurrency model kills process with all the concurrent processes (see https://github.com/phusion/passenger/issues/1850).

Passenger config, most important settings:

passenger_enabled on;
passenger_ruby /home/lysbon/.rbenv/versions/2.1.7/bin/ruby;
passenger_app_env "production";
passenger_concurrency_model thread;
passenger_thread_count 30;
passenger_max_instances 8;
passenger_min_instances 6;
passenger_max_request_time 0;
passenger_max_request_queue_size 3000;
passenger_app_group_name api_v4;
passenger_memory_limit 700;
passenger_env_var THREAD_SAFE 1;

passenger_rolling_restarts on; # global config
passenger_resist_deployment_errors on; # global config
passenger_max_pool_size 50; # global config

passenger-status shows workers are growing, I saw even more ~15 of additional workers (in the first wave), that tried to be disabled.

Requests in queue: 982
  * PID: 19955   Sessions: 30      Processed: 4       Uptime: 12s
    CPU: 35%     Memory  : 167M    Last used: 0s ago
  * PID: 19993   Sessions: 30      Processed: 9       Uptime: 12s
    CPU: 43%     Memory  : 224M    Last used: 0s ago
  * PID: 20031   Sessions: 30      Processed: 1       Uptime: 12s
    CPU: 50%     Memory  : 236M    Last used: 1s ago
  * PID: 20111   Sessions: 30      Processed: 3       Uptime: 12s
    CPU: 46%     Memory  : 228M    Last used: 1s ago
  * PID: 20150   Sessions: 30      Processed: 20      Uptime: 12s
    CPU: 43%     Memory  : 177M    Last used: 1s ago
  * PID: 20214   Sessions: 30      Processed: 2       Uptime: 7s
    CPU: 81%     Memory  : 201M    Last used: 1s ago
  * PID: 20252   Sessions: 30      Processed: 6       Uptime: 7s
    CPU: 79%     Memory  : 205M    Last used: 0s ago
  * PID: 20072   Sessions: 20      Processed: 10      Uptime: 12s
    CPU: 49%     Memory  : 222M    Last used: 12s ago
    Disabling...
  * PID: 19010   Sessions: 30      Processed: 0       Uptime: 32s
    CPU: 84%     Memory  : 748M    Last used: 32s ago
    Shutting down...
  * PID: 19044   Sessions: 30      Processed: 26      Uptime: 32s
    CPU: 84%     Memory  : 701M    Last used: 15s ago
    Shutting down...
  * PID: 19096   Sessions: 30      Processed: 48      Uptime: 32s
    CPU: 85%     Memory  : 725M    Last used: 23s ago
    Shutting down...
  * PID: 19185   Sessions: 30      Processed: 17      Uptime: 32s
    CPU: 85%     Memory  : 731M    Last used: 24s ago
    Shutting down...
  * PID: 19220   Sessions: 30      Processed: 37      Uptime: 32s
    CPU: 85%     Memory  : 747M    Last used: 29s ago
    Shutting down...
  * PID: 19139   Sessions: 29      Processed: 30      Uptime: 32s
    CPU: 88%     Memory  : 747M    Last used: 28s ago
    Shutting down...
  * PID: 19275   Sessions: 29      Processed: 38      Uptime: 31s
    CPU: 81%     Memory  : 834M    Last used: 17s ago
    Shutting down...
  * PID: 19602   Sessions: 30      Processed: 11      Uptime: 27s
    CPU: 86%     Memory  : 730M    Last used: 20s ago
    Shutting down..

PS. Is there are a suggestions how to setup thread applications?

passenger enterprise 5.0.30/nginx ubuntu 14.04 APT repo Ruby 2.1.7, rbenv, Rails 3.2.2.22 Plain hardware used

[CamJN]

passenger_memory_limit is the maximum amount of memory that an application process may use, in megabytes. Once an application process has surpassed its memory limit, Passenger will allow it to finish processing all of its current requests, then shut the process down. However, shutting down happens according to how your application is written, and can take some time, this is why you see the increased number of processes. If you profile your application while it is shutting down you may be able to speed this process up, however it is more important to avoid having to replace processes all the time, so you will want to evaluate whether your memory limit is reasonable.

Because you have many threads, you may want to consider the expected memory footprint of your application. It may be that 700MB is too low. If your single-threaded memory footprint is 200MB we estimate that each thread will use an additional ~10% of the total memory footprint or ~20MB of RAM, that means that for 30 threads you would expect to see 200_(1+(0.1_30)) = 800MB of ram quite easily.

There is some helpful information on choosing your configuration values in this article: https://www.phusionpassenger.com/library/config/nginx/optimization/.

If you find that 700MB for 30 threads should suffice for your application, then profiling your application for leaks would be the next step.

[dmitry]

Yes, I tried to profile for ruby memory leaks - but we are free of them; seems like all the leaks somewhere inside the gems, in C extensions... at the moment I don't have much time to analyze and solve those issues, so the only solutution at the moment to have passenger_memory_limit enabled.

The problem I can see at the moment, that Passenger tries to spawn new workers too early, before actually queue of other workers will be disconnected from the traffic and shutdowned. Means when you have room of 32GB on hardware machine, and you have 24 instances with 1GB each, passenger may kill your server with swapping because it can create additional 24 instances, so in total all of them could take up to 48, or even more, when memory limitation of the old processes are larger than 1GB (that's why actually they are going to be killed).

Additionally, I noticed that passenger may leave unlimited number of processes on disabled state, so in total number of processes may be upper than 48 in total.

[simonweil]

Any chance to get an option to limit the passenger memory?