passenger
passenger copied to clipboard
Published
20 hours ago •
phusion
phusion
`passenger-status --show=requests` causes passenger crash with SIGSEGV(11)
Posted on StackOverflow, but no response there: https://stackoverflow.com/questions/66777958/passenger-status-causes-sigsegv11-crashes-passenger-leading-to-502s
Issue report
Question 1: What is the problem?
passenger-status --show=requests command causes the passenger process to crash with SIGSEGV(11), leading to 502s for our clients.
Use case: We run the above-mentioned command periodically to check for any long-running requests in rails. Though it runs fine most of the time, it intermittently causes the passenger process to crash.
-
What is the expected behavior? The passenger process shouldn't crash.
-
What is the actual behavior? The passenger process crashes. Please refer to the logs below for the stack trace.
-
How can we reproduce it? The issue doesn't arise if we don't call
passenger-status --show=requestscommand. We have other scripts running which callpassenger-statuscommand (without the--show=requestsflag) and they have been working fine for years now.
Please refer to the logs here:
[ pid=4204, timestamp=1588721822 ] Process aborted! signo=SIGSEGV(11), reason=SI_KERNEL, si_addr=0x0, randomSeed=1586240133
[ pid=4204 ] Crash log dumped to /var/tmp/passenger-crash-log.1588721822
[ pid=4204 ] Date, uname and ulimits:
Tue May 5 23:37:03 UTC 2020
Linux 2.6.32-642.4.2.el6.x86_64 #1 SMP Tue Aug 23 19:58:13 UTC 2016 x86_64 x86_64
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 13695
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 13695
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
[ pid=4204 ] Phusion Passenger version: 4.0.53
[ pid=4204 ] libc backtrace available!
--------------------------------------
[ pid=4204 ] Backtrace with 31 frames:
Using backtrace sanitizer.
PassengerHelperAgent(Passenger::dumpBacktrace(Passenger::AbortHandlerState&, void*) at /opt/ruby225/lib/ruby/gems/2.2.0/gems/passenger-4.0.53/ext/common/agents/Base.cpp:545) [0x5f9a56]
PassengerHelperAgent(Passenger::runInSubprocessWithTimeLimit(Passenger::AbortHandlerState&, void (*)(Passenger::AbortHandlerState&, void*), void*, int) at /opt/ruby225/lib/ruby/gems/2.2.0/gems/passenger-4.0.53/ext/common/agents/Base.cpp:388) [0x5f9369]
PassengerHelperAgent(Passenger::dumpDiagnostics(Passenger::AbortHandlerState&) at /opt/ruby225/lib/ruby/gems/2.2.0/gems/passenger-4.0.53/ext/common/agents/Base.cpp:734) [0x5fa4ee]
PassengerHelperAgent(Passenger::abortHandler(int, siginfo*, void*) at /opt/ruby225/lib/ruby/gems/2.2.0/gems/passenger-4.0.53/ext/common/agents/Base.cpp:979) [0x5fb042]
/lib64/libpthread.so.0(?? at ??:0; +0xf7e0) [0x7f6017e607e0]
PassengerHelperAgent(__gnu_cxx::hashtable<std::pair<Passenger::StaticString const, Passenger::StaticString>, Passenger::StaticString, Passenger::StaticString::Hash, std::_Select1st<std::pair<Passenger::StaticString const, Passenger::StaticString> >, std::equal_to<Passenger::StaticString>, std::allocator<Passenger::StaticString> >::find(Passenger::StaticString const&) const at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/backward/hashtable.h:519; _ZNK9__gnu_cxx9hashtableISt4pairIKN9Passenger12StaticStringES3_ES3_NS3_4HashESt10_Select1stIS5_ESt8equal_toIS3_ESaIS3_EE4findERS4_+0x3e) [0x56a24c]
PassengerHelperAgent(__gnu_cxx::hash_map<Passenger::StaticString, Passenger::StaticString, Passenger::StaticString::Hash, std::equal_to<Passenger::StaticString>, std::allocator<Passenger::StaticString> >::find(Passenger::StaticString const&) const at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/backward/hash_map:213; _ZNK9__gnu_cxx8hash_mapIN9Passenger12StaticStringES2_NS2_4HashESt8equal_toIS2_ESaIS2_EE4findERKS2_+0x23) [0x55203b]
PassengerHelperAgent(Passenger::ScgiRequestParser::getHeader(Passenger::StaticString const&) const at /opt/ruby225/lib/ruby/gems/2.2.0/gems/passenger-4.0.53/ext/common/agents/HelperAgent/ScgiRequestParser.h:352) [0x51f4bb]
PassengerHelperAgent(void Passenger::Client::inspect<std::basic_stringstream<char, std::char_traits<char>, std::allocator<char> > >(std::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >&) const at /opt/ruby225/lib/ruby/gems/2.2.0/gems/passenger-4.0.53/ext/common/agents/HelperAgent/RequestHandler.h:592) [0x56f18e]
PassengerHelperAgent(void Passenger::RequestHandler::inspect<std::basic_stringstream<char, std::char_traits<char>, std::allocator<char> > >(std::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >&) const at /opt/ruby225/lib/ruby/gems/2.2.0/gems/passenger-4.0.53/ext/common/agents/HelperAgent/RequestHandler.h:2733) [0x5554c7]
PassengerHelperAgent(RemoteController::processRequests(Passenger::MessageServer::CommonClientContext&, RemoteController::SpecificContext*, std::vector<std::string, std::allocator<std::string> > const&) at /opt/ruby225/lib/ruby/gems/2.2.0/gems/passenger-4.0.53/ext/common/agents/HelperAgent/Main.cpp:174) [0x5381c2]
PassengerHelperAgent(RemoteController::processMessage(Passenger::MessageServer::CommonClientContext&, boost::shared_ptr<Passenger::MessageServer::ClientContext>&, std::vector<std::string, std::allocator<std::string> > const&) at /opt/ruby225/lib/ruby/gems/2.2.0/gems/passenger-4.0.53/ext/common/agents/HelperAgent/Main.cpp:206) [0x538957]
PassengerHelperAgent(Passenger::MessageServer::processMessage(Passenger::MessageServer::CommonClientContext&, std::vector<boost::shared_ptr<Passenger::MessageServer::ClientContext>, std::allocator<boost::shared_ptr<Passenger::MessageServer::ClientContext> > >&, std::vector<std::string, std::allocator<std::string> > const&) at /opt/ruby225/lib/ruby/gems/2.2.0/gems/passenger-4.0.53/ext/common/MessageServer.h:463) [0x535df2]
PassengerHelperAgent(Passenger::MessageServer::clientHandlingMainLoop(Passenger::FileDescriptor) at /opt/ruby225/lib/ruby/gems/2.2.0/gems/passenger-4.0.53/ext/common/MessageServer.h:514) [0x5366ce]
PassengerHelperAgent(boost::_mfi::mf1<void, Passenger::MessageServer, Passenger::FileDescriptor>::operator()(Passenger::MessageServer*, Passenger::FileDescriptor) const at /opt/ruby225/lib/ruby/gems/2.2.0/gems/passenger-4.0.53/ext/boost/bind/mem_fn_template.hpp:165) [0x5aca95]
PassengerHelperAgent(void boost::_bi::list2<boost::_bi::value<Passenger::MessageServer*>, boost::_bi::value<Passenger::FileDescriptor> >::operator()<boost::_mfi::mf1<void, Passenger::MessageServer, Passenger::FileDescriptor>, boost::_bi::list0>(boost::_bi::type<void>, boost::_mfi::mf1<void, Passenger::MessageServer, Passenger::FileDescriptor>&, boost::_bi::list0&, int) at /opt/ruby225/lib/ruby/gems/2.2.0/gems/passenger-4.0.53/ext/boost/bind/bind.hpp:313) [0x5a35d1]
PassengerHelperAgent(boost::_bi::bind_t<void, boost::_mfi::mf1<void, Passenger::MessageServer, Passenger::FileDescriptor>, boost::_bi::list2<boost::_bi::value<Passenger::MessageServer*>, boost::_bi::value<Passenger::FileDescriptor> > >::operator()() at /opt/ruby225/lib/ruby/gems/2.2.0/gems/passenger-4.0.53/ext/boost/bind/bind_template.hpp:21) [0x598d33]
PassengerHelperAgent(boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, boost::_mfi::mf1<void, Passenger::MessageServer, Passenger::FileDescriptor>, boost::_bi::list2<boost::_bi::value<Passenger::MessageServer*>, boost::_bi::value<Passenger::FileDescriptor> > >, void>::invoke(boost::detail::function::function_buffer&) at /opt/ruby225/lib/ruby/gems/2.2.0/gems/passenger-4.0.53/ext/boost/function/function_template.hpp:154) [0x58f4b7]
PassengerHelperAgent(boost::function0<void>::operator()() const at /opt/ruby225/lib/ruby/gems/2.2.0/gems/passenger-4.0.53/ext/boost/function/function_template.hpp:768) [0x53f6a3]
PassengerHelperAgent(oxt::dynamic_thread_group::thread_main(boost::function<void ()()>&, oxt::dynamic_thread_group::thread_handle*) at /opt/ruby225/lib/ruby/gems/2.2.0/gems/passenger-4.0.53/ext/oxt/dynamic_thread_group.hpp:109) [0x4de7a0]
PassengerHelperAgent(boost::_mfi::mf2<void, oxt::dynamic_thread_group, boost::function<void ()()>&, oxt::dynamic_thread_group::thread_handle*>::operator()(oxt::dynamic_thread_group*, boost::function<void ()()>&, oxt::dynamic_thread_group::thread_handle*) const at /opt/ruby225/lib/ruby/gems/2.2.0/gems/passenger-4.0.53/ext/boost/bind/mem_fn_template.hpp:281) [0x5a4f8a]
PassengerHelperAgent(void boost::_bi::list3<boost::_bi::value<oxt::dynamic_thread_group*>, boost::_bi::value<boost::function<void ()()> >, boost::_bi::value<oxt::dynamic_thread_group::thread_handle*> >::operator()<boost::_mfi::mf2<void, oxt::dynamic_thread_group, boost::function<void ()()>&, oxt::dynamic_thread_group::thread_handle*>, boost::_bi::list0>(boost::_bi::type<void>, boost::_mfi::mf2<void, oxt::dynamic_thread_group, boost::function<void ()()>&, oxt::dynamic_thread_group::thread_handle*>&, boost::_bi::list0&, int) at /opt/ruby225/lib/ruby/gems/2.2.0/gems/passenger-4.0.53/ext/boost/bind/bind.hpp:393) [0x59a29a]
PassengerHelperAgent(boost::_bi::bind_t<void, boost::_mfi::mf2<void, oxt::dynamic_thread_group, boost::function<void ()()>&, oxt::dynamic_thread_group::thread_handle*>, boost::_bi::list3<boost::_bi::value<oxt::dynamic_thread_group*>, boost::_bi::value<boost::function<void ()()> >, boost::_bi::value<oxt::dynamic_thread_group::thread_handle*> > >::operator()() at /opt/ruby225/lib/ruby/gems/2.2.0/gems/passenger-4.0.53/ext/boost/bind/bind_template.hpp:21) [0x5910ab]
PassengerHelperAgent(boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, boost::_mfi::mf2<void, oxt::dynamic_thread_group, boost::function<void ()()>&, oxt::dynamic_thread_group::thread_handle*>, boost::_bi::list3<boost::_bi::value<oxt::dynamic_thread_group*>, boost::_bi::value<boost::function<void ()()> >, boost::_bi::value<oxt::dynamic_thread_group::thread_handle*> > >, void>::invoke(boost::detail::function::function_buffer&) at /opt/ruby225/lib/ruby/gems/2.2.0/gems/passenger-4.0.53/ext/boost/function/function_template.hpp:154) [0x585deb]
PassengerHelperAgent(oxt::thread::thread_main(boost::function<void ()()>, boost::shared_ptr<oxt::thread_local_context>) at /opt/ruby225/lib/ruby/gems/2.2.0/gems/passenger-4.0.53/ext/oxt/implementation.cpp:358) [0x64e3d8]
PassengerHelperAgent(void boost::_bi::list2<boost::_bi::value<boost::function<void ()()> >, boost::_bi::value<boost::shared_ptr<oxt::thread_local_context> > >::operator()<void (*)(boost::function<void ()()>, boost::shared_ptr<oxt::thread_local_context>), boost::_bi::list0>(boost::_bi::type<void>, void (*&)(boost::function<void ()()>, boost::shared_ptr<oxt::thread_local_context>), boost::_bi::list0&, int) at /opt/ruby225/lib/ruby/gems/2.2.0/gems/passenger-4.0.53/ext/boost/bind/bind.hpp:313) [0x5b6760]
PassengerHelperAgent(boost::_bi::bind_t<void, void (*)(boost::function<void ()()>, boost::shared_ptr<oxt::thread_local_context>), boost::_bi::list2<boost::_bi::value<boost::function<void ()()> >, boost::_bi::value<boost::shared_ptr<oxt::thread_local_context> > > >::operator()() at /opt/ruby225/lib/ruby/gems/2.2.0/gems/passenger-4.0.53/ext/boost/bind/bind_template.hpp:21) [0x5b5ff7]
PassengerHelperAgent(boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(boost::function<void ()()>, boost::shared_ptr<oxt::thread_local_context>), boost::_bi::list2<boost::_bi::value<boost::function<void ()()> >, boost::_bi::value<boost::shared_ptr<oxt::thread_local_context> > > > >::run() at /opt/ruby225/lib/ruby/gems/2.2.0/gems/passenger-4.0.53/ext/boost/thread/detail/thread.hpp:118) [0x5b5194]
PassengerHelperAgent(thread_proxy at /opt/ruby225/lib/ruby/gems/2.2.0/gems/passenger-4.0.53/ext/boost/libs/thread/src/pthread/thread.cpp:164) [0x61744d]
/lib64/libpthread.so.0(?? at ??:0; +0x7aa1) [0x7f6017e58aa1]
/lib64/libc.so.6(?? at ??:0; clone+0x6d) [0x7f60171fdaad]
--------------------------------------
[ pid=4204 ] Dumping additional diagnostical information...
--------------------------------------
### Request handler state
2 clients:
Client 26:
host = some-host-abcd-1234.com
uri = /some/url
connected at = 2020-05-05 23:37:02 (1 sec ago)
state = FORWARDING_BODY_TO_APP
session pid = 19041 (/home/app/railsApp/current#default)
session gupid = 19406ad-tXMtf0VqNfJ
session initiated = true
requestBodyIsBuffered = false
requestIsChunked = false
requestBodyLength = 1469
requestBodyAlreadyRead = 1469
responseContentLength = -1
responseBodyAlreadyRead = 0
clientInput = 0x7f5ffc07c7e0 fd=26, state=LIVE, buffer(0)="", paused=1, socketPaused=1, nextTickInstalled=0, generation=3, error=0
clientInput started = false
clientBodyBuffer started = false
clientBodyBuffer reachedEnd = false
clientOutputPipe started = true
clientOutputPipe reachedEnd = false
clientOutputWatcher active = false
appInput = 0x7f5ffc05de50 fd=27, state=LIVE, buffer(0)="", paused=0, socketPaused=0, nextTickInstalled=0, generation=2, error=0
appInput started = true
appInput reachedEnd = false
responseHeaderSeen = false
useUnionStation = false
Client 28:
host = some-host-abcd-1234.com
uri = some/url
connected at = 2020-05-05 23:37:02 (1 sec ago)
state = FORWARDING_BODY_TO_APP
session pid = 19012 (/home/app/railsApp/current#default)
session gupid = 19406ad-efmHkBj7BxY
session initiated = true
requestBodyIsBuffered = false
requestIsChunked = false
requestBodyLength = 133
requestBodyAlreadyRead = 133
responseContentLength = -1
responseBodyAlreadyRead = 0
clientInput = 0x7f5ffc10d150 fd=28, state=LIVE, buffer(0)="", paused=1, socketPaused=1, nextTickInstalled=0, generation=3, error=0
clientInput started = false
clientBodyBuffer started = false
clientBodyBuffer reachedEnd = false
clientOutputPipe started = true
clientOutputPipe reachedEnd = false
clientOutputWatcher active = false
appInput = 0x7f5ffc104120 fd=30, state=LIVE, buffer(0)="", paused=0, socketPaused=0, nextTickInstalled=0, generation=2, error=0
appInput started = true
appInput reachedEnd = false
responseHeaderSeen = false
useUnionStation = false
### Pool state (simple)
----------- General information -----------
Max pool size : 9
Processes : 9
Requests in top-level queue : 0
----------- Application groups -----------
/home/app/railsApp/current#default:
App root: /home/app/railsApp/current
Requests in queue: 0
* PID: 18991 Sessions: 0 Processed: 13269 Uptime: 7h 23m 56s
CPU: 9% Memory : 243M Last used: 14s ago
URL : http://127.0.0.1:33735
Password: some_password
* PID: 18998 Sessions: 0 Processed: 13604 Uptime: 7h 23m 56s
CPU: 9% Memory : 244M Last used: 2s ago
URL : http://127.0.0.1:39821
Password: some_password
* PID: 19005 Sessions: 0 Processed: 13825 Uptime: 7h 23m 55s
CPU: 9% Memory : 239M Last used: 10s ago
URL : http://127.0.0.1:40429
Password: some_password
* PID: 19012 Sessions: 1 Processed: 14499 Uptime: 7h 23m 54s
CPU: 10% Memory : 236M Last used: 1s ago
URL : http://127.0.0.1:40782
Password: some_password
* PID: 19021 Sessions: 0 Processed: 13616 Uptime: 7h 23m 53s
CPU: 9% Memory : 228M Last used: 1s ago
URL : http://127.0.0.1:37763
Password: some_password
* PID: 19028 Sessions: 0 Processed: 14185 Uptime: 7h 23m 52s
CPU: 10% Memory : 219M Last used: 10s ago
URL : http://127.0.0.1:40643
Password: some_password
* PID: 19041 Sessions: 1 Processed: 13692 Uptime: 7h 23m 50s
CPU: 9% Memory : 221M Last used: 1s ago
URL : http://127.0.0.1:39801
Password: some_password
* PID: 19050 Sessions: 0 Processed: 13250 Uptime: 7h 23m 49s
CPU: 9% Memory : 203M Last used: 7s ago
URL : http://127.0.0.1:45908
Password: some_password
* PID: 19061 Sessions: 0 Processed: 13984 Uptime: 7h 23m 47s
CPU: 10% Memory : 202M Last used: 10s ago
URL : http://127.0.0.1:40767
Password: some_password
### Pool state (XML)
<?xml version="1.0" encoding="iso8859-1" ?>
<info version="3"><passenger_version>4.0.53</passenger_version><process_count>9</process_count><max>9</max><capacity_used>9</capacity_used><get_wait_list_size>0</get_wait_list_size><get_wait_list></get_wait_list><supergroups><supergroup><name>/home/app/railsApp/current</name><state>READY</state><get_wait_list_size>0</get_wait_list_size><capacity_used>9</capacity_used><secret>some_secret</secret><group default="true"><name>/home/app/railsApp/current#default</name><component_name>default</component_name><app_root>/home/app/railsApp/current</app_root><app_type>rack</app_type><environment>production</environment><uuid>0fPF8oI5HN5VqKncwgO0</uuid><enabled_process_count>9</enabled_process_count><disabling_process_count>0</disabling_process_count><disabled_process_count>0</disabled_process_count><capacity_used>9</capacity_used><get_wait_list_size>0</get_wait_list_size><disable_wait_list_size>0</disable_wait_list_size><processes_being_spawned>0</processes_being_spawned><secret>some_secret</secret><life_status>ALIVE</life_status><options><app_root>/home/app/railsApp/current</app_root><app_group_name>/home/app/railsApp/current</app_group_name><app_type>rack</app_type><start_command>/usr/bin/ruby	/opt/passenger-4.0.53//helper-scripts/rack-loader.rb</start_command><startup_file>config.ru</startup_file><process_title>Passenger RackApp</process_title><log_level>0</log_level><start_timeout>90000</start_timeout><environment>production</environment><base_uri>/</base_uri><spawn_method>smart</spawn_method><default_user>app</default_user><default_group>app</default_group><ruby>/usr/bin/ruby</ruby><python>python</python><nodejs>node</nodejs><logging_agent_address>unix:/tmp/passenger.1.0.1623/generation-7/logging</logging_agent_address><logging_agent_username>logging</logging_agent_username><logging_agent_password>some_password</logging_agent_password><debugger>false</debugger><analytics>false</analytics><group_secret>some_secret</group_secret><min_processes>9</min_processes><max_processes>0</max_processes><max_preloader_idle_time>-1</max_preloader_idle_time><max_out_of_band_work_instances>1</max_out_of_band_work_instances></options><processes><process><pid>18991</pid><sticky_session_id>1923564376</sticky_session_id><gupid>19406ad-rfF1dvAhgok</gupid><connect_password>some_password</connect_password><concurrency>1</concurrency><sessions>0</sessions><busyness>0</busyness><processed>13269</processed><spawner_creation_time>1588695165847872</spawner_creation_time><spawn_start_time>1588695187238181</spawn_start_time><spawn_end_time>1588695187245698</spawn_end_time><last_used>1588721809507854</last_used><last_used_desc>14s ago</last_used_desc><uptime>7h 23m 56s</uptime><code_revision>947e7ae01cb615c6dd823ce3e83c3cddc56bfbfe</code_revision><life_status>ALIVE</life_status><enabled>ENABLED</enabled><has_metrics>true</has_metrics><cpu>9</cpu><rss>321204</rss><pss>257607</pss><private_dirty>249748</private_dirty><swap>0</swap><real_memory>249748</real_memory><vmsize>793032</vmsize><process_group_id>4204</process_group_id><command>Passenger RackApp: /home/app/railsApp/current</command><sockets><socket><name>main</name><address>unix:/tmp/passenger.1.0.1623/generation-7/backends/ruby.1dNu5PtEfTM6xQkFajCMMZ5gumXsPX5rq3sGjr8f6F4tBlv</address><protocol>session</protocol><concurrency>1</concurrency><sessions>0</sessions></socket><socket><name>http</name><address>tcp://127.0.0.1:33735</address><protocol>http</protocol><concurrency>1</concurrency><sessions>0</sessions></socket></sockets></process><process><pid>18998</pid><sticky_session_id>1823271253</sticky_session_id><gupid>19406ad-YkKpCpnkPrp</gupid><connect_password>some_password</connect_password><concurrency>1</concurrency><sessions>0</sessions><busyness>0</busyness><processed>13604</processed><spawner_creation_time>1588695165847872</spawner_creation_time><spawn_start_time>1588695187763488</spawn_start_time><spawn_end_time>1588695187771300</spawn_end_time><last_used>1588721821858041</last_used><last_used_desc>2s ago</last_used_desc><uptime>7h 23m 56s</uptime><code_revision>947e7ae01cb615c6dd823ce3e83c3cddc56bfbfe</code_revision><life_status>ALIVE</life_status><enabled>ENABLED</enabled><has_metrics>true</has_metrics><cpu>9</cpu><rss>323976</rss><pss>258658</pss><private_dirty>250536</private_dirty><swap>0</swap><real_memory>250536</real_memory><vmsize>793128</vmsize><process_group_id>4204</process_group_id><command>Passenger RackApp: /home/app/railsApp/current</command><sockets><socket><name>main</name><address>unix:/tmp/passenger.1.0.1623/generation-7/backends/ruby.Zy2puBEwlt9Z5yDz7TAUdhPhTGA4a3XiRwJHzyujfmptlk5</address><protocol>session</protocol><concurrency>1</concurrency><sessions>0</sessions></socket><socket><name>http</name><address>tcp://127.0.0.1:39821</address><protocol>http</protocol><concurrency>1</concurrency><sessions>0</sessions></socket></sockets></process><process><pid>19005</pid><sticky_session_id>1467947789</sticky_session_id><gupid>19406ad-4Xykya8fWG6</gupid><connect_password>some_password</connect_password><concurrency>1</concurrency><sessions>0</sessions><busyness>0</busyness><processed>13825</processed><spawner_creation_time>1588695165847872</spawner_creation_time><spawn_start_time>1588695188462803</spawn_start_time><spawn_end_time>1588695188482408</spawn_end_time><last_used>1588721813170083</last_used><last_used_desc>10s ago</last_used_desc><uptime>7h 23m 55s</uptime><code_revision>947e7ae01cb615c6dd823ce3e83c3cddc56bfbfe</code_revision><life_status>ALIVE</life_status><enabled>ENABLED</enabled><has_metrics>true</has_metrics><cpu>9</cpu><rss>322232</rss><pss>253934</pss><private_dirty>245328</private_dirty><swap>0</swap><real_memory>245328</real_memory><vmsize>793224</vmsize><process_group_id>4204</process_group_id><command>Passenger RackApp: /home/app/railsApp/current</command><sockets><socket><name>main</name><address>unix:/tmp/passenger.1.0.1623/generation-7/backends/ruby.VsSE2a5TYUm2dVprhp86U2Zr9AHKwq9kzOKQEGacAw7jCPJ</address><protocol>session</protocol><concurrency>1</concurrency><sessions>0</sessions></socket><socket><name>http</name><address>tcp://127.0.0.1:40429</address><protocol>http</protocol><concurrency>1</concurrency><sessions>0</sessions></socket></sockets></process><process><pid>19012</pid><sticky_session_id>761268639</sticky_session_id><gupid>19406ad-efmHkBj7BxY</gupid><connect_password>some_password</connect_password><concurrency>1</concurrency><sessions>1</sessions><busyness>2147483647</busyness><processed>14499</processed><spawner_creation_time>1588695165847872</spawner_creation_time><spawn_start_time>1588695189413738</spawn_start_time><spawn_end_time>1588695189422658</spawn_end_time><last_used>1588721822217731</last_used><last_used_desc>1s ago</last_used_desc><uptime>7h 23m 54s</uptime><code_revision>947e7ae01cb615c6dd823ce3e83c3cddc56bfbfe</code_revision><life_status>ALIVE</life_status><enabled>ENABLED</enabled><has_metrics>true</has_metrics><cpu>10</cpu><rss>321788</rss><pss>250912</pss><private_dirty>241748</private_dirty><swap>0</swap><real_memory>241748</real_memory><vmsize>793320</vmsize><process_group_id>4204</process_group_id><command>Passenger RackApp: /home/app/railsApp/current</command><sockets><socket><name>main</name><address>unix:/tmp/passenger.1.0.1623/generation-7/backends/ruby.M38i3pc2yMBDxkBU9NdnIVLngqokZIc257ldeDeV1BTyAj8</address><protocol>session</protocol><concurrency>1</concurrency><sessions>1</sessions></socket><socket><name>http</name><address>tcp://127.0.0.1:40782</address><protocol>http</protocol><concurrency>1</concurrency><sessions>0</sessions></socket></sockets></process><process><pid>19021</pid><sticky_session_id>1384879099</sticky_session_id><gupid>19406ad-9D2s958lHWZ</gupid><connect_password>some_password</connect_password><concurrency>1</concurrency><sessions>0</sessions><busyness>0</busyness><processed>13616</processed><spawner_creation_time>1588695165847872</spawner_creation_time><spawn_start_time>1588695190358672</spawn_start_time><spawn_end_time>1588695190366368</spawn_end_time><last_used>1588721822060607</last_used><last_used_desc>1s ago</last_used_desc><uptime>7h 23m 53s</uptime><code_revision>947e7ae01cb615c6dd823ce3e83c3cddc56bfbfe</code_revision><life_status>ALIVE</life_status><enabled>ENABLED</enabled><has_metrics>true</has_metrics><cpu>9</cpu><rss>319164</rss><pss>244556</pss><private_dirty>234488</private_dirty><swap>0</swap><real_memory>234488</real_memory><vmsize>793416</vmsize><process_group_id>4204</process_group_id><command>Passenger RackApp: /home/app/railsApp/current</command><sockets><socket><name>main</name><address>unix:/tmp/passenger.1.0.1623/generation-7/backends/ruby.wvaYW8CY181QwotDPQsySzNPOVhWsXVAm0j0osCb0EOzOcE</address><protocol>session</protocol><concurrency>1</concurrency><sessions>0</sessions></socket><socket><name>http</name><address>tcp://127.0.0.1:37763</address><protocol>http</protocol><concurrency>1</concurrency><sessions>0</sessions></socket></sockets></process><process><pid>19028</pid><sticky_session_id>1987758100</sticky_session_id><gupid>19406ad-8I0y30aKthz</gupid><connect_password>some_password</connect_password><concurrency>1</concurrency><sessions>0</sessions><busyness>0</busyness><processed>14185</processed><spawner_creation_time>1588695165847872</spawner_creation_time><spawn_start_time>1588695191773960</spawn_start_time><spawn_end_time>1588695191790791</spawn_end_time><last_used>1588721813198632</last_used><last_used_desc>10s ago</last_used_desc><uptime>7h 23m 52s</uptime><code_revision>947e7ae01cb615c6dd823ce3e83c3cddc56bfbfe</code_revision><life_status>ALIVE</life_status><enabled>ENABLED</enabled><has_metrics>true</has_metrics><cpu>10</cpu><rss>318280</rss><pss>236914</pss><private_dirty>224568</private_dirty><swap>0</swap><real_memory>224568</real_memory><vmsize>793512</vmsize><process_group_id>4204</process_group_id><command>Passenger RackApp: /home/app/railsApp/current</command><sockets><socket><name>main</name><address>unix:/tmp/passenger.1.0.1623/generation-7/backends/ruby.s7QIfVQIshxj78PX6w875r1ivc3RYLvC7MHwkt8BRBcfPFL</address><protocol>session</protocol><concurrency>1</concurrency><sessions>0</sessions></socket><socket><name>http</name><address>tcp://127.0.0.1:40643</address><protocol>http</protocol><concurrency>1</concurrency><sessions>0</sessions></socket></sockets></process><process><pid>19041</pid><sticky_session_id>39836972</sticky_session_id><gupid>19406ad-tXMtf0VqNfJ</gupid><connect_password>some_password</connect_password><concurrency>1</concurrency><sessions>1</sessions><busyness>2147483647</busyness><processed>13692</processed><spawner_creation_time>1588695165847872</spawner_creation_time><spawn_start_time>1588695193258765</spawn_start_time><spawn_end_time>1588695193275842</spawn_end_time><last_used>1588721822576591</last_used><last_used_desc>1s ago</last_used_desc><uptime>7h 23m 50s</uptime><code_revision>947e7ae01cb615c6dd823ce3e83c3cddc56bfbfe</code_revision><life_status>ALIVE</life_status><enabled>ENABLED</enabled><has_metrics>true</has_metrics><cpu>9</cpu><rss>332592</rss><pss>243062</pss><private_dirty>226596</private_dirty><swap>0</swap><real_memory>226596</real_memory><vmsize>793608</vmsize><process_group_id>4204</process_group_id><command>Passenger RackApp: /home/app/railsApp/current</command><sockets><socket><name>main</name><address>unix:/tmp/passenger.1.0.1623/generation-7/backends/ruby.2oqn10PTD4MfDxWsoAJEQe76nxDAYM8933lNC3WCCxnP6TE</address><protocol>session</protocol><concurrency>1</concurrency><sessions>1</sessions></socket><socket><name>http</name><address>tcp://127.0.0.1:39801</address><protocol>http</protocol><concurrency>1</concurrency><sessions>0</sessions></socket></sockets></process><process><pid>19050</pid><sticky_session_id>484990901</sticky_session_id><gupid>19406ad-2OrDZlY2tKB</gupid><connect_password>some_password</connect_password><concurrency>1</concurrency><sessions>0</sessions><busyness>0</busyness><processed>13250</processed><spawner_creation_time>1588695165847872</spawner_creation_time><spawn_start_time>1588695194624493</spawn_start_time><spawn_end_time>1588695194659289</spawn_end_time><last_used>1588721816861571</last_used><last_used_desc>7s ago</last_used_desc><uptime>7h 23m 49s</uptime><code_revision>947e7ae01cb615c6dd823ce3e83c3cddc56bfbfe</code_revision><life_status>ALIVE</life_status><enabled>ENABLED</enabled><has_metrics>true</has_metrics><cpu>9</cpu><rss>316496</rss><pss>225703</pss><private_dirty>207980</private_dirty><swap>0</swap><real_memory>207980</real_memory><vmsize>793704</vmsize><process_group_id>4204</process_group_id><command>Passenger RackApp: /home/app/railsApp/current</command><sockets><socket><name>main</name><address>unix:/tmp/passenger.1.0.1623/generation-7/backends/ruby.ZqtcIb1it3rDAuDgR12WvzfGJVPcHfS4SRBTuLP49ElPL5X</address><protocol>session</protocol><concurrency>1</concurrency><sessions>0</sessions></socket><socket><name>http</name><address>tcp://127.0.0.1:45908</address><protocol>http</protocol><concurrency>1</concurrency><sessions>0</sessions></socket></sockets></process><process><pid>19061</pid><sticky_session_id>362224351</sticky_session_id><gupid>19406ad-uhhpo5JBxLW</gupid><connect_password>some_password</connect_password><concurrency>1</concurrency><sessions>0</sessions><busyness>0</busyness><processed>13984</processed><spawner_creation_time>1588695165847872</spawner_creation_time><spawn_start_time>1588695196209255</spawn_start_time><spawn_end_time>1588695196235242</spawn_end_time><last_used>1588721813970298</last_used><last_used_desc>10s ago</last_used_desc><uptime>7h 23m 47s</uptime><code_revision>947e7ae01cb615c6dd823ce3e83c3cddc56bfbfe</code_revision><life_status>ALIVE</life_status><enabled>ENABLED</enabled><has_metrics>true</has_metrics><cpu>10</cpu><rss>316244</rss><pss>225752</pss><private_dirty>207376</private_dirty><swap>0</swap><real_memory>207376</real_memory><vmsize>793800</vmsize><process_group_id>4204</process_group_id><command>Passenger RackApp: /home/app/railsApp/current</command><sockets><socket><name>main</name><address>unix:/tmp/passenger.1.0.1623/generation-7/backends/ruby.Mc7vCO9TVzUbqgAUWRtpoQLAR6PBTH6U1EhjEa4NCoDBFel</address><protocol>session</protocol><concurrency>1</concurrency><sessions>0</sessions></socket><socket><name>http</name><address>tcp://127.0.0.1:40767</address><protocol>http</protocol><concurrency>1</concurrency><sessions>0</sessions></socket></sockets></process></processes></group></supergroup></supergroups></info>
### Backtraces
Thread 'Main thread' (0x7f6018280720, LWP 4204):
in 'void Server::mainLoop()' (Main.cpp:571)
Thread 'MultiLibeio dispatcher' (0x7f601827e700, LWP 4204):
(empty)
Thread 'Pool garbage collector' (0x7f601822b700, LWP 4204):
in 'static void Passenger::ApplicationPool2::Pool::garbageCollect(Passenger::ApplicationPool2::PoolPtr)' (Pool.h:623)
Thread 'MessageServer thread' (0x7f60181ea700, LWP 4204):
in 'void Passenger::MessageServer::mainLoop()' (MessageServer.h:580)
Thread 'Pool event loop' (0x7f6016713700, LWP 4204):
(empty)
Thread 'Pool analytics collector' (0x7f601826c700, LWP 4204):
in 'static void Passenger::ApplicationPool2::Pool::collectAnalytics(Passenger::ApplicationPool2::PoolPtr)' (Pool.h:754)
Thread 'Request event loop' (0x7f6015d12700, LWP 4204):
(empty)
Thread 'PipeWatcher: PID 18823 stderr, fd 25' (0x7f6015103700, LWP 18822):
in 'void Passenger::ApplicationPool2::PipeWatcher::threadMain()' (Implementation.cpp:1539)
in 'static void Passenger::ApplicationPool2::PipeWatcher::threadMain(boost::shared_ptr<Passenger::ApplicationPool2::PipeWatcher>)' (Implementation.cpp:1520)
Thread 'PipeWatcher: PID 18991 stdout, fd 37' (0x7f5fe7fff700, LWP 18822):
in 'void Passenger::ApplicationPool2::PipeWatcher::threadMain()' (Implementation.cpp:1539)
in 'static void Passenger::ApplicationPool2::PipeWatcher::threadMain(boost::shared_ptr<Passenger::ApplicationPool2::PipeWatcher>)' (Implementation.cpp:1520)
Thread 'PipeWatcher: PID 18998 stdout, fd 38' (0x7f6014f68700, LWP 18822):
in 'void Passenger::ApplicationPool2::PipeWatcher::threadMain()' (Implementation.cpp:1539)
in 'static void Passenger::ApplicationPool2::PipeWatcher::threadMain(boost::shared_ptr<Passenger::ApplicationPool2::PipeWatcher>)' (Implementation.cpp:1520)
Thread 'PipeWatcher: PID 19005 stdout, fd 40' (0x7f6014f27700, LWP 18822):
in 'void Passenger::ApplicationPool2::PipeWatcher::threadMain()' (Implementation.cpp:1539)
in 'static void Passenger::ApplicationPool2::PipeWatcher::threadMain(boost::shared_ptr<Passenger::ApplicationPool2::PipeWatcher>)' (Implementation.cpp:1520)
Thread 'PipeWatcher: PID 19012 stdout, fd 42' (0x7f601502b700, LWP 18822):
in 'void Passenger::ApplicationPool2::PipeWatcher::threadMain()' (Implementation.cpp:1539)
in 'static void Passenger::ApplicationPool2::PipeWatcher::threadMain(boost::shared_ptr<Passenger::ApplicationPool2::PipeWatcher>)' (Implementation.cpp:1520)
Thread 'PipeWatcher: PID 19021 stdout, fd 44' (0x7f6014fea700, LWP 18822):
in 'void Passenger::ApplicationPool2::PipeWatcher::threadMain()' (Implementation.cpp:1539)
in 'static void Passenger::ApplicationPool2::PipeWatcher::threadMain(boost::shared_ptr<Passenger::ApplicationPool2::PipeWatcher>)' (Implementation.cpp:1520)
Thread 'PipeWatcher: PID 19028 stdout, fd 47' (0x7f60181c9700, LWP 18822):
in 'void Passenger::ApplicationPool2::PipeWatcher::threadMain()' (Implementation.cpp:1539)
in 'static void Passenger::ApplicationPool2::PipeWatcher::threadMain(boost::shared_ptr<Passenger::ApplicationPool2::PipeWatcher>)' (Implementation.cpp:1520)
Thread 'PipeWatcher: PID 19041 stdout, fd 49' (0x7f6014fa9700, LWP 18822):
in 'void Passenger::ApplicationPool2::PipeWatcher::threadMain()' (Implementation.cpp:1539)
in 'static void Passenger::ApplicationPool2::PipeWatcher::threadMain(boost::shared_ptr<Passenger::ApplicationPool2::PipeWatcher>)' (Implementation.cpp:1520)
Thread 'PipeWatcher: PID 19050 stdout, fd 41' (0x7f60180f5700, LWP 18822):
in 'void Passenger::ApplicationPool2::PipeWatcher::threadMain()' (Implementation.cpp:1539)
in 'static void Passenger::ApplicationPool2::PipeWatcher::threadMain(boost::shared_ptr<Passenger::ApplicationPool2::PipeWatcher>)' (Implementation.cpp:1520)
Thread 'PipeWatcher: PID 19061 stdout, fd 19' (0x7f6014ee6700, LWP 18822):
in 'void Passenger::ApplicationPool2::PipeWatcher::threadMain()' (Implementation.cpp:1539)
in 'static void Passenger::ApplicationPool2::PipeWatcher::threadMain(boost::shared_ptr<Passenger::ApplicationPool2::PipeWatcher>)' (Implementation.cpp:1520)
Thread 'MessageServer client thread 20' (0x7f60150ba700, LWP 4211):
in 'void RemoteController::processRequests(RemoteController::CommonClientContext&, RemoteController::SpecificContext*, const std::vector<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&)' (Main.cpp:170)
in 'void Passenger::MessageServer::clientHandlingMainLoop(Passenger::FileDescriptor)' (MessageServer.h:513)
Thread 'MessageServer client thread 29' (0x7f60180b4700, LWP 4211):
in 'bool RemoteController::processInspect(RemoteController::CommonClientContext&, RemoteController::SpecificContext*, const std::vector<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&)' (Main.cpp:121)
in 'void Passenger::MessageServer::clientHandlingMainLoop(Passenger::FileDescriptor)' (MessageServer.h:513)
--------------------------------------
[ pid=4204 ] Open files and file descriptors:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
Passenger 4204 root cwd DIR 202,64 4096 2 /
Passenger 4204 root rtd DIR 202,64 4096 2 /
Passenger 4204 root txt REG 202,64 16266360 662223 /opt/passenger-4.0.53/buildout/agents/PassengerHelperAgent
Passenger 4204 root mem REG 202,64 65960 132107 /lib64/libnss_files-2.12.so
Passenger 4204 root mem REG 202,64 1923352 132050 /lib64/libc-2.12.so
Passenger 4204 root mem REG 202,64 90880 132905 /lib64/libgcc_s-4.4.7-20120601.so.1
Passenger 4204 root mem REG 202,64 596360 132063 /lib64/libm-2.12.so
Passenger 4204 root mem REG 202,64 987096 224 /usr/lib64/libstdc++.so.6.0.13
Passenger 4204 root mem REG 202,64 43944 132123 /lib64/librt-2.12.so
Passenger 4204 root mem REG 202,64 142688 132111 /lib64/libpthread-2.12.so
Passenger 4204 root mem REG 202,64 154664 131381 /lib64/ld-2.12.so
Passenger 4204 root 0u CHR 1,3 0t0 3845 /dev/null
Passenger 4204 root 1w FIFO 0,8 0t0 1090130273 pipe
Passenger 4204 root 2w FIFO 0,8 0t0 1090130273 pipe
Passenger 4204 root 3u unix 0xffff8800e02b5400 0t0 610343985 socket
Passenger 4204 root 4u unix 0xffff8800037d0540 0t0 1090130279 /tmp/passenger.1.0.1623/generation-7/request
Passenger 4204 root 5w FIFO 0,8 0t0 1090130273 pipe
Passenger 4204 root 8u REG 0,9 0 3841 [eventfd]
Passenger 4204 root 9u REG 0,9 0 3841 [eventpoll]
Passenger 4204 root 10u REG 0,9 0 3841 [eventfd]
Passenger 4204 root 11u REG 0,9 0 3841 [eventpoll]
Passenger 4204 root 12u REG 0,9 0 3841 [eventfd]
Passenger 4204 root 13r FIFO 0,8 0t0 610344009 pipe
Passenger 4204 root 14w FIFO 0,8 0t0 610344009 pipe
Passenger 4204 root 15u unix 0xffff8800e142dc00 0t0 610344010 /tmp/passenger.1.0.1623/generation-7/request
Passenger 4204 root 16u unix 0xffff8800e142d880 0t0 610344012 /tmp/passenger.1.0.1623/generation-7/helper_admin
Passenger 4204 root 17r CHR 1,9 0t0 3850 /dev/urandom
Passenger 4204 root 18r CHR 1,9 0t0 3850 /dev/urandom
Passenger 4204 root 19u unix 0xffff880095c58440 0t0 1084423784 socket
Passenger 4204 root 20u unix 0xffff8800e13093c0 0t0 1090130264 /tmp/passenger.1.0.1623/generation-7/helper_admin
Passenger 4204 root 23u unix 0xffff8800e2254c40 0t0 1084418971 socket
Passenger 4204 root 25r FIFO 0,8 0t0 1084418972 pipe
Passenger 4204 root 26u unix 0xffff8800a4bc53c0 0t0 1090130233 /tmp/passenger.1.0.1623/generation-7/request
Passenger 4204 root 27u unix 0xffff8800e01d1100 0t0 1090130235 socket
Passenger 4204 root 28u unix 0xffff8800e1507100 0t0 1090130162 /tmp/passenger.1.0.1623/generation-7/request
Passenger 4204 root 30u unix 0xffff8800e22cb440 0t0 1090130164 socket
Passenger 4204 root 37u unix 0xffff8800e0446bc0 0t0 1084423114 socket
Passenger 4204 root 38u unix 0xffff8800b787c7c0 0t0 1084423121 socket
Passenger 4204 root 40u unix 0xffff8800e22cbb40 0t0 1084423134 socket
Passenger 4204 root 41u unix 0xffff8800b79b6180 0t0 1084423674 socket
Passenger 4204 root 42u unix 0xffff880095c580c0 0t0 1084423147 socket
Passenger 4204 root 44u unix 0xffff8800e0ad7b00 0t0 1084423164 socket
Passenger 4204 root 47u unix 0xffff8800e1dbcc00 0t0 1084423284 socket
Passenger 4204 root 49u unix 0xffff8800e0446140 0t0 1084423554 socket
--------------------------------------
[ pid=4204 ] Dumping a more detailed backtrace with crash-watch...
Crash-watch is not installed. Please install it with 'gem install crash-watch' or download it from https://github.com/FooBarWidget/crash-watch.
Question 2: Passenger version and integration mode:
- Passenger open source 4.0.53 with nginx.
Question 3: OS or Linux distro, platform (including version):
- Last year, we were on CentOS 6, now we are on CentOS 7. The issue occurred on both.
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"
Question 4: Passenger installation method:
Your answer:
[] Other, please specify: We installed passenger gem, then used passenger-install-nginx-module with nginx 1.6.3. Once built, we created a tarball and use that to install.
Question 5: Your app's programming language (including any version managers) and framework (including versions):
- Ruby 2.2.5, Rails 4.1.15, NodeJS 12.18.3
Question 6: Are you using a PaaS and/or containerization? If so which one?
- No, running on EC2 instance.
Question 7: Anything else about your setup that we should know?
Your answer: None at the moment. But please do let me know if there is anything else that I can share. FYI: the logs attached above are probably an year old. But the crash is the same. We got it recently too when we tried running the command.
We don't provide support for Passenger 4.0.53 outside of Premium support contracts. Does the issue still occur with a recent Passenger version?
